Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't download all the image from the pixiv fanbox creator #611

Closed
MingLaw30 opened this issue Jan 1, 2020 · 22 comments
Closed

Can't download all the image from the pixiv fanbox creator #611

MingLaw30 opened this issue Jan 1, 2020 · 22 comments
Labels
Enhancement

Comments

@MingLaw30
Copy link

MingLaw30 commented Jan 1, 2020 •
edited

Hello
I am going to download the images from the creator さより＠ネコぱらゲーム製作中
Here is the link https://www.pixiv.net/fanbox/creator/104409
The version of software is 20200101

And here is what I get from the software
Screenshot 2020-01-01 23-41-42

Comparing to the fanbox website
It seems that the software can only download the images stop from this post
Screenshot 2020-01-01 23-42-46

And you can see that there are still about half of the posts not yet downloaded
Screenshot 2020-01-01 23-43-08

I believe that my account has enough permission to access to the post
Screenshot 2020-01-01 23-59-43

And even I use the search function, there is still no any result
Screenshot 2020-01-01 23-59-27

I see the log here and found that the software keep saying that there is no cover image and image available in post
Screenshot 2020-01-02 00-03-06

And here is the log file
pixivutil.log

Hope that you can help me and fix the problem please
@Nandaka
Copy link
Owner

Nandaka commented Jan 1, 2020

do you have paid access to the Fanbox?

@MingLaw30
Copy link
Author

do you have paid access to the Fanbox?

Yes
I have already paid and subscribe for it
Screenshot 2020-01-02 00-35-27
Screenshot 2020-01-02 00-38-07

And I can see the post in the browser too
Screenshot 2020-01-02 00-36-52

It seems that the software can only download the posts cost $100 and can't download the posts cost $540 from 2017-09-29

@Nandaka
Copy link
Owner

Nandaka commented Jan 1, 2020

Looks like the old posts are using non-standard way of showing the images (using custom html, instead of image list).

Can you replace the library.zip with this version: library.zip

Also delete the old log file and reupload it here if there still issues.

@MingLaw30
Copy link
Author

Looks like the old posts are using non-standard way of showing the images (using custom html, instead of image list).

Can you replace the library.zip with this version: library.zip

Also delete the old log file and reupload it here if there still issues.

Happen the error here
Screenshot 2020-01-02 03-08-08
pixivutil.log

Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
Implement #611
7b092f4
Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
Update #611
2f11c09
Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
update #611 and version
d6f113a
@Nandaka
Copy link
Owner

Nandaka commented Jan 2, 2020

try https://github.com/Nandaka/PixivUtil2/releases/tag/v20200102-beta1

@MingLaw30
Copy link
Author

try https://github.com/Nandaka/PixivUtil2/releases/tag/v20200102-beta1

Still the error here
Screenshot 2020-01-02 13-34-05

@Nandaka
Copy link
Owner

Nandaka commented Jan 2, 2020

ok, can you upload the log file?

@MingLaw30
Copy link
Author

ok, can you upload the log file?

Sorry that
pixivutil.log

@Nandaka
Copy link
Owner

Nandaka commented Jan 2, 2020

try library.zip

Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
Update #611
e8ef619
@MingLaw30
Copy link
Author

try library.zip

Get the error here
Screenshot 2020-01-02 20-54-05

And it download the same thing like the above
Screenshot 2020-01-02 20-55-23

Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
update #611
68178bf
@Nandaka
Copy link
Owner

Nandaka commented Jan 2, 2020

updated library.zip

@MingLaw30
Copy link
Author

It download the same thing and the software say that no cover image and no image available

Screenshot 2020-01-02 22-23-07

Screenshot 2020-01-02 22-23-49

@MingLaw30
Copy link
Author

pixivutil.log

Nandaka added a commit that referenced this issue Jan 2, 2020
@Nandaka
update #611
1f21af4
@Nandaka
Copy link
Owner

Nandaka commented Jan 2, 2020

try https://github.com/Nandaka/PixivUtil2/releases/tag/v20200103-beta2

also delete your old log file before running, so I can get clean log.

@MingLaw30
Copy link
Author

It seems that it can get all the images at this time
Screenshot 2020-01-03 02-36-07

Screenshot 2020-01-03 02-36-17

Screenshot 2020-01-03 02-36-11

Here is the log if you need it
pixivutil.log

Also thank you for your patient and effort for solving the problem

@bluerthanever
Copy link
Contributor

Hey, Mr. Nandaka.
I found out that the alteration related to this issue caused FanboxPost.parsePost() method being called twice, once in

PixivUtil2/PixivModelFanbox.py

Line 118 in 2f11c09

self.parsePost(page)

another one in

PixivUtil2/PixivBrowserFactory.py

Line 760 in 2f11c09

post.parsePost(js["body"])

and caused that there being one cover image duplicate in post.embeddedFiles.
Not a big deal, but I think you should know.

@Nandaka
Copy link
Owner

Nandaka commented Jan 13, 2020

It purposefully calling the parsePost twice as the data source comes from different endpoints (https://www.pixiv.net/ajax/fanbox/creator?userId= and https://fanbox.pixiv.net/api/post.info?postId=), need to add check if the post already added or not.

@bluerthanever
Copy link
Contributor

bluerthanever commented Jan 14, 2020 •
edited

Urr. Sorry for assuming you are and calling you a Mister. You are one though, aren't you? If not I am soooo sorry, again.
I noticed that the difference in endpoints actually...
Maybe in parsePost, firstly we check the cover image, if it's none, then we can assign the url to it and append to embed file list?

Nandaka added a commit that referenced this issue Jan 23, 2020
@Nandaka
update #611: avoid duplicates
ed8c4a8
@bluerthanever
Copy link
Contributor

bluerthanever commented Jan 26, 2020 •
edited

Happy Lunar New Year!
Is Nandaka-san Chinese? Haha!
Thanks for the update, but I found out that the issue I mentioned kinda still exists. And I think you modified the codes in parseBody instead of parsePost.
For myself I just added:
if self.coverImageUrl is None:
in front of

PixivUtil2/PixivModelFanbox.py

Lines 138 to 140 in f11e39e

self.coverImageUrl = jsPost["coverImageUrl"]
if self.coverImageUrl is not None:
self.embeddedFiles.append(jsPost["coverImageUrl"])

@Nandaka
Copy link
Owner

Nandaka commented Jan 27, 2020

hmm, can you send a pull request? I'm not sure the changes 😛

@bluerthanever
Copy link
Contributor

Ok. I'll get to it later.

@Nandaka
Copy link
Owner

Nandaka commented Jan 28, 2020 •
edited

merged #629

@Nandaka Nandaka closed this as completed Feb 26, 2020
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
Implement Nandaka#611
c0ad763
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
Update Nandaka#611
14101b3
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
update Nandaka#611 and version
6978258
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
Update Nandaka#611
51e9b48
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
update Nandaka#611
4663501
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
update Nandaka#611
c0897ed
byjtje pushed a commit to byjtje/PixivUtil2 that referenced this issue Oct 30, 2020
@Nandaka
update Nandaka#611: avoid duplicates
ae58838
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Nandaka @bluerthanever @MingLaw30