dmarby / humblebundle-ebook-downloader Goto Github PK

I was trying to fetch all my title when a connection reset popped up. The next few attemps to lauch the tool sent me 502 errors.

Error details:

Downloading Humble Book Bundle: Programmable Boards by Make: - Making Things See: 3D vision with Kinect, Processing, Arduino, and MakerBot (EPUB) (10.5 MB)... (762/1595)
An error occured, exiting.
{ Error: read ECONNRESET
    at _errnoException (util.js:1003:13)
    at TLSWrap.onread (net.js:623:25) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
PS C:\Users\edemers\Desktop\humblebundlebooks> humblebundle-ebook-downloader -d ./ --debug --all
Starting...
Validating session...
An error occured, exiting.
Error: Could not validate session, unknown error, status code: 502
    at Request.request.get [as _callback] (C:\Users\edemers\AppData\Roaming\npm\node_modules\humblebundle-ebook-downloader\index.js:114:17)
    at Request.self.callback (C:\Users\edemers\AppData\Roaming\npm\node_modules\humblebundle-ebook-downloader\node_modules\request\request.js:185:22)
    at Request.emit (events.js:160:13)
    at Request.<anonymous> (C:\Users\edemers\AppData\Roaming\npm\node_modules\humblebundle-ebook-downloader\node_modules\request\request.js:1157:10)
    at Request.emit (events.js:160:13)
    at IncomingMessage.<anonymous> (C:\Users\edemers\AppData\Roaming\npm\node_modules\humblebundle-ebook-downloader\node_modules\request\request.js:1079:12)
    at Object.onceWrapper (events.js:255:19)
    at IncomingMessage.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)

After waiting 10 minutes or so i was able to launch the tool again without error. This happened 2 or 3 times so far and i suppose it may be my network's fault or a humhub's session expiration policy (since i have a VERY large library). This is more to inform you with the error stack than to report an actual bug since i'm still able to use the tool and resume where it crashed.

It looks like file names aren't normalised before the comparison check for an existing file, leading to the same book downloading instead of being skipped on successive runs.

e.g. "Bloom County:The Complete Digital Library Vol. 1" is normalised to "Bloom CountyThe Complete Digital Library Vol. 1", but will be downloaded each time.

Hi, just having a new pc and tried to use your app again... Now I have the problem that after logging in with the popping up electron-window, I log in, but there nothing more happens. It doesnt matter if I run the app via powershell or cmd....

Commandline just says "Authenticating"

This could be an alternative to #25 and #26. A parameter to ignore the extension when downloading would allow for downloading a bundle that contains

Book 1 [epub] [mobi] [pdf]
Book 2 [mobi] [pdf] [pdf_hd]
Book 3 [cbz] [epub] [pdf] [pdf_hd]

Current:
Download 1: -f cbz

• Book 3.cbz

Download 2: -f pdf_hd

• Book 2.pdf
• Book 3.pdf

Download 3: -f epub

• Book 1.epub
• Book 3.epub

Ignoring Extension:
Download 1: -f cbz

• Book 3.cbz

Download 2: -f pdf_hd --compare-no-ext

• Book 2.pdf

Download 3: -f epub --compare-no-ext

• Book 1.epub

I receive this now at random intervals when downloading my books. Thoughts on how to make it complete?

An error occured, exiting.
{ Error: connect ETIMEDOUT 205.185.208.62:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)
  errno: 'ETIMEDOUT',
  code: 'ETIMEDOUT',
  syscall: 'connect',
  address: '205.185.208.62',
  port: 443 }

It will be nice to have ability to sort bundles by purchase date
it slowly becoming unmanagable to pick up new bundles to download only new purches.

Run npm install then the following command:

npm version 6.13.4

humblebundle-ebook-downloader --all

electron windows pop up, I enter username and password, then paste the validation code in that was emailed to me then I get this message

{ Error: certificate has expired
at TLSSocket. (_tls_wrap.js:1116:38)
at emitNone (events.js:106:13)
at TLSSocket.emit (events.js:208:7)
at TLSSocket._finishInit (_tls_wrap.js:643:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:473:38) code: 'CERT_HAS_EXPIRED' }

Certain books include a supplement "version". For example:

In this case, it includes some runnable code samples that correlate to those in the book. humblebundle-ebook-downloader --format all handles fetching of these files perfectly. However, they are assigned the extension *.supplement.

In my limited experience (6 books in my library with supplements), these are downloaded from the Humble Bundle site as *.zip files. Rather than My Book.supplement, could we have these downloaded as My Book (supplement).zip?

(Thanks for the handy tool!)

When I purchased my last bundle, i only provided an email address.

So I received a link by email to download the bundle without the intermediate login step.

Then pasted the email link in the browser location bar, then copied the _simpleauth_sess cookie value and used that, to no avail

humblebundle-ebook-downloader --format pdf --download-limit 1 --auth-token "${_simpleauth_sess}"
Starting...
Validating session...
An error occured, exiting.
Error: Could not validate session, unknown error, status code: 401
    at Request.request.get [as _callback] (/usr/local/lib/node_modules/humblebundle-ebook-downloader/index.js:114:17)
    at Request.self.callback (/usr/local/lib/node_modules/humblebundle-ebook-downloader/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/local/lib/node_modules/humblebundle-ebook-downloader/node_modules/request/request.js:1154:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/humblebundle-ebook-downloader/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)

The only thing that looks weird in the auth token are these characters right before the first vertical bar. This could create all sorts of escaping nightmares with my download command.

...\075\075|

Has anyone managed to successfully download an 'anonymous' bundle (one purchased without a login)? Is there any trick i should know?

Besides the usual formats (pdf, epub, mobi), there sometimes is another type like 'download'

But humblebundle-ebook-downloader does not let me specify that type

$ humblebundle-ebook-downloader --format download  ...
Invalid format selected.

Is there a way to support this. Ideally I'd like to specify --format "pdf,download"

i would like to download not by direct link
i would like end with directory full of *.torrent files

As an alternative or addendum to #25, it would at be nice if the output displayed individual books that could not be downloaded in the selected format, similar to the way entire bundles are displayed now if there are no qualifying files in the entire bundle.

Since HumbleBundle supplies md5 checksums, it would be nice to be able to verify those instead of just checking for the file already being downloaded. From existing issues, I see that someone has already done that, I may see how much work it is to implement and submit a PR.

fork with md5 checking mentioned by @jlongman in #1 (comment)

The retrieving of bundles information can get quite long (a few minutes) on an account with a very big library (1500+ purchases). It would be nice to have a flag to pass one or more bundle names to fetch. The script could then check if you have these bundles / if the name you pass actually exists and by pass the retrival of the information of all you bundles/purchases.

if I want to download all formats, pdf_hd has the same name like the normal pdf... could you change this so that "(hd)" is added for pdf_hd files?

First; I'm a huge fan of the project. THANKSf!

I like being able to use the "--format all --all" flags to sync my local folder after getting a few new bundles. But, my library is getting pretty big and the time to go through all the files and figure out that the download "already exists" takes significant time and seems to put undue load on the humblebundle.com servers.

Two ideas:

1. When there's a directory for the name of a previously downloaded bundle, then assume that all files in that bundle are already downloaded. Then don't check each file. Maybe a "--skip-previous-bundles" flag to enable this different functionality?
2. Keep a local state file containing info about previously downloaded files. Then, just get the new ones. I'm thinking of something similar to the "--download-archive " flag in youtube-dl

My library has > 200 bundles and >8000 files. The humblebundle-ebook-downloader seems to be less reliable as my library is getting bigger. My most recent run crashed after 18 minutes only making it through <4000 / 8000 files. I feel like scoping it down may make the software more reliable.

On Windows 7 the reading and writing of the config file fails if the script is run while the working directory is on another drive than the users home directory.

This is because on Window process.env.HOMEPATH evaluates to a path without the drive letter. process.env.USERPROFILE on the other hand evaluates to a path with the drive letter.

A fix could be to swap the order of those two environment variables when resolving configPath, as long as this doesn't break any other platform.

Hi,

I did the whole download process to test the tools and everything is working just fine.

But when i ran it again, the authentication step is bypassed since my login is still valid. Is there any way to log out of the tool and re-login with another account? Any config file i should remove/modify?

Also a proposition to add a force-authentication flag or logout of some kind.

Thanks!

Is it possible to get ARM support for the new Apple Silicon devices?

npm install -g humblebundle-ebook-downloader
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! code 1
npm ERR! path /opt/homebrew/lib/node_modules/humblebundle-ebook-downloader/node_modules/electron
npm ERR! command failed
npm ERR! command sh -c node install.js
Downloading electron-v2.0.18-darwin-arm64.zip
npm ERR! Error: GET https://github.com/electron/electron/releases/download/v2.0.18/electron-v2.0.18-darwin-arm64.zip returned 404
npm ERR! /opt/homebrew/lib/node_modules/humblebundle-ebook-downloader/node_modules/electron/install.js:54
npm ERR!   throw err
npm ERR!   ^
npm ERR!
npm ERR! Error: Failed to find Electron v2.0.18 for darwin-arm64 at https://github.com/electron/electron/releases/download/v2.0.18/electron-v2.0.18-darwin-arm64.zip
npm ERR!     at Request.<anonymous> (/opt/homebrew/lib/node_modules/humblebundle-ebook-downloader/node_modules/nugget/index.js:169:61)
npm ERR!     at Request.emit (node:events:379:20)
npm ERR!     at Request.onRequestResponse (/opt/homebrew/lib/node_modules/humblebundle-ebook-downloader/node_modules/request/request.js:1059:10)
npm ERR!     at ClientRequest.emit (node:events:379:20)
npm ERR!     at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:668:27)
npm ERR!     at HTTPParser.parserOnHeadersComplete (node:_http_common:129:17)
npm ERR!     at TLSSocket.socketOnData (node:_http_client:534:22)
npm ERR!     at TLSSocket.emit (node:events:379:20)
npm ERR!     at addChunk (node:internal/streams/readable:313:12)
npm ERR!     at readableAddChunk (node:internal/streams/readable:288:9)

As per the title. Can I download both pdf and epub version in one call?

Hi, I used your program a lot and have some ideas, which would be helpful.

At first, could you maybe add a config file, where you can easily choose, which file fomats shall be downloaded? I'm thinking of a format like the following "code"

pdf = true
epub = true
cbz = false

Also It would be great if you could add support for the games, audio-books, programs, video-files and zip (where a config file would be a great benefit)...

Don't know if you can do it and want to change your ebook-downloader to a humblebundle downloader, but your program is the only one, which really runs well ;)

I'm getting this when trying to download everything:

Downloading Humble eBook Bundle - Signal to Noise (Read Me) (0 bytes)... (480/990)
/home/vladimir/.npm_global/lib/node_modules/humblebundle-ebook-downloader/index.js:369
        url: download.url.web
                         ^

TypeError: Cannot read property 'web' of undefined
    at checkSignatureMatch (/home/vladimir/.npm_global/lib/node_modules/humblebundle-ebook-downloader/index.js:369:26)
    at ReadStream.stream.on (/home/vladimir/.npm_global/lib/node_modules/humblebundle-ebook-downloader/index.js:333:14)
    at emitNone (events.js:110:20)
    at ReadStream.emit (events.js:207:7)
    at endReadableNT (_stream_readable.js:1059:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Note the (Read Me) as the type and 0 bytes as the size; should probably skip "weird" entries like this.

I retrieved the _simpleauth_sess from my cookie after logging into the Humble Bundle site. But I get the error:

Error using the humblebundle API, invalid session cookie?

What should I put into the auth cookie argument?

I have 43 bundles in my account, three of which are only available in pdf and one of which reports /no/ available formats. It would be nice if it could either by default or additional parameter gloss over the ones that don't work and focus on what does.

Thanks!

No downloads found matching the right format (epub) for bundle (Humble BulkyPix Mobile Bundle), available formats: (pdf)
No downloads found matching the right format (epub) for bundle (Humble Indie Bundle 11), available formats: (pdf)
No downloads found matching the right format (epub) for bundle (Humble Mobile Bundle: KEMCO X HyperDevbox), available formats: (pdf)
No downloads found matching the right format (epub) for bundle (Humble Very Positive Bundle), available formats: ()

Sometimes the download will just get stuck at some point. I can leave it for hours and it hasn't moved on. It most often happens at the beginning, after 3-5 books. It especially happens if I try to do multiple downloads at the same time. It would be nice with some progress bar (can be a %) or a "verbose mode" where it outputs what's going on so I can see why it suddenly stops.

I usually run with the -d path and -f all, downloading all formats of books I buy and putting them in Dropbox. Doing a ctrl+c in the terminal and starting again usually helps.

System with selinux will fail to run docker command from readme. They require additional ":z" option in volume mount. The ":z" option shouldn't mess with system without selinux

So should look like this.

docker run -v $(PWD)/download:/download:z --rm -it dmarby/humblebundle-ebook-downloader -d /download --auth-token "auth_string_here"

I think _simpleauth_sess shouldn't be visible in command history, so it will be probably good to have option to pass it as environment variables with --env-file option. But this will require modifying program probably…

As this program don't require elevated privileges maybe it should be run with podman instead of docker, which don't require root, and it will not mess file permissions on linux. If it will stay docker it would be nice to add "-u" option from docker. Than files will no longer belong to root. This is probably unix only…

docker run -v $(PWD)/download:/download:z -u $(id -u) --rm -it dmarby/humblebundle-ebook-downloader -d /download --auth-token "auth_string_here"

Hello again,

thanks for fixing the bug. Thought about how to improve the program. Here's my idea:

How about creating an option to sort the downloads by size (ascending / descending)? Would be good for guys like me who have slow internet to get the small book files first.

EDIT: For myself it wouldn't matter, if the downloader jumps between different bundles to download the smallest / biggest files first.

I have just started trying this out, as I purchased a big bundle and was going crazy clicking on all of the items to download them individually. One thing I did notice, if the download fails (i.e. I had to kill the docker container), it sees the incomplete downloads as done and skips them when the container is rerun.

I would suggest downloading the files to a temporary name and then renaming them on completion. If the HumbleBundle download process allows resuming the download, that would even allow you to resume incomplete files, or at the very least restart them.

I just updated from 0.0.1 to 1.0.3 and I can't figure out how to use the downloader anymore.

Nearly all parameters changed. I don't see a way to give my session cookie. Reading through the source code, the session seems to be handled by nightmare, but nothing happens.

I do not have a GUI, just console for my linux box.

humblebundle-ebook-downloader$ nodejs index
Starting...
Validating session...
Authenticating...

In a frequent case, book bundles will contain books in multiple formats, but not all books in all formats. Downloading EPUB from a bundle therefore might miss books only available as PDF or PDF_HD.

It would be nice to have a configurable fallback order, so you could specify that you want "cbz, pdf_hd, pdf, epub" or "epub, mobi, pdf"