matthuisman / gdrivedl Goto Github PK

View Code? Open in Web Editor NEW

182.0 182.0 52.0 8.39 MB

Google Drive Download Python Script

Home Page: https://www.matthuisman.nz

License: GNU General Public License v3.0

Python 100.00%

google-drive python

gdrivedl's People

Contributors

Stargazers

Watchers

gdrivedl's Issues

Updated files on google drive aren't downloaded

Hi Matt,

I must say, I am very pleased with your script.

I found an issue though.
We are using it to download mame artwork.
Now we have updated one file.

It seems the script checks the filename and if it's there it won't update the file.

You can read from here :
https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-and-mame-standalone/583
Until post 588

Would be nice if it could be solved ;-)

Cannot download shared folders

Hello, it works good for downloading a shared google drive file, but not a shared folder. Can you please show me how to download a shared folder if it can do? Thanks!

Download with authentication

Hi @matthuisman
I was wondering if there is any chance to add the authentication feature to use credentials as described here:
https://medium.com/swlh/google-drive-api-with-python-part-i-set-up-credentials-1f729cb0372b
I have to work with private data that must be accessed through credentials and I think that setting them in my google account and give them to your script should work. Checking your script I've seen that login feature is not enabled and I wonder if you can add this feature or give me some clue to code myself.
Thank you in advance
Pedro Seoane

TypeError: normalize() argument 2 must be unicode, not str

Hi, I think something went wrong with the last updates.

I use this command line wich runs ok on this commit 41d9dee but it throws an error with the latest version 0535faf

Don't know what are the new changes about, but wanted to know if something is wrong or the use case is not supported any more.

This is the error log:

python gdrivedl.py https://drive.google.com/file/d/[ID] Traceback (most recent call last): File "gdrivedl.py", line 250, in <module> main() File "gdrivedl.py", line 246, in main gdrive.process_url(url, directory=args.directory_prefix, filename=args.output_document) File "gdrivedl.py", line 133, in process_url self.process_file(id, directory, filename=filename) File "gdrivedl.py", line 182, in process_file return self.process_file(id, directory, filename=filename, confirm=confirm.group(1)) File "gdrivedl.py", line 186, in process_file file_path = os.path.join(directory, sanitize(filename)) File "gdrivedl.py", line 64, in sanitize filename = unicodedata.normalize("NFKD", filename) TypeError: normalize() argument 2 must be unicode, not str

Hope this helps! Thank you

TODO: publish to pypi

HTML

I did exactly what you said but it is downloading the HTML and saving it as .file . What to do?

does't support chinese name file

when i download chinese name file,got this
Directory: E:\gbook [Exists]
Directory: E:\gbook\200___________________________________________________ 1 [Created]
E:\gbook\200___________________________________________________ 1_miji8miji8.txt
[==================================================] 0.00MB/0.00MB
E:\gbook\200_____ 1\YP139_______________________.pdf
[==================================================] 1.00MB/1.00MB
E:\gbook\200___________________________________________________ 1\YP140____________________.pdf
[==================================================] 8.00MB/8.00MB
E:\gbook\200___________________________________________________ 1\YP141__________________________.pdf

not all files from folder are downloaded

Hi Matt,

You have made a very nice program !

I am trying to use it for downloading artwork into RetroPie.
It works but It does not get all the zip files in the folder.
The folder is https://drive.google.com/drive/folders/1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3
I tried also this https://drive.google.com/drive/folders/1NshkplPGDMGdSKN92kEZJRL8yi80aRDJ

It would be nice if you could take a look at this.
Thank you on forehand.

Issues when data downloading

Dear developer, since yesterday I have been experiencing an error when trying to download some data using a public link on Drive. When the download starts, the following error appears after some files have been downloaded:

Traceback (most recent call last):

  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 414, in <module>
    main()
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 409, in main
    gdrive.process_url(
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 165, in process_url
    self.process_folder(id, directory)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 194, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 190, in process_folder
    self.process_file(
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 260, in process_file
    with self._request(FILE_URL.format(id=id, confirm=confirm)) as resp:
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/gdrivedl/gdrivedl.py", line 146, in _request
    f = self._opener.open(req)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 557, in error
    result = self._call_chain(*args)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/mnt/home/soft/soft_bio_267/programs/x86_64/pyenv/.pyenv/versions/3.10.8/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

Could you help me understand what is happening? Thank you in advance for your time and assistance.

ERROR: Quota exceeded for this file

Hi,

I want to download a file of 166GB from google drive using the gdrivedl. However, it shows the error ERROR: Quota exceeded for this file.

I wonder if there are any solutions for it. Thanks!

Possiblity add option to read url from txt file?

is it possible to read the urls of a file .txt?

TypeError: 'in ' requires string as left operand, not bytes

Hi Matt,

When there are no files that were already downloaded then the command give a traceback error.
Command (keeping the same as always):

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - https://drive.google.com/drive/folders/15LrYXncMR9afhcauhzS6mBANCCZzHQ25 -m -P /home/pi/RetroPie/roms/konamih

Gives :
Traceback (most recent call last):
File "", line 452, in
File "", line 447, in main
File "", line 173, in process_url
File "", line 201, in process_folder
File "", line 306, in process_file
TypeError: 'in ' requires string as left operand, not bytes

If there are already files everything seems to be fine :
INFO: /home/pi/RetroPie/roms/konamih/gamelist.xml [Exists]
INFO: /home/pi/RetroPie/roms/konamih/media/emulationstation/Kbilly.mp4 [Exists]
INFO: /home/pi/RetroPie/roms/konamih/media/emulationstation/kbilly.png [Exists]
INFO: /home/pi/RetroPie/roms/konamih/media/emulationstation/kbilly1.png [Exists]
INFO: /home/pi/RetroPie/roms/konamih/media/emulationstation/Kblades.mp4 [Exists]
INFO: /home/pi/RetroPie/roms/konamih/media/emulationstation/kblades.png [Exists]
etc.

Can you have a look ?
Thanks

ERROR: content-disposition not found and confirm=t did not work

Hi Matt,

Seems something has changed on the gdrive server.
Here is an example of downloading gamelists + media :

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - https://drive.google.com/drive/folders/15LrYXncMR9afhcauhzS6mBANCCZzHQ25 -m -P /home/pi/RetroPie/roms/konamih

As of now the command doesn't work anymore and I get the error.
Debugging information is :

DEBUG: Requesting: https://drive.google.com/embeddedfolderview?id=15LrYXncMR9afhcauhzS6mBANCCZzHQ25#list
DEBUG: Requesting: https://docs.google.com/uc?export=download&id=1l4gA4qlVqbtgABBpLHZjKGnZjVmowVid&confirm=
DEBUG: Requesting: https://docs.google.com/uc?export=download&id=1l4gA4qlVqbtgABBpLHZjKGnZjVmowVid&confirm=t
ERROR: content-disposition not found and confirm=t did not work

I would appreciate if you could have a look.

support service account

TODO: Make more friendly to use programmatically (not via cli)

ERROR: content-disposition not found

Hi Matt,

Seems there is a problem that we never encountered earlier.
Perhaps google-drive has changed something.

When downloading this :
https://drive.google.com/drive/folders/1-74gaUnnVA1anVhtIo0zBDN0NeR3C_zD
It will present the error when it should download the gamelist.xml.
Or this :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m
It will present the error when it encounters the first gamelist.xml of the first directory (allin1) and then stops the download.

Can you have a look at this problem ?

REQUEST: Add "--spider" option #enhancement

Add an option to list existing contents (files/folders names, sizes, modified dates) but without downloading.
Similar to "wget --spider" option. See: https://www.gnu.org/software/wget/manual/wget.html#index-spider
Thanks!

Can't use with python 3

Hello thank you for this awesome script. It works like magic after hours of wget to google returning me html files.

On the readme it says that it supports python3 but I don't know how to run it with py3. It kept throwing the error requesting a byte object instead of a str. Sorry I haven't read the script. Maybe you can help me understand that.

Stuck no progress

The script gets stuck when I use the following link.

https://drive.google.com/drive/folders/1YT-L8lR-FVs5AqcFUdXhAf5Htw7bVppx

It doesn't crash or give me an error, and works perfectly for all other links.

Check it out please.

support for aria2

support for aria2, thanks

can't download drive which has special characters

Hi, I tried to download drive's folder which has Japanese contain special character and it got error like this

Traceback (most recent call last):
  File "G:\gdrivedl-master\gdrivedl.py", line 241, in <module>
    main()
  File "G:\gdrivedl-master\gdrivedl.py", line 237, in main
    gdrive.process_url(url, directory=args.directory_prefix, filename=args.output_document)
  File "G:\gdrivedl-master\gdrivedl.py", line 128, in process_url
    self.process_folder(id, directory)
  File "G:\gdrivedl-master\gdrivedl.py", line 150, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "G:\gdrivedl-master\gdrivedl.py", line 150, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "G:\gdrivedl-master\gdrivedl.py", line 150, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "G:\gdrivedl-master\gdrivedl.py", line 148, in process_folder
    self.process_file(id, directory, filename=sanitize(item_name))
  File "G:\gdrivedl-master\gdrivedl.py", line 184, in process_file
    os.makedirs(directory)
  File "C:\Program Files\Python39\lib\os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '.\\2019\\10-2019\\ちょっなんかコレ違くね!?'

And

 Traceback (most recent call last):
  File "G:\gdrivedl-master\gdrivedl.py", line 241, in <module>
    main()
  File "G:\gdrivedl-master\gdrivedl.py", line 237, in main
    gdrive.process_url(url, directory=args.directory_prefix, filename=args.output_document)
  File "G:\gdrivedl-master\gdrivedl.py", line 128, in process_url
    self.process_folder(id, directory)
  File "G:\gdrivedl-master\gdrivedl.py", line 150, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "G:\gdrivedl-master\gdrivedl.py", line 148, in process_folder
    self.process_file(id, directory, filename=sanitize(item_name))
  File "G:\gdrivedl-master\gdrivedl.py", line 184, in process_file
    os.makedirs(directory)
  File "C:\Program Files\Python39\lib\os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '.\\【Breaking News】Broadcast accident during live broadcast?'

Infinite loop error on folders and file links

It just keeps doing this for any link I try. Did google drive change something?

PS C:\Users\devbox\Desktop\9-2-2022\gdrivedl> py gdrivedl.py https://drive.google.com/drive/folders/1ir6wuuU8W2aver__YPnxCbiQN7demDVx?usp=sharing
Traceback (most recent call last):
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 400, in <module>
    main()
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 395, in main
    gdrive.process_url(
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 165, in process_url
    self.process_folder(id, directory)
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 194, in process_folder
    self.process_folder(id, os.path.join(directory, sanitize(item_name)))
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 190, in process_folder
    self.process_file(
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 260, in process_file
    with self._request(FILE_URL.format(id=id, confirm=confirm)) as resp:
  File "C:\Python310\lib\contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "C:\Users\devbox\Desktop\9-2-2022\gdrivedl\gdrivedl.py", line 146, in _request
    f = self._opener.open(req)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 749, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Python310\lib\urllib\request.py", line 557, in error
    result = self._call_chain(*args)
  File "C:\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Python310\lib\urllib\request.py", line 738, in http_error_302
    raise HTTPError(req.full_url, code,
urllib.error.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

Some drive links are not working

Hi,

Thanks for your great script, it has a lovely advantage, it downloads folders, and files if the format matches:

https://drive.google.com/file/d/ID/view

However when I try links with uc?id= they don't work...

Example:
https://drive.google.com/u/0/uc?id=1SuZkr3Uge4Td756K1f5IYM4vR9_zxgPG&export=download
https://drive.google.com/uc?id=1PQ10390fQxjySik3tnwxpTcuBuYAQ2gx

I get:

ERROR: That id XXXXXXXXXX returned an unknown url https://drive.google.com/uc?id=XXXXXXXXXXXXXX

Could you please fix this?

Thanks in advance

INFO: /home/rog/RetroPie/roms/all_in1/media/emulationstation/zudugo.png [Exists]
ERROR: content-disposition not found

After that it stops from downloading the other directories using this command ( edit user to your needs, here it is "pi" ):

curl https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py | python3 - https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m -m -P "/home/pi/RetroPie/roms"

Can you have a look ?

matthuisman / gdrivedl Goto Github PK

gdrivedl's People

Contributors

Stargazers

Watchers

Forkers

gdrivedl's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs