tanaikech / goodls

This is a CLI tool to download shared files and folders from Google Drive.

goodls

Overview

This is a CLI tool to download shared files and folders from Google Drive. For large file, the resumable download can be also run.

Methods

1. Download shared files from the shared URL without the authorization.

We have already known that the shared files on Google Drive can be downloaded without the authorization. But when the size of file becomes large (about 40MB), it requires a little ingenuity to download the file. It requires to access 2 times to Google Drive. At 1st access, it retrieves a cookie and a code for downloading. At 2nd access, the file is downloaded using the cookie and code. I created this process as a CLI tool.

2. Download all shared files with the folder structure from the shared folder.

This method uses API key.

There are sometimes the situation for downloading files in a shared folder. But I couldn't find the CLI applications for downloading files in the shared folder. So I implemented this. But when in order to retrieve the file list from the shared file, Drive API is required to be used. In order to use Drive API, it is required to use OAuth2, Service account and API key. So I selected to use API key which is the simplest way. This CLI tool can retrieve the file list in the shared folder using API key and download all files in the shared folder. From version 1.2.2, you can retrieve only files with the specific mimeType from the folder.

3. Run resumable download for large files.

This method uses API key.

At a recent proposal, I knew the requirement of the resumable download of shared file. So I implemented this.

How to Install

Download an executable file of goodls from the release page and import to a directory with path.

or

Use go get.

$ go install github.com/tanaikech/goodls@latest

• GO111MODULE=on

Usage

1. Download shared files

The image used for this demonstration was created by k3-studio

You can use this just after you download or install goodls. You are not required to do like OAuth2 process.

$ goodls -u [URL of shared file on Google Drive]

• Help
  • $ goodls --help
• Options
  • -e
    • Extension of output file. This is for only Google Docs (Spreadsheet, Document, Presentation). Default is pdf. When ms is used, the shared Google Docs can be downloaded as Microsoft Docs.
    • Sample :
      • $ goodls -u https://docs.google.com/document/d/#####/edit?usp=sharing -e txt
  • -f
    • Filename of file which is output. When this was not used, the original filename on Google Drive is used.
    • Sample :
      • $ goodls -u https://docs.google.com/document/d/#####/edit?usp=sharing -e txt -f sample.txt
• URL is like below.
  • In the case of Google Docs (Spreadsheet, Document, Slides)
    • https://docs.google.com/spreadsheets/d/#####/edit?usp=sharing
    • https://docs.google.com/document/d/#####/edit?usp=sharing
    • https://docs.google.com/presentation/d/#####/edit?usp=sharing
  • In the case of except for Google Docs
    • https://drive.google.com/file/d/#####/view?usp=sharing
  • In the case of webContentLink
    • https://drive.google.com/uc?export=download&id=###

File with several URLs

If you have a file including URLs, you can input the URL data using standard input and pipe as follows. If wrong URL is included, the URL is skipped.

$ cat sample.txt | goodls

or

$ goodls < sample.txt

sample.txt

https://docs.google.com/spreadsheets/d/#####/edit?usp=sharing
https://docs.google.com/document/d/#####/edit?usp=sharing
https://docs.google.com/presentation/d/#####/edit?usp=sharing

When you download shared files from Google Drive, please confirm whether the files are shared.

2. Download all files from shared folder

When above structure is downloaded, the command is like below. At that time, the folder ID is the folder ID of "sampleFolder1".

Files are downloaded from the shared folder. In this demonstration, the fake folder ID and API key are used.

Retrieve API key

In order to use this, please retrieve API key as the following flow.

1. Login to Google.

2. Access to https://console.cloud.google.com/?hl=en.

3. Click select project at the right side of "Google Cloud Platform" of upper left of window.

   • If you cannot see the project, please try to access to https://console.cloud.google.com/cloud-resource-manager?hl=en. You can also create new project at there. - When you created new project there, please click the left of "Google Cloud Platform". You can see it like 3 horizontal lines. By this, a side bar is opened. At there, please select "API & Services" -> "Library". After this, please check the following flow from Click "Library" at left side..

4. Click "NEW PROJECT"

   1. Input "Project Name".
   2. Click "CREATE".
   3. Open the created project.
   4. Click "Enable APIs and get credentials like keys".
   5. Click "Library" at left side.
   6. Input "Drive API" in "Search for APIs & Services".
   7. Click "Google Drive API".
   8. Click "ENABLE".
   9. Back to https://console.cloud.google.com/?hl=en.
   10. Click "Enable APIs and get credentials like keys".
   11. Click "Credentials" at left side.
   12. Click "Create credentials" and select API key.
   13. Copy the API key. You can use this API key.

Registering API key to environment variable

• When API key is registered to the environment variable. When GOODLS_APIKEY as the environment variable is set API key, goodls uses API key from the environment variable.
• If API key is used with the option at the command line, the priority of option is higher than the environment variable.

For example, in the case of bash, please add a following code to .bashrc.

export GOODLS_APIKEY=### your API key ###

Download

When the URL of shared folder is https://drive.google.com/drive/folders/#####?usp=sharing, you can download all files in the folder by the following command.

$ goodls -u https://drive.google.com/drive/folders/#####?usp=sharing -key [APIkey]

• Project files cannot be downloaded by API key. If you want to download the project files, you can download them by ggsrun, because ggsrun uses OAuth2.
• This new function uses the Go library of go-getfilelist.
• When the option of --NoProgres, -np is used, the progress information is not seen. This is a silent mode.
• If the files which are tried to be downloaded are existing, an error occurs. But when you use the option --overwrite and --skip, the files are overwritten and skipped, respectively.
• If you use the option -m, you can retrieve only files with the specific mimeType from the folder. For example, when you want to download PDF and PNG files, please run like below.
  • $ goodls -u [URL] -key [APIkey] -m "application/pdf,image/png"

Retrieve information of file and folder

When you want to retrieve the information of file and folder, you can do it as follows.

For file

$ goodls -u https://docs.google.com/spreadsheets/d/#####/edit?usp=sharing -key [APIkey] -i

For folder

$ goodls -u https://drive.google.com/drive/folders/#####?usp=sharing -key [APIkey] -i

3. Resumable download of shared file

When you use this option, at first, please retrieve API key. About how to retrieve API key, you can see at here.

When you want to download 100 MBytes of the shared file, you can use the following command.

$ goodls -u [URL of shared file on Google Drive] -key [APIkey] -r 100m

• Please use the option -r. In this sample, 100m means to download 100 MBytes of the shared file.
  • If you want to download 1 GB, please use -r 1g.
  • If you use -r 1000000, 1 MByte of the file will be able to be downloaded.

You can see the actual running of this option at the following demonstration movie.

In this demonstration, the following command is run 3 times.

$ goodls -u https://drive.google.com/drive/folders/abcdefg?usp=sharing -key htjklmn -r 80m

• At 1st run, the data of 0 - 80 MBytes is downloaded.
  • You can see New download at "Current status".
• At 2nd run, the data of 80 - last is downloaded.
  • You can see Resumable download at "Current status".
• At 3rd run, the download has already been done. So the checksum is shown.
  • You can see Download has already done. at "Current status".

Note

• Reason that API key is used for this.
  • When it accesses to the shared file without the authorization, the file size and md5checksum cannot be retrieved. So in order to use Drive API, I adopted to use API key.
• Reason that the download size is inputted every time.
  • When this option is run 1 time, 1 quota is used for Drive API. So I adopted this way.

Q&A

• I want to download shared projects from user's Google Drive.

  • You can download shared projects using ggsrun.
  • ggsrun can also download shared files from other user's Google Drive using Drive API which needs the access token.

• I want to download all files including the standalone projects from the shared folder and own folder.

  • You can achieve it using ggsrun.

• I want to use this with "Dockerfile build".

  • In this case, it has already been mentioned by rahulpowar at #15. Please check it.

---

Licence

MIT

Author

Tanaike

If you have any questions and commissions for me, feel free to tell me.

Update History

• v1.0.0 (January 10, 2018)

  1. Initial release.

• v1.0.1 (January 11, 2018)

  1. In order to download several files, a datafile including URLs using Standard Input and Pipe have gotten to be able to be inputted.

• v1.0.2 (May 10, 2018)

  1. Files with large size has gotten to be able to be used.
     • In order to download files with large size (several gigabytes), files are saved by chunks.

• v1.0.3 (September 4, 2018)

  1. When the files are downloaded, the progress of downloading got to be able to be displayed.
     • This demonstration can be seen at Demo.
     • If the new option of --np is used, the progress is not displayed.

• v1.1.0 (November 4, 2018)

  1. By using API key, files from the shared folder got to be able to be downloaded while keeping the folder structure.
     • This demonstration can be seen at Demo.
  2. By using API key, the information of shared file and folder can be also retrieved.
  3. About the option of --extension and -e, when -e ms is used, Google Docs (Document, Spreadsheet, Slides) are converted to Microsoft Docs (Word, Excel, Powerpoint), respectively.

• v1.1.1 (November 13, 2018)

  	1. Version of [go-getfilelist](https://github.com/tanaikech/go-getfilelist) was updated. Because the structure of ``drive.File`` got to be able to be used, I also updated this application.
  

• v1.2.0 (November 24, 2018)

  1. By using API key, the shared large files can be run the resumable download.
     • This demonstration can be seen at Demo.

• v1.2.1 (November 25, 2018)

  1. API key got to be able to be used by an environment variable. When GOODLS_APIKEY as the environment variable is set API key, goodls uses API key from the environment variable.

• v1.2.2 (December 12, 2018)

  1. When files are downloaded from a specific folder, it got to be able to select mimeType. By this, files with the specific mimeType in the specific folder can be retrieved. For this, I updated the go library go-getfilelist.
     • $ goodls -u [URL] -key [APIkey] -m "application/pdf,image/png"

• v1.2.3 (October 31, 2019)

  1. An option for inputting the directory to save the downloaded files was added. When this option is not used, the files are saved to the current working directory.
     • $ goodls -u [URL] -d [directory]

• v1.2.4 (January 3, 2020)

  1. It seems that the specification of github.com/urfave/cli was changed by the update of https://github.com/urfave/cli. By this, when go get -u github.com/tanaikech/goodls is run, an error occurred. So this error was removed.

• v1.2.5 (January 29, 2020)

  1. An option for selecting whether the top directory is created was added.
     • $ goodls -u [URL] --notcreatetopdirectory or $ goodls -u [URL] -ntd
     • When this option is NOT used (default situation), when a folder including sub-folders is downloaded, the top folder which is downloaded is created as the top directory under the working directory. When this option is used, the top directory is not created and all files and sub-folders under the top folder are downloaded under the working directory.
     • This feature request was implemented.

• v1.2.6 (February 23, 2020)

  1. When the files are downloaded from the shared folder, if an error occurs, the download was stopped. In this thread, a request for skipping this error was proposed. This request was implemented at v1.2.6. Please try the option of --skiperror like below.
     • $ goodls -u https://drive.google.com/drive/folders/###?usp=sharing --skiperror

• v1.2.7 (August 21, 2020)

  1. As the URL for downloading the files, webContentLink was added. So from this version, the URL of https://drive.google.com/uc?export=download&id=### got to be able to be used. This is the request from this thread.

• v1.2.8 (February 17, 2022)

  1. Recently, it seems that the specification the process for downloading the shared file on Google Drive has been changed. So I updated goodls for reflecting this. The usage of goodls is not changed.

• v2.0.0 (February 25, 2022)

  1. By changing the specification of methods, drive.New() and transport.APIKey were deprecated. By this, I updated go-getfilelist. In this version, I used this updated library to goodls. And also, drive.NewService() is used instead of drive.New().

• v2.0.1 (February 26, 2022)

  1. A bug for the resumable download was removed.

• v2.0.2 (February 24, 2023)

  1. Modified go.mod and go.sum.

• v2.0.3 (April 5, 2023)

  1. Forgot to update the version number and modified it. And, built the sources with the latest version. Ref

• v2.0.4 (March 9, 2023)

  1. From January 2024, it seems that the specification of the process for downloading a large shared file on Google Drive has been changed. So I updated goodls to reflect this. The usage of goodls has not changed. In the current stage, when a large share file is downloaded, it is required to click the following button.

  <form
    id="download-form"
    action="https://drive.usercontent.google.com/download"
    method="get"
  >
    <input
      type="submit"
      id="uc-download-link"
      class="goog-inline-block jfk-button jfk-button-action"
      value="このままダウンロード"
    />
    <input type="hidden" name="id" value="fileId" />
    <input type="hidden" name="export" value="download" />
    <input type="hidden" name="authuser" value="0" />
    <input type="hidden" name="confirm" value="t" />
    <input type="hidden" name="uuid" value="uuId" />
    <input type="hidden" name="at" value="some value" />
  </form>

  In this version, the URL obtained by this click is created, and the created URL is used for downloading it.

• v2.0.5 (March 10, 2023)

  1. From this version, when the API key is used, the large file is downloaded by the API key. Because the specification for downloading the shared large file is sometimes changed. When the API key is not used, the shared large file is downloaded by the current specification (v2.0.4). If the shared large file cannot be downloaded by the change of the specification on Google's side, please use the API key and test it again. The usage of goodls has not changed.

TOP

goodls's Issues

Feature request: display download speed and percentage (d -j)

It would be a very nice addition to see the current or average download speed as well as percentage of file completion.
Seeing the progress in bytes is very difficult to understand at first glance.
Moving to Mbytes and above dynamically or having an option to choose the unit would be also nice to have.
Thanks.

[bug] When downloading from a file with multiple folders they gets folded into each other

When downloading folders from a file with several URLs the next downloaded folder is created inside the last created subfolder.
In other words, the current directory gets shifted and the downloaded folders get folded into each other.

#sample.txt
https://drive.google.com/drive/folders/<PUBLIC FOLDER A>
https://drive.google.com/drive/folders/<PUBLIC FOLDER B>
https://drive.google.com/drive/folders/<PUBLIC FOLDER C>

Then run

goodls < sample.txt

Here is the actual file structure:

- PUBLIC FOLDER A
  - Subfolder A1
  - Subfolder A2
    - PUBLIC FOLDER B
      - Subfolder B1
      - Subfolder B2
	    - PUBLIC FOLDER C

Expected file structure:

- PUBLIC FOLDER A
  - Subfolder A1
  - Subfolder A2
- PUBLIC FOLDER B
  - Subfolder B1
  - Subfolder B2
- PUBLIC FOLDER C

go get error

Hi! I'm new to go, so couldn't figure this out :(

I just installed go.

$ go version
go version go1.16.3 linux/amd64

But cannot install goodls.

$ go get -u github.com/tanaikech/goodls
# github.com/tanaikech/goodls
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:380:12: cannot use []*cli.Author{...} (type []*cli.Author) as type[]cli.Author in assignment
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:388:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:393:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:399:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:404:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:409:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:414:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:419:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:424:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:429:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:429:4: too many errors

Is Shared Folder Resumable Download Feature Exists?

I provided my CLI with -r option but when my accumulated downloaded bytes exceeds my expectation it just keeps going.

Is the resumable feature only for files? Not folders?

go get error is there anyother way to install

package google.golang.org/api/drive/v3: unrecognized import path "google.golang.org/api/drive/v3" (https fetch: Get https://google.golang.org/api/drive/v3?go-get=1: dial tcp: lookup google.golang.org: no such host)
package google.golang.org/api/googleapi: unrecognized import path "google.golang.org/api/googleapi" (https fetch: Get https://google.golang.org/api/googleapi?go-get=1: dial tcp: lookup google.golang.org: no such host)

cd .; git clone -- https://github.com/urfave/cli /home/nimnim/go/src/github.com/urfave/cli

Cloning into '/home/nimnim/go/src/github.com/urfave/cli'...
fatal: unable to access 'https://github.com/urfave/cli/': Could not resolve host: github.com
package github.com/urfave/cli: exit status 128
package golang.org/x/crypto/ssh/terminal: unrecognized import path "golang.org/x/crypto/ssh/terminal" (https fetch: Get https://golang.org/x/crypto/ssh/terminal?go-get=1: dial tcp: lookup golang.org: no such host)
package google.golang.org/api/googleapi/transport: unrecognized import path "google.golang.org/api/googleapi/transport" (https fetch: Get https://google.golang.org/api/googleapi/transport?go-get=1: dial tcp: lookup google.golang.org: no such host)

Can't download a large file: 75GB

Hi,
When I tried to download 75GB shared file, I've got an error.

./goodls_linux_amd64 -u https://drive.google.com/file/d/1PcRscAlaXK9XipeCVi5RXXAYq6/view?usp=sharing -key AIzaSyAlmESKBTuj-RjbttIfc -r 65g

Error:
file ID [ 1PcRscAlaXK9XiXXXXXXXXi5RXXAYq6 ] cannot be downloaded

Would you help me out with this?

Thanks in advance.

Can this download the entire Google Drive Folder?

For example downloading a folder full of pictures. Does not seem to work.

$ goodls -u 'https://drive.google.com/drive/folders/0By-NOo-lCyxBREtydWtzR0VvcWc?usp=sharing'
Error: Error: URL is wrong.

$ goodls -u 'https://drive.google.com/open?id=0By-NOo-lCyxBREtydWtzR0VvcWc'
Error: Error: URL is wrong

[Feature Request] Implementation of parallel downloading of files

Hi,

Tried to download a folder of 24KB from my personal drive based on around ~5 files and it took it 7 seconds to download (though this folder was small and only had small size files, on large files and in much count, it'll take much time). Can we please implement the feature to download multiple files in parallel?

It would be great to speed up the process and it will also consume less time.

POC (of time taken):

[Feature Request] Download the contents of a share into a directory ignoring the share's root folder

For example:
If https://drive.google.com/drive/folders/123123123 is 'Folder_1' with the contents of 'SubFolder_1', 'file_1', 'file_2'

goodls -u https://drive.google.com/drive/folders/123123123 -d /localfolder/here/

I would like the end result to be:

/localfolder/here/SubFolder_1
/localfolder/here/SubFolder_1/file_1
/localfolder/here/SubFolder_1/file_2

and not:

/localfolder/here/Folder_1/SubFolder_1
/localfolder/here/Folder_1/SubFolder_1/file_1
/localfolder/here/Folder_1/SubFolder_1/file_2

Possible to add a switch to ignore the root folder of the share?
Or possibly if -d contains a trailing slash, ignore the root folder of the share?

[Feature Request -- declined] Can't download files from Google's shared drive

Hey,

Hope you're doing fine, I'm trying to download files from my shared drive and the following error occurs while using the tool. I'm not sure if this is a bug or this feature isn't implemented.

POC:

Output:

~$ goodls -u https://drive.google.com/drive/folders/1ww4T0Fso4DcefAT343CRr_D4xnWDLR0l?usp=sharing
Error: googleapi: Error 403: The attempted action requires shared drive membership., teamDriveMembershipRequired

---

You can use the above URL for testing (in case it's a bug or need to implement this feature). Let me know if my input is required anywhere.

unable to install

Hello, i'm unable to install the package .

i get "too many errors" while trying to install with

$ go install github.com/tanaikech/goodls@latest

and none of the executables work for me

❯ go version
go version go1.17 darwin/arm64

Feature Request : Renaming files downloaded in bulk downloads

Option to rename files downloaded via multiple URL file inputs. So while passing list of URL via file like, goodls < list.csv would it be possible to provide an option to rename the downloaded files ?

PS: All following discussion is based on previous description

Previous description

I was trying to download bulk files with `goodls` and renaming them by passing information from CSV file. But the `-f` doesn't seem to have any effect.

If I understand correctly, file input doesn't support file naming. It would be great if you could let me know what I'm missing here.

while IFS=, read -r id gdlink; do
   echo "Fetching $gdlink to $id"
   goodls -u "$gdlink" -f "$id"
done < goodls_sample_products.csv

Output

Fetching //drive.google.com/uc?export=download&id=1HGkFBVuKHx5jokRf5ToH_03UOhVB6ScX to 3911401000000874007
Downloading (bytes)... 80817
{"Filename": "1.jpg", "Type": "file", "MimeType": "image/jpeg", "FileSize": 80817}
Downloading (bytes)... 24986
{"Filename": "LATINE BEIGE.jpg", "Type": "file", "MimeType": "image/jpeg", "FileSize": 24986}

Sample data

3911401000000874007,//drive.google.com/uc?export=download&id=1HGkFBVuKHx5jokRf5ToH_03UOhVB6ScX
3911401000000874003,//drive.google.com/uc?export=download&id=1mKPzSb2_XdERUwcKYkT5DV3FVhvnPWmv
3911401000000872221,//drive.google.com/uc?export=download&id=1pplVG_66kUiAarPo98X8dfy3vCpuO9bS

Resuming interrupted downloads?

First of all, this program is lovely—before finding it whenever I needed to do this, I was always digging around my browser trying to find the right cookies to feed to curl, and eventually I'd get it, but it was a massive pain.

Anyway, I'm trying to download some VERY large files from my Google Drive. One is >250GB. I expect it to take several days, unfortunately, as I do not have a very speedy connection. Inevitably, unfortunately, there are brief interruptions in the connection, and goodls has quit when this happens, and when I restart, it starts again from byte 0. Is there no way to resume a download at a certain offset using their APIs?

Terminal check breaks use in Dockerfile

Had trouble using this in a Dockerfile build and finally figured out that the check here:

goodls/goodls.go

Line 340 in 3c78fe8

if terminal.IsTerminal(int(syscall.Stdin)) {

... meant I had to echo the URL and pipe which is a bit unexpected as below.

RUN    echo 'https://drive.google.com/file/d/13oKu33Fj0cwH661apA2_vsjn9rT1NkEb/view?usp=sharing' | /tmp/goodls_linux_amd64 --np -f vocoder.pth.tar && \
    echo 'https://drive.google.com/file/d/1qN7vQRIYkzvOX_DtiZtTajzoZ1eW1-Eg/view?usp=sharing' | /tmp/goodls_linux_amd64 --np -f vocoder.json 

Why are MIME types required for folder downloads?

Gathering MIME types for folder downloads is a huge pain in the neck, is there a reason this is required?

Is it at all possible to make this support wildcards at least so things like 'image/*' works?

Download stopped when the having quota exceeded error

Problem

I tried to download a folder from Google Drive, which contains many files. After starting the download, the following error is shown and no file is downloaded.

Download files from a folder '(removed due to privacy)'.
There are 1714 files and 0 folders in the folder.
Starting download.
{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "downloadQuotaExceeded",
    "message": "The download quota for this file has been exceeded."
   }
  ],
  "code": 403,
  "message": "The download quota for this file has been exceeded."
 }
}


real	1m23.767s
user	0m0.137s
sys	0m0.016s

Expected result

The download process shall continue, but just skipping this file, because there are many other files to download, or at least, showing which file is having the problem, so that I can manually handle problem.

Out of memory when downloading very large files

When downloading very large files from Google I get the following error:

Now downloading.
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x759761, 0x16)
        C:/Go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc620600000, 0x200000000, 0x0, 0x92ded8)
        C:/Go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x914e60, 0x200000000, 0x427812)
        C:/Go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0x914e60, 0x100000, 0x0)
        C:/Go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0x914e60, 0x100000, 0x92dee8, 0xc420041eb0)
        C:/Go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0x914e60, 0x100000, 0xc420320101, 0x1)
        C:/Go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()
        C:/Go/src/runtime/mheap.go:733 +0x4d
runtime.systemstack(0xc420041f08)
        C:/Go/src/runtime/asm_amd64.s:360 +0xab
runtime.(*mheap).alloc(0x914e60, 0x100000, 0xc420010101, 0xc420000180)
        C:/Go/src/runtime/mheap.go:732 +0xa1
runtime.largeAlloc(0x1fffffe00, 0xc420000101, 0x100000000000004)
        C:/Go/src/runtime/malloc.go:827 +0x98
runtime.mallocgc.func1()
        C:/Go/src/runtime/malloc.go:722 +0x46
runtime.systemstack(0xc420029300)
        C:/Go/src/runtime/asm_amd64.s:344 +0x79
runtime.mstart()
        C:/Go/src/runtime/proc.go:1135

goroutine 1 [running]:
runtime.systemstack_switch()
        C:/Go/src/runtime/asm_amd64.s:298 fp=0xc420075650 sp=0xc420075648 pc=0x454b40
runtime.mallocgc(0x1fffffe00, 0x6e2be0, 0xc420075701, 0x0)
        C:/Go/src/runtime/malloc.go:721 +0x7ae fp=0xc4200756f8 sp=0xc420075650 pc=0x4108ee
runtime.makeslice(0x6e2be0, 0x1fffffe00, 0x1fffffe00, 0xc520600000, 0xc4200b2528, 0xc4200b25e8)
        C:/Go/src/runtime/slice.go:54 +0x77 fp=0xc420075728 sp=0xc4200756f8 pc=0x440a07
bytes.makeSlice(0x1fffffe00, 0x0, 0x0, 0x0)
        C:/Go/src/bytes/buffer.go:231 +0x77 fp=0xc420075768 sp=0xc420075728 pc=0x46b1a7
bytes.(*Buffer).ReadFrom(0xc420075810, 0x7f76cbe1d4a8, 0xc4200b2500, 0xc42039e200, 0x0, 0x200)
        C:/Go/src/bytes/buffer.go:203 +0x291 fp=0xc4200757e0 sp=0xc420075768 pc=0x46b0e1
io/ioutil.readAll(0x7f76cbe1d4a8, 0xc4200b2500, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
        C:/Go/src/io/ioutil/ioutil.go:33 +0x12c fp=0xc420075888 sp=0xc4200757e0 pc=0x4dfbec
io/ioutil.ReadAll(0x7f76cbe1d4a8, 0xc4200b2500, 0xc4200b2500, 0x7f76cbe1d4a8, 0xc4200b2500, 0x5d, 0x5d)
        C:/Go/src/io/ioutil/ioutil.go:42 +0x3e fp=0xc4200758d8 sp=0xc420075888 pc=0x4dfcce
main.(*para).saveFile(0xc420075c98, 0xc42043a000, 0x0, 0x0)
        D:/build/goodls/goodls.go:51 +0x149 fp=0xc4200759b0 sp=0xc4200758d8 pc=0x6aac79
main.(*para).downloadLargeFile(0xc420159c98, 0xc4200fe210, 0xa9)
        D:/build/goodls/goodls.go:87 +0x22c fp=0xc420075a40 sp=0xc4200759b0 pc=0x6ab45c
main.(*para).download(0xc420159c98, 0x7ffd2e3393c6, 0x52, 0x7ffd2e3393c6, 0x52)
        D:/build/goodls/goodls.go:166 +0x2b3 fp=0xc420075b00 sp=0xc420075a40 pc=0x6abef3
main.handler(0xc4200b06e0)
        D:/build/goodls/goodls.go:192 +0x725 fp=0xc420075d30 sp=0xc420075b00 pc=0x6ac9b5
github.com/urfave/cli.HandleAction(0x6e1620, 0x769090, 0xc4200b06e0, 0xc42005e1e0, 0x0)
        C:/Go/projects/src/github.com/urfave/cli/app.go:492 +0x7c fp=0xc420075d58 sp=0xc420075d30 pc=0x699cbc
github.com/urfave/cli.(*App).Run(0xc420010d00, 0xc42000e140, 0x5, 0x5, 0x0, 0x0)
        C:/Go/projects/src/github.com/urfave/cli/app.go:264 +0x635 fp=0xc420075f38 sp=0xc420075d58 pc=0x697db5
main.main()
        D:/build/goodls/goodls.go:255 +0x84 fp=0xc420075f80 sp=0xc420075f38 pc=0x6acf54
runtime.main()
        C:/Go/src/runtime/proc.go:195 +0x226 fp=0xc420075fe0 sp=0xc420075f80 pc=0x42b7f6
runtime.goexit()
        C:/Go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420075fe8 sp=0xc420075fe0 pc=0x4576c1

goroutine 51 [running]:
        goroutine running on other thread; stack unavailable
created by net/http.(*http2Transport).newClientConn
        C:/Go/src/net/http/h2_bundle.go:7053 +0x6b9

This happens after some minutes. No data has been saved then.

No URL Data for Non-interactive session

When running goodls from GitHub actions, it runs in a non-interactive terminal. When the URL parameter is passed to the goodls in a GitHub Workflow, it is ignored and instead reads from STDIN. It displays the error "No URL data. Please check help". For now, I just have a workaround where it echoes the URL and pipes it through to goodls, but that is not ideal. It is the check on line 340 of goodls.go that is causing the error.

download folders with "resourcekey" in their URLs

Hi,

I am trying to download a folder from Google Drive using your tool. After setting up the API key, I run the following command:

./goodls_linux_amd64 -u "https://drive.google.com/drive/folders/<FOLDER_ID>?resourcekey=<RESOURCE_KEY>&usp=usp_sharing" --key <API_key>

or

./goodls_linux_amd64 -u "https://drive.google.com/drive/folders/<FOLDER_ID>&usp=sharing" --key <API_key>

but I am always getting the following error:

googleapi: Error 404: File not found: 0B2X7pBsRBLDJc05EdlhHOUVtRmM., notFound

Am I doing something wrong?

Thanks
Kostas

Unable to compile for armhf on Raspberry Pi 3B

Since there's no arm binary already uploaded I tried to compile from source on a Raspberry Pi 3B running Raspbian Stretch where the repo go version is 1.7.4. I set $GOPATH to a home subfolder and run "go get -u github.com/tanaikech/goodls" as the manual suggests but I get

package google.golang.org/api/transport/http/internal/propagation: no buildable Go source files in /home/kostis/GoLang/src/google.golang.org/api/transport/http/internal/propagation

So, is there a way to build for armhf? Is perhaps a higher Go version required for this source code? Or perhaps a binary could be uploaded for armhf?

file ID [ ... ] is not shared, while the file is existing

Basically, goodls has stopped working for me over the past few months. I always get the same error file ID [ ... ] is not shared, while the file is existing (actual file ID redacted). These are typically large files in the couple GB range. I thought maybe it was a size limit, 24-hour lock out period, but it has been days and days since I've tried a download, and probably a month or two since it actually worked.

Getting 'The requested conversion is not supported'

I ran the following command:

./goodls -u "https://drive.google.com/drive/folders/example?usp=sharing"

Download files from a folder 'ABC'.
There are 1343 files and 294 folders in the folder.

Starting download.

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "badRequest",
    "message": "The requested conversion is not supported.",
    "locationType": "parameter",
    "location": "convertTo"
   }
  ],
  "code": 400,
  "message": "The requested conversion is not supported."
 }
}

Any idea why it is happening and what can be done?

Unable to download shared files

I've a couple of thousand files with their external share link. While trying the URL pattern it always returns URL Error. I'm not familar with go, but I believe checkURL functions is skipping this URL pattern ?

Sample link : https://drive.google.com/uc?export=download&id=1pFOCG0vBrrx5wLedCLWydMoEaP_8301H

Author Unidentified when installing on ubuntu

when I use the installation command
go get -u github.com/tanaikech/goodls

I get the following error:

# github.com/tanaikech/goodls ../go/src/github.com/tanaikech/goodls/goodls.go:361:3: a.Author undefined (type *cli.App has no field or method Author) ../go/src/github.com/tanaikech/goodls/goodls.go:362:3: a.Email undefined (type *cli.App has no field or method Email) ../go/src/github.com/tanaikech/goodls/goodls.go:366:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal: cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:370:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal: cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:375:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal: cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:379:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal: cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:383:17: cannot use cli.StringFlag literal (type cli.StringFlag) as type cli.Flag in array or slice literal: cli.StringFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:387:15: cannot use cli.BoolFlag literal (type cli.BoolFlag) as type cli.Flag in array or slice literal: cli.BoolFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:391:15: cannot use cli.BoolFlag literal (type cli.BoolFlag) as type cli.Flag in array or slice literal: cli.BoolFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:395:15: cannot use cli.BoolFlag literal (type cli.BoolFlag) as type cli.Flag in array or slice literal: cli.BoolFlag does not implement cli.Flag (Apply method has pointer receiver) ../go/src/github.com/tanaikech/goodls/goodls.go:395:15: too many errors

go version:
go version go1.13.5 linux/amd64

Ubuntu version: Ubuntu 16.04.3 LTS

Not opening in windows

Hi! I would like to know why is .exe not opening in windows? thank you!

"URL is wrong" for Colab URLs

Get the public ("anyone can view") sharing URL for the file while in drive.google.com. (This is the same sharing URL one gets from within Colab too.). URL Statrs with a "colab", and goodls script flat-out refuses to try it:

$ ./goodls_linux_amd64 -u https://colab.research.google.com/drive/1SxJJc6LsKrjWAM-HhwPrLJBpUzImO5oX?usp=sharing
URL is wrong

Adding other options via -e and -f doesn't help, always gets just "URL is wrong".

Feature Request: Add "verbose" or "debug" option

I'm currently have to perform a reconciliation loop over a set of files in a directory which requires repeatedly attempting to download content. Unfortunately, while I see the fileId and the name of individual files in the output, i don't see the folder or other data.

e.g.

...
{"Filename": "keys.ogg", "Type": "", "MimeType": "audio/ogg", "FileSize": 4562227}
Downloading (bytes)... 171706 / 171706
{"Filename": "notes.mid", "Type": "", "MimeType": "audio/unknown", "FileSize": 171706}
Downloading (bytes)... 5093014 / 5093014
{"Filename": "rhythm.ogg", "Type": "", "MimeType": "audio/ogg", "FileSize": 5093014}
Downloading (bytes)... 934 / 934
{"Filename": "song.ini", "Type": "", "MimeType": "application/octet-stream", "FileSize": 934}
Downloading (bytes)... 4555503 / 4555503
{"Filename": "song.ogg", "Type": "", "MimeType": "audio/ogg", "FileSize": 4555503}
!! Downloading 'vocals.ogg' (fileId: XXXXXX48Io6JnVQz6s0EML4u8m9mc8PQk) was skipped by an error. Status code is 403.
!! Downloading 'album.png' (fileId: XXXXXXKByh2uw_aX8F-axMZlLWTP8iT1u) was skipped by an error. Status code is 403.
!! Downloading 'drums_1.ogg' (fileId: XXXXXXsWxLmUwdRRc3tbVilEnbPDFbOz1) was skipped by an error. Status code is 403.
!! Downloading 'drums_2.ogg' (fileId: XXXXXX3EClcHnYwQ1dB78x0ANoWQ2rXUt) was skipped by an error. Status code is 403.
!! Downloading 'drums_3.ogg' (fileId: XXXXXXoC2CeSg6lHaOzhHDbqFjJGTrjOO) was skipped by an error. Status code is 403.
!! Downloading 'guitar.ogg' (fileId: XXXXXX8PLtHQTklvwp51lnsaxU9Uk-bME) was skipped by an error. Status code is 403.
!! Downloading 'notes.mid' (fileId: XXXXXXeG_IlyI4bgVLcgSuhJ2q7XCAeQ1) was skipped by an error. Status code is 403.
!! Downloading 'rhythm.ogg' (fileId: XXXXXX_BZcC5i85Yg5aZbrbFIDShCk0Zv) was skipped by an error. Status code is 403.
...

In this situation it would be helpful to more easily identify content which was not retrieved. Honestly, i'm not really opinionated on the mechanism. I assume I could use the fileId information, but it would be challenging to rebuild the directory content on that alone.

Open to suggestions and flexible on how things get done. :D

Compiling to armv6

Can a binary for armv6 (RPI 0 W) can be added to releases? I tried to cross compile it myself on fedora, following instructions from user karser #4 (comment) but still getting error:

# go build /go/src/golang.org/x/net/http2/frame.go:18:2: found packages hpack (encode.go) and main (gen.go) in /go/src/golang.org/x/net/http2/hpack /go/src/google.golang.org/grpc/internal/channelz/types_linux.go:24:2: found packages unix (affinity_linux.go) and main (mkasm.go) in /go/src/golang.org/x/sys/unix /go/src/golang.org/x/text/secure/bidirule/bidirule.go:16:2: found packages bidi (bidi.go) and main (gen.go) in /go/src/golang.org/x/text/unicode/bidi /go/src/golang.org/x/net/idna/idna10.0.0.go:27:2: found packages norm (composition.go) and main (maketables.go) in /go/src/golang.org/x/text/unicode/norm /go/src/golang.org/x/net/http2/transport.go:19:2: cannot find package "io/fs" in any of: /usr/local/go/src/io/fs (from $GOROOT) /go/src/io/fs (from $GOPATH)

Can't download folder

I insert correct API Key but cant download folder. I use window 10 and try both 64 bit and 32 bit goodls ver but not working :(

How to download

When I download an executable how do I import to a directory with path?

Unable to download shared url

@tanaikech The following shared url seems not working. Could you please have a look at this?

$ goodls -v
goodls version 1.2.7

$ goodls -u 'https://drive.google.com/uc?export=download&id=1wON0d7v_wxi0wKc59wM4W_D19vblLwVG'
file ID [ 1wON0d7v_wxi0wKc59wM4W_D19vblLwVG ] is not shared, while the file is existing

Originally posted by @pengbins in #13 (comment)

Proxy support

Super useful to bypass the school or enterprise traffic control mechanisms.

TY

Failed to compile goodls on Ubuntu 20.04.2 LTS.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

$ go version
go version devel go1.17-fd3612e433 Sat Apr 17 00:02:01 2021 +0000 linux/amd64

$ git clone https://github.com/tanaikech/goodls.git goodls.git 
$ cd goodls.git 
$ go install github.com/tanaikech/goodls@latest
go: finding module for package google.golang.org/api/googleapi/transport
go: finding module for package golang.org/x/crypto/ssh/terminal
go: finding module for package google.golang.org/api/drive/v3
go: finding module for package github.com/urfave/cli
go: finding module for package google.golang.org/api/googleapi
go: finding module for package github.com/tanaikech/go-getfilelist
go: found github.com/tanaikech/go-getfilelist in github.com/tanaikech/go-getfilelist v1.0.2
go: found github.com/urfave/cli in github.com/urfave/cli v1.22.5
go: found golang.org/x/crypto/ssh/terminal in golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
go: found google.golang.org/api/drive/v3 in google.golang.org/api v0.56.0
go: found google.golang.org/api/googleapi in google.golang.org/api v0.56.0
go: found google.golang.org/api/googleapi/transport in google.golang.org/api v0.56.0
# github.com/tanaikech/goodls
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:380:12: cannot use []*cli.Author{...} (type []*cli.Author) as type []cli.Author in assignment
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:388:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:393:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:399:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:404:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:409:4: unknown field 'Aliases' in struct literal of type cli.StringFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:414:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:419:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:424:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:429:4: unknown field 'Aliases' in struct literal of type cli.BoolFlag
../../go/pkg/mod/github.com/tanaikech/[email protected]/goodls.go:429:4: too many errors

you are awesome! ❤️

goodls saved my day, it just works.
I want to buy you a beer, so the issue is: the repository is missing a link to send donations 🔥

download to customized folder?

Hi,

I would like to download a folder from my Google-Drive to another directory than my current directory.

The command $ ./goodls -u [url/to/folder/Google_Drive] starts downloading in my current directory!

Is there any option giving a customized path to new directory?

Cheers,

--skip an --overwrite doens't work with -d option

When in use the option -d, the check for existing file seems not working, the file will be downloaded even if exist.
I think the check is made in the current directory not in the output directory

Make a version of this for proton?

this not an issue but rather a request to make a version of gools for proton? https://proton.me/drive protls?

if you do this will be very nice