tanaikech / downloadlargefilesbyurl

DownloadLargeFilesByUrl is a GAS library for downloading large files from URL to Google Drive using Google Apps Script (GAS).

• About Me
• Contact
• Donate

downloadlargefilesbyurl's Issues

Feature request: set the type of output file

I'm using this code to download large csv files.
By default, they are saved to Drive as "text" files, and Drive doesn't know to open them in Sheets. It wants to open them in a text editor instead.
Could we either:
a. designate the type of output file in a parameter? (Gdoc, Sheets, CSV, etc)
b. have the code automatically detect that the data is of a certain type? (harder to do!)

Always "Downloading file is not completed. Please download next chunk."

I tried to download http://v00992.mainsoft.it/r/AlVolante/2205/1Fi/files/assets/common/downloads/AlVolante_2022005.pdf but after second iteration I always have

status={message=Downloading file is not completed. Please download next chunk.}

Here the json, there are 3 chunks even if only 2 are needed
{fileSize=8.1344298E7, status={message=Downloading file is not completed. Please download next chunk.}, fileName=1649833085061, startDate=1.649833085061E12, chunks=[{start=0.0, fileId=1U-E2yKZchOEvq-zzay0GJczQVZsyi1DD, end=5.1380223E7}, {start=5.1380224E7, fileId=15iKRIzD1h_Q-sVd7PYtdWLKR5zxUFfr1, end=8.1344297E7}, {}], exportFolderId=0AEswpU0WsW8mUk9PVA, mimeType=application/pdf, nextJoin=-1.0, expirationOfLocation=0.0, totalElapsedTime=52.0, totalJoinTime=0.0, totalDownloadTime=52.0, location=, url=http://v00992.mainsoft.it/r/AlVolante/2205/1Fi/files/assets/common/downloads/AlVolante_2022005.pdf, nextDownload=1.0}

ReferenceError: "blob" is not defined.

When I try to download a small file (file size < chunk size) with script I get an error:

ReferenceError: "blob" is not defined. (line 116, file "Code", project "DownloadLargeFilesByUrl")

Error: DownloadLargeFilesByUrl.cfg file cannot be found in the folder of

I provided the folder ids i.e temp folder and export folder ids, and url to your sample script.
I am getting this error.

Error: DownloadLargeFilesByUrl.cfg file cannot be found in the folder of

Here is the code (modified from your sample script)

function driveUpload(){ var tempid = DriveApp.createFolder('tempp').getId(); var exportid = DriveApp.createFolder('export').getId(); var resource = { url: 'https://nptel.ac.in/content/storage2/MP4/106106197/mod01lec02.mp4', exportFolderId: exportid, tempFolderId: tempid, chunkSize: 51380224, downloadPerExecution: 4, }; var res = DownloadLargeFilesByUrl.download(resource); }

What did I miss?

prop.setProperty is failing in Chrome v8

This line:

prop.setProperty("tempFolderId", r.tempFolderId);

is failing, when running in the new Chrome v8 runtime environment.
I disabled v8 and ran in legacy apps script environment, and that seems to fix it.

bug: error on 2nd iteration

First iteration runs fine. Downloads the complete 18MB target file.
Then it sets trigger to run second time. (Why?)
The second run throws an error. It's trying to run the function chkFolder, but tempFolderId is undefined.

The target file I'm downloading is this:
https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv

And I'm using the sample code you gave on the home page for this project. It starts with:

function main() {
console.log("*** starting Main ");
var url = "https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv";
var exportFolderId = "";
var email = "***";
var functionName = "main"; // If you want to modify the function name, please also modify this.

try {...

Feature request: designate the output file name

Would be nice to be able to give the output file a useful name. Could make it a parameter, along with designating the source URL and the export Folder ID.

Download File from URL Requiring Authentication

Hi, I'm trying to download a large file from the Atlassian Jira API, which requires an authentication key to be passed through the headers. The DownloadLargeFilesByURL method will fail, because it is only taking the URL and getting denied at the server. Normally I would use UrlFetchApp.fetch(url, params) to get the file. Is there a way to pass parameters/headers through the method call?

I'm currently using the sample script to attempt this:

function main() {
var url = ""; // Please set URL you want to download. This URL is required to be a direct link of the file.
var exportFolderId = "*****"; // Please set folder ID for saving the downloaded file.
var email = ""; // Please set email address. This is used for sending email when the download is completed and an error occurs.
var functionName = "main"; // If you want to modify the function name, please also modify this.

try {
clearTrigger(functionName);
var prop = PropertiesService.getScriptProperties();
var tempFolderId = prop.getProperty("tempFolderId");
var resource = {
url: url,
exportFolderId: exportFolderId,
tempFolderId: tempFolderId || "",
};
var r = DownloadLargeFilesByUrl.download(resource); // Here, this library is used.
if (!resource.tempFolderId) {
prop.setProperty("tempFolderId", r.tempFolderId);
}
var t = new Date();
t.setMinutes(t.getMinutes() + 3); // This means that the script run again after 3 minutes later.
ScriptApp.newTrigger(functionName).timeBased().at(t).create();
if (r.nextDownload == -1 && r.nextJoin == -1) {
clearEnv(prop, functionName);
MailApp.sendEmail({
to: email,
subject: "Download was completed.",
body: JSON.stringify(r),
});
}
} catch (e) {
clearEnv(prop, functionName);
MailApp.sendEmail({
to: email,
subject: "Error occurred.",
body: JSON.stringify(e),
});
}
}

Javascript runtime exited unexpectedly

Using the library as it is I ran into the titled error using the V8 runtime. I identified the source of the problem in the two payload definitions in the request header in the lines

DownloadLargeFilesByUrl/DownloadLargeFilesByUrl.js

Line 376 in 8d55db0

payload: DriveApp.getFileById(this.cfgVal.chunks[this.cfgVal.nextJoin].fileId).getBlob().getBytes(),

and

DownloadLargeFilesByUrl/DownloadLargeFilesByUrl.js

Line 435 in 8d55db0

payload: DriveApp.getFileById(e.fileId).getBlob().getBytes(),

Using the methods getBlob().getBytes() seems to cause this problem. Removing the two methods fixes the problem for me. Assigning an instance of the DriveApp File to the payload resolves its content and the request is submitted correctly.

How to handle if link expired

I am trying to download youtube video but it's stuck at 9th chunk, possibly due to expire link (neither show error nor success messages)

URL :- https://r6---sn-qxoedn7z.googlevideo.com/videoplayback?expire=1598662237&ei=_VFJX4XEAcjgigSj-q6YAw&ip=35.187.132.108&id=o-AHn-lDVRW8XY_YHfO-gX8GTAMWXEcEXEAbW26bTRUWHP&itag=18&source=youtube&requiressl=yes&mh=Ec&mm=31%2C26&mn=sn-qxoedn7z%2Csn-qxo7rn7...........

Error: Exception: Limit Exceeded: Drive

Hi there, thanks for the library! Is there a way to tell what exact limit it exceeded? I'm not sure whether it exceeded the Drive's space limit or execution limit or any other restrictions. Here is the error log I got. Thanks in advance!

{
"insertId": "lnmjc9f3kxys4",
"jsonPayload": {
"serviceContext": {
"version": "5",
"service": "AKfycbzS-s3XZb9CKeipv0jWBZLYD_DyEr3psSsVzOJNUtYShlO6TA"
},
"context": {
"reportLocation": {
"lineNumber": 329,
"filePath": "Code",
"functionName": "saveChunkToFile"
}
},
"message": "Error: Exception: Limit Exceeded: Drive.\n at saveChunkToFile(Code:329:15)\n at unknown function\n at resumableDl(Code:266:11)\n at downloadMain(Code:130:26)\n at DownloadLargeFilesByUrl.download(Code:66:27)\n at download(Code:9:14)"
},
"resource": {
"type": "app_script_function",
"labels": {
"function_name": "doGet",
"project_id": "thermal-formula-283609",
"invocation_type": "apps script api"
}
},
"timestamp": "2021-04-21T23:01:12.230Z",
"severity": "ERROR",
"labels": {
"script.googleapis.com/process_id": "EAEA1GOxxKzw-CR6rrnU0eLDMi7lUjd_HmlusL4ihG1Opv20s5eTPfdW_HdsQjtICwAhTThqRrXBfKdxZT7bBgkE1c0pSwbrHXi1FdRC7BVuRiUlONJBSPlRwxjrxil7t5lqEn6caOydjaw9WX6HO0j0IaukcLJ1g3jWWTwHMearHVQpwKsG2yusQfeJiexLBZ6DCA5Boottij574pW9XiMRj7ef09asuGMHx0CZGZFIeqXudJpzj8cRMDBwAdeBE2LldZsmYHodADE3ru1LA09gy9LerbkyNmtng",
"script.googleapis.com/deployment_id": "AKfycbzS-s3XZb9CKeipv0jWBZLYD_yEr3psSFsVzOJNUtYShlO6TA",
"script.googleapis.com/user_key": "APrZfeqJLeKIfKino4q+iMjQGSTO00T3gYluLqvuv5ZOq9dKx31pmCCTxHiMwA9hkjvTSUC7cUS",
"script.googleapis.com/project_key": "Mgijff1XtCB3iXtJWt3rmXhxIoJWoTElh"
},
"logName": "projects/thermal-formula-283609/logs/script.googleapis.com%2Fconsole_logs",
"receiveTimestamp": "2021-04-21T23:01:12.526278699Z"
}

Download to Shared Google Drive Folder

Is it possible to modify this script to download a large file to a Shared Google Drive Folder?

Cheers,

Camille