yakovkhalinsky / backblaze-b2 Goto Github PK
View Code? Open in Web Editor NEWNode.js Library for the Backblaze B2 Storage Service
License: MIT License
Node.js Library for the Backblaze B2 Storage Service
License: MIT License
Related to #32. Applies to uploadPart
and uploadFile
.
If hash
is not passed and data
is a stream, the hash can be computed on the fly and appended to the output, while providing the header X-Bz-Content-Sha1: hex_digits_at_end
. It would be nice if the client would wrap up this logic itself.
This change is simpler than it seems at first. I wrote the following transform stream that hashes the content as it passes through, then emits the hash before the stream ends. We are using this in production successfully.
const crypto = require('crypto');
const stream = require('stream');
function makeSha1AppendingStream() {
const d = crypto.createHash('sha1');
return new stream.Transform({
transform(chunk, encoding, cb) {
d.update(chunk, encoding);
this.push(chunk, encoding);
cb();
},
flush(cb) {
this.push(d.digest('hex'));
cb();
},
});
}
Used simply like (adjust variable names as needed):
if (hash === undefined && typeof data.pipe === 'function') {
const hashStream = makeSha1AppendingStream();
data.on('error', err => { hashStream.emit('error', err); });
data = data.pipe(hashStream);
hash = 'hex_digits_at_end';
contentLength += 40;
}
Side note: if streams are used, all retrying/redirect-following should be disabled. This is either unsafe since the stream has been consumed, or will likely consume a large amount of memory as the entire request body is buffered in memory in case the request needs to be replayed. We had to pass maxRedirects: 0
to axios or process memory would balloon (we're uploading several-hundred-MB files and this was killing us).
How can I integrate the this API with GraphQl. Since node will reside on EC2 If I use getDownlaodAutherization it will be valid only for that id and wont work for client device. Similarly upload url might not work for client or I may be wrong.
Please do advise on how to go about this.
I need help uploading an image from a canvas. Right now this is what I have.
var imageDataURL = canvas.toDataURL();
var b2 = new B2({
accountId: B2_ACCOUNT_ID,
applicationKey: B2_APPKEY
});
b2.authorize().then(function() {
console.log(".authorize-DONE");
return b2.getUploadUrl(B2_BUCKET_ID);
}).then(function(response) {
console.log(".uploadUrl:" + JSON.stringify(response));
return b2.uploadFile({
uploadUrl: response.uploadUrl,
uploadAuthToken: response.authorizationToken,
mime: 'image/png',
filename: 'sample.png',
data: imageDataURL // You mentioned to use a Buffer but I am clueless how to do it. Example please.
});
}).then(function(response) {
console.log(JSON.stringify(response));
}).catch(function(error) {
console.log("A problem has occurred:" + error.stack);
});
I store files using "folders". So as an example, my structure looks like this:
{folder}/{folder}/{file.ext}
When using deletefileversion, I get back the following:
{ code: 'file_not_present',
message: 'file not present: -K82QoL4BCVSRqv3u3r4%2Fcollages%2Fd3f81d06-973d-4478-d19e-0677d8bdc681.jpg 4_zd55dcf6fe56cedf45d130010_f11275d83e9982f7a_d20160115_m035841_c001_v0001012_t0043',
status: 400 }
The actual filename is:
-K82QoL4BCVSRqv3u3r4/collages/d3f81d06-973d-4478-d19e-0677d8bdc681.jpg
So, it appears that the filename is being urlencoded when perhaps it should not be? Or perhaps something else is broken.
Thanks!
-Matt
B2 does not return any redirect codes under any circumstances, so there is no need to handle the redirect case. When uploading a stream, axios will buffer the entire request body if maxRedirects
is not 0 (the default is 5) so it can correctly handle HTTP codes 307 and 308. When uploading files, this causes axios to hold the entire stream contents in memory to handle a situation that will never occur. This is wasteful, particularly for very large uploads.
uploadFile()
and uploadPart()
should set maxRedirects:0
on the axios request config. (It should be safe to set this everywhere, as a global default, but these two functions are the most critical.)
Hey, I'm having troubles with finishLargeFile()
and I was wondering if my structure for partSha1Array
is wrong. At the moment it's basic array of [hash1, hash2, ...]
but I can't get any response from finishLargeFile()
and it doesn't seem to throw errors either.
Any ideas?
cant get a valid file (test zip and mp3)
b2.downloadFileById(fileid).then(function(file){
// tried var bitmap = new Buffer(file.data,'base64');
//fs.writeFile('testload.txt',bitmap)
fs.writeFile('testload.mp3',file.data, function(err) {
console.log(err);
});
});
Any idea? thnx
Related to #1
We ran into an issue where binary downloads were not working.
This can be addressed by adding the responseType attribute as an 'arraybuffer'.
There is no documentation on passing through options to the axios library used
e.g. this is the correct way to download binary files.
b2.downloadFileById({
fileId: fileId,
responseType: 'arraybuffer'
See a stackoverflow answer related to this issue here:
https://stackoverflow.com/questions/41846669/download-an-image-using-axios-and-convert-it-to-base64
I put the keys in the app and also in Backblaze bucket i configure cors with all origins and in google console:
Access to XMLHttpRequest at 'https://api.backblazeb2.com/b2api/v2/b2_authorize_account' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Might not be a bad idea to have the SHA that was generated and used for the upload returned.
Just putting this in so we can clean up for an npm release.
I think since we've tagged some PR's already for 1.1.0, we should move those that aren't in 1.0.4 into the 1.1.0 release in preparation for making a changelog and doing the final version bump.
This would go a long way to refresh the package.
I am open to anyone who would like to contribute, I can add any interested parties to the repo and the npm library.
Im missing copy
and b2_copy_part
feature
https://www.backblaze.com/b2/docs/b2_copy_file.html
Perhaps we should have backblaze-b2
automatically call authorize_account
when it encounters a 401 error, or have it as an option at least.
I encountered an issue in a long-running application where after 24 hours all calls stopped working.
https://www.backblaze.com/b2/docs/application_keys.html
Authorization tokens are only good for 24 hours. You can use the application key to make new authorization tokens as they expire.
It would be nice if the module automatically did this rather than the user having to: 1. Call authorize
before every method (incurs a Class C transaction cost), 2. Add an error handler to every call to check for 401s, or 3. Add a setInterval
every 24 hours to authorize (a bit hacky).
Thoughts?
It says the lib doesn't support large file methods.
Looks like the switch to Axios has also limited upload parts to 10MB, since there's no way to access the underlying Axios object to set a larger maxBodyLength.
If you try to send an info key that contains an underscore, you get back this error: Info header keys contain invalid characters: <header>
Backblaze suggests using X-Bz-Info-src_last_modified_millis
to send the file's last modification time (and in fact the b2 sync
command in their Python CLI application does this). I had to use axios.headers
to manually set the header as a workaround.
Any chance of getting an implementation for b2_create_key?
Any plans to support setting X-Bz-Info headers?
Thanks!
Hello,
When I am downloading arbitrary data stored in B2 bucket, the content-length header has the correct byte length of 208374 but when I call:
Buffer.byteLength(resp.data)
where resp is the await return value from b2.downloadFileById()
, I get 378415. Even the backblaze UI shows that the size is 208.4 kB. What is the correct way to decode the data sent from the downloadFileById()
method to get the correct-size Buffer or Uint8 array?
B2 returns the bucket id on authorisation if the auth key is for a particular bucket. Thus, one can store the bucket id in the instance too and use it if bucket id is not passed in a function call.
I am open to implementing this myself. Looking for feedback initially.
Hi I am trying to use backblaze-b2 plugin in my application but when I used b2.authorize() I am getting this error : Error: Invalid authorizationToken
I debugged it and found that b2.authorizationToken is null in utils.js
Could somebody tell me what could have went wrong?
Code :
var B2 = require('backblaze-b2');
// create B2
// create b2 object instance
var b2 = new B2({
accountId: 'accId',
applicationKey: 'appkey'
});
b2.authorize();
It would be interesting if they heard more operations that could be performed without the file ID, just with the filename, like:
Hello,
Thanks for this project, have used it already with great satisfaction thus far.
I'm wondering though: is there anyway to pass in an option and retry on failure? I'm specifically wanting to
retry once or maybe twice if I get a 503, service_unavailable
Is it possible to get file download url or do I have to download on server and handle downloadUrl my self?
When uploading large Files i need to get UploadPartUrl for each part / chunk first.
Sending my Chunks like this:
b2.getUploadPartUrl({fileId:bbFileId}).then(
function(){
b2.uploadPart({
partNumber: chunk.cnt,
data: chunk.data,
partSha1: chunk.partSha1,
uploadUrl: uploadUrl,
uploadAuthToken: token
});
},
reject);
results in getting
Error: socket hang up
at createHangUpError (_http_client.js:331:15)
at TLSSocket.socketOnEnd (_http_client.js:423:23)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickCallback (internal/process/next_tick.js:181:9)`
for some parts / chunks.
Seems like calling getUploadPartUrl many times in a row will cause this error.
Line 54 in a13380b
When authorising the account, this lines overwrites the accountId that was provided as argument on init with the accountId returned from the authorize_account api.
This doesn't work when using other application keys, because the API returns the accountId of the master key and not the one provided.
This means that you cannot renew the authorizationToken when using keys other than the master key.
I'm now using a workaround (patched the authorize action function with my own).
Why is saving the accountId from the API actually needed if we need to provide it on init?
Hey, so I am trying to upload a file but am kind of confused how I will go about it, so I have
b2.uploadFile({
uploadUrl: 'my url is here',
uploadAuthToken: 'my token is here',
)}
Now, how do I go about uploading a file, because the only other param I see is fileName
, and not like a specific file or path where I can get the image and upload it, because I have an api and want to post the uploaded users image to backblaze,
So, Example for uploading an image from a canvas #17 kind of helped me, but it's just upload buffer, and I am unable to view it, just shows a black box.
Hi.
I was trying to use the module for uploading files within an Electron app, but when uploading a file, I constantly encountered an issue with the SHA1 checksum:
{code: "bad_request", message: "sha1 is wrong length", status: 400}
Any ideas how to solve that?
Thank's a lot
Heiko
my code look like :
const uploadPartObj = {
partNumber: orderedChunks[i + 1].id,
uploadUrl: tokenInfo.data.uploadUrl,
uploadAuthToken: tokenInfo.data.authorizationToken,
data: orderedChunks[i + 1].buffer,
hash: orderedChunks[i + 1].sha1,
onUploadProgress: (event) => console.log(event),
};
await this.#b2.uploadPart(uploadPartObj);
but i never see the console.log, the onUploadProgress is never trigger ...
Add functionality for large file API
Hi,
I'm trying to set up the library. However, I am receiving
Error getting bucket: Error: Request failed with status code 401
const b2 = new B2({
applicationKeyId: "sadfasdffad", // or accountId: 'accountId'
applicationKey: "masterApplicationKey", // or masterApplicationKey
});
export const uploadCreationImage = async (
) => {
try {
await b2.authorize(); // must authorize first (authorization lasts 24 hrs)
let response = await b2.getBucket({
bucketName: "bobbyhill",
});
console.log(response.data);
} catch (err) {
console.log("Error getting bucket:", err);
}
};
What is causing this issue?
The method b2.uploadFile()
seems to use filename
as an arg, while most (if not all) others like b2.deleteFileVersion()
use fileName
.
Perhaps we could change b2.uploadFile()
to use fileName
by default, but fallback to filename
so we don't break the library for everyone?
I'm not sure if this error is the fault of this package, or the b2 API. I'm just trying to upload a file to a bucket, but it seems that it's failing because the sha1 didn't match up with what the server received.
I'm unsure if I am able to provide a sha1, or if the library is doing this for me.
Here is the code:
b2.authorize().then( /* authorized */
(response) => {
b2.getUploadUrl(bucket).then( /* get url to upload to */
(response) => {
fs.readFile('test.mp3', 'utf8', (err, data) => { /* get data object from fs */
b2.uploadFile( { /* upload data to bucket */
uploadUrl: response.uploadUrl,
uploadAuthToken: response.authorizationToken,
filename: 'test.mp3',
data: data /* maybe a problem? */
}).then(
(response) => {console.log(response)},
(err) => {console.log(err)} /* error gets logged here )': */
);
});
},
(error) => { console.log(error); }
);
},
(error) => { console.log(error); }
);
Is there any way to know how much data has been uploaded?
missing a useful optional field to specify what fileId to start from.
The axios version in the package.json of the released package is still 0.18. This always leads to npm complaining about security issues. Setting this to 0.21.1 would be cool.
Calling authorize second time fails when application credentials are used, because b2.accountId
gets re-written with actual accountId.
Related to #64
It appears 1.6.0 has not been published yet... Any good reason for this?
Hi,
Sorry to ask this here, but b2 is new so not a lot of people are responding on stackoverflow.
Do you know any solution to delete all files in a folder (so with name starting with "folder/") ?
I guess I have to do a listFile starting with "folderUrl" and then for each file a delete file version call.
Per https://www.backblaze.com/b2/docs/string_encoding.html any strings should be URIEncoded.
If you do b2.uploadFile(...'test file.txt'...);
you receive the error:
{ code: 'bad_request',
message: 'Bad character in percent-encoded string: 32',
status: 400 }
This is because the filename does not get URIEncoded.
(There's probably other areas in this implementation where this happens, too)
The documentation and the README needs a bit of improvement.
Hello,
I've noticed the prefix and delimiter options are missing from the listFileVersions
function.
This raises an issue when trying to filter specific files from backblaze.
Proposed change:
exports.listFileVersions = function(b2, args) {
const bucketId = args.bucketId;
const startFileName = args.startFileName;
const startFileId = args.startFileId;
const maxFileCount = args.maxFileCount;
const prefix = args.prefix;
const delimiter = args.delimiter;
const options = {
url: endpoints(b2).listFileVersionsUrl,
method: 'POST',
headers: utils.getAuthHeaderObjectWithToken(b2),
data: {
bucketId: bucketId,
startFileName: startFileName || '',
prefix: prefix || '',
delimiter: delimiter || null,
startFileId: startFileId,
maxFileCount: maxFileCount || 100
}
};
// merge order matters here: later objects override earlier objects
return request.sendRequest(_.merge({},
_.get(args, 'axios', {}),
options,
_.get(args, 'axiosOverride', {})
));
};
I've gotten this to work with node and typescript with the associated @types/backblaze-b2 package but I have not been able to figure out how to use the interfaces defined in the index.d.ts file in my node js program. For example, I am trying to define a BucketInfo interface that can be used to extract data from a getBucket() call. In the BucketInfo interface, I would like to define a bucketType: BucketType property, but I cannot figure out how to get access to the BucketType type definition in the index.d.ts file in @types/backblaze-b2. Can anyone suggest a solution?
Currently, I am importing the b2 definition like this:
import B2 = require("./node_modules/@types/backblaze-b2");
This allows me to call B2 as a constructor, but I don't seem to be able to use it as a namespace.
I'm not sure if this is appropriate to post as an issue, but it would be helpful to add this to the examples in the doc, so I suppose it is valid.
Hi there @yakovkhalinsky! Thanks so much for your work on this lib!
What are the chances of adding a method for the b2_get_download_authorization
operation? Looks like it's listed in the API docs, but not available in this lib.
API link: https://www.backblaze.com/b2/docs/b2_get_download_authorization.html
I just did a run-through of all the available operations, and this is the only one that's missing, as of 2017-11-14.
There's no docs relating to streams, are they able to be used with this library?
The AWS S3 SDK for JavaScript has an upload
function that does not correspond to any particular API request. You can give it a buffer or a stream, and it will automatically perform either a single PutObject call or a multi-part upload.
It would be a great benefit to this library to provide something similar. Right now, large file uploads are unnecessarily cumbersome, especially when the input is a stream. Authorization token management is a giant pain.
I am working on such a function right now for our own internal use. I'm writing it as a module that exposes a single function that can be attached to the prototype of the B2
class provided by this library (B2.prototype.uploadAny = require('backblaze-b2-upload-any');
).
This issue is intended to convey my intent to integrate this function into this library and submit a PR. Therefore, I would very much appreciate any feedback on my proposal so that I can accommodate any necessary design changes as early as possible.
The current planned features of this function (many of which are already done) are:
There is a difference between the local file and stream cases. When uploading a local file, no content is buffered in memory. Rather, multiple read streams are created (and re-created as necessary if a part upload must be retried).
Stream support necessarily requires some buffering in memory to facilitate retries since node streams cannot be seeked (and not all stream types would be seekable, anyway).
Note that I currently introduce two new dependencies:
I'm working on a 3D model viewer feature for my app and the client side of the application which is written in Reactjs has a component that renders the .obj model and the only way to successfully load a model is to pass a url paramater which can be either a locally stored file in the computer or a download url (this is how I want to tackle it), the thing is that to download files from backblaze cloud you need to add the authorization headers in order to get the permissions, is there a way to go around this as I need a url that is usable by itself?
b2.authorize().then(() => {
console.log('auth success');
return b2.getUploadUrl('b19aa53a2f0f1c4256da021f').then(response => {
console.log('get uplaod url success');
this.auth = response.data;
// Get the file Content-Length
return fp.stat(filePath).then(stat => {
this.stat = stat;
return b2.uploadFile({
uploadUrl: this.auth.uploadUrl,
uploadAuthToken: this.auth.authorizationToken,
filename: fileID,
mime: file.type,
data: fs.createReadStream(filePath),
info: {
'Content-Length': this.stat.size,
}
}).then(response => {
console.log('upload success')
console.log(response)
})
})
})
})
.catch(error => {
console.log(error);
})
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.