storj-archived / storj.js Goto Github PK
View Code? Open in Web Editor NEWDeprecated. Browser library for interacting with Storj.
Home Page: https://storj.io
License: GNU Lesser General Public License v3.0
Deprecated. Browser library for interacting with Storj.
Home Page: https://storj.io
License: GNU Lesser General Public License v3.0
No longer need to use the fork for svgs
The C library will soon support reed-solomon erasure coding for file chunks. We need to ensure that storj.js is compatible with RS coded files produced by libstorj, and that uploads can be RS coded in storj.js and rebuilt by the C library.
Related:
storj-archived/libstorj#279
storj-archived/libstorj#278
Need a way of fetching a list of public keys currently stored on the bridge for a user. Needs tests and documentation.
Note, this is for keys uploaded via
registerKey
, notmakePublic
. To get keys registered viamakePublic
, usegetBucket
.
renderTo
, appendTo
, and blob operations do not have tests.
These tests should follow the same pattern as test/index.js
, ideally named test/browser.js
. Would prefer if the package.json
handled figuring out which tests to run in which environment (i.e. node-test
picks it's own tests and testling
includes browser.js
) as opposed to doing feature detection in the tests themselves.
Just tried to install and use:
npm install --save storj
test.js:
const Storj = require('storj');
Run:
truffle test
Output:
Error: Cannot find module 'storj'
at Function.Module._resolveFilename (module.js:513:15)
at Function.Module._load (module.js:463:25)
at Module.require (module.js:556:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/Users/k06a/Projects/pre-alpha-base-contract/test/BaseContract.js:16:15)
at Module._compile (module.js:612:30)
at loader (/Users/k06a/Projects/pre-alpha-base-contract/node_modules/babel-register/lib/node.js:144:5)
at Object.require.extensions.(anonymous function) [as .js] (/Users/k06a/Projects/pre-alpha-base-contract/node_modules/babel-register/lib/node.js:154:7)
at Module.load (module.js:531:32)
at tryModuleLoad (module.js:494:12)
render-media will request a stream from us, optionally including "seek" parameters indicating where the stream should begin and end.
Right now, we ignore these, leading to undefined behaviour when a user attempts to seek in a file.
When browserifying the latest commits I get this error initially destroying the storj module.
Uncaught Error: More than one instance of bitcore-lib found. Please make sure to require bitcore-lib and check that submodules do not also include their own bitcore-lib dependency.
Bitcore lib has a version gaurd coded into that won't allow submodules to require it.
Storj core already requires bitcore-lib so when it is required for the Mnemonic in storj.js it is interfering. The simple fix is to generate a utility in core and require that through storj.js.
Or have storj-crypto module which started as a shim to allow webcrypto (which it cant do) contain all of the crypto modules so we don't run into this again.
Currently stack traces look like this:
12:44:29.966 Error: Not enough entropy. Minimum is: 192 bits
assert() storj.js:6377
HmacDRBG() storj.js:6373
sign() storj.js:6359
[528]</</KeyPair.prototype.sign() storj.js:20111
[521]</</BridgeClient.prototype._authenticate() storj.js:19978
_request() storj.js:19970
retryAttempt() storj.js:3004
retry() storj.js:3004
[521]</</BridgeClient.prototype._request() storj.js:19970
[521]</</BridgeClient.prototype.createToken() storj.js:19823
createFile() storj.js:8
upload() 03:19
onchange() 03:1
1 storj.js:6377:26596
Looking at storj.js:6377
gives us:
This is hardly useful. Having dev bundles with source maps and un-minified source would help a lot.
Travis isn't injecting environment variables into the tests.
Currently implemented in https://github.com/Storj/storj.js/blob/NewAPI/lib/api/get-file-pointers.js
This needs to be updated to match the new API, exposed as storj.getFilePointers
, and documented in https://github.com/Storj/storj.js/blob/NewAPI/README.md
We should consider having getFile
invoke getFilePointers
as opposed to duplicating logic.
Long term, getFile
should accept a pointers
object which can be used to bypass the need for a client to talk to the bridge. This would allow a server to handle all bridge auth and allow clients to upload/download data directly from farmers, that way clients wouldn't need to handle private keys or mnemonics.
The file sizes for storj.es5.js
and storj.es6.js
are both quite large (several MBs). We should pipe them through uglify
or something similar to reduce their final size.
Note: we need to maintain our sourcemaps when doing the transform.
This method is documented but not implemented.
When streaming data from farmers, we use a lot of Buffer.concat operations to buffer up the file into memory. Firefox appears to be de-optimizing this function. Since all of the crypto logic is shared with the rest of the browser's thread, the user experience for downloading a large file is that their browser locks up until the browser pauses the script with a prompt.
You can see concat
invocations (and descendant functions) are plateaus.
Concat is the in call chain of >75% of the on-thread time for our script
We are exporting the browserified bundle as storj
as opposed to Storj
Per the browserify documentation, it looks like the -s
flag will automagically be camel cased, so switching to -s Storj
may not be enough to properly export the constructor.
Need to document and expose makePublic
Since the methods that operate on the file are either stream or callback based, they should be safe to call before the file enters into a ready
state.
Callback based functions should either callback with an error if the file
object errors before ready
is emitted, or should invoke their callback as soon as they have the data necessary to complete the request.
Stream based functions should return a stream immediately. The stream should error when file
errors, and should provide data as soon as it becomes readable. NOTE: The stream emitting an error should not save a file
object from throwing an unhandled
error if the user does not register an error
listener directly on the file
object. In a similar manner, a readable stream coming from a file
should not block the file
from downloading it's data from a farmer into a blob store. I.E. calling createReadStream
and not consuming data from it should not fill the TCP socket's stream's internal buffer, data should still be able to flow into the underlying blob store.
getBuffer
getBlob
getBlobUrl
appendTo
renderTo
getReadStream
Stack trace:
12:44:29.966 Error: Not enough entropy. Minimum is: 192 bits
assert() storj.js:6377
HmacDRBG() storj.js:6373
sign() storj.js:6359
[528]</</KeyPair.prototype.sign() storj.js:20111
[521]</</BridgeClient.prototype._authenticate() storj.js:19978
_request() storj.js:19970
retryAttempt() storj.js:3004
retry() storj.js:3004
[521]</</BridgeClient.prototype._request() storj.js:19970
[521]</</BridgeClient.prototype.createToken() storj.js:19823
createFile() storj.js:8
upload() 03:19
onchange() 03:1
1 storj.js:6377:26596
In browser when uploading.
Websockets are so 2016. Get with the hip HTTP transfer.
See docs here: http://storj.github.io/core/tutorial-data-transfer.html
Requiring that a user instantiate an entire storj
object just to generate a mnemonic
is silly.
Same for generateKeyPair
although registering the keypair still requires auth.
The use case for generateKeyPair
isn't there. If you aren't authenticated against the bridge, having generateKeyPair
doesn't offer anything. Just focusing on mnemonic
for now.
Currently implemented in https://github.com/Storj/storj.js/blob/NewAPI/lib/api/get-data.js
This method doesn't expose the file
object to the user, instead it returns a readable stream of the data.
This should be documented in https://github.com/Storj/storj.js/blob/NewAPI/README.md
'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Unable to download libstorj for platform: win32 and arch: x64
npm WARN [email protected] requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @types/googlemaps@^3.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @types/markerclustererplus@^2.1.29 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] preinstall: node ./download.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Caprice\AppData\Roaming\npm-cache_logs\2018-08-24T06_31_30_970Z-debug.log
https://github.com/Storj/storj.js#fileondata-function-cbdata-
Will require a PR to core, currently planned but not implemented.
request
is not correctly handling 302 redirects from http
to https
, we need to correctly handle status codes like this.
var stream = storj.download(fileId, bucketId)
Return a readable stream of decrypted data being pulled directly from farmers. This method bypasses the File API entirely, and doesn't use a backend abstract-blob-store. If you only need to download a file once, and don't want to hold onto it in memory, this is the method for you.
storj.download(fileId, bucketId).pipe(fs.createWriteStream('cat.jpg'));
The correct function should be storj.download(bucketId,fileId)
Will someone with access fix it, too small for me to file a pull request
libstorj has migrated to referring to the mnemonic as an encryptionKey (since we use it to derive the key used to encrypt/decrypt files).
Storj.js should follow suite.
Currently, core does not expose a way to track the upload progress of a file, which prevents storj.js from emitting on('data'
events and updating file.progress
.
Things to think about:
data
event may not match the order of those chunks in the file. What implications does this carry for userland?data
event? When we finish encrypting the chunk, when we send the data over the wire, when we get confirmation that the farmer received that data, etc? What implications will the timing of the data
event carry for userland?Storj.js is CPU heavy compared to most JS on the web. This creates a less-than-ideal user experience in many browsers.
The solution is Web Workers, but we need a little more exploration before we jump to an implementation.
Constraints:
storj.js
script when initializing. (Should construct a blob via a string containing the JS, pull it's URL, and use that to initialize the Web Worker)Whenever we have an error-first callback, the callback should be required to force error acknowledgment. In cases where the callback is registered to the done
event, the callback should be optional.
Those are secret, currently we write them to stdout.
TAP version 13
# Valid environment
# Running with [email protected]:password@http://127.0.0.1:32770
From @dylanlott at cpollard1001/storj.js#2
After trying to merge in billing changes to Bridge-GUI today and realizing that it's not going to (easily or consistently) work with the most recent storj-lib versions, it is most likely going to be come necessary to pull bridge client out of core, or create a better browser-only library for talking to Bridge instead of using the BridgeClient in storj-lib.
I've started work on this, but wanted to open up a thread for discussion between all parties already involved, since I saw that Storj.js requires a BridgeClient to work as well.
Any thoughts or ideas on this before we get too far in on it? I'm scaffolding out the library right now, and will be working on it tomorrow with @bryanchriswhite
Default when signing is { compact: true }
. I've found that this needs to be { compact: false }
in order to generate the correct signature.
Suggested change to lines 395-397 in README.md:
var signature = keypair.sign(message, [opts])
Sign a message with this key. message
should be either a String
or a Buffer
, and the returned signature will be a String
. Default options is { compact: true }
.
or
var signature = keypair.sign(message, { compact: false })
Sign a message with this key. message
should be either a String
or a Buffer
, and the returned signature will be a String
. Default options is { compact: true }
.
KeyPair.prototype.sign = function(message, options) {
var sign = null;
var opts = merge({ compact: true }, options);
if (opts.compact) {
sign = Message(message).sign(this._privkey);
} else {
if (!Buffer.isBuffer(message)) {
message = new Buffer(message, 'utf8');
}
sign = ecdsa.sign(
crypto.createHash('sha256').update(message).digest('hex'),
this.getPrivateKey()
).toDER('hex');
}
file.name
, file.mimetype
, and file.length
don't appear to be getting set in the browser.
The documentation calls for a ready
event to be emitted off of the Storj object during setup.
Currently there isn't any async setup necessary. In the original design of this API, there was a round trip for auth validation during setup. We decided to keep the async constructor for forward support of async operations during setup.
We can either:
A) drop the ready
event and risk having to make a breaking change in the future
B) implement the ready
event
I'm ๐ on (B)
It is implemented but not documented
Right now stack traces are super noisy in tests. Would be handy to have a source-map aware error object.
Look into:
I've made a browserified version of Storj.js using:
browserify lib/ -o storj.js
and when including it into the HTML like this:
<script src="storj.js"></script>
I get this error:
Uncaught ReferenceError: Storj is not defined
at index.html:68
Here is the layout of my files:
storj-client
|-- index.html
`-- storj.js
Am I doing anything wrong or does it just not work?
this
refers to the created file
object, which defines createReadStream
. So when createReadStream
returns this.createReadStream()
it is calling itself.
https://github.com/Storj/storj.js/blob/master/lib/file.js#L113
Crypto in JS is painfully slow. @nginnever has kicked off work exploring the WebCrypto API.
Requesting corresponding updateBucketById
method to update bucket information
https://github.com/Storj/core/blob/master/lib/bridge-client/index.js#L256-L258
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.