storj-archived / storj.js Goto Github PK

No longer need to use the fork for svgs

The C library will soon support reed-solomon erasure coding for file chunks. We need to ensure that storj.js is compatible with RS coded files produced by libstorj, and that uploads can be RS coded in storj.js and rebuilt by the C library.

Related:
storj-archived/libstorj#279
storj-archived/libstorj#278

Need a way of fetching a list of public keys currently stored on the bridge for a user. Needs tests and documentation.

Note, this is for keys uploaded via registerKey, not makePublic. To get keys registered via makePublic, use getBucket.

renderTo, appendTo, and blob operations do not have tests.

These tests should follow the same pattern as test/index.js, ideally named test/browser.js. Would prefer if the package.json handled figuring out which tests to run in which environment (i.e. node-test picks it's own tests and testling includes browser.js) as opposed to doing feature detection in the tests themselves.

Just tried to install and use:

npm install --save storj

test.js:

const Storj = require('storj');

Run:

truffle test

Output:

Error: Cannot find module 'storj'
    at Function.Module._resolveFilename (module.js:513:15)
    at Function.Module._load (module.js:463:25)
    at Module.require (module.js:556:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/k06a/Projects/pre-alpha-base-contract/test/BaseContract.js:16:15)
    at Module._compile (module.js:612:30)
    at loader (/Users/k06a/Projects/pre-alpha-base-contract/node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (/Users/k06a/Projects/pre-alpha-base-contract/node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:531:32)
    at tryModuleLoad (module.js:494:12)

render-media will request a stream from us, optionally including "seek" parameters indicating where the stream should begin and end.

Right now, we ignore these, leading to undefined behaviour when a user attempts to seek in a file.

When browserifying the latest commits I get this error initially destroying the storj module.

Uncaught Error: More than one instance of bitcore-lib found. Please make sure to require bitcore-lib and check that submodules do not also include their own bitcore-lib dependency.

Bitcore lib has a version gaurd coded into that won't allow submodules to require it.

Storj core already requires bitcore-lib so when it is required for the Mnemonic in storj.js it is interfering. The simple fix is to generate a utility in core and require that through storj.js.

Or have storj-crypto module which started as a shim to allow webcrypto (which it cant do) contain all of the crypto modules so we don't run into this again.

Currently stack traces look like this:

12:44:29.966 Error: Not enough entropy. Minimum is: 192 bits
assert() storj.js:6377
HmacDRBG() storj.js:6373
sign() storj.js:6359
[528]</</KeyPair.prototype.sign() storj.js:20111
[521]</</BridgeClient.prototype._authenticate() storj.js:19978
_request() storj.js:19970
retryAttempt() storj.js:3004
retry() storj.js:3004
[521]</</BridgeClient.prototype._request() storj.js:19970
[521]</</BridgeClient.prototype.createToken() storj.js:19823
createFile() storj.js:8
upload() 03:19
onchange() 03:1
1 storj.js:6377:26596

Looking at storj.js:6377 gives us:

This is hardly useful. Having dev bundles with source maps and un-minified source would help a lot.

Travis isn't injecting environment variables into the tests.

Currently implemented in https://github.com/Storj/storj.js/blob/NewAPI/lib/api/get-file-pointers.js

This needs to be updated to match the new API, exposed as storj.getFilePointers, and documented in https://github.com/Storj/storj.js/blob/NewAPI/README.md

We should consider having getFile invoke getFilePointers as opposed to duplicating logic.

Long term, getFile should accept a pointers object which can be used to bypass the need for a client to talk to the bridge. This would allow a server to handle all bridge auth and allow clients to upload/download data directly from farmers, that way clients wouldn't need to handle private keys or mnemonics.

The file sizes for storj.es5.js and storj.es6.js are both quite large (several MBs). We should pipe them through uglify or something similar to reduce their final size.

Note: we need to maintain our sourcemaps when doing the transform.

This method is documented but not implemented.

When streaming data from farmers, we use a lot of Buffer.concat operations to buffer up the file into memory. Firefox appears to be de-optimizing this function. Since all of the crypto logic is shared with the rest of the browser's thread, the user experience for downloading a large file is that their browser locks up until the browser pauses the script with a prompt.

Firefox User Experience:

Flamegraph:

You can see concat invocations (and descendant functions) are plateaus.

Function breakout:

Concat is the in call chain of >75% of the on-thread time for our script

We are exporting the browserified bundle as storj as opposed to Storj

Per the browserify documentation, it looks like the -s flag will automagically be camel cased, so switching to -s Storj may not be enough to properly export the constructor.

Need to document and expose makePublic

Since the methods that operate on the file are either stream or callback based, they should be safe to call before the file enters into a ready state.

Callback based functions should either callback with an error if the file object errors before ready is emitted, or should invoke their callback as soon as they have the data necessary to complete the request.

Stream based functions should return a stream immediately. The stream should error when file errors, and should provide data as soon as it becomes readable. NOTE: The stream emitting an error should not save a file object from throwing an unhandled error if the user does not register an error listener directly on the file object. In a similar manner, a readable stream coming from a file should not block the file from downloading it's data from a farmer into a blob store. I.E. calling createReadStream and not consuming data from it should not fill the TCP socket's stream's internal buffer, data should still be able to flow into the underlying blob store.

• getBuffer
• getBlob
• getBlobUrl
• appendTo
• renderTo
• getReadStream

The code here doesn't seem to match up with the documentation here.

Expected Behavior:
Generate a deterministic key from a Mnemonic and a file ID.

Actual Behavior:
Generate a deterministic key from a private signing key and a file ID.

Stack trace:

12:44:29.966 Error: Not enough entropy. Minimum is: 192 bits
assert() storj.js:6377
HmacDRBG() storj.js:6373
sign() storj.js:6359
[528]</</KeyPair.prototype.sign() storj.js:20111
[521]</</BridgeClient.prototype._authenticate() storj.js:19978
_request() storj.js:19970
retryAttempt() storj.js:3004
retry() storj.js:3004
[521]</</BridgeClient.prototype._request() storj.js:19970
[521]</</BridgeClient.prototype.createToken() storj.js:19823
createFile() storj.js:8
upload() 03:19
onchange() 03:1
1 storj.js:6377:26596

In browser when uploading.

Websockets are so 2016. Get with the hip HTTP transfer.

See docs here: http://storj.github.io/core/tutorial-data-transfer.html

Requiring that a user instantiate an entire storj object just to generate a mnemonic is silly.

Same for generateKeyPair although registering the keypair still requires auth.

The use case for generateKeyPair isn't there. If you aren't authenticated against the bridge, having generateKeyPair doesn't offer anything. Just focusing on mnemonic for now.

Currently implemented in https://github.com/Storj/storj.js/blob/NewAPI/lib/api/get-data.js

This method doesn't expose the file object to the user, instead it returns a readable stream of the data.

This should be documented in https://github.com/Storj/storj.js/blob/NewAPI/README.md

'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Unable to download libstorj for platform: win32 and arch: x64
npm WARN [email protected] requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @types/googlemaps@^3.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of @types/markerclustererplus@^2.1.29 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] preinstall: node ./download.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Caprice\AppData\Roaming\npm-cache_logs\2018-08-24T06_31_30_970Z-debug.log

https://github.com/Storj/storj.js#fileondata-function-cbdata-

Will require a PR to core, currently planned but not implemented.

request is not correctly handling 302 redirects from http to https, we need to correctly handle status codes like this.

Note this particular portion in readme.md

var stream = storj.download(fileId, bucketId)

Return a readable stream of decrypted data being pulled directly from farmers. This method bypasses the File API entirely, and doesn't use a backend abstract-blob-store. If you only need to download a file once, and don't want to hold onto it in memory, this is the method for you.

storj.download(fileId, bucketId).pipe(fs.createWriteStream('cat.jpg'));

The correct function should be storj.download(bucketId,fileId)

Will someone with access fix it, too small for me to file a pull request

libstorj has migrated to referring to the mnemonic as an encryptionKey (since we use it to derive the key used to encrypt/decrypt files).

Storj.js should follow suite.

Currently, core does not expose a way to track the upload progress of a file, which prevents storj.js from emitting on('data' events and updating file.progress.

Things to think about:

• If a file is being uploaded across multiple shards concurrently, the order in which chunks of data are emitted via the data event may not match the order of those chunks in the file. What implications does this carry for userland?
• When do we emit a data event? When we finish encrypting the chunk, when we send the data over the wire, when we get confirmation that the farmer received that data, etc? What implications will the timing of the data event carry for userland?

Storj.js is CPU heavy compared to most JS on the web. This creates a less-than-ideal user experience in many browsers.

The solution is Web Workers, but we need a little more exploration before we jump to an implementation.

Constraints:

• Multiple shards should be able to be decrypted simultaneously for the same file. (Web Worker pool)
• User shouldn't have to pass in a URL to their local storj.js script when initializing. (Should construct a blob via a string containing the JS, pull it's URL, and use that to initialize the Web Worker)
• Should implement a streaming interface.

Whenever we have an error-first callback, the callback should be required to force error acknowledgment. In cases where the callback is registered to the done event, the callback should be optional.

Those are secret, currently we write them to stdout.

TAP version 13
# Valid environment
# Running with [email protected]:password@http://127.0.0.1:32770

From @dylanlott at cpollard1001/storj.js#2

After trying to merge in billing changes to Bridge-GUI today and realizing that it's not going to (easily or consistently) work with the most recent storj-lib versions, it is most likely going to be come necessary to pull bridge client out of core, or create a better browser-only library for talking to Bridge instead of using the BridgeClient in storj-lib.

I've started work on this, but wanted to open up a thread for discussion between all parties already involved, since I saw that Storj.js requires a BridgeClient to work as well.

Any thoughts or ideas on this before we get too far in on it? I'm scaffolding out the library right now, and will be working on it tomorrow with @bryanchriswhite

Default when signing is { compact: true }. I've found that this needs to be { compact: false } in order to generate the correct signature.

Suggested change to lines 395-397 in README.md:

var signature = keypair.sign(message, [opts])
Sign a message with this key. message should be either a String or a Buffer, and the returned signature will be a String. Default options is { compact: true }.

or

var signature = keypair.sign(message, { compact: false })
Sign a message with this key. message should be either a String or a Buffer, and the returned signature will be a String. Default options is { compact: true }.

Keypair Docs

KeyPair.prototype.sign = function(message, options) {
  var sign = null;
  var opts = merge({ compact: true }, options);

  if (opts.compact) {
    sign = Message(message).sign(this._privkey);
  } else {
    if (!Buffer.isBuffer(message)) {
      message = new Buffer(message, 'utf8');
    }

    sign = ecdsa.sign(
      crypto.createHash('sha256').update(message).digest('hex'),
      this.getPrivateKey()
    ).toDER('hex');
  }

file.name, file.mimetype, and file.length don't appear to be getting set in the browser.

The documentation calls for a ready event to be emitted off of the Storj object during setup.

Currently there isn't any async setup necessary. In the original design of this API, there was a round trip for auth validation during setup. We decided to keep the async constructor for forward support of async operations during setup.

We can either:

A) drop the ready event and risk having to make a breaking change in the future
B) implement the ready event

I'm 👍 on (B)

It is implemented but not documented

Right now stack traces are super noisy in tests. Would be handy to have a source-map aware error object.

Look into:

• https://github.com/evanw/node-source-map-support
• https://www.npmjs.com/package/nyc#accurate-stack-traces-using-source-maps

I've made a browserified version of Storj.js using:

browserify lib/ -o storj.js

and when including it into the HTML like this:

<script src="storj.js"></script>

I get this error:

Uncaught ReferenceError: Storj is not defined
    at index.html:68

Here is the layout of my files:

storj-client
    |-- index.html
    `-- storj.js

Am I doing anything wrong or does it just not work?

this refers to the created file object, which defines createReadStream. So when createReadStream returns this.createReadStream() it is calling itself.

https://github.com/Storj/storj.js/blob/master/lib/file.js#L113

Crypto in JS is painfully slow. @nginnever has kicked off work exploring the WebCrypto API.

Requesting corresponding updateBucketById method to update bucket information

https://github.com/Storj/core/blob/master/lib/bridge-client/index.js#L256-L258