arablocks / ara-filesystem Goto Github PK
View Code? Open in Web Editor NEWThe Ara FileSystem
License: GNU Lesser General Public License v3.0
The Ara FileSystem
License: GNU Lesser General Public License v3.0
The estimateCommitGasCost
will default to deploying a proxy contract if one doesn't exist for the afs, even before estimating the gas, costing the user ether immediately.
When prompted for password on CLI, command line should hide password.
When prompted for password on CLI, command line doesn't hide the password until the password is submitted.
I should be able to commit via the CLI using a force
flag (-f, --force) to skip any confirmations prompts, specifically committing or setting a price.
Gas cost prompts always appear regardless of a force
flag.
When a user commits updates to an AFS, they should be sent as a single transaction.
Each individual change to the metadata/tree
and metadata/signatures
file each requires their own write, and thus, their own individual transactions.
Correctly catch issue if user creates AFS that is readonly but there is no current proxy.
Error is not caught / is not thrown correctly, and causes crash. In storage.js
, the create function can potentially pass a null value to _decode on read, which does not handle nulls.
call create({ did, password })
will clone/replicate/download home
and metadata
dirs of an AFS
call create({ did, password })
will write a staged.json
file in the local afs directory
staged.json
in directory)create({ did, password })
create({ did, password })
: ara-filesystem:storage metadata/signatures staged write at offset 0 size 32 +0ms
ara-filesystem:storage writing (and therefore appendingg ~~~~~. ) +0ms
ara-filesystem:commit APPENDING ~~~ +0ms
ara-filesystem:commit could not make dir at /Users/vipyne/.ara/afs/d58cb5c6921d3dd7e2bd462acc9d64b56f3ba713a115986755b4660dbf006e98/staged.json +1ms
ara-filesystem:commit BUT COULD YOU THO +0ms
ara-filesystem:commit { Error: EEXIST: file already exists, mkdir '/Users/vipyne/.ara/afs/d58cb5c6921d3dd7e2bd462acc9d64b56f3ba713a115986755b4660dbf006e98'
at Object.fs.mkdirSync (fs.js:905:18)
at _makeStagedFile (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/commit.js:216:8)
at generateStagedPath (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/commit.js:138:5)
at append (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/commit.js:106:16)
at RandomAccess.write [as _write] (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/storage.js:61:7)
at Request._run (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/random-access-storage/index.js:199:40)
at Request._unqueue (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/random-access-storage/index.js:155:50)
at Request.callback (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/random-access-storage/index.js:160:8)
at nextTickCallback (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/random-access-storage/index.js:249:7)
at process._tickCallback (internal/process/next_tick.js:152:19)
errno: -17,
code: 'EEXIST',
syscall: 'mkdir',
path: '/Users/vipyne/.ara/afs/d58cb5c6921d3dd7e2bd462acc9d64b56f3ba713a115986755b4660dbf006e98' } +0ms
ara-filesystem:storage metadata/tree staged write at offset 0 size 32 +3ms
ara-filesystem:storage writing (and therefore appendingg ~~~~~. ) +0ms
ara-filesystem:commit APPENDING ~~~ +2ms
ara-filesystem:storage metadata/signatures read at offset 160 size 64 +8ms
ara-filesystem:storage metadata/tree read at offset 72 size 40 +25ms
ara-filesystem:storage metadata/tree read at offset 192 size 40 +1ms
ara-filesystem:storage metadata/tree read at offset 32 size 40 +40ms
I think the issue is here:
https://github.com/AraBlocks/ara-filesystem/blob/master/commit.js#L199-L205
Use OpenZeppelin Ownable contract. It would be contract Storage is Ownable
or create a function that makes possible to update contract owner.
ara-filesystem/contracts/Storage.sol
Line 5 in b089cfe
File is added.
Command hangs because the file doesn't exist.
afs add <did> <file>
maddie@lscomputer:~$ afs add 7b65940677c88cd072616f323156c89b3ad28cd416852c19fbaf1e5602607fde ~/doggo.mp4 -D
ara: info: id: 7b65940677c88cd072616f323156c89b3ad28cd416852c19fbaf1e5602607fde
ara: info: paths: [ '/home/maddie/doggo.mp4' ]
? Please provide the passphrase for your identity. This is needed to complete this action.
Passphrase: [hidden]
(node:5289) DeprecationWarning: ara-crypto: crypto.decrypt() is deprecated. Please use 'ara-secret-storage' instead.
ara-filesystem:storage metadata/signatures staged write at offset 0 size 32 +0ms
ara-filesystem:storage metadata/tree staged write at offset 0 size 32 +2ms
(node:5289) DeprecationWarning: ara-crypto: crypto.encrypt() is deprecated. Please use 'ara-secret-storage' instead.
ara-filesystem:add copy start: /home/maddie/doggo.mp4 +0ms
ara-filesystem:add adding path /home/doggo.mp4 +1ms
When afs.add
is called subsequently toafs.create
, in the same node process, it should close the AFS.
Seems that unless the node process ends, the AFS remains open after adding. This is causing a problem for the content-manager, in where the publish function creates, adds, commits, and then broadcasts in a single step. It breaks when someone tries to download from the swarm broadcast.
Adding afs.close()
to the onfinish
function of add.js
will fix this.
$ afs -h
usage: afs: [-hDV] [--help] [--version]
[--debug] <command> [<args>]
Below are some useful afs commands:
afs create Create a new AFS
afs add Adds file(s) and/or directories to an AFS
afs remove Removes file(s) and/or directores from an AFS
afs commit Publishes an AFS to the network
afs history Prints AFS history
afs price Sets, gets the price (in ARA) of an AFS
afs destroy Removes an AFS from the network
afs unarchive Unarchive an AFS to a directory
afs metadata CRUD operation interface for an AFS' metadata
afs ownership Ownership management of an AFS
afs deploy Deploy an AFS proxy to the network
afs hr
Options:
--help, -h Show this message
--version, -V Show AFS CLI version
--verbose, -v Show verbose output
--debug, -D Enable debug output (Sets 'DEBUG=ara-filesystem:*')
--secret, -s Shared secret for the keyring [string]
--network, -n Network name of the key for the DID resolver in the keyring
[string]
--keyring, -k Path to the keyring [string]
$ afs hr
prompt:
(user types in up, up, down, down, left, right, left, right)
[ print out of hood rat ascii art ]
Encrypt/decrypt within AFS should not throw deprecation warnings, and should use ara-secret-storage
for these operations.
Encrypt/decrypt throws deprecation warnings as ara-crypto
encrypt/decrypt functions are still being used.
In https://github.com/AraBlocks/ara-filesystem/blob/master/contracts/Storage.sol#L58, there is a loop that iterates over the keys
array, which is an array that can grow indefinitely. It is recommended to never use unbounded loops unless you are 100% sure that the array will be kept small.
Ability to add files/directories to an AFS from non-cwd.
Only able to add from cwd.
Below is an example using mirror-folder
that allows the adding of files/directories to an afs from the non-cwd.
const { join, basename } = require('path')
const mirror = require('mirror-folder')
async function add(opts){
if (null === opts.did || 'string' !== typeof opts.did || !opts.did) {
throw new TypeError('ara-filesystem.add: Expecting non-empty did.')
}
if (null === opts.password || 'string' !== typeof opts.password || !opts.password) {
throw new TypeError('ara-filesystem.add: Password required to continue')
}
if (null === opts.paths || (!(opts.paths instanceof Array)
&& 'string' !== typeof opts.paths) || 0 === opts.paths.length) {
throw new TypeError('ara-filesystem.add: Expecting one or more filepaths to add')
}
const {
did, paths, password, force
} = opts
let afs
try {
({ afs } = await create({ did, password }))
} catch (err) {
throw err
}
for (let i = 0; i < paths.length; i++){
await mirrorPath(paths[i])
}
debug('full copy complete')
debug(await afs.readdir(afs.HOME))
return afs
async function mirrorPath(path){
debug(`copy start: ${path}`)
let name = afs.HOME
if (await pify(isFile)(path)) {
name = join(afs.HOME, basename(path))
}
const error = await pify(mirror)({ name: path }, {
name,
fs: afs
}, { keepExisting: true })
if (error){
debug(`copy error: ${path}: ${error}`)
} else {
debug(`copy complete: ${path}`)
}
}
}
The size of each entry in the buffer is passed to ara-contracts
storage.write()
, when it is never actually used. The calculation of sizes should be removed.
An AFS file accessed by it's Ara ID is read back from storage
ara:network:node:cdn received request for content AID 04aff33f4c4a3f45847aa9674e0ce8d96e6ea768c91f1c21b779ca8a7eccd1d8 from undefined +35s
ara:network:node:cdn afs list :: [ 'jSGeSHD.gif' ] +2ms
ara:network:node:cdn streaming content +0ms
ara:network:node:cdn received request for content AID 6b24991c55e0d21add967067321e1fddd48742320dadd84cac770da9147c194a from undefined +2s
ara-filesystem:storage metadata/signatures staged write at offset 0 size 32 +37s
ara-filesystem:storage metadata/tree staged write at offset 0 size 32 +1ms
ara-filesystem:storage metadata/signatures read at offset 352 size 64 +8ms
ara-filesystem:storage metadata/tree read at offset 152 size 40 +29ms
ara-filesystem:storage metadata/tree read at offset 392 size 40 +1ms
ara-filesystem:storage metadata/tree read at offset 32 size 40 +40ms
ara-filesystem:storage content/signatures staged write at offset 0 size 32 +20ms
ara-filesystem:storage content/tree staged write at offset 0 size 32 +1ms
ara-filesystem:storage content/signatures read at offset 4832 size 64 +3ms
ara-filesystem:storage content/tree read at offset 2552 size 40 +17ms
ara-filesystem:storage content/tree read at offset 5432 size 40 +1ms
ara-filesystem:storage content/tree read at offset 5912 size 40 +1ms
ara-filesystem:storage metadata/tree read at offset 352 size 40 +42ms
ara-filesystem:storage metadata/tree read at offset 432 size 40 +14ms
ara-filesystem:storage metadata/tree read at offset 112 size 40 +15ms
ara:network:node:cdn afs list :: [ 'states-of-a-programmer.png', 'jSGeSHD.gif' ] +382ms
ara:network:node:cdn streaming content +1ms
ara-filesystem:storage content/tree read at offset 152 size 40 +24ms
ara-filesystem:storage content/tree read at offset 392 size 40 +2ms
ara-filesystem:storage content/tree read at offset 512 size 40 +27ms
ara:network:node RangeError [ERR_INDEX_OUT_OF_RANGE]: Index out of range
ara:network:node at checkOffset (buffer.js:1020:11)
ara:network:node at Buffer.readUInt32BE (buffer.js:1096:5)
ara:network:node at Object.exports.decode (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/uint64be/index.js:24:17)
ara:network:node at Request._callback (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/hypercore/lib/storage.js:160:25)
ara:network:node at Request.callback (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/random-access-storage/index.js:161:8)
ara:network:node at RandomAccess.read [as _read] (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/storage.js:55:11)
ara:network:node at <anonymous>
ara:network:node at process._tickCallback (internal/process/next_tick.js:160:7) +53s
ara-network-node-cdn
develop
ara-filesystem branch
local blockchain running via Truffle
Version: 1.0.0
Platform: Mac OS
Subsystem: 10.12.6
id
and key
of afs should be equivalent
id
and key
of afs are different
identifier
and key
of AFSInstead of all AFSs living in ~/.ara/afs, allow a user to pass in a custom path ie.
$ afs create <araid> --path ./asdf/asdf/asdf
default path would still be ~/.ara/afs
Add an AFS CLI status command that functions similarly to git status
To remain consistent across the various Ara
modules, AFS mnemonics should be displayed similarly to ara-identity
when creating a new identity via the command line.
AFS displays AFS identity mnemonics differently.
Add an afs ls <did> <path>
command which prints out the current contents of the afs in the cli where did
is the did
of the afs and the optional path
parameter allows to ls for different sub-directories of the specified afs.
Creating read-only programmatic instance of AFS on different computers should have the same public key.
Creating read-only programmatic instance of AFS on different computers have different public keys.
AFS Create will successfully create all of the required folders
AFS Create throws uncaught error due to using a non-synchronous function and not passing a callback
ara-filesystem create <did>
If the owner's keystore is not local, it should fetch is remotely and be able to replicate it locally prior to needing to decrypt and re-encrypt.
Right now claiming ownership requires the keystore to be local to the machine that needs to do the decryption.
Be able to create an AFS with sparse replication.
Cannot create an AFS with sparse replication.
archiverNetwork and resolverNetwork should be passed from afs create cli
It isn't because it only destructures network
archiverNetwork
and/or resolverNetwork
Instead of using arithmetic operations directly, it is recommended to use something like the SafeMath library from OpenZeppelin. This will make the function revert if there is an overflow or an underflow.
At the beginning of the contract you can do using SafeMath for uint256
, and then you can do things like: return largest_key.add(buffer_mappings[identity][file].buffers[largest_key].length);
.
resolveBufferIndex should split using the '' character on windows systems
resolveBufferIndex splits using the '/' character on all platforms.
Error: Path is not properly formatted
at resolveBufferIndex (D:\Development\EnvironmentA\ara-filesystem\storage.js:103:11)
A user should be prompted to input their identity's password before being allowed to change metadata for a particular AFS.
A user can change any metadata of any AFS as long as they have the DID for that AFS.
require
instead of if
. It is a better practice. Using if it will not throw. It should stop fast and earlier.ara-filesystem/contracts/Storage.sol
Line 28 in b089cfe
Changing the rc file or passing corresponding options will change the behavior of ara-filesystem accordingly (change what keyring file is used, have a default DID, pass a resolver network name)
Inconsistent behavior. In some cases, options are correctly used but fallback on hard-coded values. In some cases, passed options will be ignored and only use the hard-coded values
keyringOpts
into ara create (or a few other functions)because ara-identity no longer requires a context object passed into it.
Currently, writeKey
only takes a single key and value, so you have to call it every time you want to add a new key, which means a new call to _readMetadataFile
and _writeMetadataFile
for every key pair. Since we'd be using this function programmatically in the DApp, and adding a handful of keys for every publish, a batchWriteKey
would be nice to have.
Happy to write it ourselves! API seems straightforward.
Ref : https://github.com/AraBlocks/ara-filesystem/blob/master/create.js#L274
metadata publicKey is being sent to create()
of ara-identity as,
const publicKey = {
"id" : "metdata",
"value" : "ec84465527bea0f8c54ce6c42c0d7549a7163336655a8bcaf731f07eb2997c73"
}
metadata publicKey should be sent as,
const publicKey = {
"id" : "metadata",
"publicKeyHex": "ec84465527bea0f8c54ce6c42c0d7549a7163336655a8bcaf731f07eb2997c73",
"publicKeyBase58": "GvGCfZpuJapzxoo3rgyaYR39XReFxaWpZZUvoaq1o8qg",
"publicKeyBase64": "OyERlUnvqD4xUzmxCwNdUmnFjM2ZVqLyvcx8H6ymXxz"
}
Code contains no semi-colons
Code contains two semi-colons when using aid.create.
The reasoning behind it is seemingly not by mistake. The issue was that if you removed those semi-colons, scoping issues would began happening when using ({ x } = y) statements. My assumption is that surrounding a statement in parenthesis creates a child scope which might cause issue if node is inferring when statements are complete. With the semi-colon, it was obvious to node but without, things get more hazy (My assumptions for future readers :))
It is considered best practice to use uint256
instead of uint
.
-h / --help
will always work without passing any options and exits successfully
If a command uses positionals, the command exits with code 1
Run any command with required options or positionals
Run echo $?
-V
prints out the current version of afs that is being used on the CLI.
Passing -V
as a flag should
Fix is to use yargs'
version()
function on parent and children processes.
afs.create
should gracefully throw if a did
does not exist, or no longer exists, even if multidrive has a reference on disk to where it thinks it should be.
Sometimes, still investigating when, afs.create
will crash without properly throwing an error that can be caught. The leading thought is that if a local afs folder is deleted, and potentially if there's no proxy, but there still exists a ref to it in the nodes folder.
Note: this isn't a huge issue, it's just a pain when testing and switching networks often.
The price of the AFS is checked before making a transaction to update it if it's already set the price that's being requested.
The transaction will be sent regardless, overwriting the existing price.
An error should be thrown when trying to call setPrice
with a value of 0.
The transaction will complete and cost gas, even if you are trying to set a price of 0.
When creating an AFS, the CLI needs the actual passphrase for your identity (rather than asking you to create a new one), so it should say "Please provide the passphrase for your identity."
When creating an AFS, the CLI responds with the statement "Please provide a passphrase for your identity."
$ afs add fa161f865d8a9... ~/Desktop/jSGeSHD.gif
ara: info: id: fa161f865d8a9620069455a758b21676bbb87a93ea374983a5e0efb43a1c4020
ara: info: paths: [ '/Users/vipyne/Desktop/jSGeSHD.gif' ]
afs: info: Reading [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 1.23 mbps 100% 0.0s
afs: info: Writing [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 1.23 mbps 100% 0.0s
$ afs add fa161f865d8a9... ~/Desktop/jSGeSHD.gif
ara: info: id: fa161f865d8a9620069455a758b21676bbb87a93ea374983a5e0efb43a1c4020
ara: info: paths: [ '/Users/vipyne/Desktop/jSGeSHD.gif' ]
/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/hyperdrive/index.js:669
if (name !== '/' && (err || !list.length)) return cb(err || new Error(name + ' could not be found'))
^
TypeError: cb is not a function
at /Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/hyperdrive/index.js:669:55
at onnodes (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/append-tree/index.js:116:33)
at onget (/Users/vipyne/Documents/littlstar/ara/ara-filesystem/node_modules/append-tree/index.js:358:21)
afs add
a file not located in current working directoryand test accordingly.
$ afs add b88592e85e99b790f22b21227aa6a1eecb2a2a738dd54c38e599978d538323ec /d/Littlstar/Test/web.mp4
Should successfully add file at path.
$ afs add b88592e85e99b790f22b21227aa6a1eecb2a2a738dd54c38e599978d538323ec /d/Littlstar/Test/web.mp4
results in:
Path: metadata\tree
Path: metadata\signatures
Path: content\tree
Path: content\signatures
D:\Development\EnvironmentA\ara-filesystem\node_modules\hyperdrive\index.js:668
if (name !== '/' && (err || !list.length)) return cb(err || new Error(name + ' could not be found'))
^
TypeError: cb is not a function
at D:\Development\EnvironmentA\ara-filesystem\node_modules\hyperdrive\index.js:668:55
at onnodes (D:\Development\EnvironmentA\ara-filesystem\node_modules\append-tree\index.js:116:33)
at onget (D:\Development\EnvironmentA\ara-filesystem\node_modules\append-tree\index.js:358:21)
at _combinedTickCallback (internal/process/next_tick.js:141:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
$ afs add b88592e85e99b790f22b21227aa6a1eecb2a2a738dd54c38e599978d538323ec web.mp4
Running any command that does not exist shows an error message.
An incorrect command will cause the CLI to hang.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.