GithubHelp home page GithubHelp logo

textury / arkb Goto Github PK

View Code? Open in Web Editor NEW
266.0 7.0 61.0 661 KB

Upload your files to the Arweave network.

License: MIT License

JavaScript 0.68% TypeScript 99.20% Shell 0.12%
arweave blockchain cli cli-tool

arkb's Introduction

arkb

Arweave Deploy that saves you data costs.

Features

  • No file size limit.
  • No amount of files limit.
  • Doesn't upload files that you have already uploaded.

How to use

arkb runs using NodeJS and NPM. You must have both installed on your machine for it to work.

Install arkb:

yarn global add arkb # recommended

or

npm install -g arkb

Note: The installation of arkb needs node >=15.11.0 or you will get an error when using arkb. To manage multiple active nodejs, we recommend you have nvm installed on your computer, then you can switch different node in a simple command.

And run:

arkb help

Note: If you are planning to upload large batches of data transactions to the Arweave network, it is strongly advised that you use the --use-bundler option instead of regular deploy to avoid transaction failures. You can read about bundles and their advantages on the Arwiki.

                    d8b        d8b
                    ?88        ?88
                     88b        88b
 d888b8b    88bd88b  888  d88'  888888b
d8P' ?88    88P'  `  888bd8P'   88P `?8b
88b  ,88b  d88      d88888b    d88,  d88
`?88P'`88bd88'     d88' `?88b,d88'`?88P'



Usage: arkb [options] [command]

Options                                 Description
--auto-confirm                          Skips the confirm screen
--concurrency -c <number>               Multi thread, default is 5
--content-type <content type>           Set the files content type
--debug                                 Display log messages
--fee-multiplier -m <number>            Set the fee multiplier for all transactions
--force -f                              Force a redeploy of all the files
--gateway -g <host_or_ip>               Set the gateway hostname or ip address
--help -h                               Show usage help for a command
--license                               Specify the license of your upload with an spdx li
--no-colors                             Print to terminal without fancy colors
--tag-name <name>                       Set a tag name
--tag-value <value>                     Set a tag value
--timeout -t <number>                   Set the request timeout
--bundle                                Locally bundle your files and deploy to Arweave
--use-bundler <host_or_ip>              Use an ans104 bundler service
--wallet -w <wallet_path>               Set the key file path

Commands (alias)                        Description
balance (b)                             Get the current balance of your wallet
deploy (d) <folder_or_file>             Deploy a directory or file
fund-bundler <amount>                   Fund your bundler account
help (h)                                Show usage help for a command
network (n)                             Get the current network info
status (s) <txid>                       Check the status of a transaction ID
transfer <address> <amount>             Send funds to an Arweave wallet
version (v)                             Show the current arkb version number
wallet-export (we)                      Exports a previously saved wallet
wallet-forget (wf)                      Removes a previously saved wallet
wallet-save (ws) <wallet_path>          Saves a wallet, removes the need of the --wallet option
withdraw-bundler <amount>               Withdraw from your bundler balance

Contributing

  1. Create a fork
  2. Create your feature branch: git checkout -b my-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request 🚀

arkb's People

Contributors

bianyuanop avatar bidhanroy avatar cedriking avatar christopheradams avatar fuadop avatar hlolli avatar hristo-mavrodiev avatar jfbeats avatar kouraf avatar rosmcmahon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

arkb's Issues

Possible Syntax error

arkb commands fail to run. I have tried updating the package, removing node modules then reinstalling dependencies, didn't work for me. I see the following error,

% arkb help
/usr/local/lib/node_modules/arkb/bin/commands.js:49
                    const name = tagNames[i]?.toString();
                                             ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/arkb/bin/app.js:9:36)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

Cache data uploaded elsewhere

Hi,

I think it could be helpful if arkb had some way to add to its cache of what's already deployed without uploading content to do so, for example by providing a txid or url to scan, from a previous run on another machine.

This could give users a way to deduplicate their uploads across machines without having to understand the cache internals.

Just a thought!

Is it save to upload pfp 10k images?

Welcome to the 21st century where everyone has a need to upload 10k image files to the permaweb. Can you tell me if arkb would be capable of doing this in one go? I'm just a bit scared of spending several hundred dollars of AR to just test it out.
Had anyone some experience with this?

10.000 images each 2-3MB each.

Bundler funds not fetched correctly

Hello! I've been unable to upload data to the bundler because no funds on the bundler have been recognized.
After some digging, I've discovered that the cause is in the Accept HTTP header.
Arkb is sending application/json, text/plain, */*, but in this case, the server responds with zero balance.
When I sent the request manually with Accept set to */* only, the server responded with the correct balance.
Not sure, if this is a problem with the arkb utility or with a configuration of that particular server.
Server used: http://bundler.arweave.net:10000/.

problem with arweave dep that's 1.11.4 (and potentially 1.11.1 - 1.11.4)

Error when running arkb with recent changes to arweave package.

/Users/philip/.nvm/versions/node/v16.15.1/lib/node_modules/arkb/node_modules/arweave/web/lib/crypto/webcrypto-driver.js:10
            throw new Error("SubtleCrypto not available!");
            ^

Error: SubtleCrypto not available!
    at new WebCryptoDriver (/Users/philip/.nvm/versions/node/v16.15.1/lib/node_modules/arkb/node_modules/arweave/web/lib/crypto/webcrypto-driver.js:10:19)
    at Object.<anonymous> (/Users/philip/.nvm/versions/node/v16.15.1/lib/node_modules/arkb/node_modules/arweave/web/common.js:120:18)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/philip/.nvm/versions/node/v16.15.1/lib/node_modules/arkb/node_modules/arweave/web/lib/deepHash.js:3:18)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)

This seems to be the case with a fresh install of arkb with a resolved dependency of [email protected]. In order to avoid this error, I forced the dependency to [email protected] and to [email protected] (the latter is needed because [email protected] has a dependency on [email protected]). I haven't tried arweave 1.11.1 - 1.11.3, but I suspect it's similar to 1.11.4.

This happened to me on node 18.3.0 and 16.15.1

Add --no-colors option

I'm integrating arkb into a deployment script where I'd like to be able to easily grep out the IPFS CID and Arweave ID of the uploaded file, but the various escape sequences in the output of arkb deploy make this challenging. I'm filtering the escape sequences out with sed right now, but if there were a --no-colors option for arkb it would be a lot easier to compose with other tools. For my purposes filtering out the colors with sed works, but the escape sequences used for the progress indicators still remain in the output file e.g. ^[[0G ^[[96m⣾ ^[[90mDeploying 2 files...^[[0m^[[0G ^[[96m⣽ ^[[90mDeploying 2 files... so maybe something more general than --no-colors would be appropriate (--simple? which could also remove the progress indicators).

need some help "Not Found"

so i uploaded 1000 files but i cant access them with the web api. is that normal? i never had this problem before.
for example:
"22.png":{"id":"Vl668XGb0bzyzaR-O1o13oKdGwFTbCJvQ5HtPNvvq-M"}
this is from the manifest file arkb generated.
can it happen that i payed for it but it didn't get mined? or that it's not hosted anymore?

Failed to upload file: Cannot read properties of undefined (reading 'toArray')

arkb deploy --debug
? Type your password [hidden]

TypeError: Cannot read properties of undefined (reading 'toArray')
at DERNode.encodeInt [as _encodeInt] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\encoders\der.js:198:26)
at DERNode.encodePrimitive [as _encodePrimitive] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:623:17)
at DERNode.encode [as _encodeValue] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:570:22)
at DERNode.encode [as _encode] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:490:23)
at DERNode.<anonymous> (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:543:25)
at Array.map (<anonymous>)
at DERNode.encode [as _encodeValue] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:532:30)
at DERNode.encode [as _encode] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:490:23)
at DERNode.encode [as _encodeValue] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:505:30)
at DERNode.encode [as _encode] (C:\Users\user\AppData\Roaming\npm\node_modules\arkb\node_modules\asn1.js\lib\asn1\base\node.js:490:23)

Arweave Wallet for testing:

{"kty":"RSA","d":"a4KC6dwI5USovoMMEnbu2T-eax4wJfedSRDoiS_tcsGs5McpcCaLiCK7-9y_dEk-6Q-qwGdDgauSqG2m_b-l8HRXUb4N4GEIpK4ilSinZWcJ_dK070GblmjNxxzqIedTAfWoRP7RNvLcxgZnpqC36wt4cEtBx81-S4IQJE7-i2YSdmz1Gb9Xdgy2suBGClYOeQPGavisKEnGpPDYL7KPpnOGC_iPyqh4UL8-oILo57CrfFHeLC2i-j2MLZyHAQPf5kFlSCJ2kXhWe71hOW6YAveXvN8rveWQTfLifTm_yXlWG0BrMAhdNK1nyvgsmNfk48l2aqUozcn57Qm72ug9C6xso1m8S58GAmrXDu3Zr6_Vgyym_Neo4QgTvBvL31qRzBa8rGUpiZBx3uo0eQy7gC7RZbgaJ3lBrT07Ec8wZflqBGmHjwKHngHsBdcFqSff-DCWS4q1_4lfmlQi2CVkKx560bNAn7L8QsmBf92KyrZTWQnhJHDS1U3tT_oiwM-60d7DHuEtOs5xX1Zj91JDSnpjHyV5LW82aC5OwhB0eamgsEm394dkVb48MDnCArfi4IzH3dNlml7SDcrlCBHFQfRSa3Tozdh4njOiGc9BbymC136Pr7ZFDQQ4C5HgHuYiCOEJHsAKExAqbQCIs13li7HaaaGh1NWu7GV3sO-RCgE","n":"qxQcZXYJnijgI3qS9vMil6UY5ioaBUHT1Zi9X3KExmE56nsmXb9XOAmfdYPqM4JJGbGXj8dvZnAS68tPkMCqKnftY-dO6bjI-0_WjFyYRsBm3YZl0MNLhDt5dqik1k6Px9zOqAikLyQ7AaRgU8rCJQFiB16_rb9OVQwsuBu3MJ43r0orvTg3rYd0UJNyHuS2Ea3vS-I7qm6fhzXXr_T04tlFqG9P4zW0b2kGgajst0cvYI8t15uyGiWrsgIBXVlVcA-vmejBz1kRE0_S4-8tKvrtECxPeFpxI1HVnvjTtQs0X0rV3ZLlAtHQGlristVNN7PVCWO3sM1reOPanAGQ7bPrMJ6NyvSrwbTmmuNavz9jFhCCf9HjfHr3qvtrZEn6fZhpjhqyUf8IoVSsW7y3zYk6rpHs7UF5tQLwgdhsmaaMiGC3wP6O3RtsGytXx1H_hCoyUcdAUg13-HbgHn-yKkKIOLIzdEpg7yVkbB2peGTJZc6GgzwXBtFTh5xxTXEHicusnaUUnQUP9hiUN1WRZK1dLiyofe6Yv8fCjicN3saaV9Nkbn4rxosQW4OMjWpzbJBVaMoH9nJIPyrck1eraKATs7EjTYPA1qBcCvyVE7FJxdpnXhUCA-Fh_ug7t-qm6AHy5CvEySgOleppMzRxjMZLAzm0R5TpoVZKFx-si80","e":"AQAB"}

Concurrent use?

Is it safe to use arkb in many concurrent processes?

I briefly visited the cache code and did not see locks around its state. I'm worried cache entries could be lost.

Incorrect Content-Types with folder deploy

When deploying a website with 69 files + manifest, many of the file transactions were set with Content-Type that belonged to a different file, causing the website to not display properly.

The deploy command:
arkb deploy ./silanael.com --ipfs-publish --tag-name=page:url --tag-value=http://www.silanael.com --tag-name=page:title --tag value=Silanael.com

Example transaction - this is a HTML file, yet it has Content-Type of another file:
https://viewblock.io/arweave/tx/IDDj_ftMcK5GPUa_u_CNjbY8js04YKmfnh5dmwfc25I

It shows properly in the ARBK deploy confirmation listing:
Screenshot from 2021-10-10 13-24-26

This is some kind of array indexing issues, as another HTML-file got the content type
"application/pgp-signature" which belongs to file "silanael-F3E9A1F85B37D0FE-pubkey.asc".
(The PGP-key has image/png ...)

Thanks to you, I now have a lot of garbage data in my address with incorrect types, hopefully not extending to IPFS.

ReferenceError: TextDecoder is not defined

Fresh install, MacOS Mojave with npm install @textury/arkb

/.../arweave-deploy-testing/node_modules/multibase/src/util.js:3
const textDecoder = new TextDecoder()

package.json:

{
  "name": "arweave-deploy-testing",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "deploy": "arkb deploy deploy/index.html --wallet assume/this/is/correct.json",

  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@textury/arkb": "^1.1.4"
  }
}

Install Issue

Hello! First off, really great stuff with this tool, thank you for your work.

I have a script which downloads the .json data for all parcels from Cryptovoxels (metaverse), and makes a backup of the .zip file using this tool. My script worked about 3-4 weeks without any issues, but about 12 days ago it started throwing the following error:
image

Any thoughts around this error? Nothing about that repo and automation through GitHub Actions has changed at all between when it was working and now.

I am running npm install -g @textury/arkb and it doesn't error out for the runner, but it does give a warning that was not there while it was still working, which seems to be the source of the problem, but I'm not sure how to navigate/fix it:
image

I tried reproducing on my M1 Macbook Air but I can't get the install to work, and not really an expert in this particular area quite yet.

Would love any thoughts on the topic! Thank you for this tool!

Static website deployed with arkb doesn't work

I deployed my static website here
But assets not loaded.

I deployed it with this command.

arkb deploy dist --use-bundler https://node2.bundlr.network --tag-name DEPLOY --tag-value foobar

It seems manifest.arkb was not created in dist folder.
Maybe it's the problem i guess.

Fails using https://ar-io.net/ as gateway with bundler

For some reason ar-io.net is truncated to ar for the hostname.

$ arkb deploy --gateway https://ar-io.net --use-bundler https://node2.bundlr.network --debug deploy README.md
�[31mSave a wallet with `arkb wallet-save file-path.json`.�[39m
�[0G�[2K�[0G�[2KRequesting: https://ar:443/tx_anchor
AxiosError: getaddrinfo ENOTFOUND ar
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
  hostname: 'ar',
  syscall: 'getaddrinfo',
  code: 'ENOTFOUND',
  errno: -3008,
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [],
    timeout: 20000,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: 536870912,
    maxBodyLength: -1,
    env: { FormData: [Function] },
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'axios/0.27.2'
    },
    baseURL: 'https://ar:443',
    method: 'get',
    url: 'tx_anchor',
    data: undefined
  },
  request: <ref *1> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError],
      socket: [Array]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'https:',
      path: '/tx_anchor',
      method: 'GET',
      headers: [Object],
      agent: undefined,
      agents: [Object],
      auth: undefined,
      hostname: 'ar',
      port: '443',
      nativeProtocols: [Object],
      pathname: '/tx_anchor'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'GET /tx_anchor HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.27.2\r\n' +
        'Host: ar\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/tx_anchor',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'ar',
      protocol: 'https:',
      _redirectable: [Circular *1],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: true,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(kUniqueHeaders)]: null
    },
    _currentUrl: 'https://ar:443/tx_anchor',
    _timeout: null,
    [Symbol(kCapture)]: false
  }
}

manifest URL points to uploaded file

i uploaded 3 txt files for testing purposes.

what i would have expected:

the manifest ID would point to the manifest json file, containing the infos for the uploaded files.

actual behaviour:

the manifest ID points to file test2.txt (1st uploaded file)

the other 3 IDs point to the txt files as expected.

is that a bug, or am i missing something here?

actual output (minus the wallet part):

arkb deploy ./test --wallet keyfile.json --no-colors 
ID                                           Size           Fee              Type                          Path
CYkaOpn0AAf-pVeBsFJHIOJyEInXbdyWIIgDwRm79Nw  4.00 Bytes     0.000073237704   text/plain                    test2.txt
XQdNiyayy37kJWriRZktJC-XuBzy5hg2AzJzSz_02us  4.00 Bytes     0.000073237704   text/plain                    test1.txt
fk3_1TjnFiy94P-3BiqQ_G2NHI2ySka_Yf-ktw2BCh4  4.00 Bytes     0.000073237704   text/plain                    test3.txt
RqYfrkI89o6CHZTFTz42uVNZAFi9_3_PWHNJf5t2Bkc  280.00 Bytes   0.000073237704   application/x.arweave-manifest

Summary
All items will be deployed in a single bundle
Total size: 292.00 Bytes
Fees: 0.000292950816 + 0.000029295081 (10% arkb fee )
Total fee: 0.000322245897

[wallet info]

? Carefully check the above details are correct, then confirm to complete this upload Yes

Files deployed! Visit the following URL to see your deployed content:
https://arweave.net/RqYfrkI89o6CHZTFTz42uVNZAFi9_3_PWHNJf5t2Bkc
✨  Done in 33.78s.

How can I upload a folder from browser?

Is it possible to use arkb to upload a folder to Arweave. My use case is to allow the user to select a folder that contains a static website and associated assets and deploy it to Arweave.

Bundlr Fees

node1.bundlr.network apparently charges a 30% fee.

it would be great if arkb were to report and include this

Ability to specify custom content type during uploads

Hey Textury team! Thank you for the great product.

I'm wondering if there's to add another option, something like --content-type to the CLI. I say this because I'm attempting to upload JSON files without the .json file extension and they are being uploaded as application/octet-stream. I would like to be able to upload with the application/json content type instead.

Needs upgrade to new arbundles + arweave libraries.

The arbundles and arweave libraries are significantly out of date.

The bundlr interface has been changing slowly, so it seems valuable to keep arbundles up to date.

I ran into some small breaking interface changes importing from arbundles when I attempted a naive upgrade.

I found this looking for workarounds to #87 .

Index path set

Would be nice if we could somehow set index.path value, currently it goes by default to the first item in the list, sometimes it could be useful to set it also?

add an --auto-confirm option

It is currently not possible to call the arkb cli for uploading in a script because the CLI will hang at the CONFIRM step

No matching version found for arbundles@^0.6.19

➜ npm --version
8.11.0
➜ node --version
v16.15.1
➜ npm install -g arkb
npm WARN config global --global, --local are deprecated. Use --location=global instead.
npm ERR! code ETARGET
npm ERR! notarget No matching version found for arbundles@^0.6.19.
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

Guidance - Few files fail to upload when handling with large amount of data

This is not a bug report, rather a request of guidance

I'm trying to upload via arkb a large amount of data (around 6k files, with a total size of 2GB). This is the command I'm using

arkb deploy path_to_files/ --ipfs-publish --use-bundler https://node1.bundlr.network --wallet ~/my-wallet.json

The upload works but a (very small) number of files fails (on average, 0.4% of the total).

How can I fix those that did not upload? Cause I got an error in the console (a JSON output), with a red string saying
Failed to deploy data item: /file_name.csv

Of course even if just one file is not uploaded, for me is a broken upload, and I have to re-run everything. Appreciate any suggestion, thanks.

The full error I see in console is the following

Deploying 336 files...Error: Request failed with status code 402
    at createError (/usr/local/share/.config/yarn/global/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/local/share/.config/yarn/global/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/local/share/.config/yarn/global/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) 

{ ... a big json with the axios request / response dump ... }

Failed to deploy data item: /file_name.csv

The only interesting part I can see about the Axios JSON is a field in the response saying

{
 "response": {
   "status": 402,
   "statusText": "Payment Required",
   ...
 }
}

adding tags to arkb deploy

after calling arkb with --tag-name name --tag-value value, when inspecting the completed transaction, this user-specified tag does not appear on the transaction tags list

Doesn't work on macbook m1 chip

Steps to replicate

install arkb on Macbook M1
OS Monterey
Node v14.18.2

run command arkb help

expected:
help output

actual:

/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/utils/crypto.js:16
            this.driver = crypto_1.webcrypto.subtle;
                                             ^

TypeError: Cannot read property 'subtle' of undefined
    at new CryptoDriver (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/utils/crypto.js:16:46)
    at Function.<static_fields_initializer> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/blockweave.js:53:21)
    at Object.<anonymous> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/blockweave.js:37:18)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:93:18)
    at Object.<anonymous> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/bin/utils/utils.js:10:38)
r@rs-MacBook-Pro nft-marriage % arkb     
/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/utils/crypto.js:16
            this.driver = crypto_1.webcrypto.subtle;
                                             ^

TypeError: Cannot read property 'subtle' of undefined
    at new CryptoDriver (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/utils/crypto.js:16:46)
    at Function.<static_fields_initializer> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/blockweave.js:53:21)
    at Object.<anonymous> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/node_modules/blockweave/dist/blockweave.js:37:18)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:93:18)
    at Object.<anonymous> (/Users/r/.nvm/versions/node/v14.18.2/lib/node_modules/arkb/bin/utils/utils.js:10:38)

how to check the status of txid that being with a '-' character?

PS C:\Users\ros> arkb status -hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ
v1.1.4
PS C:\Users\ros> arkb status -"hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ"
v1.1.4
PS C:\Users\ros> arkb status -- -hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ
v1.1.4
PS C:\Users\ros> npx arweave-deploy status -- -hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ
Trasaction ID: -hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ

Status: 200 Accepted 🚀

 - Block: 418511
 - Block hash: qNnun20oWVqsVFcWxK9EwfojcZXXPww-rYK27xQoOlCWNSYp0dCjPkNOI3qfwBVp
 - Confirmations: 347338

URL: https://arweave.net/-hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ
Block URL: https://arweave.net/block/hash/qNnun20oWVqsVFcWxK9EwfojcZXXPww-rYK27xQoOlCWNSYp0dCjPkNOI3qfwBVp

Block explorer URL: https://viewblock.io/arweave/block/418511

PS C:\Users\ros> arkb status -- -hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ
v1.1.4
PS C:\Users\ros> arkb status -- "-hvARuCZcnPxcdiqhuSx_qwVKDTsBv3aq6Inz5NiheQ"
v1.1.4

Arkb reports success, but url not found on arweave

We use arkb from automation to upload claims files, but found this past week that the file uploaded is not available on arweave.

The url reported by arkb is https://arweave.net/PX2jXfxTlWaIWkiwKKRtjgwfX7NaIoXSvddRPf61YzE.

  • Is there any way to trace why this didn't become active?
  • Is there a way to prevent this situation in arkb?

Issues using the bundler option

Hey, thanks a lot for this amazing package!

When trying to use the bundler I am facing 402 issues even tho I have enough AR on my wallet, uploading without the bundler is working fine.
At the end of the log after all the errors there is a success message pointing to the gateway even tho it failed.

arkb deploy --wallet /path-to-my-wallet.json . --use-bundler http://bundler.arweave.net:10000

Failed to deploy data item: /Users/.../build/images/3.png
Error: Request failed with status code 402
    at createError (/Users/leo/.nvm/versions/node/v16.11.1/lib/node_modules/arkb/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/leo/.nvm/versions/node/v16.11.1/lib/node_modules/arkb/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Users/leo/.nvm/versions/node/v16.11.1/lib/node_modules/arkb/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (node:events:402:35)
    at endReadableNT (node:internal/streams/readable:1343:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    url: 'http://bundler.arweave.net:10000/tx',
    method: 'post',
    data: ReadStream {
      fd: null,
      path: '/var/folders/4s/7jb4mzk50kg012zrf3qzghyw0000gn/T/tmp-47857-ERuQI9HNKA1A',
      flags: 'r',
      mode: 438,
      start: undefined,
      end: Infinity,
      pos: undefined,
      bytesRead: 208404,
      closed: true,
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      [Symbol(kFs)]: [Object],
      [Symbol(kIsPerformingIO)]: false,
      [Symbol(kCapture)]: false
    },
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/octet-stream',
      'User-Agent': 'axios/0.21.4'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 100000,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: Infinity,
    validateStatus: [Function: validateStatus],
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    }
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: true,
    shouldKeepAlive: false,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: null,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'bundler.arweave.net',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      timeout: 100000,
      [Symbol(async_id_symbol)]: 1599,
      [Symbol(kHandle)]: [TCP],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: Timeout {
        _idleTimeout: 100000,
        _idlePrev: [TimersList],
        _idleNext: [TimersList],
        _idleStart: 43643,
        _onTimeout: [Function: bound ],
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: false,
        [Symbol(refed)]: false,
        [Symbol(kHasPrimitive)]: false,
        [Symbol(asyncId)]: 1629,
        [Symbol(triggerId)]: 1601
      },
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'POST /tx HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/octet-stream\r\n' +
      'User-Agent: axios/0.21.4\r\n' +
      'Host: bundler.arweave.net:10000\r\n' +
      'Connection: close\r\n' +
      'Transfer-Encoding: chunked\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype],
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/tx',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 402,
      statusMessage: 'Not enough balance for transaction',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://bundler.arweave.net:10000/tx',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 18,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'bundler.arweave.net',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 208404,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://bundler.arweave.net:10000/tx',
      _timeout: null,
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: true,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      'user-agent': [Array],
      host: [Array]
    }
  },
  response: {
    status: 402,
    statusText: 'Not enough balance for transaction',
    headers: {
      date: 'Sun, 31 Oct 2021 08:57:58 GMT',
      'content-type': 'text/plain; charset=utf-8',
      'content-length': '34',
      connection: 'close',
      server: 'nginx/1.18.0 (Ubuntu)',
      vary: 'Origin',
      'x-ratelimit-limit': '10000',
      'x-ratelimit-remaining': '9990',
      'x-ratelimit-reset': '1635670732'
    },
    config: {
      url: 'http://bundler.arweave.net:10000/tx',
      method: 'post',
      data: [ReadStream],
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 100000,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      transitional: [Object]
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [Socket],
      _header: 'POST /tx HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/octet-stream\r\n' +
        'User-Agent: axios/0.21.4\r\n' +
        'Host: bundler.arweave.net:10000\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/tx',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'bundler.arweave.net',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: 'Not enough balance for transaction'
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}
Failed to deploy data item: /Users/.../build/images/2.png

Data items deployed! Visit the following URL to see your deployed content:
https://arweave.net/AgC_RBWsNy6g4gc5DJmqewi0fYnH0-Ed-XNK8Z501IE

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.