ethereumjs / rlp

Project is in active development and has been moved to the EthereumJS monorepo.

rlp

Recursive Length Prefix encoding for Node.js and the browser.

INSTALL

npm install rlp

install with -g if you want to use the CLI.

USAGE

import assert from 'assert'
import RLP from 'rlp'

const nestedList = [[], [[]], [[], [[]]]]
const encoded = RLP.encode(nestedList)
const decoded = RLP.decode(encoded)
assert.deepEqual(nestedList, decoded)

API

RLP.encode(plain) - RLP encodes an Array, Uint8Array or String and returns a Uint8Array.

RLP.decode(encoded, [stream=false]) - Decodes an RLP encoded Uint8Array, Array or String and returns a Uint8Array or NestedUint8Array. If stream is enabled, it will just decode the first rlp sequence in the Uint8Array. By default, it would throw an error if there are more bytes in Uint8Array than used by the rlp sequence.

CLI

rlp encode <JSON string>
rlp decode <0x-prefixed hex string>

Examples

• rlp encode '5' -> 0x05
• rlp encode '[5]' -> 0xc105
• rlp encode '["cat", "dog"]' -> 0xc88363617483646f67
• rlp decode 0xc88363617483646f67 -> ["cat","dog"]

TESTS

Tests use mocha.

To run tests and linting: npm test

To auto-fix linting problems run: npm run lint:fix

CODE COVERAGE

Install dev dependencies: npm install

Run coverage: npm run coverage

The results will be at: coverage/lcov-report/index.html

EthereumJS

See our organizational documentation for an introduction to EthereumJS as well as information on current standards and best practices.

If you want to join for work or do improvements on the libraries have a look at our contribution guidelines.

rlp's Issues

Rewrite in TypeScript

Part of ethereumjs/organization#28

Should update to safe-buffer

e.g. const Buffer = require('safe-buffer').Buffer and use Buffer.from / Buffer.alloc instead of new Buffer

rlp encodes 0 to 00

Hey,

Here's a problems with encoding of 0:

console.log(rlp.encode(0).toString('hex'));

prints 00 but should print 80.

Because of that ethereumjs-vm generates wrong new address if contract was created from an account with nonce 0.

npm test does not work in Windows

You can change package.json removing explicit use of node_modules

from

  "scripts": {
    "test": "standard ./index.js ./bin/rlp &&  ./node_modules/mocha/bin/mocha --reporter spec ./test/",
    "webtest": "browserify test/max.js | testling -u"
  },

to

  "scripts": {
    "test": "standard ./index.js ./bin/rlp &&  mocha --reporter spec ./test/",
    "webtest": "browserify test/max.js | testling -u"
  },

Cannot find module 'bn.js' error when requiring after clean install

To reproduce:

1. Run npm init -y and npm install rlp
2. In that directory, run node and try to import rlp module with require('rlp')
3. Following error appears:

$ node
> require('rlp')
Error: Cannot find module 'bn.js'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
>

This is probably due to bn.js listed as a devDependency in package.json, instead of as a regular dependency. Moving bn.js to dependencies should fix this error.

[BUG] rlp.encode([]) throws invalid type Error in the Truffle console

Describe the bug
Encoding an array does not work in truffle console while it works in the node console.

To Reproduce
Steps to reproduce the behavior:

$ mkdir testDir && cd testDir
$ truffle init
$ npm install --save-dev rlp
$ node
> const rlp = require('rlp')
> rlp.encode()
<Buffer 80>
> rlp.encode([])
<Buffer c0>
> .exit
$ truffle develop
trufle(develop)> const rlp = require('rlp')
trufle(develop)> rlp.encode([])
<Buffer 80>
trufle(develop)> rlp.encode([])
Error: invalid type
    at toBuffer (/home/james/Temp/test/node_modules/rlp/dist/index.js:232:19)
    at Object.encode (/home/james/Temp/test/node_modules/rlp/dist/index.js:20:24)
...

Expected behavior
Encode array without any error also in the truffle console

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu18.04
• zsh: v5.4.2
• Node: v10.15.0
• Truffle: v5.0.2
• rlp: v2.2.2

Run tests on node 4, 6, 8

Encoding and Decoding RLP lists when part of a buffer

I am encountering a problem in which I try to RLP decode some data that was a part of a larger buffer. Here's an example to illustrate

let buf1 = Buffer.from('a')
let buf2 = RLP.encode([1, 2, 3])
let buf3 = Buffer.from('b')
let final_buf = Buffer.concat([buf1, buf2, buf3])

// Slicing
let letter_a = final_buf.slice(0,1)
let rlp_list = final_buf.slice(1,4)
let letter_b = final_buf.slice(4,5)

// Error occurs when trying to index the decoded rlp_list.
// rlp_list[i] is a number instead of an array element since rlp_list,
// upon being decoded is a buffer and not an array of buffers.
rlp_list = RLP.decode(rlp_list)

I am using typescript but the above example applies. How do I remedy this?

make streamable

its possible. we should do it

Add Ethereum name or address to package.json

I would like to see an Ethereum name or address added to the ethereum field of the package.json file. This will allow consumers of the project to know a cannonical address to send funds to, for example donations.

I'm building an application that will allow developers to automatically donate ETH to all of their npm dependencies that have Ethereum addresses in their package.json files. I'm not quite ready to share the project, but I'd love to see an Ethereum address in this project's package.json file.

ES5 Compatibility

Hi there,

Downstream projects using RLP that try to target ES5 cannot use this library since it's published code is not compiled using Babel.

In the case of 0x, we are unable to minify our webpack bundle using UglifyJs since it only supports ES5. By publishing this library in ES6 syntax rather then compiling it to the lowest denominator, downstream projects are forced to either fork & compile to ES5, or use something else.

Please consider keeping your source code in ES6 syntax, but compiling it to ES5 before publishing the library to NPM.

Improve type definitions for decode

Thinking abstractly about decode I'd expect a function signature such as:

export function decode(input: Buffer): NestedBufferList

where NestedBufferList could be any nested composition of buffers and arrays such as Buffer, or Buffer[], or [Buffer, Buffer[]], etc. (haven't tried expressing this in TS).

Currently the decode method has the following definitions:

rlp/src/index.ts

Lines 60 to 63 in f06a96f

	export function decode(input: Buffer, stream?: boolean): Buffer
	export function decode(input: Buffer[], stream?: boolean): Buffer[]
	export function decode(input: Input, stream?: boolean): Buffer[] | Buffer | Decoded
	export function decode(input: Input, stream: boolean = false): Buffer[] | Buffer | Decoded {

1. First definition takes Buffer and returns a Buffer. Not sure why, it could very well be that the decoding is a NestedBufferList. This was also the problem stated in #77
2. Second def. takes a Buffer[]. Is this to support decoding multiple serialized items simultaneously? the code doesn't seem to do that. I suggest we remove this
3. The last two take Input as argument. I imagine this is for backwards-compatibility. I suggest we tighten the API and only accept a Buffer

On the stream flag: we could keep it, or add a new function decodeStreaming.

Reach 95%+ code coverage with tests

Missing type exports: RLPInput, RLPDecoded

The types RLPInput, RLPDecoded cannot be used in caller code because the index.d.ts does not export them.

A workaround is

// tslint:disable-next-line:no-submodule-imports
import { RLPInput } from "rlp/dist/types";

Having the types available is needed when you want to build a wrapper in TS like

export function toRlp(data: RLPInput): Uint8Array {
  const dataBuffer = rlp.encode(data);
  return Uint8Array.from(dataBuffer);
}

Some love for CLI tool

The CLI tool could really need some love here with:

• Some simple parameter check
• Some help/example output on wrong usage + error cases
• An explicit example for encoding and decoding in the README (e.g. from the tests: rlp encode "[ 5 ]")

Consider having TypeScript/Flow interface definition

Do you have any plan to have TypeScript/Flow interface definition added? There are a few benefits:

• That would help a lot to improve safety
• Enable intellisense for RLP using VS Code
• Convert to BuckleScript/ReasonML ( https://rrdelaney.github.io/ReasonablyTyped/ )

Update TypeScript to turn off allowSyntheticDefaultImports

The TypeScript configuration currently used allows for the incorrect use of libraries that don't publish ES6 modules via the allowSyntheticDefaultImports compiler option. We should disable that.

Is the Dictionary type necessary?

The Input definition includes a Dictionary type, even though Dictionary doesn't seem to be used in the code and toBuffer doesn't explicitly handle it. The RLP reference hasn't mentioned anything about objects either. I was wondering if it can be removed, or there's a reason why it's there?

Add code coverage and coveralls support

Make a release

After #16 is merged.

encode differs from pyrlp

below is my py code:

info = ['hffpcointomoon', 256, 41, 'fractal.account', 10000000, '0x1', '0xf8649068666670636f696e746f6d6f6f6e31398e68666670636f696e746f6d6f6f6eb841049112dda5f665da48458d3be328feed0ab6a897afb1c2c28ac8fa0b856c4ea867f01883f066ebb2229e6d373419896bf64ca1417fa30d0c5b9c8f6d63c7d9684e80', 1, 0, 0]

info = rlp.encode(info)

for x in info:
    print(x)

i got this:
248 254 142 104 102 102 112 99 111 105 110 116 111 109 111 111 110 130 ....

below is my js code:

const {encode} = require('rlp')

let info = ['hffpcointomoon', 256, 41, 'fractal.account', 10000000, '0x1', '0xf8649068666670636f696e746f6d6f6f6e31398e68666670636f696e746f6d6f6f6eb841049112dda5f665da48458d3be328feed0ab6a897afb1c2c28ac8fa0b856c4ea867f01883f066ebb2229e6d373419896bf64ca1417fa30d0c5b9c8f6d63c7d9684e80', 1, 0, 0]

let enc = encode(info)

for(let v of enc){
     console.log(v)
}

i got this:
248 147 142 104 102 102 112 99 111 105 110 116 111 109 111 111 110 130 1 0 41 143 ...

you can see that the results are different. I hope anyone can give me some help.

below is the code address:

pyrlp
ethereumjs/rlp

Question, why is decode(encode([0])) empty buffer?

Description

I wonder why decode(encode([0])) is empty buffer.

> RLP.decode(RLP.encode([0, 1, 2]))
[ <Buffer >, <Buffer 01>, <Buffer 02> ]

Consider following the API spec of abstract-encoding

See: https://github.com/mafintosh/abstract-encoding

Error using [email protected] as a CLI command

We used to use [email protected] as a CLI command having installed it with npm i -g rlp. Now that we've updated to [email protected]:

$ sudo npm i -g rlp
/usr/bin/rlp -> /usr/lib/node_modules/rlp/bin/rlp
/usr/lib
└── [email protected]

we've faced an error:

$ rlp encode <whatever>
module.js:549
    throw err;
    ^

Error: Cannot find module './dist/index.js'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/usr/lib/node_modules/rlp/bin/rlp:3:13)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

If we vim /usr/lib/node_modules/rlp/bin/rlp we'll see that there's ./dist/index.js included in the beginning which wasn't the case for [email protected]:

#!/usr/bin/env node

const rlp = require('./dist/index.js')
const command = process.argv[2]
var raw = process.argv[3]
...

I'm submitting this as a bug but maybe I'm supposed to install it differently or something to make it work?

Should rlp support bigint or mediumint in ethereum/tests?

Recently I run ethereal/tests/RLPtests with this library, the encoded medium int and big int were different because the number type was String.
Because of loss of precision of number, the encoded value were also different if number type was Number.
There was no further information In RLP wiki, so I post here.

Should rlp support bigint or mediumint in ethereum/tests?
Thanks!

rlp.decode ressources exhaustion

Hello,
While playing with thelonious and ethereumjs-lib, i came to this case

var  rlp = require('rlp');
//json in order to be copy/paste compliant
var data={"type":"Buffer","data":[239,191,189,239,191,189,239,191,189,239,191,189,239,191,189,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,239,191,189,29,239,191,189,77,239,191,189,239,191,189,239,191,189,93,122,239,191,189,239,191,189,239,191,189,103,239,191,189,239,191,189,239,191,189,26,239,191,189,18,69,27,239,191,189,239,191,189,116,19,239,191,189,239,191,189,66,239,191,189,64,212,147,71,239,191,189,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,239,191,189,11,222,155,122,54,42,194,169,239,191,189,70,239,191,189,72,239,191,189,239,191,189,54,53,239,191,189,100,73,239,191,189,55,239,191,189,239,191,189,59,1,239,191,189,109,239,191,189,239,191,189,93,239,191,189,208,128,239,191,189,239,191,189,0,239,191,189,239,191,189,239,191,189,15,66,64,239,191,189,239,191,189,239,191,189,239,191,189,4,239,191,189,79,103,239,191,189,85,239,191,189,239,191,189,239,191,189,74,239,191,189,239,191,189,239,191,189,239,191,189,54,239,191,189,239,191,189,239,191,189,239,191,189,239,191,189,83,239,191,189,14,239,191,189,239,191,189,239,191,189,4,63,239,191,189,63,239,191,189,41,239,191,189,239,191,189,239,191,189,67,28,239,191,189,239,191,189,11,239,191,189,31,239,191,189,239,191,189,104,96,100,239,191,189,239,191,189,12,239,191,189,239,191,189,206,152,239,191,189,239,191,189,31,112,111,239,191,189,239,191,189,65,239,191,189,41,239,191,189,239,191,189,53,84,11,239,191,189,239,191,189,12,102,24,12,42,105,109,239,191,189,58,239,191,189,4,239,191,189,104,82,9,239,191,189,6,66,91,43,38,102,117,239,191,189,105,239,191,189,239,191,189,239,191,189,89,127,239,191,189,114]};

var ble = new Buffer(JSON.parse(data))
console.log(ble);
var zor =  rlp.decode(ble);

At this point the node process exhausted its own ressource until being killed by oom.
I only tried on debian, but it afaik the same string can be decoded by pyethereum.

Best :)

Bug. Decoding 0x80 / Encoding and decoding of 0

The rlp encoding of 0 is 0x80.

decoding 0x80 should return 0x0. The source returns an empty buffer which is incorrect.

rlp.decode(rlp.encode(0)) returns Buffer.from([]) instead of 0.

Analyze new listofbinaries RLP test case

Running a new test in discussion on the tests (see ethereum/tests#554 respectively ethereum/tests#562) repo fails, it should be analyzed why.

Remove ethereumjs-testing dependency ?

Should we remove ethereumjs-testing dependency here? This is a pretty heavy dependency with 600+MB for just a single file in use which hardly ever changes.

We could just copy this over into the tests/ folder.

Add test case title output on official test runs

Currently on running the official RLP tests from the generic test suite there is just a single line of output on the result like:

offical tests
  ✓ pass all tests

There should be one test case named one line of output for every testcase to make it explicit what tests cases are actually run.

This is an easy fix and helps to avoid issues on eventually having outdated tests running (caching stuff or the like) and also helps on accompanying changes on the tests library like ethereum/tests#554.

Handle negative numbers

Encoding BN instances of negative numbers just doesn't work, it seems to just convert them to unsigned integers.

Any way to get around this? perhaps we can do something similar to solidity int256.

License: is it MPL or GPL?

• package.json says MPL2.0
• README has no information about this
• but the LICENSE file contains GPL2.0

rlp.encode(-1) should throw an Error.

Currently the intToBuffer -function would convert the -1 to an empty Buffer, since (-1).toString(16) creates -1 and Buffer.from('-1','hex') creates an empty Buffer since this is an invalid hex-input. But currently no error is thrown, which simply creates incorrect rlp-representation.

https://github.com/ethereumjs/rlp/blob/master/src/index.ts#L221

how to docode hex to string

hi every one, how to transfer hex to string,like site at https://codechain-io.github.io/rlp-debugger/,it has four type.

export encode/decode for types

In RLPx context we need read code from message and then read content.
For example ping message: <Buffer 02 c0> if we trying decode with current implementation we get AssertionError: invalid remainder.
We can read only first byte and then decode content with RLP, it works, but only for message codes that less than 0x80.

Reading message code in go-ethereum.

Move from Travis to Circleci

Recursive Typings

There seems to be an issue using this lib in Typescript. Consider this example that works perfectly fine in the console:

> rlp.encode([[[[[0]]]]])
<Buffer c5 c4 c3 c2 c1 80>

Now, writing the same in Typescript won't work:

Argument of type 'number[][][][][]' is not assignable to parameter of type 'string | number | Buffer | (string | number | Buffer)[]'.
  Type 'number[][][][][]' is not assignable to type '(string | number | Buffer)[]'.
    Type 'number[][][][]' is not assignable to type 'string | number | Buffer'.
      Type 'number[][][][]' is not assignable to type 'Buffer'.
        Property 'write' is missing in type 'number[][][][]'.

I guess there needs to be some way to support recursive types, if that is possible?