chainsafe / ssz Goto Github PK

View Code? Open in Web Editor NEW

46.0 8.0 17.0 8.7 MB

Typescript implementation of Simple Serialize (SSZ)

Home Page: https://simpleserialize.com/

License: Other

TypeScript 54.86% JavaScript 0.84% WebAssembly 44.02% HTML 0.01% SCSS 0.17% Shell 0.07% Dockerfile 0.03%

simple-serialize eth2 merkle-tree ssz typescript

ssz's Introduction

@Chainsafe/ssz

Typescript Simple Serialize (SSZ) libraries

Intro

This monorepo is part of ChainSafe's Lodestar project. It contains packages related to Simple Serialize.

Packages

Package	Version	Description
@chainsafe/as-sha256		SHA256 optimized for 64 byte input
@chainsafe/persistent-merkle-tree		Merkle tree using structural sharing
@chainsafe/persistent-ts		List using structural sharing
@chainsafe/ssz		Simple serialize
simpleserialize.com	N/A	Simple serialize demo website

ssz's People

Contributors

Stargazers

Watchers

Forkers

dapplion scottypoi protolambda g11tech hujw77 alexxuyang lan956 twoeths nflaig mpetrunic pinkdiamond1 acolytec3 nicobao jflo boogh46 ark-us akshatgada

ssz's Issues

Consume HashObject in ssz and new array method

Motivation

HashObject

The new as-sha256 and persistent-merkle-tree implementation uses HashObject internally, we need to expose that
ssz would use it to probably extract/set value faster (for uint type) without having to use an intermediate Uint8Array

In order to update an array (list/vector) the consumer has to get array item, +/-delta and update back the item to the array

   state.balances[i] = state.balances[i] + delta;

we should be able to implement a new "tree_" method in the array like

  tree_updateUint64Delta(index: number, delta: number): number;

it'd save 1 time to access/traverse the tree.

Add spec tests in CI

We need the @chainsafe/lodestar-types package to run spec tests.

A fix for this will likely look like:

a new github action to run on pull_request
clone lodestar monorepo, from master
yarn links the current ssz repo
build lodestar (if possible, just up to lodestar-config)
run lerna run test:ssz from the lodestar monorepo

BasicListType - element type check

BasicListType should allow only Basic types as "elementType" option

Documentation needs to be expanded

Description

Currently, the documentation doesn't convey how to make use of this package, examples (as requested in #2), simple explanations and some context should be added.

Inconsistent hashTreeRoot() of BitVectorType

This unit test failed

it.only("SyncCommitteeBits", function () {
    const SyncCommitteeBits = new BitVectorType({
      length: 512,
    });
    const biStr = "00001110011100101010100110111001111011110111001110110010101000010010011110000110001101111100100100011011001001010000111010010011100100111010111101110110001000000011011001011000011101010111111011000110000101100111111000110011110010010110101011111110111010101111110010011111101001011110001101111110111001100110110001100010100010101110110010100100001011000101101000011010111010111000100100101000101100001100011001110100100111110011100111001100101001011011111001111010111011000100100000010000000111010010100000000111";
    const arr = Array.from({length: 512}, (_, i) => biStr.charAt(i) === "0" ? false: true);
    const rootByStruct = SyncCommitteeBits.hashTreeRoot(arr);
    const bytes = SyncCommitteeBits.serialize(arr);
    const rootByTreeBacked = SyncCommitteeBits.createTreeBackedFromBytes(bytes).hashTreeRoot();
    expect(toHexString(rootByStruct)).to.be.equal(toHexString(rootByTreeBacked), "Inconsistent hashTreeRoot");
  });

with

Inconsistent hashTreeRoot
      + expected - actual

      -0x293e7a2daf052e84155cbc618f5ab1873cac531310ea0ecbd5d6ac927797f3a5
      +0xdeda3ed719579278aaab269fbe7bec2ffa69f2d4d488d6ce50925424b6e7354c
      
      at Context.<anonymous> (test/unit/regression.test.ts:15:45)
      at processImmediate (internal/timers.js:461:21)

seems like the tree backed implementation was good while the struct hashTreeRoot() was not, see ChainSafe/lodestar#2648

[Crash/Fuzzing] TypeError in ssz library during BeaconBlock deserialize

Describe the bug

During fuzzing with beaconfuzz, I found this TypeError crash inside ssz library when trying to deserialize a beaconblock.

Expected behavior

Should throw a custom Error.

Steps to Reproduce

crash_TypeError_block_lodestar.js:

var mainnet_1 = require("@chainsafe/lodestar-types/lib/ssz/presets/mainnet");

buf = Buffer.from('0100000000000000ae000000000000000a0a0a0a2a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a000000000000000000000000000000000000000000000000000000000000000054000000b69753a1c80a81b630fedc668c19ac093cbcc44fe1e294ff7164de52ef0f109a602b7065cb7b21f14a65cbcbbd4455960cef385d552e393e2d47b340cab50291ce81256c1b8239d6ebc1ab84ae2410f6b9fb3dc989e15a9fe9a4431393e1da5d0000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000dc000000dc000000dc000000dc000000dc00ddff', 'hex')

mainnet_1.types.BeaconBlock.deserialize(buf);

Run:

$ npm i @chainsafe/lodestar-types

$ node crash_TypeError_block_lodestar.js
XXX/lodestar/node_modules/@chainsafe/ssz/lib/types/basic/uint.js:176
      output += BigInt(data[offset + i]) << BigInt(8 * i);
                ^

TypeError: Cannot convert undefined to a BigInt
    at BigInt (<anonymous>)
    at BigIntUintType.fromBytes (XXX/lodestar/node_modules/@chainsafe/ssz/lib/types/basic/uint.js:176:17)
    at XXX/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:133:40
    at Array.forEach (<anonymous>)
    at ContainerStructuralHandler.fromBytes (XXX/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:112:39)
    at XXX/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:135:51
    at Array.forEach (<anonymous>)
    at ContainerStructuralHandler.fromBytes (XXX/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:112:39)
    at XXX/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/array.js:209:54
    at Function.from (<anonymous>)

Desktop (please complete the following information):

OS: Ubuntu 18.04
@chainsafe/lodestar-types: 0.8.0
node -v: v12.9.1

Fix deserialization error cases

See test cases added in #42
These tests should pass when un-skipped.

createTreeBackedFromStruct performance

To research potential improvements I've written a manual createTreeBackedFromStruct() function for the validator object

    const pubkey_n4 = new LeafNode(validator.pubkey.slice(0, 32));
    const pubkey_n5 = new LeafNode(validator.pubkey.slice(32, 64));
    const pubkey_n6 = new LeafNode(validator.pubkey.slice(64, 96));
    const pubkey_n2 = new BranchNode(pubkey_n4, pubkey_n5);
    const pubkey_n3 = new BranchNode(pubkey_n6, zeroNode(0));
    const pubkey_n1 = new BranchNode(pubkey_n2, pubkey_n3);

    const v_8 = pubkey_n1;
    const v_9 = new LeafNode(validator.withdrawalCredentials);
    const v_10 = new LeafNode(bigintToBuffer(validator.effectiveBalance));
    const v_11 = new LeafNode(bigintToBuffer(validator.activationEligibilityEpoch));
    const v_12 = new LeafNode(bigintToBuffer(validator.activationEpoch));
    const v_13 = new LeafNode(bigintToBuffer(validator.exitEpoch));
    const v_14 = new LeafNode(bigintToBuffer(validator.withdrawableEpoch));

    const v_4 = new BranchNode(v_8, v_9);
    const v_5 = new BranchNode(v_10, v_11);
    const v_6 = new BranchNode(v_12, v_13);
    const v_7 = new BranchNode(v_14, zeroNode(0));

    const v_2 = new BranchNode(v_4, v_5);
    const v_3 = new BranchNode(v_6, v_7);

    const v_1 = new BranchNode(v_2, v_3);
    const tree = new Tree(v_1);

Also a variation where the epoch fields are Numbers

...
    const v_10 = new LeafNode(numToBuffer(validatorNum.effectiveBalance));
    const v_11 = new LeafNode(numToBuffer(validatorNum.activationEligibilityEpoch));
    const v_12 = new LeafNode(numToBuffer(validatorNum.activationEpoch));
    const v_13 = new LeafNode(numToBuffer(validatorNum.exitEpoch));
    const v_14 = new LeafNode(numToBuffer(validatorNum.withdrawableEpoch));
....

Results show that createTreeBackedFromStruct can be significantly improved.

    ✓ Validator - createTreeBackedFromStruct                              193760.9 ops/s    6.161000 us/op        -     183164 runs   1.01 s
    ✓ Validator - custom to tree bigint                                   376222.7 ops/s    2.658000 us/op        -     343743 runs   1.02 s
    ✓ Validator - custom to tree number                                   658761.5 ops/s    1.518000 us/op        -     557524 runs   1.02 s

Looking at profiles of this custom to tree functions most of the time is spent:

slicing the Buffer
doing bit ops on the numbers to convert them to buffers

I'm not sure if we can improve them any more, but occasionally updating a few validators (~100) would cost less than 1 ms.

CC: @wemeetagain

Sort class methods

General methods (without prefix)
struct_*
tree_*
bytes_*

Reduce Union-type selector to 1 byte and other union type cleanups

This is part of altair1.1.0-alpha.7 https://github.com/ethereum/eth2.0-specs/releases

Refer to ethereum/consensus-specs#2445 ethereum/consensus-specs#2456 ethereum/consensus-specs#2458

Improve intToBytes, bytesToInt for number

Is your feature request related to a problem? Please describe.
Right now these utils convert to BigInt unnecessarily and we know that we have performance issue with using BigInt

Describe the solution you'd like
Reimplement these functions without going through bigint-buffer

Add Karma to test in browser env

Is your feature request related to a problem? Please describe.

We have no assurance in CI that code doesn't break in the browser, and it does.

Describe the solution you'd like

Add Karma like in bls repo with sample webpack config https://github.com/ChainSafe/bls/blob/master/karma.conf.js

Add strict mode > throw on non-existing value

Add a strict mode that applies to Eth2.0 data structures. It should make accessing a value in a list that does not contain a value that should throw an exception.

Rationale

While accessing non-existing keys in javascript objects returns undefined, this behavior is very problematic with Typescript. As of now, Typescript has no way to tell if list[I] is defined but assumes it is defined (which may be false). This can make large parts of the codebase not type-safe. See ChainSafe/lodestar#1092 as an example of this non-type-safety in action.

Making SSZ data structures ensure that list[I] is defined by throwing an error otherwise, would fix that problem making Typescript types true.

Implementation

Revert f94ad9d

Please, update documentation or provide me with some examples of how to unserialize

I need to somehow convert aggregation bits from hex-string format to any other format (preferable array of bits or similar)

However, I can't find a documentation how to do that.

Example:
0x00000004810000001c8114004100002012
(this is aggregation bits of attestation of a slot in mainnet)

should be converted to this:
[ 0 0 0 100 10000001 0 0 0 11100 10000001 10100 0 1000001 0 0 100000 10]

I have already tried to use BitListType in different ways, however it does not help.

Can you give me more examples how to use this library?

Fix bitfield_bytes type error

ethereum/consensus-specs#1806

readonlyValues for TreeBacked<CompositeListType>

This unit test

import { expect } from "chai";
import { ListType, readonlyValues } from "../../src";
import { bytes32Type, number16List100Type } from "./objects";

const bytes32ListType = new ListType({
  elementType: bytes32Type,
  limit: 2
});

describe.only("readOnlyValues", function () {
  const testCases: {value: any; type: any;}[] = [
    {value: [0, 1, 2], type: number16List100Type},
    {value: [Buffer.alloc(32, 0), Buffer.alloc(32, 1)], type: bytes32ListType},
  ];

  for (const {type, value} of testCases) {
    it(`should work with ${type.constructor.name}`, function () {
      expect(Array.from(readonlyValues(type.createTreeBackedFromStruct(value)))).to.be.deep.equal(value);
    });
  }

});

failed as below

readOnlyValues
    ✓ should work with BasicListType
    1) should work with CompositeListType

Benchmark transformations

Add benchmarks to understand the cost of

binary -> struct
binary -> tree
struct -> tree
tree -> struct

Also benchmark accessing properties in

struct
tree

This result will guide some important design choices Lodestar

Document how to create types

Show some examples and explain how to create new types

Investigate inconsistent performance of hashTreeRoot()

part of ChainSafe/lodestar#2046

this is a simple test to calculate hashTreeRoot

it.only("set validator valances", function () {
    this.timeout(0);
    const originalState = state.getOriginalState();
    // cache hashTreeRoot
    config.types.BeaconState.hashTreeRoot(originalState);
    const balances = Array.from({length: originalState.validators.length}, () => BigInt(31217089836));
    let minTime = Number.MAX_SAFE_INTEGER;
    let maxTime = 0;
    let average = {duration: 0, count: 0};
    const MAX_TRY = 10000;
    for (let i = 0; i < MAX_TRY; i++) {
      const state = config.types.BeaconState.clone(originalState);
      state.balances = balances as List<Gwei>;
      const start = Date.now();
      config.types.BeaconState.hashTreeRoot(state);
      const duration = Date.now() - start;
      const totalDuration = average.duration * average.count + duration;
      const totalCount = average.count + 1;
      average.count = totalCount;
      average.duration = totalDuration / totalCount;
      if (duration < minTime) minTime = duration;
      if (duration > maxTime) maxTime = duration;
    }
    console.log("hashTreeRoot minTime:", minTime, "maxTime:", maxTime, "average:", average.duration, "MAX_TRY:", MAX_TRY);
  });

this prints out hashTreeRoot minTime: 65 maxTime: 507 average: 80.46129999999977 MAX_TRY: 10000. I notice the max time is very similar to the one we have during a long epoch transition.

we need to investigate why the performance is inconsistent and if we can improve it.

Add minSize(), maxSize() api to all types

The demand comes from eth2 spec https://github.com/ethereum/eth2.0-specs/pull/1800/files

Standardize Uint toJson

We currently serialize BigInt to a string, Number to a number. We should standardize to string, across the board for all Uint w byteLength < 4.
Eg:

const Number64 = new NumberUintType({byteLength: 8})
Number64.toJson(5) // => "5"

Inconsistent hashTreeRoot()

Motivation:

Currently the struct hashTreeRoot() returns a Buffer while a TreeBacked returns Uint8Array, we want both of them to return Uint8Array

[Crash/Fuzzing] RangeError in ssz library when parsing empty BeaconBlock

Describe the bug

During fuzzing with beaconfuzz, I found this RangeError crash inside the ssz library when providing an empty beaconblock.

Expected behavior

Should detect and throw an Error.

Steps to Reproduce

crash_RangeError_block_lodestar.js:

var mainnet_1 = require("@chainsafe/lodestar-types/lib/ssz/presets/mainnet");

buf = Buffer.from('', 'hex')

mainnet_1.types.BeaconBlock.deserialize(buf);

Run:

$ npm i @chainsafe/lodestar-types

$ node crash_RangeError_block_lodestar.js
/home/scop/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:99
        offsets.push(start + fixedSection.getUint32(index, true));
                                          ^

RangeError: Offset is outside the bounds of the DataView
    at DataView.getUint32 (<anonymous>)
    at /home/scop/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:99:43
    at Array.reduce (<anonymous>)
    at ContainerStructuralHandler.fromBytes (/home/scop/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:97:33)
    at ContainerStructuralHandler.deserialize (/home/scop/node_modules/@chainsafe/ssz/lib/backings/structural/abstract.js:55:17)
    at ContainerType.deserialize (/home/scop/node_modules/@chainsafe/ssz/lib/types/composite/abstract.js:112:28)
    at Object.<anonymous> (XXX/crash_RangeError_block_lodestar.js:6:29)
    at Module._compile (internal/modules/cjs/loader.js:936:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:947:10)
    at Module.load (internal/modules/cjs/loader.js:790:32)

Desktop (please complete the following information):

OS: Ubuntu 18.04
@chainsafe/lodestar-types: 0.8.0
nodejs: v12.9.1

BitVector deserialize bug

Attempting to deserialize an altair block results in an SSZ error

Jun-01 13:16:57.566 [SYNC]          verbose: Batch download error id=Finalized, startEpoch=9, status=Downloading code=SSZ_SNAPPY_ERROR_DESERIALIZE_ERROR, deserializeError={"message":"Invalid deserialized bitvector length"}
Error: Invalid deserialized bitvector length
    at BitVectorType.tree_deserializeFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/bitVector.ts:124:13)
    at ContainerType.tree_deserializeFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/container.ts:369:21)
    at ContainerType.tree_deserializeFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/container.ts:369:21)
    at ContainerType.tree_deserializeFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/container.ts:369:21)
    at ContainerType.tree_deserializeFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/container.ts:369:21)
    at ContainerType.tree_deserialize (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/abstract.ts:119:17)
    at ContainerType.createTreeBackedFromBytes (/home/lion/Code/eth2.0/lodestar/node_modules/@chainsafe/ssz/src/types/composite/abstract.ts:396:39)
    at deserializeSszBody (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encodingStrategies/sszSnappy/decode.ts:139:24)
    at readSszSnappyPayload (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encodingStrategies/sszSnappy/decode.ts:34:12)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
Error: SSZ_SNAPPY_ERROR_DESERIALIZE_ERROR
    at deserializeSszBody (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encodingStrategies/sszSnappy/decode.ts:144:11)
    at readSszSnappyPayload (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encodingStrategies/sszSnappy/decode.ts:34:12)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at readEncodedPayload (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encodingStrategies/index.ts:25:14)
    at chunk (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/encoders/responseDecode.ts:62:13)
    at /home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/utils/onChunk.ts:7:22
    at response (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/request/responseTimeoutsHandler.ts:39:7)
    at /home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/request/collectResponses.ts:25:22
    at withTimeout (/home/lion/Code/eth2.0/lodestar/packages/utils/src/timeout.ts:20:12)
    at sendRequest (/home/lion/Code/eth2.0/lodestar/packages/lodestar/src/network/reqresp/request/index.ts:139:25)

Config: eth-clients/eth2-networks#47

Block:

0x640000008539e6c27cbca13373395583fb3b2b324669db949582cbcd4abf7bd81c2b289690c564799b74884ec99d8c501d690fea05df41a8a93bff241208bf03a94e15e4a56ba35d6d669ba188dfcfec267f4c4dd11235f81f20510da8cf23b1e37f810841010000000000001f02000000000000b422275e97f488979b7f85735370952ea3b88fba39175a689e54a2d4b379d4b16f0761935eae7d354c215549e06ee14d3b98f4eff470cc470e8b563bfafbb91654000000ad9ac936202876254219fae2b6bffa96968c21360040413ae4a6197c9b83fe0c3d77f136eb10127b415d59f7e6ec622d186d8b76fbd502badb80cac59eb2315f2b137eb2095b097857a95a229b532911e7c9d75d83ff98859d01fcfb53927149d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e00000000000000006342bde06c37c72e50401c7c6ca3a898c129da44cb647c1de60584c3b3414faa74656b752f7632312e352e302b35342d673633376631316400000000000000007c0100007c0100007c0100006e0200006e02000017ccd8d26fd7b9375fbdd6cf69b7370857bfa5c78d4937d53f181132b448268432f34248cbb7f54e3730278014170f9e27af2ba3ecbcadf5017e6161abe9a7b980c0117e338f5db745d1ff4233e313a9f9f04b20bee6f80cd68ef5fce77e3ac9888913438274bd7be6c1739744bc84ea15c4058b5825b1008e9e65f899b17e49ca60e224a3ca211ad8410364aeb71d00396b397d2fb8eaae6e3bfca18fb7f31704000000e400000040010000000000000000000000000000b422275e97f488979b7f85735370952ea3b88fba39175a689e54a2d4b379d4b108000000000000008da011351e515f87967e99e9b7f0a6c0e48cade59826c065969a055b290af9dc0a00000000000000b422275e97f488979b7f85735370952ea3b88fba39175a689e54a2d4b379d4b1b9f8367e9ea68a5db480abf345785ed5b240d81053ef2d0e2bcaa3339a2bac52ed0c66059f9eb93199e8f1a646ea063f1723a6076d361992ac14578f0f83da63f9d30af9259ec5eaf40c84f96f38ed79a3e014663603b565afc5f4fbf6d940a2ffffffffffffffffff0f

Replace iterables throughout

Using the iterable interface instead of Arrays is much slower, even a for of of an array is slower than a simple for(;;) iterating over an array.
Since ssz is a low level library, meant to be used in higher-performance applications (such as Lodestar), we need to squeeze as much performance as possible.

There's still some iterator nonsense in ssz that should be removed:

use of iterateAtDepth, not to be confused with Tree#iterateNodesAtDepth which is resolved with #167, needs a getGindicesAtDepth alternative
use of generator functions in public interface, needs an array alternative, can be introduced in a non-breaking change

toHexString performance issue

Running a Lodestar node in Prater with validators 3.25% of total CPU time is spent on toHexString()

get() Root should not return a sub Tree

When getting a property from a tree backed ssz type this if decides to return the "raw" value or tree backed.

ssz/src/backings/tree/container.ts

Line 138 in d6efe10

if (!isCompositeType(fieldType)) {

In my opinion I would assume that a Root will not return a sub Tree, but just the raw value. However this is not the case

const stateTree = config.types.phase0.BeaconState.tree.deserialize(stateBytes);
console.log({
      epoch: stateTree.finalizedCheckpoint.epoch,
      root: stateTree.finalizedCheckpoint.root,
});

// stdout of above code
{
  epoch: 0,
  root: Tree { _node: LeafNode { _root: [Uint8Array] }, hook: [Function] }
}

Due to the Tree hook anyone that holds any root part of a state will prevent to garbage collect that state, causing this memory issues ChainSafe/lodestar#2181 (comment)

Container Representation [Fields as an Array]

import {ContainerType, ByteVectorType} from "@chainsafe/ssz";

// Creates a "Keypair" SSZ data type (a private key of 32 bytes, a public key of 48 bytes)
const Keypair = new ContainerType({
  fields: {
    priv: new ByteVectorType({
      length: 32,
    }),
    pub: new ByteVectorType({
      length: 48,
    }),
  },
});

First off, happy holidays.

Second, does the ContainerType have plans to support the fields property as an array. I though properties of containers had to be in specific ordering, and thus an object which is non-deterministic or not always deterministically ordered (see: https://stackoverflow.com/questions/5525795/does-javascript-guarantee-object-property-order) I thought, would not be appropriate.

Wouldn't this be more appropriate?

const Keypair = new ContainerType({
  fields: [
    { name: 'priv', value: new ByteVectorType({
      length: 32,
    }) },
    { name: 'pub', value: new ByteVectorType({
      length: 48,
    }) },
  ],
});

Fast conversion from decimal to bytes

Lodestar spends significant CPU time converting uint types from decimal to bytes and back. For example, converting the balance tree to flat and back consist of doing 200_000 ops of those.

SSZ is the one hosting the code that does the conversions. This is one of the implementations that participate in this conversions.

ssz/src/types/basic/uint.ts

Lines 68 to 82 in be03a33

 struct_serializeToBytes(value: number, output: Uint8Array, offset: number): number { 

 if (this.byteLength > 6 && value === Infinity) { 

 for (let i = offset; i < offset + this.byteLength; i++) { 

 output[i] = 0xff; 

 } 

 } else { 

 let v = value; 

 const MAX_BYTE = 0xff; 

 for (let i = 0; i < this.byteLength; i++) { 

 output[offset + i] = v & MAX_BYTE; 

 v = Math.floor(v / 256); 

 } 

 } 

 return offset + this.byteLength; 

 }

ssz/src/types/basic/uint.ts

Lines 84 to 98 in be03a33

 struct_deserializeFromBytes(data: Uint8Array, offset: number): number { 

 this.bytes_validate(data, offset); 

 let isInfinity = true; 

 let output = 0; 

 for (let i = 0; i < this.byteLength; i++) { 

 output += data[offset + i] * 2 ** (8 * i); 

 if (data[offset + i] !== 0xff) { 

 isInfinity = false; 

 } 

 } 

 if (this.byteLength > 6 && isInfinity) { 

 return Infinity; 

 } 

 return Number(output); 

 }

We should investigate faster alternatives

Implement ByteList type

SSZ generic spec tests include a ByteList, like a ByteVector.

We will want to implement the ByteList, which practically is indistinguishable from a ListType({..., elementType: byteType}) other than json serialization/deserialization, which matches ByteVector (serialize to hex string).

create ssz cli

create cli for serialize, deserialize, hash-tree-root

[non-substantive] Clarify offsets serialization

ethereum/consensus-specs#2043

Type Errors in Integration into Lodestar

@chainsafe/lodestar-types: src/altair/sszTypes.ts:116:30 - error TS2322: Type '() => ContainerType<altair.BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<altair.BeaconBlock>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Call signature return types 'ContainerType<BeaconBlock>' and 'CompositeType<CompositeValue>' are incompatible.
@chainsafe/lodestar-types:       The types of 'struct_equals' are incompatible between these types.
@chainsafe/lodestar-types:         Type '(value1: BeaconBlock, value2: BeaconBlock) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:           Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:             Type 'CompositeValue' is not assignable to type 'BeaconBlock'.
@chainsafe/lodestar-types:               Type 'Record<string, unknown>' is missing the following properties from type 'BeaconBlock': body, slot, proposerIndex, parentRoot, stateRoot
@chainsafe/lodestar-types: 116   elementType: new RootType({expandedType: () => typesRef.get().BeaconBlock}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/altair/sszTypes.ts:121:30 - error TS2322: Type '() => ContainerType<altair.BeaconState>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<altair.BeaconState>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Call signature return types 'ContainerType<BeaconState>' and 'CompositeType<CompositeValue>' are incompatible.
@chainsafe/lodestar-types:       The types of 'struct_equals' are incompatible between these types.
@chainsafe/lodestar-types:         Type '(value1: BeaconState, value2: BeaconState) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:           Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:             Type 'CompositeValue' is not assignable to type 'BeaconState'.
@chainsafe/lodestar-types:               Type 'Record<string, unknown>' is missing the following properties from type 'BeaconState': previousEpochParticipation, currentEpochParticipation, inactivityScores, currentSyncCommittee, and 20 more.
@chainsafe/lodestar-types: 121   elementType: new RootType({expandedType: () => typesRef.get().BeaconState}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/altair/sszTypes.ts:144:31 - error TS2322: Type '() => ContainerType<altair.BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<altair.BeaconBlock>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Type 'ContainerType<BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types: 144     parentRoot: new RootType({expandedType: () => typesRef.get().BeaconBlock}),
@chainsafe/lodestar-types:                                   ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/altair/sszTypes.ts:145:30 - error TS2322: Type '() => ContainerType<altair.BeaconState>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<altair.BeaconState>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Type 'ContainerType<BeaconState>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types: 145     stateRoot: new RootType({expandedType: () => typesRef.get().BeaconState}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/altair/sszTypes.ts:170:34 - error TS2322: Type 'ContainerType<HistoricalBatch>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type 'ContainerType<HistoricalBatch>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types: 170       elementType: new RootType({expandedType: HistoricalBatch}),
@chainsafe/lodestar-types:                                      ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:107:30 - error TS2322: Type 'ContainerType<DepositData>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type 'ContainerType<DepositData>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Types of property 'struct_equals' are incompatible.
@chainsafe/lodestar-types:       Type '(value1: DepositData, value2: DepositData) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:         Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:           Type 'CompositeValue' is not assignable to type 'DepositData'.
@chainsafe/lodestar-types:             Type 'Record<string, unknown>' is missing the following properties from type 'DepositData': pubkey, withdrawalCredentials, amount, signature
@chainsafe/lodestar-types: 107   elementType: new RootType({expandedType: DepositData}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:160:30 - error TS2322: Type '() => ContainerType<phase0.BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<phase0.BeaconBlock>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Call signature return types 'ContainerType<BeaconBlock>' and 'CompositeType<CompositeValue>' are incompatible.
@chainsafe/lodestar-types:       The types of 'struct_equals' are incompatible between these types.
@chainsafe/lodestar-types:         Type '(value1: BeaconBlock, value2: BeaconBlock) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:           Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:             Type 'CompositeValue' is not assignable to type 'BeaconBlock'.
@chainsafe/lodestar-types:               Type 'Record<string, unknown>' is missing the following properties from type 'BeaconBlock': slot, proposerIndex, parentRoot, stateRoot, body
@chainsafe/lodestar-types: 160   elementType: new RootType({expandedType: () => typesRef.get().BeaconBlock}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:165:30 - error TS2322: Type '() => ContainerType<phase0.BeaconState>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<phase0.BeaconState>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Call signature return types 'ContainerType<BeaconState>' and 'CompositeType<CompositeValue>' are incompatible.
@chainsafe/lodestar-types:       The types of 'struct_equals' are incompatible between these types.
@chainsafe/lodestar-types:         Type '(value1: BeaconState, value2: BeaconState) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:           Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:             Type 'CompositeValue' is not assignable to type 'BeaconState'.
@chainsafe/lodestar-types:               Type 'Record<string, unknown>' is missing the following properties from type 'BeaconState': genesisTime, genesisValidatorsRoot, slot, fork, and 17 more.
@chainsafe/lodestar-types: 165   elementType: new RootType({expandedType: () => typesRef.get().BeaconState}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:291:31 - error TS2322: Type '() => ContainerType<phase0.BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<phase0.BeaconBlock>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Type 'ContainerType<BeaconBlock>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types: 291     parentRoot: new RootType({expandedType: () => typesRef.get().BeaconBlock}),
@chainsafe/lodestar-types:                                   ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:292:30 - error TS2322: Type '() => ContainerType<phase0.BeaconState>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type '() => ContainerType<phase0.BeaconState>' is not assignable to type '() => CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Type 'ContainerType<BeaconState>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types: 292     stateRoot: new RootType({expandedType: () => typesRef.get().BeaconState}),
@chainsafe/lodestar-types:                                  ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'
@chainsafe/lodestar-types: src/phase0/sszTypes.ts:324:34 - error TS2322: Type 'ContainerType<HistoricalBatch>' is not assignable to type 'CompositeType<CompositeValue> | (() => CompositeType<CompositeValue>)'.
@chainsafe/lodestar-types:   Type 'ContainerType<HistoricalBatch>' is not assignable to type 'CompositeType<CompositeValue>'.
@chainsafe/lodestar-types:     Types of property 'struct_equals' are incompatible.
@chainsafe/lodestar-types:       Type '(value1: HistoricalBatch, value2: HistoricalBatch) => boolean' is not assignable to type '(struct1: CompositeValue, struct2: CompositeValue) => boolean'.
@chainsafe/lodestar-types:         Types of parameters 'value1' and 'struct1' are incompatible.
@chainsafe/lodestar-types:           Type 'CompositeValue' is not assignable to type 'HistoricalBatch'.
@chainsafe/lodestar-types:             Type 'Record<string, unknown>' is missing the following properties from type 'HistoricalBatch': blockRoots, stateRoots
@chainsafe/lodestar-types: 324       elementType: new RootType({expandedType: HistoricalBatch}),
@chainsafe/lodestar-types:                                      ~~~~~~~~~~~~
@chainsafe/lodestar-types:   ../../../ssz/lib/types/composite/root.d.ts:10:5
@chainsafe/lodestar-types:     10     expandedType: CompositeType<T> | (() => CompositeType<T>);
@chainsafe/lodestar-types:            ~~~~~~~~~~~~
@chainsafe/lodestar-types:     The expected type comes from property 'expandedType' which is declared here on type 'IRootOptions<CompositeValue>'

Add prettier

Apply same config as ChainSafe/lodestar#1008

Invalid deserialized bitvector length error

This is to reproduce ssz issue in lodestar
ChainSafe/lodestar#2580

const SyncCommitteeBits = new BitVectorType({
  length: 32,
});

it.only("should serialize and deserialize SyncCommitteeBits", function () {
  const data = Array.from({length: 32}, () => true);
  const bytes = SyncCommitteeBits.serialize(data);
  const tb = SyncCommitteeBits.createTreeBackedFromBytes(bytes);
  for (let i = 0; i < 32; i ++) {
    expect(tb[i]).to.be.true;
  }
});

it prints out

 1) should serialize and deserialize SyncCommitteeBits:
     Error: Invalid deserialized bitvector length
      at BitVectorType.tree_deserializeFromBytes (src/types/composite/bitVector.ts:6:78)
      at BitVectorType.tree_deserialize (src/types/composite/abstract.ts:6:54)
      at BitVectorType.createTreeBackedFromBytes (src/types/composite/abstract.ts:75:76)

Error: Too many nodes

This test failed

export const CommitteeBits = new BitListType({
  limit: 2048,
});

const bitList = Array.from({length: 2048}, () => true);
 const bitListTree = CommitteeBits.createTreeBackedFromStruct(bitList);

with error

  create a new tree from tree backed properties:
     Error: Too many nodes
      at Object.subtreeFillToContents (node_modules/@chainsafe/persistent-merkle-tree/lib/subtree.js:45:15)
      at BitListType.struct_convertToTree (src/types/composite/array.ts:108:21)
      at BitListType.struct_convertToTree (src/types/composite/list.ts:87:24)
      at BitListType.createTreeBackedFromStruct (src/types/composite/abstract.ts:333:39)

it's fine if we create CommitteeBits from an array of 128 boolean.

Chained error messages

We would like errors in processing sub-fields of an ssz object to 'bubble up' to provide an error message that locates the error to the sub-field in question.

Eg:

interface BeaconBlock {
  body: {
    attestations: {
      data: {
        slot: number;
      }
    }[]
  }
}

If there was an error in when running assertValidValue/deserialize of a slot inside an attestation data inside a beacon block, as in the object above, the error message would look something like: body: attestations: index ${i}: data: slot: ${error message}

See

ssz/src/core/assertValidValue.ts

Line 120 in bbdae82

throw new Error(`Invalid container, field ${fieldName}: ${e.message}`);

as an example of how this was accomplished in our old ssz implementation.
Resolving this issue likely involves adding try/catch to our array and container structural/tree backing classes for a few key methods. Likely assertValidValue and deserialize/fromBytes are the most important.

Be on par with remerkleable test coverage

Proto noted that we are missing some tests from the python implementation

ssz/test/unit/deserialize.test.ts

Line 39 in fe890be

{value: "ffffff0f", type: number32Type, expected: 2 ** 28 - 1},

https://github.com/protolambda/remerkleable/blob/master/remerkleable/test_impl.py

Type identifier

If you debug and come across SSZ type, you aren't sure what type that it.
Example:

Proposed solution would be to add public property typeName to "ContainerType" which would force each type to declare it's name.

Type revision

Since my attempt to improve types just scratched surface, I would suggest some bottom to top type revision

I would like to confirm some basic presumptions:

type BasicValue = bigint | number | boolean | Uint8Array:

type CompositeValue = Record<string, unknown> | ArrayLike<BasicValue | CompositeValue> | {}; // last one is hack for eth2 type interfaces to work. We could maybe just add it to Container generic?

Use raw for loops where possible

Generators and callback based iterators are shown to be slower than raw for loops. Replace where possible.

Create new type root "SSZType"

I would suggest creating new interface "SSZType" with following properties:

BasicType and Container type should extend from it
should contain all common ssz methods like
- hashTreeRoot
- serialize
- deserialize
- fromJson

This would help with writing generic code that requies any ssz type.

.equals() benchmark

Since calling .equals on SSZ objects is a frequent operation in the ETH2 client I've benchmarked alternatives

// Generate random data
const num = 1000000;
const dataArr: {a: Uint8Array; b: Uint8Array}[] = [];
const len = 32;
for (let i = 0; i < num; i++) {
  const nums: number[] = [];
  for (let j = 0; j < len; j++) {
    nums[j] = j;
  }
  const a = new Uint8Array(nums);
  const b = new Uint8Array(nums);
  dataArr.push({a, b});
}

function equalsFor(a: Uint8Array, b: Uint8Array): boolean {
  for (let i = 0; i < a.length; i++) {
    if (a[i] !== b[i]) return false;
  }
  return true;
}

function equalsEvery(a: Uint8Array, b: Uint8Array): boolean {
  return a.every((v, i) => v === b[i]);
}

function equalsNode(a: Uint8Array, b: Uint8Array): boolean {
  return Buffer.compare(Buffer.from(a), Buffer.from(b)) === 0;
}

// Run benchmarks
for (const [id, fn] of Object.entries({equalsFor, equalsEvery, equalsNode})) {
  const label = `${num} - ${id}`;
  console.time(label);
  for (const {a, b} of dataArr) fn(a, b);
  console.timeEnd(label);
}

// 1000000 - equalsFor:   132.577ms
// 1000000 - equalsEvery: 579.814ms
// 1000000 - equalsNode:  486.077ms

Not sure it the savings are relevant but there shouldn't be any downside to save a few ms overall by not using .every()

ProposerSlashing is not correctly cased

Keys with a trailing number like signedHeader1 are not correctly cased:

signed_header1

this breaks compatibility with the eth2 api spec

Improve `balances` setter for BeaconState

part of ChainSafe/lodestar#2046

this is a quick performance test I have

import { BeaconState } from "./objects";

describe("fromStructural", function () {
  it.only("reproduce the event loop lag", function () {
    this.timeout(0);
    const balances = Array.from({length: 200000}, () => BigInt(31217089836));
    let minTime = Number.MAX_SAFE_INTEGER;
    let maxTime = 0;
    let average = {duration: 0, count: 0};
    const MAX_TRY = 10000;
    for (let i = 0; i < MAX_TRY; i++) {
      const state = BeaconState.tree.defaultValue();
      const start = Date.now();
      state.balances = balances;
      const duration = Date.now() - start;
      const totalDuration = average.duration * average.count + duration;
      const totalCount = average.count + 1;
      average.count = totalCount;
      average.duration = totalDuration / totalCount;
      if (duration < minTime) minTime = duration;
      if (duration > maxTime) maxTime = duration;
    }
    console.log("fromStructural minTime:", minTime, "maxTime:", maxTime, "average:", average.duration, "MAX_TRY:", MAX_TRY);
  });
});

this prints out

1st time fromStructural minTime: 262 maxTime: 999 average: 307.5632000000007 MAX_TRY: 10000
2nd time fromStructural minTime: 258 maxTime: 676 average: 291.47639999999956 MAX_TRY: 10000

Add design doc to README

@wemeetagain wrote this great write-up that should be added to the README upon revision https://hackmd.io/KuV9Ca1dTyaIRkKNKzwWQA

Avoid using BigInt whenever possible

v8 has to allocate an object for each BigInt under the cover: https://v8.dev/blog/bigint and it causes memory/performance issue in lodestar, so we need to use Number instead whenever possible.

also we need to add a performance test for uint.ts toBytes(), fromBytes()

  it.only("reproduce readOnlyMap issue", async function () {
    this.timeout(0);
    let minTime = Number.MAX_SAFE_INTEGER;
    let maxTime = 0;
    const originalState = state.getOriginalState();
    for (let i = 0; i < 20000; i++) {
      const start = Date.now();
      readOnlyMap(originalState.balances, (balance) => balance);
      const duration = Date.now() - start;
      if (duration < minTime) minTime = duration;
      if (duration > maxTime) maxTime = duration;
    }
    console.log("@@@ minTime", minTime, "maxTime", maxTime);
  });

this currently prints out

2021-02-09 08:15:42 []                 info: Loaded state slot=756416, numValidators=114038
@@@ minTime 140 maxTime 1109

Equals on AggregateAndProof throws

Ideally, equals method should catch stuff and return false. But curious what could have caused this

(node:17070) UnhandledPromiseRejectionWarning: Error: Boolean value must be true or false
    at BooleanType.assertValidValue (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/types/basic/boolean.js:35:13)
    at BooleanType.equals (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/types/basic/abstract.js:74:10)
    at BitListStructuralHandler.equals (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/array.js:57:35)
    at /home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:83:37
    at Array.every (<anonymous>)
    at ContainerStructuralHandler.equals (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/backings/structural/container.js:79:46)
    at ContainerType.equals (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/types/composite/abstract.js:54:30)
    at /home/mpetrunic/projects/eth2/lodestar/packages/lodestar/lib/db/api/beacon/repositories/aggregateAndProof.js:51:46
    at CompositeListTreeHandler.findIndex (/home/mpetrunic/projects/eth2/lodestar/node_modules/@chainsafe/ssz/lib/backings/tree/array.js:393:11)
    at /home/mpetrunic/projects/eth2/lodestar/packages/lodestar/lib/db/api/beacon/repositories/aggregateAndProof.js:50:27

Tree retains memory in global? scope - causes OOM :warning:

This gist demostrates a case where memory is retained in some upper scope such that it's not garbage collected.

https://gist.github.com/dapplion/e079d8faf795f758f1f3b341ee590dc2

import {ssz} from "@chainsafe/lodestar-types";

let i = 0;
const heapUsed = process.memoryUsage().heapUsed;

while (true) {
  getBigStateTreeBacked();
  global.gc();
  console.log(i++, (process.memoryUsage().heapUsed - heapUsed) / 1e6, "MB");
}

function getBigStateTreeBacked(): any {
  const stateTB = ssz.phase0.BeaconState.defaultTreeBacked();
  const validator = ssz.phase0.Validator.defaultValue();
  for (let i = 0; i < 250_000; i++) {
    stateTB.validators.push(validator);
  }
}

lodestar$ LODESTAR_PRESET=mainnet node --expose-gc ts-node lodestar-ssz-oom.ts
0 456.3956 MB
1 919.7116 MB
2 1388.220224 MB
3 1846.017496 MB

<--- Last few GCs --->

[26583:0x5ab6bc0]    37176 ms: Mark-sweep (reduce) 2046.8 (2053.8) -> 2045.9 (2053.8) MB, 1838.1 / 0.0 ms  (+ 0.0 ms in 29 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1845 ms) (average mu = 0.150, current mu = 0.016)[26583:0x5ab6bc0]    38694 ms: Mark-sweep (reduce) 2047.0 (2050.8) -> 2046.1 (2052.0) MB, 1515.4 / 0.0 ms  (average mu = 0.081, current mu = 0.001) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa25510 node::Abort() [node]
 2: 0x9664d3 node::FatalError(char const*, char const*) [node]
 3: 0xb9a8be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ac37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd56ca5  [node]
 6: 0xd5782f  [node]
 7: 0xd6566b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xd6922c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xd3790b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x107fbef v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1426919  [node]
Aborted (core dumped)

List interface type error

List.pop should return T|undefined instead of just T to satisfy strictNullChecks in typescript

	struct_serializeToBytes(value: number, output: Uint8Array, offset: number): number {
	if (this.byteLength > 6 && value === Infinity) {
	for (let i = offset; i < offset + this.byteLength; i++) {
	output[i] = 0xff;
	}
	} else {
	let v = value;
	const MAX_BYTE = 0xff;
	for (let i = 0; i < this.byteLength; i++) {
	output[offset + i] = v & MAX_BYTE;
	v = Math.floor(v / 256);
	}
	}
	return offset + this.byteLength;
	}

	struct_deserializeFromBytes(data: Uint8Array, offset: number): number {
	this.bytes_validate(data, offset);
	let isInfinity = true;
	let output = 0;
	for (let i = 0; i < this.byteLength; i++) {
	output += data[offset + i] * 2 ** (8 * i);
	if (data[offset + i] !== 0xff) {
	isInfinity = false;
	}
	}
	if (this.byteLength > 6 && isInfinity) {
	return Infinity;
	}
	return Number(output);
	}