umegaya / pb3sol

protobuf3 compiler plugin for solidity and protobufjs integration support

pb3sol

• protobuf3 solidity code generator, inpired by https://github.com/shmookey/solpb (only supports pb2) enhanced with:
  • soltype-pb, which enhances inter-operability with protobuf.js
  • support native solidity types like address, uint256, by including special protobuf definition Solidity.proto
• unimplemented protobuf feature
  • true enum generated by protobuf enum declaration. because solidity does not allow us to set arbiter values to enum declaration.
  • float/double. no floating type in solidity.

Motivation

• recently, some kind of dapp like game need to treat complex, unstable data structure.
• but for now, only solidity code can describe data structure and solidity code is immutable if it once deployed.
• actually, code can be replaced by zos or similar technology, but data related with old contract cannot move to new one automatically
• thus migrate data schema in smart contract is often with full database copy. its un-acceptable from the view point of performance and cost.
• storing data as bytes and parse with protobuf could solve this problem with small increase of gas comsumption for each contract call, by separating actual data storage and its schema definition (replacable as solidity library)
• also, if we describe data as protobuf encoded byte array, contract only need to return byte array and parsing can be done on each client, which is huge save for contract code size and execution fee

Try it out

• try to run test first.
• easiest way to run test is using docker. clone this repository and run make tsh to enter shell, then cd test && make test_setup test_on_host
• it does:
  • launch docker container and enter shell
  • install necessary node module by using npm install
  • compile proto files to solidity source
  • run truffle test
    • create storage contract (contract which just holds byte array with string key)
    • store byte array encoded by protobuf
    • load byte array and decoded by protobuf
    • check encoded value is correctly restored
    • load byte array and decode on js side
    • decode it with new version of proto file and its correctly migreated

Caveat with solidity <= 0.4.24

• at the time I wrote this (2018/06/05), latest released solidity version is 0.4.24, which cannot allow us to:
  • pass/return arbiter struct value to/from non-internal contract call
• so still we cannot use generated library for linking, same reason of 0.4.20.

Caveat with solidity <= 0.4.20

• at the time I wrote this (2018/03/03), latest released solidity version is 0.4.20, which cannot allow us to:
  • return bytes from non-internal contract call
  • pass arbiter struct value to non-internal contract call
• as a result, read and parse protobuf encoded value from storage contract will be more expensive than it should be, because we cannot avoid do like this to read arbiter length bytes from external contract for now.
• also this problem prevent our contract from linking with library generated by pb3sol, because we have to declare internal modifier for all encoder/decoder, which seems to prevent solc from generating delegatecall instruction.

Basic Usage

• simplest way is using docker. for example, under truffle project do like following.

mkdir -p `pwd`/contracts/libs/pb 		# output directory for pb3sol
mkdir -p `pwd`/proto 					# input directory (proto files)
touch /proto/test.proto 				# create protofile (and edit) 

# compile test.proto as test_pb.sol with dependent library, runtime.sol and put into `pwd`/contracts/libs/pb 
docker run --rm -ti -v `pwd`/contracts/libs/pb:/out -v `pwd`/proto:/in umegaya/pb3sol protoc -I/ -I/protoc/include --plugin=protoc-gen-sol=gen_runtime=runtime.sol:/out /in/test.proto

• for actually usage in solidity codes, see here and here

Handling native solidity type

• if you use docker image, built-in proto file "Solidity.proto" can import straight forward like import "Solidity.proto". then you can use .solidity.$typename to declare variable which directly convert to $typename variable in solidity codes.

  • if you don't want to use docker, copy above file into somewhere in your proto compiler $path.

• basically these solidity type is bytes variable boxed with message. but convert these bytes into correct number or bigint, is not trivial work.

  • we create node module soltype-pb to add small support for handling these native solidity types with protobufjs.
  • installation to your project: npm install soltype-pb
  • see here and here for usage.

pb3sol's Issues

Error running tests

Hi I have just tried to run the tests as documented, i.e. using docker running commands:

make tsh
cd test && make test_setup test_on_host

However this does not run successfully:

root@a31dc20795c5:/test# make test_setup test_on_host
cd src/soltype-pb && npm install
up to date in 0.065s
cd test && npm install
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

up to date in 0.116s
cd test/tools/testrpc && npm install
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

added 3 packages in 6.333s
make -C test run CONTAINER_RUNARG="" INPUT_DIR=`pwd`/test/proto OUTPUT_DIR=`pwd`/test/contracts/libs/pb
make[1]: Entering directory '/test/test'
protoc -I/ -I/protoc/include --plugin=protoc-gen-sol=/protoc/plugin/gen_sol.py --sol_out="gen_runtime=True:/test/test/contracts/libs/pb" /test/test/proto/TaskList.proto
sed -e 's/.solidity.int128 f4 = 5;/.solidity.int128 f4 = 5;.solidity.uint256 new_id = 6;/g' proto/TaskList.proto > proto/TaskList2.proto
protoc -I/ -I/protoc/include --plugin=protoc-gen-sol=/protoc/plugin/gen_sol.py --sol_out="pb_libname=pb2:/test/test/contracts/libs/pb" /test/test/proto/TaskList2.proto
bash ./tools/run_chain.sh
truffle test
net_version
eth_accounts
eth_accounts
Error parsing /test/test/contracts/libs/pb/TaskList2_pb.sol: ParsedContract.sol:351:5: ParserError: Expected type name
    enum prog_type;             
    ^--^
Compilation failed. See above.
makefile:23: recipe for target 'run' failed
make[1]: *** [run] Error 1
make[1]: Leaving directory '/test/test'
makefile:13: recipe for target 'test_on_host' failed
make: *** [test_on_host] Error 2
root@a31dc20795c5:/test# sl
bash: sl: command not found
root@a31dc20795c5:/test# ls
lREADME.md  makefile  src  test
root@a31dc20795c5:/test# vi makefile 
bash: vi: command not found
root@a31dc20795c5:/test# vim makefile 
bash: vim: command not found
root@a31dc20795c5:/test# nano makefile 
bash: nano: command not found
root@a31dc20795c5:/test#

thanks in advance for your help :)