iqlusioninc / veriform

Security-oriented protobuf-like serialization format with "Merkleized" content hashing support

Veriform is a cryptographically verifiable data serialization format inspired by Protocol Buffers, useful for things like credentials, transparency logs, and "blockchain" applications.

Specification

Rationale (a.k.a. "Oh no! Not another serialization format!")

If you look at another engineer's work and think, "That's dumb. Why don't you just..." Take a breath. Find out why the problem is hard. —Adrienne Porter Felt

Obligatory xkcd

Veriform is similar in design to Protocol Buffers, which is at its core a schema-driven format. It also includes a small amount of information which makes it partly self-describing, "wire types", however they are too limited to comprehend serialized messages in absence of a schema.

Without a fully self-describing wire format, it isn't possible to implement Veriform's primary distinguishing feature: "Merkleized" content hashing which functions even if some of the fields in a message aren't known by the schema. This means that the sender and receiver of a message don't need to be working with the same version of a schema to agree on a message hash, which enables schema evolution.

Veriform is not intended to be a general purpose serialization format: for that we recommend something like Protocol Buffers or Cap'n Proto. While it's fine to use Veriform for general purpose serialization if it fits your needs, it's lacking many features such as an associated RPC protocol. Instead, Veriform is the sort of thing you might use for the credentials passed as part of an RPC protocol.

Another interesting use case for Veriform is Certificate Transparency or otherwise "blockchain"-like systems that heavily rely on cryptographic integrity and Merkle proofs.

Veriform's data model is isomorphic with a subset of TJSON, a microformat which extends JSON with richer types. All Veriform documents can be bidirectionally transcoded to/from TJSON with no data loss. Furthermore, TJSON documents can be authenticated with the same Merkleized hashing scheme as Veriform, meaning signatures for one encoding will validate in the other.

Comparison with other serialization formats

The table below compares Veriform to the other formats:

Name	Schemas	Self-Describing	Integers	Authentication	Standardization
Veriform	💚†	💚	vint64	Structured Hashing	None
ASN.1 DER	💔	💛	Fixed-Width	Canonicalization	ITU/IETF
Cap'n Proto	💚	💚	Fixed-Width	Canonicalization	None
CBOR	💔	💚	Fixed-Width	Canonicalization	IETF
csexp	💔	💚	Fixed-Width	Canonicalization	IETF
MessagePack	💔	💚	Fixed-Width	None	None
Protobuf	💚	💔	LEB128	Canonicalization	None
XDR	💚	💔	Fixed-Width	None	IETF

†NOTE: Coming soon!

Status

Veriform does not yet provide the minimum viable functionality it needs to be useful. The table below covers the current implementation state:

Status	Feature	Notes
🚧	Message Decoding	In progress
⛔	Message Encoding	Not started
🚧	Structured Hashing	In progress
⛔	Schemas	Not started
🚧	TJSON Transcoding	In progress

NOTE: Veriform is a multi-language monorepo: all implementations in all languages within the repo are intended to implement the spec in its current state and share a consistent feature set. The progress above applies equally to all language implementations currently within the repo.

Copyright

Copyright © 2017-2020 Tony Arcieri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be licensed as above, without any additional terms or conditions.

veriform's Issues

Support TAI64 timestamps

Currently, only TAI64N timestamps are supported, which represent time as 96 bits (64 for seconds and 32 for nanoseconds). The specs erroneously define timestamps as using the TAI64 format.

There should be support for TAI64 in addition to TAI64N. The former uses only 64 bits to represent time at a resolution of seconds, and is 33% smaller than TAI64N (in other words, TAI64N is 50% larger than TAI64). This is more than sufficient for almost all blockchains, which do not have an external clock and have blocktimes much longer than a single second.

Add comparison with NoProto

It would be nice to add NoProto to the comparisons list.

Fuzz testing

This issue tracks the current state of fuzz testing across zser's various implementations. It will be closed when all implementations are fuzzed.

Language	Varints	Messages	Method
Go	⛔	⛔	go-fuzz?
Python	⛔	⛔	kitty?
Ruby	⛔	⛔	FuzzBert?
Rust	✅	⛔	cargo fuzz
TypeScript	⛔	⛔	js-fuzz?

RUSTSEC-2020-0146: arr! macro erases lifetimes

arr! macro erases lifetimes

Details
Package	generic-array
Version	0.12.3
URL	fizyk20/generic-array#98
Date	2020-04-09
Patched versions	>=0.14.0
Unaffected versions	<0.8.0

Affected versions of this crate allowed unsoundly extending
lifetimes using arr! macro. This may result in a variety of
memory corruption scenarios, most likely use-after-free.

See advisory page for additional details.

RUSTSEC-2020-0146: arr! macro erases lifetimes

arr! macro erases lifetimes

Details
Package	generic-array
Version	0.13.2
URL	fizyk20/generic-array#98
Date	2020-04-09
Patched versions	>=0.14.0
Unaffected versions	<0.8.0

Affected versions of this crate allowed unsoundly extending
lifetimes using arr! macro. This may result in a variety of
memory corruption scenarios, most likely use-after-free.

See advisory page for additional details.

Question: State and future of veriform

Hey @tarcieri,

In search for what's out there in regards to a replacement for json/cbor that can be consistently hashed I bumped into TJSON and Veriform (as well as object hash) all of which are amazing projects.

They also all seem to stop dead in their tracks last year.
The most interesting to me is Veriform as it brings TJSON and content aware (structured I think you call it here) hashing.

So I was wondering if there is any plan to continue work on either of these projects and if not if you wouldn't mind pointing me to any difficulties or roadblocks in this attempt.

Thank you very much for your work.

Regards,
George

vint64: failure to roundtrip 1048576

This number is encoded as [0x8, 0x0, 0x0, 0x1], but decoding that again fails with a LeadingZeroes error.

Decoding [0x8, 0xbb, 0x3, 0x1] fails with the same error, which might be the same issue (this one I found while trying to make rustc use vint64 for metadata encoding).

Support additional hashing functions

Additional hashing algorithms should be added to implementations.

id	name	reason
sha-256d	2-SHA-256	Prevents length extension attacks, and can otherwise leverage the same benefits (e.g. hardware implementations) as SHA-256.
keccak-256	Keccak-256	There are concerns that the changes to Keccak when finalizing it to SHA3 made it more vulnerable.
sha3-256	SHA3-256	SHA3 is the next standard hash function, and unlike SHA2 is not known to be vulnerable to length extension attacks.