Fast and space efficiency JSON serialization golang library. It is a schema oriented design which leverages schema definition to encode JSON document into compact binary encoded format, and decodes back into JSON document.
When we want to exchange data between services or over network, the JSON is most popular format to do it.
In golang world, the most convenient way is using official encoding/json
package to marshal and unmarshal JSON document from/to struct or map. For usual RESTful web service scenario, JSON format is quite convenience and representative, but for real-time message exchanging or other scenarios that has small footprint data and low-latency requirement, JSON is a bit too heavy weight, not only data footprint is not space saving, but also has heavy loading in encode/decode procedure.
So if we want to a compact, small footprint data for exchanging over network, and also leverages the convenience of JSON, we need an encoding format that removes "property name" and other notations likes ':', '[', '{'...etc from original JSON document, and leaves "value" only.
To achieve this goal, we need a schematic to define our JSON document and provide enough information for serialization engine to know sequence and data type of every properties in document. it's the reason why JSONPack
is a schema oriented design library.
- Similar Marshal / Unmarshal API to standard
encoding/json
package. - Space saving encoded format, the size of encoded data is similar to Protocol Buffers, can be 30-80% of original JSON document, depends on data.
- Blazing fast, provides about 3.x decoding speed compared to
protobuf
and many times than other JSON packages. - Memory saving design, avoids any un-necessary memory allocations, suitable for embedded environment.
- Has production ready javascript implementation Buffer Plus, can be used in node.js and Web browser environment.
- No need to write schema definition by hand,
jsonpack
will generate schema definition from golang struct automatically.
go get github.com/arloliu/jsonpack
Example of add schema definition
import "github.com/arloliu/jsonpack"
type Info struct {
Name string `json:"name"`
Area uint32 `json:"area"`
// omit this field
ExcludeField string `-`
}
jsonPack := jsonpack.NewJSONPack()
sch, err := jsonPack.AddSchema("Info", Info{}, jsonpack.LittleEndian)
Example of encoding data with Info
struct
infoStruct := &Info{
Name: "example name",
Area: 888,
}
// encodedResult1 contains encoded data,
encodedResult1, err := jsonPack.Marshal("Info", infoStruct)
Example of encoding data with golang map
infoMap := map[string]interface{} {
"name": "example name",
"area": 888,
}
encodedResult2, err := jsonPack.Marshal("Info", infoMap)
Example of decoding data
decodeInfoStruct = Info{}
err := jsonPack.Decode("Info", encodedResult1, &decodeInfoStruct)
decodeInfoMap = make(map[string]interface{})
err := jsonPack.Decode("Info", encodedResult2, &decodeInfoMap)
The benchmark result is a important reference but not always suitable for every scenarios.
Test environment: Intel i7-9700K [email protected].
Benchmark code is here
Sorts from fastest to slowest in the following.
ns/op | allocation bytes | allocation times | |
---|---|---|---|
jsonpack | 1933 ns/op | 752 B/op | 2 allocs/op |
jsoniter | 10134 ns/op | 3320 B/op | 46 allocs/op |
std. json | 23560 ns/op | 8610 B/op | 171 allocs/op |
goccy | 75298 ns/op | 82639 B/op | 651 allocs/op |
ns/op | allocation bytes | allocation times | |
---|---|---|---|
jsonpack | 6461 ns/op | 6512 B/op | 96 allocs/op |
jsoniter | 17436 ns/op | 9666 B/op | 290 allocs/op |
std. json | 18949 ns/op | 8864 B/op | 228 allocs/op |
goccy | 19985 ns/op | 15900 B/op | 316 allocs/op |
ns/op | allocation bytes | allocation times | |
---|---|---|---|
jsonpack | 1834 ns/op | 800 B/op | 3 allocs/op |
protobuf | 1972 ns/op | 896 B/op | 1 allocs/op |
goccy | 2166 ns/op | 1280 B/op | 1 allocs/op |
jsoniter | 3372 ns/op | 1296 B/op | 3 allocs/op |
std. json | 3578 ns/op | 1280 B/op | 1 allocs/op |
ns/op | allocation bytes | allocation times | |
---|---|---|---|
jsonpack | 1475 ns/op | 96 B/op | 2 allocs/op |
goccy | 3284 ns/op | 2215 B/op | 5 allocs/op |
jsoniter | 4680 ns/op | 1072 B/op | 79 allocs/op |
protobuf | 5075 ns/op | 3152 B/op | 84 allocs/op |
std. json | 18378 ns/op | 1232 B/op | 69 allocs/op |
The benchmark result indicates jsonpack keeps constant performance on both encoding and encoding side, and keeps very low memory allocation size and times.
The benchmark result also delivers some important messages.
- The performance of operating with golang map sucks.
- The pointer of type likes
*string
,*int32
cost twice of time. - Golang reflection is expensive,
reflect.ValueOf(...)
is very expensive.
According to the explanation of benchmarking and some tests during the implementation stage of this library, there are some suggestions in the following.
- It's better to use struct instead of map for encode/decode if possible.
- When declares struct fields, declares it as value instead of pointer to value gives performance gain.
- Designs a small and compacted structure, uses number type to represent enum. fields instead of declares it as string type.