bnclabs / gson Goto Github PK

While collating large uint64 and int64 that are outside the bounds of floating point precision for integers, we are self composing a e-notation for such numbers. While doing so, we are supposed to use -1 precision https://golang.org/pkg/strconv/#FormatFloat.

Solution: Use math/big in such cases.

Refer: json5

Let it be a step by step approach highlighting common use case. The challenge though, would be to figure out the common use cases.

travis-ci might bail out if gson-tools is going to randomly thrash gson functions for a long time, find a reasonable time slot and execute gson-tools/make check as part of travis-ci

If golang type json.Number is encounter while transforming
value -> {JSON, CBOR, Binary-collate} treat them in the following manner:

While transforming to JSON:

Do nothing, json.Number is just a string and encode them as JSON string.

While transforming to CBOR:

Check if json.Number is negative by checking for a leading - . If so encode it as CBOR-int64, else encode it as CBOR-uint64.

While transforming to Binary-collate:

Check if json.Number is negative by checking for a leading -. Either case, eventually it gets encoded as float64.

Note that collate algorithm transparently treats int, uint, float as
number and make it comparable as same type, as number.

BenchmarkVal2CollMap 50000 26696 ns/op 16416 B/op 2 allocs/op
BenchmarkVal2CollTyp 10000 174548 ns/op 49250 B/op 6 allocs/op

BenchmarkJson2CollStr 2000 635015 ns/op 1048647 B/op 5 allocs/op

BenchmarkCbor2CollMap 2000 787087 ns/op 1089654 B/op 8 allocs/op
BenchmarkCbor2CollTyp 500 2500523 ns/op 3269067 B/op 25 allocs/op

When key used in a kv storage is shaped as JSON string, then we need some additional support besides what is available in JSON spec

• Min-Unbounded, to denote that we want to iterate from the first item.
• Max-Unbounded, to denote that we want to iterate till the last item.
• -Infinity, to denote smallest number.
• +Infinity, to denote largest number.
• NaN, since it is included in IEEE 754 spec.
• Missing, what is the sort position for missing in the supported list of types.

under function value_cbor.go:valtime2cbor()

v.Format(time.RFC3339) // TODO: make rfc3339 as config.

Similarly, under function cbor_value.go:cbor2dtval()

time.Parse(time.RFC3339, item.(string))

JSON-pointer is relative new compared to JSON, so it is good to have an introductory page on JSON-pointer and have references to advanced materials.

Values in int64, uint64 and float64 should be comparable as if they belong to the same type -Number. Like suggested elsewhere, sorting is a human perception.

Also while converting Binary-collate back to Value, or CBOR, or JSON make sure that precision and accuracy is not lost.

Finally, add test cases to validate them same.

Collating strings is a complex affair. To describe some cases where gson's collation algorthim might fail are:

• When applications want to sort smaller case letters (like 'a', 'b', 'c') to sort before capital case letters (like 'A', 'B', 'C').
• When application want to sort latin characters in the context of different languages like german or swedish. Refer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Collator
• When applications want to follow ICU collation standard for unicode characters.
  • http://userguide.icu-project.org/collation/architecture
  • http://site.icu-project.org/design/collation/v2
• Databases can used different collation standard.
  • https://www.ibm.com/support/knowledgecenter/en/SSGU8G_11.50.0/com.ibm.glsug.doc/ids_gug_026.htm

Refer: collation

Right now val.Tojson() supports the following types:

• map[string]interface{}
• [][2]interface{}

Both of them are converted to JSON property object. Support for:

• map[string]uint64.

As of now, Gson supports following CBOR tagged data types:

tagDateTime, tagEpoch, tagPosBignum, tagNegBignum,
tagDecimalFraction, tagBigFloat, tagCborEnc, tagRegexp,
tagCborPrefix

I don't think JSON supports any of them in its present form. If we
can find some innovative way to still represent these tagged data
types within the limited set of JSON types, we can explore further.

Refer: test-vectors

Behind the scenes, support load balancing based on random / moving-average of latency.

Ref: https://github.com/bradleyfalzon/apicompat

This implementation assigns tags 37 and 38 with different meanings than in the document http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml#tags . Using these tags should be avoided, as CBOR data that makes use of them with the meanings other than specified in that document would make them less interoperable; instead, tags starting from 266 should be used; they're currently not assigned according to that document.

And how it affects different operations.

NewValue(map).ToJson(jsn).Bytes() should return JSON string with keys sorted.

Make this repeatable with future version of gson, and alternate tools similar to gson. Collate the details and numbers that can be consumed via slides, wiki et al.

(a) Replace sync.Pool for objects with simple linked list of objects.
(b) Remove defer() calls that can happen in tight loops.

(a) will have to be tested and benchmarked for

• Converting map objects form CBOR to collation format.
• Converting collated string to JSON format.
• Converting JSON string data to Collation formation.
• Converting map objects from golang value to collation format.
• Converting map objects from golang value to JSON format.
• Converting map[string]uint64 from golang value to JSON format.
• Converting map objects from golang value to CBOR format.
• List JSON pointers in a given golang value map..

(b) will have to be tested and benchmarked for

• Converting CBOR array to Collation format (both definite and indefinite CBOR variant).
• Converting CBOR map to Collation format (both definite and indefinite CBOR variant).
• List JSON pointers in a given golang value map.
• Converting map objects from golang value to CBOR.
• Converting map objects and map[string]uint64 golang value to JSON format.
• Converting map objects from golang value to Collation format.
• Converting string and map objects from JSON to Collation format.
• Converting string and map objects from JSON to golang value.
• Converting from Collated string to JSON format.
• Converting from Collated string to CBOR format.