Comments (6)
The extension code used by msgp is 5
, not -1
, so it doesn't have to follow the predefined time type.
According to the docs you can annotate with msg:"name,extension"
and register your new type.
This package predates the time extension proposal so that is probably why it is like this.
from msgp.
I've bumped into this and I would like to understand. The current implementation does not provide interoperability between other libraries such as https://www.npmjs.com/package/msgpack5, for instance. In this case, msgpack5 for node yields Error: unable to find ext type 5
when trying to deserialize time.Time
types serialized by msgp. I am very surprised that this is happening... and not so people are complaining. Other libs like github.com/vmihailenco/msgpack/v5 do not incur in this error.
I understand that the implementation chosen here might be different in order to strongly adhere to specs, but how is the global situation and the motivation to do this?
from msgp.
//msgp:ignore Time
// Time is a time.Time that serializes to/from Msgpack Extension -1.
type Time struct {
time.Time
}
func (t Time) ExtensionType() int8 {
return -1
}
func (t Time) Len() int {
// Time round towards zero time.
secPrec := t.Round(time.Second)
remain := t.Sub(secPrec)
asSecs := secPrec.Unix()
if remain == 0 && asSecs > 0 && asSecs <= math.MaxUint32 {
return 4
}
if asSecs < 0 || asSecs >= (1<<34) {
return 12
}
return 8
}
func (t Time) MarshalBinaryTo(bytes []byte) error {
// Time rounded towards zero.
secPrec := t.Truncate(time.Second)
remain := t.Sub(secPrec).Nanoseconds()
asSecs := secPrec.Unix()
if remain == 0 && asSecs > 0 && asSecs <= math.MaxUint32 {
if len(bytes) != 4 {
return fmt.Errorf("expected length 4, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes, uint32(asSecs))
return nil
}
if asSecs < 0 || asSecs >= (1<<34) {
if len(bytes) != 12 {
return fmt.Errorf("expected length 12, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes[:4], uint32(remain))
binary.BigEndian.PutUint64(bytes[4:], uint64(asSecs))
return nil
}
if len(bytes) != 8 {
return fmt.Errorf("expected length 8, got %d", len(bytes))
}
binary.BigEndian.PutUint64(bytes, uint64(asSecs)|(uint64(remain)<<34))
return nil
}
func (t *Time) UnmarshalBinary(bytes []byte) error {
switch len(bytes) {
case 4:
secs := binary.BigEndian.Uint32(bytes)
t.Time = time.Unix(int64(secs), 0)
return nil
case 8:
data64 := binary.BigEndian.Uint64(bytes)
nsecs := int64(data64 >> 34)
if nsecs > 999999999 {
// In timestamp 64 and timestamp 96 formats, nanoseconds must not be larger than 999999999.
return fmt.Errorf("nsecs overflow")
}
secs := int64(data64 & 0x3ffffffff)
t.Time = time.Unix(secs, nsecs)
return nil
case 12:
nsecs := int64(binary.BigEndian.Uint32(bytes[:4]))
if nsecs > 999999999 {
// In timestamp 64 and timestamp 96 formats, nanoseconds must not be larger than 999999999.
return fmt.Errorf("nsecs overflow")
}
secs := int64(binary.BigEndian.Uint64(bytes[4:]))
t.Time = time.Unix(secs, nsecs)
return nil
}
return fmt.Errorf("unknown time format length: %v", len(bytes))
}
The spec doesn't specify explicitly that numbers are big endian, but since the other msgpack are, I assume they are.
Annotate with ",extension", eg:
Modtime Time `msg:"Modtime,extension"`.
Roundtrip Test
func TestTime_MarshalBinaryTo(t *testing.T) {
rng := rand.New(rand.NewSource(0))
for i := 0; i < 10000; i++ {
in := Time{
Time: time.Unix(rng.Int63()-1<<62, rng.Int63n(999999999+1)),
}
dst := make([]byte, in.Len())
err := in.MarshalBinaryTo(dst)
if err != nil {
t.Fatal(err)
}
var got Time
err = got.UnmarshalBinary(dst)
if err != nil {
t.Fatal(err)
}
if !in.Equal(got.Time) {
t.Errorf("%v != %v", in.Time, got.Time)
}
}
for i := 0; i < 10000; i++ {
in := Time{
Time: time.Unix(rng.Int63n(1<<32), 0),
}
dst := make([]byte, in.Len())
if len(dst) != 4 {
t.Errorf("unexpected length: %v", len(dst))
}
err := in.MarshalBinaryTo(dst)
if err != nil {
t.Fatal(err)
}
var got Time
err = got.UnmarshalBinary(dst)
if err != nil {
t.Fatal(err)
}
if !in.Equal(got.Time) {
t.Errorf("%v != %v", in.Time, got.Time)
}
}
for i := 0; i < 10000; i++ {
in := Time{
Time: time.Unix(rng.Int63n(1<<34), rng.Int63n(999999999+1)),
}
dst := make([]byte, in.Len())
if len(dst) > 8 {
t.Errorf("unexpected length: %v", len(dst))
}
err := in.MarshalBinaryTo(dst)
if err != nil {
t.Fatal(err)
}
var got Time
err = got.UnmarshalBinary(dst)
if err != nil {
t.Fatal(err)
}
if !in.Equal(got.Time) {
t.Errorf("%v != %v", in.Time, got.Time)
}
}
}
from msgp.
@klauspost you certainly were quick in answering and providing this solution! Is there any repo with extensions like this?
Thanks so much!
from msgp.
@klauspost I made as you pointed, but now I get this error when running tests (and thus running msgp autogenerated tests): EOF
. Debugging reached me to this function of the fwd
package.
// ReadMapHeader reads the next object
// as a map header and returns the size
// of the map and the number of bytes written.
// It will return a TypeError{} if the next
// object is not a map.
func (m *Reader) ReadMapHeader() (sz uint32, err error) {
}
My time struct is defined as follows:
//msgp:ignore Time
// Time is a time.Time that serializes to/from Msgpack Extension -1.
type Time struct {
time.Time
}
func (t Time) Equal(other Time) bool {
return t.Time.Equal(other.Time)
}
func (t Time) FromTime(native time.Time) Time {
t.Time = native
return t
}
func (t Time) Truncate(d time.Duration) Time {
return Time{Time: t.Time.Truncate(d)}
}
func (t Time) Add(d time.Duration) Time {
return Time{Time: t.Time.Add(d)}
}
func (t Time) ExtensionType() int8 {
return -1
}
func (t Time) Len() int {
// Time round towards zero time.
secPrec := t.Round(time.Second)
remain := t.Sub(secPrec)
asSecs := secPrec.Unix()
if remain == 0 && asSecs > 0 && asSecs <= math.MaxUint32 {
return 4
}
if asSecs < 0 || asSecs >= (1<<34) {
return 12
}
return 8
}
func (t Time) MarshalBinaryTo(bytes []byte) error {
// Time rounded towards zero.
secPrec := t.Time.Truncate(time.Second)
remain := t.Sub(secPrec).Nanoseconds()
asSecs := secPrec.Unix()
if remain == 0 && asSecs > 0 && asSecs <= math.MaxUint32 {
if len(bytes) != 4 {
return fmt.Errorf("expected length 4, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes, uint32(asSecs))
return nil
}
if asSecs < 0 || asSecs >= (1<<34) {
if len(bytes) != 12 {
return fmt.Errorf("expected length 12, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes[:4], uint32(remain))
binary.BigEndian.PutUint64(bytes[4:], uint64(asSecs))
return nil
}
if len(bytes) != 8 {
return fmt.Errorf("expected length 8, got %d", len(bytes))
}
binary.BigEndian.PutUint64(bytes, uint64(asSecs)|(uint64(remain)<<34))
return nil
}
func (t *Time) UnmarshalBinary(bytes []byte) error {
switch len(bytes) {
case 4:
secs := binary.BigEndian.Uint32(bytes)
t.Time = time.Unix(int64(secs), 0)
return nil
case 8:
data64 := binary.BigEndian.Uint64(bytes)
nsecs := int64(data64 >> 34)
if nsecs > 999999999 {
// In timestamp 64 and timestamp 96 formats, nanoseconds must not be larger than 999999999.
return fmt.Errorf("nsecs overflow")
}
secs := int64(data64 & 0x3ffffffff)
t.Time = time.Unix(secs, nsecs)
return nil
case 12:
nsecs := int64(binary.BigEndian.Uint32(bytes[:4]))
if nsecs > 999999999 {
// In timestamp 64 and timestamp 96 formats, nanoseconds must not be larger than 999999999.
return fmt.Errorf("nsecs overflow")
}
secs := int64(binary.BigEndian.Uint64(bytes[4:]))
t.Time = time.Unix(secs, nsecs)
return nil
}
return fmt.Errorf("unknown time format length: %v", len(bytes))
}
func init() {
// Registering an extension is as simple as matching the
// appropriate type number with a function that initializes
// a freshly-allocated object of that type
msgp.RegisterExtension(-1, func() msgp.Extension { return new(Time) })
}
and the model is
Model struct {
Time Time `json:"time" msg:"time,extension"`
//... some other fields
}
from msgp.
Seems like MarshalBinaryTo
cannot expect to be given the exact output size. Fixed:
func (t Time) MarshalBinaryTo(bytes []byte) error {
// Time rounded towards zero.
secPrec := t.Time.Truncate(time.Second)
remain := t.Sub(secPrec).Nanoseconds()
asSecs := secPrec.Unix()
if remain == 0 && asSecs > 0 && asSecs <= math.MaxUint32 {
if len(bytes) < 4 {
return fmt.Errorf("need length 4, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes, uint32(asSecs))
return nil
}
if asSecs < 0 || asSecs >= (1<<34) {
if len(bytes) < 12 {
return fmt.Errorf("need length 12, got %d", len(bytes))
}
binary.BigEndian.PutUint32(bytes[:4], uint32(remain))
binary.BigEndian.PutUint64(bytes[4:], uint64(asSecs))
return nil
}
if len(bytes) < 8 {
return fmt.Errorf("need length 8, got %d", len(bytes))
}
binary.BigEndian.PutUint64(bytes, uint64(asSecs)|(uint64(remain)<<34))
return nil
}
from msgp.
Related Issues (20)
- map[string]struct{} unused variable
- `map[int]int` failed to parse
- Compilation error about unused variable in generated *_gen.go file if the struct only has a single property
- Output size configuration
- unnecessary wrapping HOT 2
- new cmd option to generate 1 file per package HOT 1
- Can we use the `json` tag to generate code
- Add Linux and Windows binaries into the release section HOT 2
- Add example for marshaling array of structs into documentation HOT 1
- Can we have a new release tag? HOT 3
- "no definitions in issue185_test.go" HOT 1
- Proposal: Add check via interface for IsEmpty() method, allows treating unknown types as empty HOT 6
- Support for go generics HOT 8
- Add generator option to always produce pointer receivers HOT 3
- Cut v1.1.9 or v1.2.0 release HOT 1
- Extension interface is too generic
- Flattening not working with non-local types HOT 1
- How to using UUID ? (GitHub.com/google/uuid) HOT 1
- Make `shim` directive handle map key HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from msgp.