crystal-community / msgpack-crystal Goto Github PK

View Code? Open in Web Editor NEW

133.0 133.0 18.0 435 KB

MessagePack implementation in Crystal msgpack.org[Crystal]

Crystal 97.28% Ruby 2.30% Shell 0.42%

msgpack-crystal's People

Contributors

Stargazers

Watchers

Forkers

xanders asterite maiha exabyt3s hinrik fryguy xqyww123 vladfaust onyxframework didactic-drunk digitalextremist will jgaskins wstein dan-do descholar-ceo

msgpack-crystal's Issues

lexer can block before returning last token

If reading from an IO, say a socket, that is not closed the lexer will block on next_byte if there is no more data currently available. F.ex if one is sending a stream of integers.

Here is a snippet from next_token

    when 0xE0..0xFF
      @token.type = :INT
      @token.int_value = current_byte.to_i8
      next_byte

If that int is the last byte available next_byte blocks and the just read token is not returned.

I think it is also a problem that next_byte is called from the lexer constructor. This means that constructing an Unpacker can actually block if there is no data available.

how to use it?

any example on decode, encode crystal objects

not correct handle union type in mapping

class A
  MessagePack.mapping({
    bla: {type: String, nilable: true}
  })
end

p A.from_msgpack({"bla" => nil}.to_msgpack) # => #<A:0x100e2af20 @bla=nil>

class B
  MessagePack.mapping({
    bla: String?
  })
end

p B.from_msgpack({"bla" => nil}.to_msgpack) # => Nil assertion failed (Exception)

Bad process on nil

data = ({aa: "\0\1\2", bb: nil}).to_msgpack # => 
# Bytes[130, 162, 97, 97, 163, 0, 1, 2, 162, 98, 98, 192]
# The serialized data is correct.
NamedTuple(aa: String, bb: String?).from_msgpack data # => 
# Unhandled exception: Missing msgpack attribute: bb at 0 (MessagePack::UnpackException)
#  from lib/msgpack/src/message_pack/from_msgpack.cr:218:3 in 'new'
#  from lib/msgpack/src/message_pack/from_msgpack.cr:5:3 in 'from_msgpack'

Type discriminator

In the standard library, JSON and YAML have defined a pattern for switching on the deserialized types using a "discriminator" field. For example, in this class:

require "json"

abstract struct Message
  include JSON::Serializable

  use_json_discriminator "type", {
    created: Created,
    updated: Updated,
  }

  getter id : String
end

struct Created < Message
end

struct Updated < Message
end

The use_json_discriminator macro sets type: "created" in the JSON to deserialize to the Created type and type: "updated" to Updated:

Message.from_json({type:"created",id:"a"}.to_json)
# => Created(@id="a")
Message.from_json({type:"updated",id:"a"}.to_json)
# => Updated(@id="a")

It'd be awesome to get that in MessagePack, as well. The implementation for use_json_discriminator is here.

Parsing an unknown data structure.

Consider the following class:

class Foo
  def initialize(@name = "")
  end

  MessagePack.mapping({
    name: String,
    attributes: {type: MessagePack::Table}
  })
end

Attempting to decode the following results in an error:

{
 "name" => "bar",
 "attributes" => {
    "foo" => "bar"
  }
}.to_msgpack

in ./libs/msgpack/message_pack/from_msgpack.cr:114: undefined method 'new' for MessagePack::Type:Class

      hash[k] = V.new(pull)

So what I'm trying to figure out is if I don't know the structure of the data that I might be recieving how do I decode it from message pack?

Thanks!

Support for nested Hash

Can we add support to pack and unpack

Hash(String, Hash(Symbol, (Int32 | String | MyClass)))

etc..

The fix of #49 doesn't cover type restriction.

data = ({aa: "a", bb: nil}).to_msgpack
typeof(NamedTuple(aa: String, bb: Nil).from_msgpack(data)[:aa]) # => (String | Nil) but should be just `String`

And this breaks my compiler type-check.

Originally posted by @xqyww123 in https://github.com/crystal-community/msgpack-crystal/issue_comments#issuecomment-441992829

different encoding ints

ruby -e 'require "msgpack"; p [1,2,3].to_msgpack.bytes'
[147, 1, 2, 3]

crystal eval 'require "./src/msgpack"; p [1,2,3].to_msgpack'
[147, 210, 0, 0, 0, 1, 210, 0, 0, 0, 2, 210, 0, 0, 0, 3]

much bigger.

Allow to write hash and array by parts

is it possible to write like this? to save us from create temp hashes, arrays of Type.

packer.write_hash_start
packer.write_hash_key("bla")
packer.write_hash_value(123)
packer.write_hash_key("bla2")
packer.write_hash_value(true)
packer.write_hash_end

packer.write_array_start
packer.write_array_element("bla")
packer.write_array_element(123)
packer.write_array_element("bla2")
packer.write_array_element(true)
packer.write_array_end

Support tarantool serializable

Tarantool send response as array of array (not hash), but I would like to mapping this response.
How can I get something like this?

class MessagePackPersonArray
  include MessagePack::Serializable
  include MessagePack::Serializable::Unmapped

  @[MessagePack::Field(key: 0)]
  property name : String
  
  @[MessagePack::Field(key: 1)]
  property age : Int32?
end

person = MessagePackPersonArray.from_msgpack(["John", 30, "1", 2, [1, 2, 3]].to_msgpack)
person.name.should eq("John")
person.age.should eq(30)

https://github.com/vladfaust/tarantool-crystal (use msgpack-crystal)

Packing array of hashes

Hello,

I want to write in a MessagePack::Packer.new a bunch of hashes
to do this I use

array_of_hash.map {|h| packer.write(h)

but I have this error

Error in ./src/zaqar.cr:15: no overload matches 'MessagePack::Packer#write' with type Hash(String, Nil | String | Float32 | Float64 | Time | JSON::Any | Bool | Int32 | Int64 | Int16 | Slice(UInt8))
Overloads are:
 - MessagePack::Packer#write(value : Nil)
 - MessagePack::Packer#write(value : Bool)
 - MessagePack::Packer#write(value : String)
 - MessagePack::Packer#write(value : Symbol)
 - MessagePack::Packer#write(value : Float32 | Float64)
 - MessagePack::Packer#write(value : Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64)
 - MessagePack::Packer#write(value : Hash(Type, Type))
 - MessagePack::Packer#write(value : Array(Type))
 - MessagePack::Packer#write(value : Tuple)

However

array_of_hash.map {|h| packer.write(h : Hash(Type, Type)) }

is not yet supported by crystal

DO you know when this will be available ?

Regards,

Support for missing or optional

Hi, does the project provide any feature like missing or optional property in a class?

Typo

msgpack-crystal/src/message_pack/serializable.cr

Line 64 in 84fa461

 # * **presense**: if `true`, a `@{{key}}_present` instance variable will be generated when the key was present (even if it has a `null` value), `false` by default 

presense -> presence

Crash in mapping

I use this data structure:

class MyData
  MessagePack.mapping({
    # [{'id' => {:payload => Slice(UInt8), :alive => Bool}}]
    array: Array(Hash(String, Hash(Symbol, (Slice(UInt8) | Bool)))),
  })
end

When I tried to use it, my compiler crashed with:

in lib/msgpack/src/message_pack/from_msgpack.cr:5: instantiating 'new(MessagePack::Unpacker)'

  new parser
  ^~~

in src/msg_pack_types.cr:4: expanding macro

  MessagePack.mapping({
  ^

in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 23:

   1.     
   2.       
   3.     
   4. 
   5.     
   6.       @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) 
   7. 
   8.       def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
   9.         @array = _array
  10.       end
  11. 
  12.       def array
  13.         @array
  14.       end
  15.     
  16. 
  17.     def initialize(__temp_23 : MessagePack::Unpacker)
  18.       
  19.         __temp_24 = nil
  20.         __temp_25 = false
  21.       
  22. 
> 23.       __temp_23.read_hash(false) do
  24.         case __temp_26 = Bytes.new(__temp_23)
  25.         
  26.           when "array".to_slice
  27.             __temp_25 = true
  28.             __temp_24 =
  29.               
  30. 
  31.               
  32.                 Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
  33.               
  34. 
  35.             
  36.         
  37.         else
  38.           
  39.             __temp_23.skip_value
  40.           
  41.         end
  42.       end
  43. 
  44.       
  45.         
  46.           if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
  47.             raise MessagePack::UnpackException.new("missing msgpack attribute: array")
  48.           end
  49.         
  50.       
  51. 
  52.       
  53.         
  54.           @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
  55.         
  56.       
  57.     end
  58. 
  59.     def to_msgpack(packer : MessagePack::Packer)
  60.       packer.write_hash_start(1)
  61.       
  62.         _array = @array
  63.         packer.write("array")
  64.         
  65.           _array.to_msgpack(packer)
  66.         
  67.       
  68.     end
  69.   

instantiating 'MessagePack::Unpacker#read_hash(Bool)'
in /msg_pack_types.cr:4: expanding macro

  MessagePack.mapping({
  ^

in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 23:

   1.     
   2.       
   3.     
   4. 
   5.     
   6.       @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) 
   7. 
   8.       def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
   9.         @array = _array
  10.       end
  11. 
  12.       def array
  13.         @array
  14.       end
  15.     
  16. 
  17.     def initialize(__temp_23 : MessagePack::Unpacker)
  18.       
  19.         __temp_24 = nil
  20.         __temp_25 = false
  21.       
  22. 
> 23.       __temp_23.read_hash(false) do
  24.         case __temp_26 = Bytes.new(__temp_23)
  25.         
  26.           when "array".to_slice
  27.             __temp_25 = true
  28.             __temp_24 =
  29.               
  30. 
  31.               
  32.                 Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
  33.               
  34. 
  35.             
  36.         
  37.         else
  38.           
  39.             __temp_23.skip_value
  40.           
  41.         end
  42.       end
  43. 
  44.       
  45.         
  46.           if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
  47.             raise MessagePack::UnpackException.new("missing msgpack attribute: array")
  48.           end
  49.         
  50.       
  51. 
  52.       
  53.         
  54.           @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
  55.         
  56.       
  57.     end
  58. 
  59.     def to_msgpack(packer : MessagePack::Packer)
  60.       packer.write_hash_start(1)
  61.       
  62.         _array = @array
  63.         packer.write("array")
  64.         
  65.           _array.to_msgpack(packer)
  66.         
  67.       
  68.     end
  69.   

instantiating 'MessagePack::Unpacker#read_hash(Bool)'
in msg_pack_types.cr:4: expanding macro

  MessagePack.mapping({
  ^

in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 32:

   1.     
   2.       
   3.     
   4. 
   5.     
   6.       @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) 
   7. 
   8.       def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
   9.         @array = _array
  10.       end
  11. 
  12.       def array
  13.         @array
  14.       end
  15.     
  16. 
  17.     def initialize(__temp_23 : MessagePack::Unpacker)
  18.       
  19.         __temp_24 = nil
  20.         __temp_25 = false
  21.       
  22. 
  23.       __temp_23.read_hash(false) do
  24.         case __temp_26 = Bytes.new(__temp_23)
  25.         
  26.           when "array".to_slice
  27.             __temp_25 = true
  28.             __temp_24 =
  29.               
  30. 
  31.               
> 32.                 Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
  33.               
  34. 
  35.             
  36.         
  37.         else
  38.           
  39.             __temp_23.skip_value
  40.           
  41.         end
  42.       end
  43. 
  44.       
  45.         
  46.           if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
  47.             raise MessagePack::UnpackException.new("missing msgpack attribute: array")
  48.           end
  49.         
  50.       
  51. 
  52.       
  53.         
  54.           @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
  55.         
  56.       
  57.     end
  58. 
  59.     def to_msgpack(packer : MessagePack::Packer)
  60.       packer.write_hash_start(1)
  61.       
  62.         _array = @array
  63.         packer.write("array")
  64.         
  65.           _array.to_msgpack(packer)
  66.         
  67.       
  68.     end
  69.   

instantiating 'Array(Hash(String, Hash(Symbol, Bool | Slice(UInt8)))):Class#new(MessagePack::Unpacker)'
in lib/msgpack/src/message_pack/from_msgpack.cr:86: instantiating 'new(MessagePack::Unpacker)'

  new(pull) do |element|
  ^~~

in lib/msgpack/src/message_pack/from_msgpack.cr:93: instantiating 'MessagePack::Unpacker#read_array()'

  pull.read_array do
       ^~~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:93: instantiating 'MessagePack::Unpacker#read_array()'

  pull.read_array do
       ^~~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:94: instantiating 'Hash(String, Hash(Symbol, Bool | Slice(UInt8))):Class#new(MessagePack::Unpacker)'

    yield T.new(pull)
            ^~~

in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'

  pull.read_hash(false) do
       ^~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'

  pull.read_hash(false) do
       ^~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:114: instantiating 'Hash(Symbol, Bool | Slice(UInt8)):Class#new(MessagePack::Unpacker)'

      hash[k] = V.new(pull)
                  ^~~

in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'

  pull.read_hash(false) do
       ^~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'

  pull.read_hash(false) do
       ^~~~~~~~~

in lib/msgpack/src/message_pack/from_msgpack.cr:109: undefined method 'new' for Symbol:Class

    k = K.new(pull)
          ^~~

Can't infer the type of instance variable '@var_present'

require "msgpack"

struct Foo
  include MessagePack::Serializable

  @[MessagePack::Field(presence: true)]
  getter foo : Int32
end

foo = Foo.from_msgpack({foo: 42}.to_msgpack)

Can't infer the type of instance variable '@foo_present' of Foo

I understand that I could add one more getter to Foo, but I don't want to pollute it. Doubling the amount of getters is too much for a simple object.

@[MessagePack::Field(ignore: true)]
getter? foo_present : Bool

Reduced problem (https://carc.in/#/r/6qks):

class Foo
  def initialize
    foo = true
    @bar = foo
  end
end

Foo.new

Can't infer the type of instance variable '@bar' of Foo

It can be fixed with @bar = uninitialized Bool (https://carc.in/#/r/6qkt):

class Foo
  def initialize
+   @bar = uninitialized Bool
    foo = true
    @bar = foo
  end
end

Foo.new

However, the same approach doesn't work with Serializable#initialize 😢

Missing Timestamp extension type

https://github.com/msgpack/msgpack/blob/master/spec.md#formats-timestamp

I suggest emitting the ext timestamp format when serializing UTC Time rather than a custom format potentially solving issues with #58.

Broken stream reading, when type mismatch

example failed code

    TCPServer.open("localhost", 5000) do |server|
      TCPSocket.open("localhost", server.local_address.port) do |client|
        sock = server.accept

        {1, 2, 3.5, 4, 5}.to_msgpack(client)
        1.to_msgpack(client)

        pull = MessagePack::Unpacker.new(sock)
        expect_raises(MessagePack::Error) do
          Array(Int32).new(pull)
        end

        Int32.new(pull).should eq 1
      end
    end

it got 4, instead of 1

What we should do in this case, i think we should skip all values for every container types, after exception. @benoist what you think?

have Serializable override inspect?

I'm not sure, but would it be a good idea to have it so including Serializable also overwrites inspect(io : IO) to include the properties?

The upside, is as a client instead of getting something like this

server

record Details, name : String, count : Int32 do
    include MessagePack::Serializable
end

def add_thing(id : String, details : Details) : Nil
end

client

c.add_thing  "hi", {name: "will"}
Traceback (most recent call last):
        6: from ./repl:32:in `<main>'
        5: from (irb):2
        4: from (irb):2:in `rescue in irb_binding'
        3: from ./repl:12:in `method_missing'
        2: from /Users/will/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/msgpack-rpc-0.6.0/lib/msgpack/rpc/session.rb:54:in `call'
        1: from /Users/will/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/msgpack-rpc-0.6.0/lib/msgpack/rpc/future.rb:54:in `get'
MessagePack::RPC::RuntimeError (bad arguments, expected [id : String, details : Details], but got details: HashT[1])

the error could maybe be nicer to better see what you were missing

MessagePack::RPC::RuntimeError (bad arguments, expected [id : String, details : Details[name : String, count : Int32], but got details: HashT[1])

There might be downsides though, but I'm not aware of what they would be, so I wanted to check first if this would be good.

I tried doing a PR to show this, but I can't figure out the macros :( in the macro included section, @type.instance_variables seems to be empty, and I don’t have a ton of experience with macros

Error while building with Crystal 0.21

instantiating 'MessagePack::Lexer#next_token(Tuple(), NamedTuple())'
in lib/msgpack/src/message_pack/lexer.cr:96: instantiating 'prefetch_token()'

    token = prefetch_token
            ^~~~~~~~~~~~~~

in lib/msgpack/src/message_pack/lexer.cr:51: instantiating 'read(UInt16:Class)'

      consume_binary(read UInt16)
                     ^~~~

in lib/msgpack/src/message_pack/lexer.cr:154: undefined constant T

  private def read(type : T.class)

Add support for type extension

Detailed here: https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types
And the binary format here: https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family

MessagePack allows applications to define application-specific types using the Extension type.
Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data.

Applications can assign 0 to 127 to store application-specific type information. An example usage is that application defines type = 0 as the application's unique type system, and stores name of a type and values of the type at the payload.

MessagePack reserves -1 to -128 for future extension to add predefined types. These types will be added to exchange more types without using pre-shared statically-typed schema across different programming environments.
[0, 127]: application-specific types
[-128, -1]: reserved for predefined types
Because extension types are intended to be added, old applications may not implement all of them. However, they can still handle such type as one of Extension types. Therefore, applications can decide whether they reject unknown Extension types, accept as opaque data, or transfer to another application without touching payload of them.

I was thinking we could pass some informations about custom types to the (un)packer (and to/from_msgpack) ?

Note: I haven't put much thoughts about it, and I don't really know how the current msgpack implementation works

not compiled with crystal 0.17

Failed giving version

Hello,

I use msgpack to communicate between my apis.

I have added

  msgpack:
    github: benoist/msgpack-crystal
    version: ~> 0.3

into my shards.yml but it lead me to an error

Error resolving msgpack (~> 0.3)

Is there a way to precise version (e.g : not to use master) ?
Regards,

Shall not include Time conversion by default

It's even stated in the docs:

msgpack-crystal/src/message_pack/from_msgpack.cr

Lines 202 to 204 in 4c123df

 # The MsgPack format itself does not specify a time data type, this method just 

 # assumes that a string holding a ISO 8601 time format can be # interpreted as a 

 # time value.

I suggest that proper implementation does not "assume" anything. There should be a converter for anything which is out of MessagePack specification scope.

A Time can be transferred either as ISO 8601 string or epoch seconds or even epoch milliseconds. There should not be a default constructor for Time IMO.

	# The MsgPack format itself does not specify a time data type, this method just
	# assumes that a string holding a ISO 8601 time format can be # interpreted as a
	# time value.

crystal-community / msgpack-crystal Goto Github PK

msgpack-crystal's People

Contributors

Stargazers

Watchers

Forkers

msgpack-crystal's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs