crystal-community / msgpack-crystal Goto Github PK
View Code? Open in Web Editor NEWMessagePack implementation in Crystal msgpack.org[Crystal]
MessagePack implementation in Crystal msgpack.org[Crystal]
presense
-> presence
I'm not sure, but would it be a good idea to have it so including Serializable also overwrites inspect(io : IO)
to include the properties?
The upside, is as a client instead of getting something like this
server
record Details, name : String, count : Int32 do
include MessagePack::Serializable
end
def add_thing(id : String, details : Details) : Nil
end
client
c.add_thing "hi", {name: "will"}
Traceback (most recent call last):
6: from ./repl:32:in `<main>'
5: from (irb):2
4: from (irb):2:in `rescue in irb_binding'
3: from ./repl:12:in `method_missing'
2: from /Users/will/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/msgpack-rpc-0.6.0/lib/msgpack/rpc/session.rb:54:in `call'
1: from /Users/will/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/msgpack-rpc-0.6.0/lib/msgpack/rpc/future.rb:54:in `get'
MessagePack::RPC::RuntimeError (bad arguments, expected [id : String, details : Details], but got details: HashT[1])
the error could maybe be nicer to better see what you were missing
MessagePack::RPC::RuntimeError (bad arguments, expected [id : String, details : Details[name : String, count : Int32], but got details: HashT[1])
There might be downsides though, but I'm not aware of what they would be, so I wanted to check first if this would be good.
I tried doing a PR to show this, but I can't figure out the macros :( in the macro included
section, @type.instance_variables
seems to be empty, and I donβt have a ton of experience with macros
In the standard library, JSON
and YAML
have defined a pattern for switching on the deserialized types using a "discriminator" field. For example, in this class:
require "json"
abstract struct Message
include JSON::Serializable
use_json_discriminator "type", {
created: Created,
updated: Updated,
}
getter id : String
end
struct Created < Message
end
struct Updated < Message
end
The use_json_discriminator
macro sets type: "created"
in the JSON to deserialize to the Created
type and type: "updated"
to Updated
:
Message.from_json({type:"created",id:"a"}.to_json)
# => Created(@id="a")
Message.from_json({type:"updated",id:"a"}.to_json)
# => Updated(@id="a")
It'd be awesome to get that in MessagePack, as well. The implementation for use_json_discriminator
is here.
Hi, does the project provide any feature like missing or optional property in a class?
instantiating 'MessagePack::Lexer#next_token(Tuple(), NamedTuple())'
in lib/msgpack/src/message_pack/lexer.cr:96: instantiating 'prefetch_token()'
token = prefetch_token
^~~~~~~~~~~~~~
in lib/msgpack/src/message_pack/lexer.cr:51: instantiating 'read(UInt16:Class)'
consume_binary(read UInt16)
^~~~
in lib/msgpack/src/message_pack/lexer.cr:154: undefined constant T
private def read(type : T.class)
is it possible to write like this? to save us from create temp hashes, arrays of Type.
packer.write_hash_start
packer.write_hash_key("bla")
packer.write_hash_value(123)
packer.write_hash_key("bla2")
packer.write_hash_value(true)
packer.write_hash_end
packer.write_array_start
packer.write_array_element("bla")
packer.write_array_element(123)
packer.write_array_element("bla2")
packer.write_array_element(true)
packer.write_array_end
data = ({aa: "a", bb: nil}).to_msgpack
typeof(NamedTuple(aa: String, bb: Nil).from_msgpack(data)[:aa]) # => (String | Nil) but should be just `String`
And this breaks my compiler type-check.
Originally posted by @xqyww123 in https://github.com/crystal-community/msgpack-crystal/issue_comments#issuecomment-441992829
It's even stated in the docs:
msgpack-crystal/src/message_pack/from_msgpack.cr
Lines 202 to 204 in 4c123df
I suggest that proper implementation does not "assume" anything. There should be a converter for anything which is out of MessagePack specification scope.
A Time
can be transferred either as ISO 8601 string or epoch seconds or even epoch milliseconds. There should not be a default constructor for Time
IMO.
Hello,
I use msgpack
to communicate between my apis.
I have added
msgpack:
github: benoist/msgpack-crystal
version: ~> 0.3
into my shards.yml
but it lead me to an error
Error resolving msgpack (~> 0.3)
Is there a way to precise version (e.g : not to use master) ?
Regards,
https://github.com/msgpack/msgpack/blob/master/spec.md#formats-timestamp
I suggest emitting the ext timestamp format when serializing UTC Time
rather than a custom format potentially solving issues with #58.
Detailed here: https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types
And the binary format here: https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family
MessagePack allows applications to define application-specific types using the Extension type.
Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data.Applications can assign
0
to127
to store application-specific type information. An example usage is that application definestype = 0
as the application's unique type system, and stores name of a type and values of the type at the payload.MessagePack reserves
-1
to-128
for future extension to add predefined types. These types will be added to exchange more types without using pre-shared statically-typed schema across different programming environments.[0, 127]: application-specific types [-128, -1]: reserved for predefined types
Because extension types are intended to be added, old applications may not implement all of them. However, they can still handle such type as one of Extension types. Therefore, applications can decide whether they reject unknown Extension types, accept as opaque data, or transfer to another application without touching payload of them.
I was thinking we could pass some informations about custom types to the (un)packer (and to/from_msgpack) ?
Note: I haven't put much thoughts about it, and I don't really know how the current msgpack implementation works
ruby -e 'require "msgpack"; p [1,2,3].to_msgpack.bytes'
[147, 1, 2, 3]
crystal eval 'require "./src/msgpack"; p [1,2,3].to_msgpack'
[147, 210, 0, 0, 0, 1, 210, 0, 0, 0, 2, 210, 0, 0, 0, 3]
much bigger.
Hello,
I want to write in a MessagePack::Packer.new
a bunch of hashes
to do this I use
array_of_hash.map {|h| packer.write(h)
but I have this error
Error in ./src/zaqar.cr:15: no overload matches 'MessagePack::Packer#write' with type Hash(String, Nil | String | Float32 | Float64 | Time | JSON::Any | Bool | Int32 | Int64 | Int16 | Slice(UInt8))
Overloads are:
- MessagePack::Packer#write(value : Nil)
- MessagePack::Packer#write(value : Bool)
- MessagePack::Packer#write(value : String)
- MessagePack::Packer#write(value : Symbol)
- MessagePack::Packer#write(value : Float32 | Float64)
- MessagePack::Packer#write(value : Int8 | Int16 | Int32 | Int64 | UInt8 | UInt16 | UInt32 | UInt64)
- MessagePack::Packer#write(value : Hash(Type, Type))
- MessagePack::Packer#write(value : Array(Type))
- MessagePack::Packer#write(value : Tuple)
However
array_of_hash.map {|h| packer.write(h : Hash(Type, Type)) }
is not yet supported by crystal
DO you know when this will be available ?
Regards,
class A
MessagePack.mapping({
bla: {type: String, nilable: true}
})
end
p A.from_msgpack({"bla" => nil}.to_msgpack) # => #<A:0x100e2af20 @bla=nil>
class B
MessagePack.mapping({
bla: String?
})
end
p B.from_msgpack({"bla" => nil}.to_msgpack) # => Nil assertion failed (Exception)
any example on decode, encode crystal objects
example failed code
TCPServer.open("localhost", 5000) do |server|
TCPSocket.open("localhost", server.local_address.port) do |client|
sock = server.accept
{1, 2, 3.5, 4, 5}.to_msgpack(client)
1.to_msgpack(client)
pull = MessagePack::Unpacker.new(sock)
expect_raises(MessagePack::Error) do
Array(Int32).new(pull)
end
Int32.new(pull).should eq 1
end
end
it got 4, instead of 1
What we should do in this case, i think we should skip all values for every container types, after exception. @benoist what you think?
Consider the following class:
class Foo
def initialize(@name = "")
end
MessagePack.mapping({
name: String,
attributes: {type: MessagePack::Table}
})
end
Attempting to decode the following results in an error:
{
"name" => "bar",
"attributes" => {
"foo" => "bar"
}
}.to_msgpack
in ./libs/msgpack/message_pack/from_msgpack.cr:114: undefined method 'new' for MessagePack::Type:Class
hash[k] = V.new(pull)
So what I'm trying to figure out is if I don't know the structure of the data that I might be recieving how do I decode it from message pack?
Thanks!
data = ({aa: "\0\1\2", bb: nil}).to_msgpack # =>
# Bytes[130, 162, 97, 97, 163, 0, 1, 2, 162, 98, 98, 192]
# The serialized data is correct.
NamedTuple(aa: String, bb: String?).from_msgpack data # =>
# Unhandled exception: Missing msgpack attribute: bb at 0 (MessagePack::UnpackException)
# from lib/msgpack/src/message_pack/from_msgpack.cr:218:3 in 'new'
# from lib/msgpack/src/message_pack/from_msgpack.cr:5:3 in 'from_msgpack'
Tarantool send response as array of array (not hash), but I would like to mapping this response.
How can I get something like this?
class MessagePackPersonArray
include MessagePack::Serializable
include MessagePack::Serializable::Unmapped
@[MessagePack::Field(key: 0)]
property name : String
@[MessagePack::Field(key: 1)]
property age : Int32?
end
person = MessagePackPersonArray.from_msgpack(["John", 30, "1", 2, [1, 2, 3]].to_msgpack)
person.name.should eq("John")
person.age.should eq(30)
https://github.com/vladfaust/tarantool-crystal (use msgpack-crystal)
If reading from an IO, say a socket, that is not closed the lexer will block on next_byte if there is no more data currently available. F.ex if one is sending a stream of integers.
Here is a snippet from next_token
when 0xE0..0xFF
@token.type = :INT
@token.int_value = current_byte.to_i8
next_byte
If that int is the last byte available next_byte blocks and the just read token is not returned.
I think it is also a problem that next_byte is called from the lexer constructor. This means that constructing an Unpacker can actually block if there is no data available.
Can we add support to pack and unpack
Hash(String, Hash(Symbol, (Int32 | String | MyClass)))
etc..
require "msgpack"
struct Foo
include MessagePack::Serializable
@[MessagePack::Field(presence: true)]
getter foo : Int32
end
foo = Foo.from_msgpack({foo: 42}.to_msgpack)
Can't infer the type of instance variable '@foo_present' of Foo
I understand that I could add one more getter to Foo
, but I don't want to pollute it. Doubling the amount of getters is too much for a simple object.
@[MessagePack::Field(ignore: true)]
getter? foo_present : Bool
Reduced problem (https://carc.in/#/r/6qks):
class Foo
def initialize
foo = true
@bar = foo
end
end
Foo.new
Can't infer the type of instance variable '@bar' of Foo
It can be fixed with @bar = uninitialized Bool
(https://carc.in/#/r/6qkt):
class Foo
def initialize
+ @bar = uninitialized Bool
foo = true
@bar = foo
end
end
Foo.new
However, the same approach doesn't work with Serializable#initialize
π’
I use this data structure:
class MyData
MessagePack.mapping({
# [{'id' => {:payload => Slice(UInt8), :alive => Bool}}]
array: Array(Hash(String, Hash(Symbol, (Slice(UInt8) | Bool)))),
})
end
When I tried to use it, my compiler crashed with:
in lib/msgpack/src/message_pack/from_msgpack.cr:5: instantiating 'new(MessagePack::Unpacker)'
new parser
^~~
in src/msg_pack_types.cr:4: expanding macro
MessagePack.mapping({
^
in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 23:
1.
2.
3.
4.
5.
6. @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))
7.
8. def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
9. @array = _array
10. end
11.
12. def array
13. @array
14. end
15.
16.
17. def initialize(__temp_23 : MessagePack::Unpacker)
18.
19. __temp_24 = nil
20. __temp_25 = false
21.
22.
> 23. __temp_23.read_hash(false) do
24. case __temp_26 = Bytes.new(__temp_23)
25.
26. when "array".to_slice
27. __temp_25 = true
28. __temp_24 =
29.
30.
31.
32. Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
33.
34.
35.
36.
37. else
38.
39. __temp_23.skip_value
40.
41. end
42. end
43.
44.
45.
46. if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
47. raise MessagePack::UnpackException.new("missing msgpack attribute: array")
48. end
49.
50.
51.
52.
53.
54. @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
55.
56.
57. end
58.
59. def to_msgpack(packer : MessagePack::Packer)
60. packer.write_hash_start(1)
61.
62. _array = @array
63. packer.write("array")
64.
65. _array.to_msgpack(packer)
66.
67.
68. end
69.
instantiating 'MessagePack::Unpacker#read_hash(Bool)'
in /msg_pack_types.cr:4: expanding macro
MessagePack.mapping({
^
in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 23:
1.
2.
3.
4.
5.
6. @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))
7.
8. def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
9. @array = _array
10. end
11.
12. def array
13. @array
14. end
15.
16.
17. def initialize(__temp_23 : MessagePack::Unpacker)
18.
19. __temp_24 = nil
20. __temp_25 = false
21.
22.
> 23. __temp_23.read_hash(false) do
24. case __temp_26 = Bytes.new(__temp_23)
25.
26. when "array".to_slice
27. __temp_25 = true
28. __temp_24 =
29.
30.
31.
32. Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
33.
34.
35.
36.
37. else
38.
39. __temp_23.skip_value
40.
41. end
42. end
43.
44.
45.
46. if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
47. raise MessagePack::UnpackException.new("missing msgpack attribute: array")
48. end
49.
50.
51.
52.
53.
54. @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
55.
56.
57. end
58.
59. def to_msgpack(packer : MessagePack::Packer)
60. packer.write_hash_start(1)
61.
62. _array = @array
63. packer.write("array")
64.
65. _array.to_msgpack(packer)
66.
67.
68. end
69.
instantiating 'MessagePack::Unpacker#read_hash(Bool)'
in msg_pack_types.cr:4: expanding macro
MessagePack.mapping({
^
in macro 'mapping' /lib/msgpack/src/message_pack/mapping.cr:58, line 32:
1.
2.
3.
4.
5.
6. @array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))
7.
8. def array=(_array : Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))) )
9. @array = _array
10. end
11.
12. def array
13. @array
14. end
15.
16.
17. def initialize(__temp_23 : MessagePack::Unpacker)
18.
19. __temp_24 = nil
20. __temp_25 = false
21.
22.
23. __temp_23.read_hash(false) do
24. case __temp_26 = Bytes.new(__temp_23)
25.
26. when "array".to_slice
27. __temp_25 = true
28. __temp_24 =
29.
30.
31.
> 32. Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))).new(__temp_23)
33.
34.
35.
36.
37. else
38.
39. __temp_23.skip_value
40.
41. end
42. end
43.
44.
45.
46. if __temp_24.is_a?(Nil) && !__temp_25 && !Union(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool)))).nilable?
47. raise MessagePack::UnpackException.new("missing msgpack attribute: array")
48. end
49.
50.
51.
52.
53.
54. @array = __temp_24.as(Array(Hash(String, Hash(Symbol, Slice(UInt8) | Bool))))
55.
56.
57. end
58.
59. def to_msgpack(packer : MessagePack::Packer)
60. packer.write_hash_start(1)
61.
62. _array = @array
63. packer.write("array")
64.
65. _array.to_msgpack(packer)
66.
67.
68. end
69.
instantiating 'Array(Hash(String, Hash(Symbol, Bool | Slice(UInt8)))):Class#new(MessagePack::Unpacker)'
in lib/msgpack/src/message_pack/from_msgpack.cr:86: instantiating 'new(MessagePack::Unpacker)'
new(pull) do |element|
^~~
in lib/msgpack/src/message_pack/from_msgpack.cr:93: instantiating 'MessagePack::Unpacker#read_array()'
pull.read_array do
^~~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:93: instantiating 'MessagePack::Unpacker#read_array()'
pull.read_array do
^~~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:94: instantiating 'Hash(String, Hash(Symbol, Bool | Slice(UInt8))):Class#new(MessagePack::Unpacker)'
yield T.new(pull)
^~~
in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'
pull.read_hash(false) do
^~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'
pull.read_hash(false) do
^~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:114: instantiating 'Hash(Symbol, Bool | Slice(UInt8)):Class#new(MessagePack::Unpacker)'
hash[k] = V.new(pull)
^~~
in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'
pull.read_hash(false) do
^~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:108: instantiating 'MessagePack::Unpacker#read_hash(Bool)'
pull.read_hash(false) do
^~~~~~~~~
in lib/msgpack/src/message_pack/from_msgpack.cr:109: undefined method 'new' for Symbol:Class
k = K.new(pull)
^~~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.