[Support request] Serializing and Deserializing between different libraries (javascript msgpackr) about messagepack-csharp

[ividyon]

This seems like a great solution. Funnily enough, I did cobble together something the way you described in the second method, in the dead of night:

public class PacketFormatter : IMessagePackFormatter<IPacket>
{
    public void Serialize(ref MessagePackWriter writer, IPacket value, MessagePackSerializerOptions options)
    {
        throw new NotImplementedException();
    }

    public IPacket Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
    {
        var peek = reader.CreatePeekReader();

        while (peek.NextMessagePackType != MessagePackType.Extension && !peek.End)
        {
            if (peek.NextMessagePackType == MessagePackType.Map) peek.ReadMapHeader();
            else peek.ReadRaw();
        }

        if (peek.NextMessagePackType != MessagePackType.Extension) throw new Exception("No opcode found");

        var ext = peek.ReadExtensionFormat();
        var type = (PacketType)ext.Data.ToArray().First();
        switch (type)
        {
            case PacketType.HelloWorld:
                return options.Resolver.GetFormatterWithVerify<HelloWorldPacket>().Deserialize(ref reader, options);
            default:
                throw new ArgumentOutOfRangeException();
        }
    }
}

public class PacketTypeFormatter : IMessagePackFormatter<PacketType>
{
    public void Serialize(ref MessagePackWriter writer, PacketType value, MessagePackSerializerOptions options)
    {
        writer.WriteExtensionFormat(new ExtensionResult(1, new ReadOnlySequence<byte>([ (byte)value ])));
    }

    public PacketType Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
    {
        var ext = reader.ReadExtensionFormat();
        return (PacketType)ext.Data.ToArray().First();
        // return Enum.Parse<PacketType>(BitConverter.ToInt32(new byte[][ 0 ]));
        throw new NotImplementedException();
    }
}

[MessagePackFormatter(typeof(PacketTypeFormatter))]
public enum PacketType
{
    HelloWorld = 0,
}

[MessagePackFormatter(typeof(PacketFormatter))]
public interface IPacket
{
    [JsonProperty("o")]
    [Key("o")]
    public PacketType Type { get; }
}

But now that you described what Union does, it seems very clean and elegant. I suggest adding this "faux json" snippet to the README, as it helped me understand the concept much better.

I've rewritten the JS side to use the Union format.

My last issue was that the objects didn't re-serialize into this format, but that appears to be fixed by using Serialize<IPacket>(packet) rather than the non-generic Serialize.

Thank you very much!

[ividyon]

So, I found the Union part of the README and that seems to address my question.

But now I'm finding that the exact same object (JSON: {"o":0,"hello":"nyellorgh","world":123456}) serializes into different byte arrays between the two libraries I'm using.

MessagePack-CSharp:

byte[31]
[0] = {byte} 131
[1] = {byte} 161
[2] = {byte} 111
[3] = {byte} 0
[4] = {byte} 165
[5] = {byte} 104
[6] = {byte} 101
[7] = {byte} 108
[8] = {byte} 108
[9] = {byte} 111
[10] = {byte} 169
[11] = {byte} 110
[12] = {byte} 121
[13] = {byte} 101
[14] = {byte} 108
[15] = {byte} 108
[16] = {byte} 111
[17] = {byte} 114
[18] = {byte} 103
[19] = {byte} 104
[20] = {byte} 165
[21] = {byte} 119
[22] = {byte} 111
[23] = {byte} 114
[24] = {byte} 108
[25] = {byte} 100
[26] = {byte} 206
[27] = {byte} 0
[28] = {byte} 1
[29] = {byte} 226
[30] = {byte} 64

JavaScript msgpackr:

{
    "0": 222,
    "1": 0,
    "2": 3,
    "3": 161,
    "4": 111,
    "5": 0,
    "6": 165,
    "7": 104,
    "8": 101,
    "9": 108,
    "10": 108,
    "11": 111,
    "12": 169,
    "13": 110,
    "14": 121,
    "15": 101,
    "16": 108,
    "17": 108,
    "18": 111,
    "19": 114,
    "20": 103,
    "21": 104,
    "22": 165,
    "23": 119,
    "24": 111,
    "25": 114,
    "26": 108,
    "27": 100,
    "28": 206,
    "29": 0,
    "30": 1,
    "31": 226,
    "32": 64
}

The difference seems to be that the first byte 131 in the CSharp output becomes three bytes 222,0,3 in msgpackr instead. The CSharp library cannot parse this. What could it be?

All I found is that 222 is 0xDE, which seems to stand for map 16 in MessagePack, and 131 is 0x83 which seems to be part of fixmap. Unfortunately I don't really know what to do with this information.

[ividyon]

I managed to resolve the above by toggling the "variable map size" option in the JavaScript library msgpackr. The byte array outputs are now identical.

However, now I can confirm that the way I use the Union attribute is apparently not working; I can deserialize into HelloWorldPacket fine, but not into the base class Packet.

My goal is to be able to parse any incoming serialized Packet subclass at once.

[Union(0, typeof(HelloWorldPacket))]
public abstract class Packet
{
    [JsonProperty("o")]
    [Key("o")]
    public abstract PacketType Type { get; }
}

[MessagePackObject]
public class HelloWorldPacket : Packet
{
    public override PacketType Type => PacketType.HelloWorld;

    [JsonProperty("hello")]
    [Key("hello")]
    public required string Hello { get; set; }

    [JsonProperty("world")]
    [Key("world")]
    public required int World { get; set; }
}

MessagePack.MessagePackSerializationException: Failed to deserialize REI05.Packets.Packet value.
 ---> MessagePack.MessagePackSerializationException: Unexpected msgpack code 131 (fixmap) encountered.
   at MessagePack.MessagePackReader.ThrowInvalidCode(Byte code)
   at MessagePack.MessagePackReader.ReadArrayHeader()
   at MessagePack.Formatters.REI05_Packets_PacketFormatter1.Deserialize(MessagePackReader&, MessagePackSerializerOptions)
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   --- End of inner exception stack trace ---
   at MessagePack.MessagePackSerializer.Deserialize[T](MessagePackReader& reader, MessagePackSerializerOptions options)
   at MessagePack.MessagePackSerializer.Deserialize[T](ReadOnlyMemory`1 buffer, MessagePackSerializerOptions options, CancellationToken cancellationToken)
   at REI05.Server.Session.Receive() in G:\Creative\Indie\REI05\REI05\Server\Session.cs:line 85

[AArnott]

[Union] in MessagePack-CSharp doesn't work the way you're defining the schema in typescript. [Union] is applied to the base type (as you're doing) and replaces the need for a field inside the type that identifies it. When you serialize or deserialize as the abstract base type, Union lights up and adds a new array around the runtime object:

[0, {"hello": "andrew", "world": 3}]

Note the array around the map. The array is always 2 elements long and the first element contains the union constant that identifies the type and the second element is the object itself.
You can probably imagine how you could reconstruct this on the typescript side by changing how your types are defined, if you want to go that route.

But if you want to go the other route, of leaving TypeScript exactly as it is and fixing the C# side to match, Union isn't going to cut it. You'll need to write your own custom IMessagePackFormatter<Packet> class that can read the o field from the map and then pass deserialization off to the more derived type's formatter given what it learned by reading o. In this way, you won't have to try deserializing multiple times, and no exceptions will be thrown in the happy path.
I could write a sample for you for how to make C# behave like your existing typescript code in about an hour, at the rates described here: https://blog.nerdbank.net/for-hire