GithubHelp home page GithubHelp logo

wire's Introduction

[Archived]

Due to how Wire handles type information on the wire, malicious payloads can be passed. e.g. using a surrogate on the sender end, an attacker can pass information about a different type for the receiving end. And by doing so allowing the serializer to create any type on the deserializing end.

This is the same issue that exists for BinaryFormatter https://docs.microsoft.com/en-us/visualstudio/code-quality/ca2300?view=vs-2019

Any future forks or derivates of Wire will have to account for this.

Security advisory: https://github.com/AsynkronIT/Wire/security/advisories/GHSA-hpw7-3vq3-mmv6

Wire

A high performance polymorphic serializer for the .NET framework.

Wire is still in beta and may have breaking changes to both the API and serialization format on the road to 1.0

Polymorphic serializations

Wire was designed to safely transfer messages in distributed systems, for example service bus or actor model based systems. In message based systems, it is common to receive different types of messages and apply pattern matching over those messages. If the messages does not carry over all the relevant type information to the receiveing side, the message might no longer match exactly what your system expect.

Consilder the following case:

public class Envelope
{
     //untyped property
     public object Payload {get;set;}
     //other properties..
     ...
}

...

var envelope = new Envelope { Payload = (float)1.2 };

If you for example are using a Json based serializer, it is very likely that the value 1.2 will be deserialized as a double as Json has no way to describe the type of the decimal value. While if you use some sort of binary serializer like Google Protobuf, all messages needs to be designed with a strict contract up front. Wire solves this by encoding a manifest for each value - a single byte prefix for primitive values, and fully qualified assembly names for complex types.

Surrogates

Sometimes, you might have objects that simply can't be serialized in a safe way, the object might be contextual in some way. Wire can solve those problems using "Surrogates", surrogates are a way to translate an object from and back to the context bound representation.

var surrogate = Surrogate.Create<IMyContextualInterface,IMySurrogate>(original => original.ToSurrogate(), surrogate => surrogate.Restore(someContext));
var options = new SerializerOptions(surrogates: new [] { surrogate });
var serializer = new Serializer(options);

This is essential for frameworks like Akka.NET where we need to be able to resolve live Actor References in the deserializing system.

Version Tolerance

Wire has been designed to work in multiple modes in terms of version tolerance vs. performance.

  1. Pre Register Types, when using "Pre registered types", Wire will only emit a type ID in the output stream. This results in the best performance, but is also fragile if different clients have different versions of the contract types.
  2. Non Versioned, this is largely the same as the above, but the serializer does not need to know about your types up front. it will embed the fully qualified typename in the outputstream. this results in a larger payload and some performance overhead.
  3. Versioned, in this mode, Wire will emit both type names and field information in the output stream. This allows systems to have slightly different versions of the contract types where some fields may have been added or removed.

Durable persistence and version tolerance

Wire has been designed as a wire format, point to point for soft realtime scenarios. If you need a format that is durable for persistence over time. e.g. EventSourcing or for message queues, then Protobuf, Thrift, Flatbuffers or MS Bond will be a better choise as those formats have been designed for true versiom tolerance.

Performance

Wire has been designed with a performance first mindset. It is not the most important aspect of Wire, Surrogates and polymorphism is more critical for what we want to solve. But even with it's rich featureset, Wire performs extremely well.

Wire - preregister types
   Serialize                      312 ms
   Deserialize                    261 ms
   Size                           38 bytes
   Total                          573 ms
Wire - no version data
   Serialize                      327 ms
   Deserialize                    354 ms
   Size                           73 bytes
   Total                          681 ms
Wire - preserve object refs
   Serialize                      400 ms
   Deserialize                    369 ms
   Size                           73 bytes
   Total                          769 ms
MS Bond
   Serialize                      429 ms
   Deserialize                    404 ms
   Size                           50 bytes
   Total                          833 ms
Wire - version tolerant
   Serialize                      423 ms
   Deserialize                    674 ms
   Size                           195 bytes
   Total                          1097 ms
Protobuf.NET
   Serialize                      638 ms
   Deserialize                    721 ms
   Size                           42 bytes
   Total                          1359 ms
Jil
   Serialize                      1448 ms
   Deserialize                    714 ms
   Size                           123 bytes
   Total                          2162 ms
Net Serializer
   Serialize                      1289 ms
   Deserialize                    1113 ms
   Size                           39 bytes
   Total                          2402 ms
Json.NET
   Serialize                      3767 ms
   Deserialize                    5936 ms
   Size                           187 bytes
   Total                          9703 ms
Binary formatter
   Serialize                      10784 ms
   Deserialize                    11374 ms
   Size                           362 bytes
   Total                          22158 ms

This test was run using the following object definition:

public class Poco
{
    public string StringProp { get; set; }      //using the text "hello"
    public int IntProp { get; set; }            //123
    public Guid GuidProp { get; set; }          //Guid.NewGuid()
    public DateTime DateProp { get; set; }      //DateTime.Now
}

Big disclaimer: The above results change drastically depending on your contracts, e.g. using smaller messages favor both NetSerializer and Jil. There is no "best" or "fastest" serializer, it all depends on context and requirements.

wire's People

Contributors

bariloce avatar cpx86 avatar danthar avatar eutech avatar gitter-badger avatar horusiath avatar itadapter avatar jaben avatar lukasz-pyrzyk avatar mattwarren avatar neuecc avatar rogeralsing avatar scooletz avatar stuarthillary avatar tornhoof avatar veblush avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wire's Issues

Improve exception message when type cannot be loaded on deserialization

By mistake I recently tried to deserialize a type which was unknown. The deserializer throws an exception because it could not load the ForeignAssembly 1.0.0.0 or one of its dependencies. It would be very good to known in the cases when a type cannot be loaded/created, which type that has been. Switching back to the JSON serializer I got an exception message containing the name (and assembly) of the type trying to deserialize. That is very helpful and Wire should do the same.

Version tolerance rquirements

The current plan for version tolerance is to support added and removed fields.
If there is a type missing, there is nothing to generate the reader from, and thus we cannot support this today.

User feedback is welcome

ObjectSerializer SpinWaits forever when BuildSerializer fails

I've come across a situation where the BuildSerializer method on some unknown or odd type causes a failure in the expression compiler. This causes the serializer to remain in the _serializers dictionary but without being initialized. This by itself is not an issue.

However the problem comes if you later try to serialize the same type - instead of getting an error, the serializer essentially spinwaits forever, waiting indefinitely for _isInitialized to be set to true, which will never happen.

Several solutions:

  1. Add the object to _serializers only after its successfully initialized. Downside is that concurrent calls will end up creating wasted extra serializers.
  2. Add a catch block and remove the serializer from the dictionary if it couldn't be initialized. Also set some flag in the serializer so if any concurrent code is accessing it before it could get removed, the spinwait can check for this error flag and raise an exception - Requires changes in every place where the dictionary gets a serializer added to it - need to check for exceptions and remove the item.
  3. Just add a timeout to the spinwait - easiest solution. If it spins for more than a second, raise an error.

I was able to reproduce this scenario by accidentally trying to serialize a BsonDocument (from MongoDB). I think its ok to throw an error on unknown types, but it shouldn't spinwait indefinitely.

I'll be happy to submit a pull request for any of these three methods - would like some feedback on which is the preferred approach?

Wire problem with Generic Option from LanguageExt

Hi,
First of all, I must say that I am new to the Wire topic, maybe because of documentation lack I can not find the answer to my question.

Im using Akka with Wire Serialization, in one of the persisted messages I'm using Option from Funcional Language Extensions library. Contrary to Newtonsoft Json, Wire has a problem with serialization. How I can affect the serialization by Wire. Can I use a custom attribute, constructor ?

Object type 'LanguageExt.Option`1[System.String]' can not convert to type 'System.String'.

Thanks for help!

Support type aliases

When I use Newtonsoft JSON.NET, I usually end up with a custom binder that allows me to configure short type names for the types I know will be serialized frequently.

This results in reduced overhead, as e.g. a simple Date value type otherwise serialized as

{
    "$type": "SomeProject.SomeComponent.SomeNamespace.Values.Date, SomeProject.SomeComponent",
    "Day": 3,
    "Month": 3,
    "Year": 2016
}

can become something like this:

{
    "$type": "*Date",
    "Day": 3,
    "Month": 3,
    "Year": 2016
}

If I understand correctly, Wire would serialize my Date type like this:

SomeProject.SomeComponent.SomeNamespace.Values.Date, SomeProject.SomeComponent
08 # int
0003
08 # int
0003
08 # int
07E0

which - if I could configure a type alias called *Date would be much more compact:

*Date
08 # int
0003
08 # int
0003
08 # int
07E0

How do you like this idea?

Support custom layout

Similar to how #19 can be used to reduce the overhead of frequently occurring values with very little actual data, another mechanism can reduce the overhead of value types even more.

The example given in #19 is actually a little bit artificial, since I usually end up adding a custom converter for my (usually) tiny value types.

With a custom converter, the data otherwise serialized as

{
    "$type": "SomeProject.SomeComponent.SomeNamespace.Values.Date, SomeProject.SomeComponent",
    "Day": 3,
    "Month": 3,
    "Year": 2016
}

is serialized as

"2016/3/3"

(which can be done because JSON.NET can infer the type Date and the custom converter has all the necessary serialization roundtripping code).

Could it be an idea for Wire to support something similar, so that the data otherwise serialized as

SomeProject.SomeComponent.SomeNamespace.Values.Date, SomeProject.SomeComponent
08 # int
0003
08 # int
0003
08 # int
07E0

could be reduced to

*Date
07 # string
??? # some length stuff
20160303 # the actual string

?

Problem with deserializing byte arrays

The following is a simple LINQPad example serializing / deserializing a byte array using version 0.0.6:

var msg = new byte[] { 1, 2, 3, 4 };
var serializer = new Serializer(new SerializerOptions(versionTolerance : true, preserveObjectReferences : true));

byte[] serialized;
using (var ms = new MemoryStream())
{
    serializer.Serialize(msg, ms);
    serialized = ms.ToArray();
}

byte[] deserialized;
using (var ms = new MemoryStream(serialized))
{
    deserialized = serializer.Deserialize<byte[]>(ms);
}

msg.SequenceEqual(deserialized).Dump("SequenceEqual");
(msg == deserialized).Dump("msg == deserialized");
msg.Equals(deserialized).Dump("msg.Equals(deserialized)");

msg.Dump("msg");
deserialized.Dump("deserialized");

Which results in the following:

capture

Serialization of F# Map

Wire appears to have a problem to serialize a F# Record that contains a Map. For example:

F#

type RecordWithString = {Name:string}
type RecordWithMap = {SomeMap: Map<int,string>}
let createRecordWithMap = {SomeMap = Map.empty }

C#

var s = new Wire.Serializer();

var mem = new MemoryStream();
var r1 = new RecordWithString("somestring");
s.Serialize(r1, mem); //this works!

mem = new MemoryStream();
var r2 = SomeBank.Accounts.Domain.createRecordWithMap;
s.Serialize(r2, mem); //this throws the exception below

This is the exception message.

An exception of type 'System.Exception' occurred in Wire.dll but was not handled in user code
Additional information: Unable to write all fields of RecordWithMap

Performance results

Here are the latest benchmarks using the current Dev branch bits.
I've made some serious progress on improving the KnownType scenario but also some nice fixes to the standard path.

These are from our own test suite.

This does not give a complete picture.
e.g. NetSerializer have completely different tests which focus on primitives and structs| in which NetSerializer is mostly faster. (but less so with the current Dev code)

TypicalMessageArray

test roundtrip serialize deserialize size
Wire - Default 569 286 283 3911
Wire - KnownTypes 573 286 286 3911
Wire - Object Identity 672 373 298 3911
Wire - Version Tolerant 759 413 346 4430
Net Serializer 769 377 392 3864
Jil 1303 508 794 0
NFX Slim Serializer - KnownTypes 1443 814 628 4162
NFX Slim Serializer 1564 891 672 4479
Protobuf.NET 1867 933 933 4400
NET-JSON 3345 733 2611 12801
ServiceStack.Text 6601 2054 4546 11001
Json.NET 6760 2840 3919 18875
Binary formatter 12208 6362 5846 5981

TypicalMessage

test roundtrip serialize deserialize size
Wire - KnownTypes 632 365 267 38
Net Serializer 818 425 392 39
Wire - Default 854 373 480 89
Wire - Object Identity 957 457 500 89
Wire - Version Tolerant 1359 497 861 211
NFX Slim Serializer - KnownTypes 1397 650 747 47
Jil 1460 676 783 0
Protobuf.NET 1663 775 888 42
NET-JSON 3193 745 2448 127
NFX Slim Serializer 5368 2624 2743 152
ServiceStack.Text 5930 2038 3891 110
Json.NET 7400 2835 4565 188
Binary formatter 22518 11069 11449 378

LargeStruct

test roundtrip serialize deserialize size
Net Serializer 435 250 184 23
Wire - KnownTypes 469 282 187 35
Wire - Default 682 320 361 83
Wire - Object Identity 808 430 377 83
Protobuf.NET 935 376 558 24
NFX Slim Serializer - KnownTypes 1206 522 683 33
Wire - Version Tolerant 1232 452 779 128
Json.NET 2410 1271 1139 58
NFX Slim Serializer 4750 2282 2467 135
Binary formatter 7694 3879 3815 205

GuidRoundrip

test roundtrip serialize deserialize size
Wire - KnownTypes 301 196 104 17
Wire - Default 301 195 106 17
Wire - Version Tolerant 325 191 134 17
Wire - Object Identity 332 210 122 17
Net Serializer 362 252 110 20
Jil 510 251 258 0
NFX Slim Serializer - KnownTypes 764 403 361 29
NET-JSON 865 188 676 38
ServiceStack.Text 966 211 755 32
Json.NET 1147 544 603 38
Protobuf.NET 2449 1286 1163 20
NFX Slim Serializer 4753 2405 2347 118
Binary formatter 14272 7036 7236 110

GuidArrayRoundtrip

test roundtrip serialize deserialize size
Wire - Default 466 254 212 1739
Wire - KnownTypes 473 254 219 1739
Wire - Version Tolerant 490 248 241 1740
Wire - Object Identity 612 364 247 1739
Net Serializer 616 367 249 1875
Jil 1209 373 836 0
NFX Slim Serializer - KnownTypes 1439 783 655 1820
NFX Slim Serializer 1480 808 672 2000
Protobuf.NET 2611 1469 1142 2000
Json.NET 4885 1396 3488 3947
ServiceStack.Text 5052 1050 4001 3301
Binary formatter 28169 14715 13453 2612

TypicalPersonRoundtrip

test roundtrip serialize deserialize size
Protobuf.NET 370 146 224 235
Wire - KnownTypes 389 213 176 305
NFX Slim Serializer - KnownTypes 407 186 220 254
Wire - Default 439 226 213 359
Wire - Object Identity 456 238 217 359
Net Serializer 544 269 275 245
Wire - Version Tolerant 688 267 420 834
NFX Slim Serializer 835 404 431 362
Jil 849 416 433 0
NET-JSON 878 346 531 438
ServiceStack.Text 1682 902 779 419
Json.NET 2578 979 1599 834
Binary formatter 5227 3101 2126 2040

TypicalPersonArrayRoundtrip

test roundtrip serialize deserialize size
Wire - Object Identity 393 220 173 28693
Wire - KnownTypes 394 219 175 28891
Wire - Default 395 218 177 28891
Protobuf.NET 433 197 235 27240
NFX Slim Serializer - KnownTypes 433 212 220 28152
NFX Slim Serializer 447 223 223 28478
Wire - Version Tolerant 465 258 206 30918
Net Serializer 567 280 287 27792
NET-JSON 1012 440 571 48323
Jil 1055 607 447 0
ServiceStack.Text 2006 1017 988 45734
Binary formatter 2676 1767 908 30775
Json.NET 2788 1202 1586 86653

Tags for Releases

I'm trying to fix some bugs and test against the latest Akka.Net release (1.1.1).
This requires Wire 0.0.6.
The latest version has a breaking binary change in the constructor of SerializerOptions so would like to try and first test my fixes against the 0.0.6 version. Which branch/tag/changeset should I use?
Is there any place to see which where the 0.0.6 release was done?

thanks!

Add support for ISerializable

I think it would be possible to add support for ISerializable in Wire as ISerializable pretty much just requires that we can serialize dictionaries that can then be used to reconstruct the target object.

e.g.

public void GetObjectData(SerializationInfo info, StreamingContext context)
    {
        info.AddValue("props", myProperty_value, typeof(string));
        //after this method ends, write the contents of SerializationInfo into the output stream as a Dictionary
    }

    // The special constructor is used to deserialize values.
    public MyItemType(SerializationInfo info, StreamingContext context)
    {
        //before this is called, read a dictionary from the stream and transform the dictionary to a serializationinfo and pass it in here
        // Reset the property value using the GetValue method.
        myProperty_value = (string) info.GetValue("props", typeof(string));
    }

Description of what Wire can/can't do

I think, serializer description should clearly state about the its possibilities and conditions. Something like FAQ would be great. Example questions:

  • Is Wire able to retain objects references?
  • Does it need some type annotations or other compile-time mechanics?
  • Is it able to serialize/deserialize custom generic types?
  • Is it able to serialize/deserialize internal or private types?
  • Is it able to serialize/deserialize private fields or properties with private getter?
  • Is it able to use type constructors on deserialization? If so:
    • Can those constructors be internal/private?
    • In case of multiple constructors for target type, how will it resolve correct constructor?
  • Is it able to serialize/deserialize delegates? What about anonymous lambdas?
  • Is it able to serialize C# LINQ expressions?

Browser support

Any plans to make a JavaScript port to support communication between .Net servers and web browsers?

Add strong schema support

Protobuf and MS Bond have a better version tolerance story as they use ID tagging on properties.
I'm thinking about doing the same for Wire.

e.g.

[WireType(0)]
public class MyMessage
{
    [WireField(0)]
    public string SomeProperty {get;set;}

    [WireField(1)] 
    public int SomeOtherProperty {get;set;}
}

Thoughts about this?

Deserializing objects when a type is missing results in unexpected exception type

When deserializing a payload that reffers to a typename that is not present, there is:

arameter name: key] [ at System.Collections.Concurrent.ConcurrentDictionary`2.TryGetValue(TKey key, TValue& value)
at Wire.Serializer.GetCustomDeserialzer(Type type)
at Wire.Serializer.GetDeserializerByManifest(Stream stream, SerializerSession session)
at Wire.Serializer.DeserializeT
at Akka.Serialization.WireSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Remote.Serialization.MessageContainerSerializer.Deserialize(ByteString bytes, Type type, Int32 serializerId)

This needs to be handled with a better exception

Blacklist harmful types

We should make a list of blacklisted types which may not be deserialized.
There are some classic examples that have been used for hacker attacks by for example creating instances of harmful types.

We need a list of those types.

Deserialize from Type

How can I deserialize from a Type? I am looking to integrate Wire into an existing library for Redis. The method I need to use has this signature.

object DeserializeCore(Type type, byte[] value)

I thought this might work but it doesn't.

        using (var memStream = new MemoryStream(value))
        {
            var serializer = new Serializer(new SerializerOptions(true));
            var d = serializer.GetDeserializerByType(type);   
            return d.ReadValue(memStream, new DeserializerSession(serializer));
        }

Version 0.7.0 not supported on .NET 4.5

I'm creating a .NET 4.5 application with Akka.NET and I'm unable to update to the latest NuGet package version because it says it does not support .NET 4.5.

If this is the recommended serializer for Akka.NET please ensure that it supports the same versions.

Datetime Kind property is not serialized properly

         var now = DateTime.UtcNow;
        Console.WriteLine(now);
        Console.WriteLine($"Kind : {now.Kind}");

        var source = new MemoryStream();
        var w = new StreamWriter(source);
        var s = new Wire.Serializer(new SerializerOptions());
        s.Serialize(now, w.BaseStream);
        source.Position = 0;
        var newdate = (DateTime)s.Deserialize(source);
        //is unspecified here, should be Utc
        Console.WriteLine($"Kind : {newdate.Kind}");

        Console.ReadLine();

Considering this fails. DateTimeOffset probably also has issues.

Shorten type manifest

Currently, the type manifest for custom types consist of the assembly qualified type name.
This could be shortened by spitting typename and assembly name.
instead of foo.bar.baz, my.asm we could write the two separately, and introduce an assembly lookup much like the current type id lookup we have.
that way, only the first occurance of an assembly would be emitted, all others would emit an assembly ID.
This would likely speed up both lookup time and reduce payload size.

Add nullable serializer

Nullable types (that is, nullable value types) are currently not supported.
I'm not sure what the best approach for this would be, either to add a NullableSerializer that writes header info for null or not null and then internally use the correct element serializer output any non value conent

Problem with deserializing data structures from System.Collections.Immutable

At the present moment Wire is not able to correctly deserialize some of the collections from System.Collections.Immutable - checked on version 1.1.37 for immutable collection and 0.0.5 for Wire.

Sample to reproduce:

var map = ImmutableDictionary<string, object>.Empty;
var serializer = new Wire.Serializer();

using (var stream = new MemoryStream())
{
    serializer.Serialize(map, stream);
    stream.Position = 0;
    var map2 = serializer.Deserialize(stream);  // exception
}

Exception message: System.MissingMethodException - No parameterless constructor defined for this object

Stack trace:

at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at Wire.Converters.EnumerableSerializerFactory.<>c__DisplayClass3_0.b__1(Stream stream, SerializerSession session)
at Wire.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, SerializerSession session)
at Wire.Serializer.Deserialize(Stream stream)

Serializing a string with same pattern breaks Wire

Hi!, Just one day at work with Wire and I feel really satisfied.
Today I've tested it again with a stupid unit test of mine which I've collected over three years of testing serializers and.. it fails! Well, at least for me.
What I'm doing wrong?, if I'm causing the issue.

This is the Helper class for Wire:

[System.Diagnostics.DebuggerStepThrough]
public class WireFormatter : IBinarySerializer
{

    static Wire.SerializerOptions options = new Wire.SerializerOptions
        (versionTolerance: false, knownTypes: new List<Type>() { typeof(Communication.Message_Wire) }, preserveObjectReferences: true
        );

    static Wire.Serializer formatter;

    static WireFormatter()
    {
        formatter = new Wire.Serializer(options);
    }


    public WireFormatter() { }

    /// <summary>
    /// Serializza un oggetto in un array di bytes.
    /// </summary>
    /// <param name="obj"></param>
    /// <returns></returns>
    [System.Diagnostics.DebuggerStepThrough]
    static public byte[] Serialize(object obj)
    {

        // SANITIZE
        if (obj == null)
            return null;

        //Wire.Serializer formatter = new Wire.Serializer(options);
        using (MemoryStream stream = new MemoryStream())
        {
            // Scrive il contenuto
            formatter.Serialize(obj, stream);
            return stream.ToArray();
        }
    }


    /// <summary>
    /// Trasforma un array di bytes nell'oggetto originario.
    /// </summary>
    /// <param name="obj"></param>
    /// <returns></returns>
    [System.Diagnostics.DebuggerStepThrough]
    static public object Deserialize(byte[] obj)
    {
        // SANITIZE
        if (obj == null)
            return null;

        //Wire.Serializer formatter = new Wire.Serializer(options);
        using (MemoryStream stream = new MemoryStream(obj))
        {
            // Legge il contenuto
            return formatter.Deserialize(stream);
        }
    }


    byte[] IBinarySerializer.Serialize<T>(T value)
    {
        return WireFormatter.Serialize(value);
    }


    T IBinarySerializer.Deserialize<T>(byte[] bytes)
    {
        return (T)WireFormatter.Deserialize(bytes);
    }
}

This is the Unit Test (with Types to be serialized included):

    [Serializable]
    public class ComplexType
    {
        public ComplexType()
        {
            this.Numero = 100;
            this.Stringa = "Contenuto";
        }
        public int Numero { get; set; }
        public string Stringa { get; set; }
    }

[Serializable]
public class Message_Wire : IMessage
{
    public Message_Wire()
    {
        this.Options = new Dictionary<string, string>();
        this.Arguments = new Dictionary<string, object>();
        this.Serializer = SerializationEngine.Formatter.WireFormatter;
    }

    public bool IsEmpty { get; set; }

    public MessageCommand Command { get; set; }

    public SerializationEngine.Formatter Serializer { get; }

    public Dictionary<string, string> Options { get; set; }

    /// <summary>
    /// Gli Arguments del parser sono sempre KeyValue. Qual'ora mancasse il Value viene inserito null.
    /// </summary>
    public Dictionary<string, object> Arguments { get; set; }


    /*
     * Public methods
     */

    public void AddOption(string option, string value)
    {
        if (!this.Options.ContainsKey(option))
            this.Options.Add(option, value);
    }

    public void AddArgument(string key, object value)
    {
        this.Arguments.Add(key, value);
    }

    public byte[] ToArray()
    {
        return WireFormatter.Serialize(this);
    }

    /*
     * Conversions
     */

    public static explicit operator Message_Wire(byte[] source)
    {
        return (Message_Wire)WireFormatter.Deserialize(source);
    }
}

    [TestMethod]
    public void Message_Wire()
    {
        ComplexType complexArgument = new ComplexType();

        /* ------------------------ */

        byte[] serializedComplexArgument = WireFormatter.Serialize(complexArgument);
        ComplexType deserializedComplexArgument = (ComplexType)WireFormatter.Deserialize(serializedComplexArgument);

        deserializedComplexArgument.ShouldDeepEqual(complexArgument);

        /* ------------------------ */

        Dictionary<string, ComplexType> complexTypeDictionary = new Dictionary<string, ComplexType>();
        complexTypeDictionary.Add("Key3", complexArgument);

        byte[] serializedDic = WireFormatter.Serialize(complexTypeDictionary);
        Dictionary<string, ComplexType> deserializeDictionary = (Dictionary<string, ComplexType>)WireFormatter.Deserialize(serializedDic);

        deserializeDictionary.ShouldDeepEqual(complexTypeDictionary);

        /* ------------------------ */

        Message_Wire message = new Message_Wire();
        message.Command = MessageCommand.Set;
        message.AddOption("keys", null);
        message.AddArgument("Key1", "Contenuto");
        message.AddArgument("Key2", 100);
        message.AddArgument("Key3", complexArgument);
        message.AddArgument("key", "test");
        message.AddArgument("value", "asdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasas");

        // Send over the wire.
        byte[] serialized = message.ToArray();

        // Get the Message sent.
        Message_Wire deserialized = (Message_Wire)serialized;

        deserialized.ShouldDeepEqual(message);
    }

I use DeepEqual (DeepEqual on NUGet) to compare objects.
The problem I encounter is at line with explicit operator

Message_Wire deserialized = (Message_Wire)serialized;

And the exception is:

System.ArgumentOutOfRangeException: Richiesto numero non negativo.
Nome parametro: count

On a side note... If i left the line
message.AddArgument("value", "asdfasdfasasdfasdfasasd etc
out of the code IT WORKS.

Thank you for your great piece of software.

Support Full AOT (Ahead of Time compilation).

In full AOT environment like iOS, runtime jitting is not allowed, which means no runtime code generation at all.
For example, it's impossible to emit code and use a generic type which parameter types are not determined at compile time.

To support this, we need to put workaround code for every dynamic code. For example, CodeGenerator.GenerateWriteAllFieldsDelegate can be changed like:

private static ValueWriter GenerateWriteAllFieldsDelegate(
    List<ValueWriter> fieldWriters)
{
    if (fieldWriters == null)
        throw new ArgumentNullException(nameof(fieldWriters));

    if (use_dynamic_code) // CHECK SERIALIZER OPTION TO USE DYNAMIC CODE
    {
        // fast but not working code on AOT.
        var streamParam = Parameter(typeof (Stream));
        var objectParam = Parameter(typeof (object));
        var sessionParam = Parameter(typeof (SerializerSession));
        var xs = fieldWriters
            .Select(Constant)
            .Select(
                fieldWriterExpression =>
                    Invoke(fieldWriterExpression, streamParam, objectParam, sessionParam))
            .ToList();
        var body = Block(xs);
        var writeallFields =
            Lambda<ValueWriter>(body, streamParam, objectParam,
                sessionParam)
                .Compile();
        return writeallFields;
    }
    else
    {
        // slow but working code on AOT.
        return (stream, obj, session) => fieldWriters.ForEach(w => w(stream, obj, session));
    }
}

How about this?

Deserialize surrogates

I meet a problem when deserialize a list of surrogate objects if some of them refer the same object.

Say, I have serialized List(){ original_1, original_1 }

Mechanism of deserialization:
For the 1st item:
Deserializes surrogate_1.
Adds surrogate_1 into DeserializerSession._objectById.
FromSurrogateSerializer translates the surrogate_1 into original_1.

For the 2nd item:
Deserializes surrogate_1 again. Indeed it is a reference of surrogate1 ,hence returns result of DeserializerSession.GetDeserializedObject , which is surrogate_1.
But there, it is supposed to be an object of Orginal type.

Did I do anything wrong?

Can't convert RemoteActorRef to ISurrogate

Hi,

I am using Akka.Net in combination with Akkling (0.2.1). I have finally established a connection between two remote actor-systems. From the actor system which has the task of a state server, I want to send an IActorRef to the client (who is asking for this). The type is: Akkling.ActorRefs.IActorRef<'MySharedDU>

This results in the following exception at the client side:

[ERROR][26-12-2015 14:13:42][Thread 0011][[akka://ThisIsMyClient/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fThisIsMyServer%40localhost%3a8080-1/endpointWriter]] AssociationError [akka.tcp://ThisIsMyClient@localhost:8090] -> akka.tcp://ThisIsMyServer@localhost:8080: Error [Cannot convert object of type Akka.Remote.RemoteActorRef into the type Akka.Util.ISurrogate.] [ at lambda_method(Closure , Object , Object )
at Wire.CodeGenerator.<>c__DisplayClass4_0.b__1(Stream stream, Object o, DeserializerSession session)
at Wire.CodeGenerator.<>c__DisplayClass1_0.b__0(Stream stream, DeserializerSession session)
at Wire.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
at Wire.ValueSerializers.FromSurrogateSerializer.ReadValue(Stream stream, DeserializerSession session)
at Wire.Serializer.Deserialize(Stream stream)
at Akkling.Serialization.WireSerializer.FromBinary(Byte[] bytes, Type _arg1) in D:\projects\dev\Akkling\src\Akkling\Serialization.fs:regel 33
at Akka.Remote.MessageSerializer.Deserialize(ActorSystem system, SerializedMessage messageProtocol)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, SerializedMessage message, IActorRef senderOption)
at Akka.Remote.EndpointReader.OnReceive(Object message)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
Cause: Unknown

I think the issue is in Wired? I just upgraded to Wired v0.0.6 (Akkling was stil dependent on v0.0.1 which gave even bigger issues). Note, I translated the exception manually from my own language, so there can be textual mismatches with the official english version.

Is Serializer thread safe?

Are serialize/deserialize methods thread safe?

I want to create SerializationWrapper and register it as a singleton in NServicebus

class SerializationWrapper
{
        private readonly Serializer serializer = new Serializer();
        public void Serialize(object obj, Stream stream)
        {
            serializer.Serialize(obj, tempStream);
        }
        [...]
}

Should I use ThreadStatic or ThreadLocal for serializer field?

Unable to deserialize anonymous object

Hi,

I was wondering if this was a supported case:

  • On computer 1: serialize anonymous object
  • Send the serialized bytes to another computer (using a file, Redis, database, etc)
  • On computer 2: deserialize the serialized bytes

I get the following error while deserializing:

Additional information: Could not load type '<>f__AnonymousType0`2' from assembly XYZ

The message make sense since the anonymous type was never created on computer 2.

On an unrelated note: for run I plugged in Wire in my own binary serializer unit tests and I was wondering if you would like to know the issues I found.

Wire doesn't pass Akka.Serialization tests

The latest Wire, which was built from sources (dotnetcore branch) doesn't pass Akka.Serialization tests, half of them failed:

System.MissingMethodException

No parameterless constructor defined for this object.

   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.Activator.CreateInstance(Type type)
   at Wire.TypeExtensions.GetEmptyObject(Type type)
   at Wire.CodeGenerator.<>c__DisplayClass1_0.<MakeReader>b__0(Stream stream, DeserializerSession session)
   at Wire.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
   at Wire.Serializer.Deserialize[T](Stream stream)
   at Akka.Serialization.WireSerializer.FromBinary(Byte[] bytes, Type type) in C:\Projects\akka.net\src\contrib\serializers\Akka.Serialization.Wire\WireSerializer.cs:line 62
   at Akka.Tests.Serialization.AkkaSerializationSpec.AssertEqual[T](T message) in C:\Projects\akka.net\src\contrib\serializers\Akka.Serialization.TestKit\AkkaSerializationSpec.cs:line 276
   at Akka.Tests.Serialization.AkkaSerializationSpec.CanSerializeResizer() in C:\Projects\akka.net\src\contrib\serializers\Akka.Serialization.TestKit\AkkaSerializationSpec.cs:line 269

ILEmit serializers like protobuf-net does

A performance gain can be provided not only by reducing allocations or providing a better wire format but also by optimizing the serializer methods themselves. It has been applied in protobuf-net (they have their own il emit context) as in JIL which uses Sigil. Is this kind of optimizations considered for the future releases of the wire? What's your opinion on this kind of optimizations ?

NetSerializer LargeStruct test perf

cc @Scooletz

When serializing the "LargeStruct" from NetSerializer test suite, Wire performs considerably worse than NetSerializer, about twice as slow.

I see no obvious reason why we do so as our IL generated writer for the struct should be just as slim as the one in NetSerializer.

cc @tomba
Are there any specific tricks NetSerializer uses in terms of structs?

Our generated Serializer for the struct would look pretty much like this

public virtual void Serialize(Stream stream, object value, SerializerSession session)
{
        var x = (LargeStruct)value;
        var buffer = session.GetBuffer(8); //pre allocated
        NoAllocBitConverter.GetBytes(x.field1,buffer);
        Stream.Write(buffer,0,8);
        NoAllocBitConverter.GetBytes(x.field2,buffer);
        Stream.Write(buffer,0,8);
        NoAllocBitConverter.GetBytes(x.field3,buffer);
        Stream.Write(buffer,0,8);
        NoAllocBitConverter.GetBytes(x.field4,buffer);
        Stream.Write(buffer,0,8);
}

We do have a virtual call and boxing of the struct ATM. but can that really make up for as much as twice much overhead?

I've tried using NetSerializer VarInts for ulong too, but no positive performance change.

String length serialization bug

There is a bug in the string serialization code. For strings longer than 254 bytes you write a marker with a value of 254. Shouldn't it be 255? Also you add 1 to the length before writing it to the stream and subtract 1 after reading the length from the stream (source). I don't know why you did this in the first place, but this also collides with the marker and produces some bugs for me.

Index and count must refer to a location within the string

Apologies for the lack of a repro here. On a customer site, trying Wire out for an array of C# POCOs with some nullable fields, and properties that use object hierarchies, getting the following error:

Index and count must refer to a location within the string: Parameter name: s

Afraid I can't give you the actual type structure or sample data - I've tried reproducing myself but couldn't so far. Have you seen this before?

   at System.Text.UTF8Encoding.GetBytes(String s, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex)
   at Wire.NoAllocBitConverter.GetBytes(String str, SerializerSession session, Int32& byteCount)
   at Wire.ValueSerializers.StringSerializer.WriteValueImpl(Stream stream, String s, SerializerSession session)
   at Wire.ValueSerializers.StringSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Wire.ValueSerializers.ObjectSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
   at Wire.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method(Closure , Stream , Object , SerializerSession )
   at Wire.ValueSerializers.ObjectSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
   at Wire.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Wire.SerializerFactories.ArraySerializerFactory.WriteValues[T](T[] array, Stream stream, Type elementType, ValueSerializer elementSerializer, SerializerSession session)
   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid6[T0,T1,T2,T3,T4,T5](CallSite site, T0 arg0, T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5)
   at Wire.SerializerFactories.ArraySerializerFactory.<>c__DisplayClass4_0.<BuildSerializer>b__1(Stream stream, Object arr, SerializerSession session)
   at Wire.ValueSerializers.ObjectSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
   at Wire.Serializer.Serialize(Object obj, Stream stream)

Crash when serializing a string

Hi,

I am using the latest version from Nuget (0.8, .Net 4.6.1) and I got an exception while serializing a string with more than 254 characters:

System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
    at System.IO.MemoryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Wire.Extensions.StreamEx.ReadString(Stream stream, DeserializerSession session)
   at Wire.ValueSerializers.StringSerializer.ReadValue(Stream stream, DeserializerSession session)

Is it related to issue #21?
I was able to replicate the bug with the following code:

        [TestMethod]
        public void MyTestMethod()
        {
            byte[] wireBytes;
            var wireSerializer = new Wire.Serializer();
            using (var ms = new MemoryStream())
            {
                var obj = new string('c', 1000);
                wireSerializer.Serialize(obj, ms);
                wireBytes = ms.ToArray();
            }
            using (var ms = new MemoryStream(wireBytes))
            {
                wireSerializer.Deserialize(ms);
            }
        }

Compiler abstraction

The current code generation is based on Linq Expressions.
These are nice to work with, but suffer from two major issues for us.

  1. You can not assign readonly fields
  2. There is a performance overhead in terms of security checks and whatnot

I want to move towards an IL Emit solution instead.
In order to do this w/o doing a big huge upfront change of the code, which would be hard.
I've introduced the Compiler<TDel> abstraction.
It still uses Expressions internally. but at least it should be easier to rip out the internals from that and replace with e.g. Sigil.

Add surrogate support

For Akka.NET, we need surrogate support.
This means that we need to be able to transform some types into other types, or rather let the serializer transform some objects into other objects before writing them to the output stream.

Support for delegates.

This could be really appreciated in some cases. While it's probably an edge case for standard message serialization, it would really simplify process of restoring behavior from events/snapshots of actor's state in persistence scenarios.

Support .NET 3.5.

After getting .net core done, it will be nice to support .NET 3.5 which Unity3D still uses. There are two things to suppor this:

  • Add ConcurrentDictionary which was introduced at .NET 4. But it's trivial because there is an alternative one.
  • Add workaround for dynamic code which was also introduced at .NET 4. But with #38, there is nothing to do for this.

How about this?

[Question] - Test framework

I see that MSTest is the test framework. This framework is really out of date in comparison to its competitors in many ways. What do you think about NUnit or XUnit? Asking because I have some plans to increase code coverage

Null reference bug in `WriteAllFields`

There is a null reference bug somewhere in the code generated code when serializing some types.
I've only seen this in a few of the Akka.NET tests so far.

An exception is thrown when trying to serialize an object with enclosed AggregateException instance

It seems that Wire is not able to serialize an instance of AggregateException created by TPL engine.
Here is the complete example to reproduce the problem:

    public abstract class Status
    {
        public class Failure : Status
        {
            public readonly Exception Cause;

            public Failure(Exception cause)
            {
                this.Cause = cause;
            }

            public override string ToString()
            {
                return "Failure: " + this.Cause.ToString();
            }
        }
    }

    public class Serializer
    {
        private readonly Wire.Serializer _wireSerializer;

        public Serializer(Wire.Serializer wireSerializer)
        {
            _wireSerializer = wireSerializer;
        }

        public byte[] ToBinary(object obj)
        {
            using (var ms = new MemoryStream())
            {
                _wireSerializer.Serialize(obj, ms);
                return ms.ToArray();
            }
        }

        public object FromBinary(byte[] bytes, Type type)
        {
            using (var ms = new MemoryStream(bytes))
            {
                var res = _wireSerializer.Deserialize<object>(ms);
                return res;
            }
        }
    }

    public class Worker
    {
        public async Task<object> DoWorkAsync()
        {
            var res = await DoWork();
            return res;
        }

        public Status.Failure WaitFailure()
        {
            var task = DoWorkAsync();
            try
            {
                task.Wait();
            }
            catch (Exception e)
            {
                return new Status.Failure(e);
            }

            return null;
        }

        private Task<object> DoWork()
        {
            throw new InvalidOperationException();
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var serializer = new Serializer(new Wire.Serializer(new SerializerOptions()));

            var worker = new Worker();
            var failure = worker.WaitFailure();
            var bytes = serializer.ToBinary(failure);

            // Exception is here
            var backObj = (Status.Failure)serializer.FromBinary(bytes, typeof(Status.Failure));
        }
    }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.