rikimaru0345 / ceras

Universal binary serializer for a wide variety of scenarios https://discord.gg/FGaCX4c

Ceras

Ceras is a binary serializer. It converts any object into a byte[] and back. It goes above and beyond in terms of features, speed, and comfort. Supports reference loops, large/complicated inheritance chains, splitting objects into parts, ...

Quick start

class Person { public string Name; public int Age; }
var p = new Person { Name = "riki", Age = 5 };

var ceras = new CerasSerializer();

var bytes = ceras.Serialize(p);

1. >> Many more examples in the code tutorial
2. >> Detailed guides for specific scenarios on my blog
3. >> Read 'Optimization & Usage Pitfalls'

• >> Changelog v3.0 (rikidev.com/new-features-in-ceras-3-0-2)
• >> New features in v4.0 (https://github.com/rikimaru0345/Ceras/releases/tag/4.0)

Features

• Very fast, small binary output (see Performance)
• Supports pretty much any type:
  • Hand-written formatters for all common .NET types
  • Comes with formatters for all common Unity types
  • Generates new formatters at runtime for any new/user type
  • Very easy to extend and customize
• Full support for reference persistence (including circular references!)
• Full support for polymorphism / inheritance / interfaces
• Can serialize objects into parts as "ExtenalObjects" (useful in many many scenarios)
• Automatic version-tolerance, no need to place any attributes on your classes!
• full feature list (and planned features)

Performance benchmarks

Ceras generally ranks at the top end of the performance spectrum, together with NetSerializer and MessagePack-CSharp. To get an idea of how Ceras performs here are the preliminary benchmark results. The resulting binary size is about the same as MessagePack-CSharp.

The shown results are obtained from this code and I encourage you to not only try it yourself, but to also provide feedback about scenarios you had good and bad results with.

Don't forget to tune the settings in SerializerConfig for your specific situation. Using Ceras to read/write network packets might require different settings than, lets say, saving a settings-object to a file, or persisting items/spells/monsters in a game, or ...

The project is still heavily work-in-progress, meaning that over time more optimizations will get implemented (your feedback is important here!).

What can this be used for?

Example usages

The primary goal is to make an universal serializer that can be used in every situation. Personally my primary intentions were easy object persistance and network communication. I've added many features over time and whenever someone can think of a good scenario that should be supported as well I'll make it happen.

Examples:

• Settings: Saving objects to disk quickly without much trouble: settings, savegames, whatever it is. With pretty much zero config. See steps 1 and 2 in the Usage Guide

• Splitting: So your Person has references to other Person objects, but each one should be serialized individually!? (without the references quickly dragging in essentially your whole program). Maybe you want to be able to put each Person into its own file, or send them over the network one-by-one as needed? No problem! Using IExternalRootObject it's not an issue! See External Objects Guide (Game DB example)).

• Network: Because of its simple API and vast set of features Ceras is uniquely suited to implement a full 'network-protocol' for you. I wrote a short guide that shows off how a basic TCP implementation could look like: Just Send(myObject); it, then var obj = await Receive(); on the other side, that's it! It literally can't get any easier than that. At the moment the guide only has 2 parts, but when I have some (and if there are requests for it) I'd like to continue the series, eventually building that sample into a full-fledged, robust, and battle-tested networking system.

• More: The above are just examples, Ceras is made so it can be used in pretty much every situation...

When should I not use this?

• If you need human readable output for some reason. For example some file that you want to be able to edit in a text-editor. For those usages JSON or XML are likely better suited.

• You plan to use this on a platform that does not support code generation. Serializers for user-types are created at runtime through code-generation. And if that isn't allowed (for example on iOS) Ceras won't be able to generate arbitrary object-formatters. Built-in types will still work though. There are ways to fix this though... (pre-generating the formatters) Ceras now has a dedicated AotMode in the config and a code-generator (quick guide for it here) for IL2CPP/Unity/AoT.

Support

• Open an issue
• Join my Discord (probably the best for direct one-on-one help)
• Make sure you've read FAQ and Optimization & Pitfalls

>> FAQ

>> Development Blog

ceras's Issues

BigInteger Not Supported?

Hey. I swapped out BinaryFormatter for Ceras the other day and the performance gains are incredible, but I have run into one issue. It appears that BigInteger isn't currently supported (it serializes without exception, but deserializes into an empty version of the object). Would it be possible to get support added for it?

Null exception in deserialize

Hi,

I create a List (150,000 objects) ;
I try to serialize and deserialize but there is a NULL EXCEPTION

            var datas = Trade.Generate(150000);
            var serializer = new CerasSerializer();
            var serialize = serializer.Serialize(datas);
            var deserialize = serializer.Deserialize<List<Trade>>(serialize);



public class Trade
{
    public string Ticker;
    public bool IsValid;
    public DateTime DateAndTime;
    public float Price;
    public uint Size;
    public EExchange Exchange;
    public ETradeCondition Condition;
    public bool HasQuotes;
    public Nbbo Nbbo;

    public Trade() { }
}

public class Quote
{
    public string Ticker;
    public DateTime DateAndTime;
    public float Bid;
    public uint BidSize;
    public EExchange BidExchange;
    public float Ask;
    public uint AskSize;
    public EExchange AskExchange;

    public Quote() { }
}

The serialialization is OK, but in deserialize there is an exception NULL

	public void Deserialize(byte[] buffer, ref int offset, ref TCollection value)
	{
		// How many items?
		var itemCount = SerializerBinary.ReadUInt32(buffer, ref offset);

/////------------------------------------ EXCEPTION, VALUE IS NULL
value.Clear();
/////------------------------------------ EXCEPTION

		for (int i = 0; i < itemCount; i++)
		{
			TItem item = default(TItem);
			_itemFormatter.Deserialize(buffer, ref offset, ref item);
			value.Add(item);
		}
	}

HashSet Deserialization loses IEqualityComparer

I'm creating a hashset with a custom IEqualityComparer and while the values within the hashset serialize and deserialize fine, the IEqualityCompareris missing on the other side. It's a read-only field so I tried with the read-only branch but that still failed. I might be doing something wrong? I have targetmembers set to all.

Feature: VersionTolerance + AotMode

At the moment VersionTolerance and AotMode can't be used together.
This issue is tracking the progress towards fixing this.

Why was this not implemented already?

The main problem here is that the DynamicFormatter and SchemaFormatter (the version tolerant variant of dynamic formatter) are pretty tightly linked. They're very similar, so they share a lot of code.

In Aot scenarios the DynamicFormatter cannot be used, the AotGenerator fixes this by generating the source code for all types that would normally be handled by the DynamicFormatter.

When not using VersionTolerance that is relatively simple, the code the AotGenerator writes is pretty much exactly what a normal user would write to customize Ceras (a custom IFormatter<> implementation).

However, if VersionTolerance is activated the code that would have to be emulated is a lot trickier!
It would be unreasonable for the AotGenerator to generate the same code the SchemaFormatter uses internally for two reasons:

• The SchemaFormatter is able to switch the Schema (basically just a ordered list of members) on the fly, meaning that it can change the way it is reading the data if the data says so. The two things that would cause a binary incompatibility are:
  • Skipped Members: if the schema defines a "skip entry" (a field/prop that once existed but was removed now) the SchemaFormatter can skip over the data and continue with the next member.
  • Reordered members: some changes can cause the order of the members to change, the SchemaFormatter solves this by generating a specialized version of the reader.

The second part is inherently dynamic; a different format (order) requires different reading code (unless you want to introduce branching, which would make things slower).

Bad solution

We could have some sort of "universal" SchemaFormatter that would dynamically switch how to read the data based on the given Schema. It would essentially be a big loop that iterates over all members described by the Schema.

Advantage: Relatively easy to do, quick to make.

Disadvantage: It would definitely be horribly slow... (looping, branching, dynamic formatter lookups, must-have for reflection because we can't call generic methods if we don't know the generic arguments in advance, ...)

Best solution

We can expect the "primary schema" to match the "data schema" most of the time.

(Primary Schema = the Schema that Ceras generates for a given Type, aka the Schema as it is in the currently running assembly)
(Data Schema = Schema that was constructed/read from some given serialized data)

The most common case by far will probably always be that PrimarySchema == DataSchema.
Meaning the data has the same "version", so we don't have to do any sort of skipping or reordering (or later: type conversion) of any members.

That means we could make use of the AotGenerator here.
It would just pre-generate a SchemaFormatter for the current type and schema.

For the other case we're definitely in trouble however.
The data is in a format none of our formatters understands, and we're not allowed to generate one that does, oof.

The only thing left that we can do is "emulation", aka a formatter that loops over all the members described by the Schema, dynamically obtains the matching formatter-instance, and then uses reflection to dispatch the call Deserialize.

Maintainability

The "primary schema" case however (that we want the AotGenerator to generate) is more difficult.
We shouldn't just copy the whole code of the SchemaFormatter.

A much better approach could work like this:
When the AotGenerator runs, it could obtain the expression-trees constructed by DynamicFormatter/SchemaFormatter and then traverse them to convert them into C# code.

That way, no matter how they change internally, the AotGenerator will always work perfectly.
Any refactorings will automatically be applied to formatters the generator writes.

And all future optimizations in DynamicFormatter are also automatically propagated and available in Aot builds!

Advantage: Best possible solution

Disadvantage: Requires a lot of work. Needs the "emulated" formatted described above, and a non-trivial rewrite of the AotGenerator as well.

Usage considerations

If the user deserializes data of an older format, they should immediately serialize it again and replace the old data. That is because Serialization always uses the "current" (newest) schema. By replacing the old data, the next deserialization can use the built-in formatter again (so maximum speed again next time).

How do we communicate to the user that when combining Aot+VersionTolerance reading might sometimes be slower than expected?
How to easily explain why that is unavoidable without going into much detail?
How to make sure they treat deserialization of old data as the kind of "data upgrade" it really is?

Todo

• Make SchemaEmulatorFormatter
• AotGenerator should generate code from expressions

Bug: Argument out of bounds exception

Describe the bug
I'm trying to deserialize the same type I was in issue #29, and it fails with an ArgumentOutOfRangeException.

How to reproduce the bug
I'm using the default serializer configuration, and stepping through the debugger what's happening is Ceras is attempting to access an object with id 118 in ObjectCache._deserializationCache (inside the method ObjectCache.GetExistingObject()) and _deserializationCache has a Count of 0.

Platform
I'm on .NET Framework 4.7.2, and Ceras version 4.0.27

Bug: AOT generator Deserialize functions don't compile when target class has serialized properties

Remember: The more faster I can reproduce your problem, the faster I can fix it :P

Describe the bug

• What are you trying to do?
  Generate formatters for a class type with serialized properties, e.g.:

[CerasAutoGenFormatter]
public class TestWithProperty
{
	int anInt;
	public int AnInt { get => anInt; set => anInt = value; }
}

The Deserialize function of the generated formatter ends up looking like:

public void Deserialize(byte[] buffer, ref int offset, ref TestWithProperty value)
{
	_intFormatter.Deserialize(buffer, ref offset, ref value.AnInt);
}

which doesn't compile since properties can't be passed as ref parameters.

• What do you think should happen?
  I'm not actually sure. If there's a way to support this easily without passing the property as a ref parameter, then that! Otherwise, omit the property and warn the user?

How to reproduce the bug
See above; use the provided class example with the AotGenerator tool (using the [CerasAutoGenFormatter] attribute to tag the class) and then try compile the generated output in a project.

Platform

• Which version of Ceras are you using? net4.5, net4.7+, or netstandard2.0?
  netstandard2.0

• Are you using Unity? If so, what version? (2018.1, 2018.2, ...)
  2018.3.14f1

• If you are using Unity, are you using IL2CPP?
  I intend to!

Planned features and improvements for Ceras v5

This issue tracks ideas for new features and improvements for the next version of Ceras (v5).

Breaking Changes

The following are changes that can't be implemented in the current version (v4) because they change the binary format Ceras uses.

ReferenceFormatter (Done!)

Together with all the other changes, there is a chance to optimize the most common code-paths taken by the ReferenceFormatter<>!

It turns out that back-references are relatively rare, so we can make them into a special case!
So we can switch from a VarInt to a fixed 1-byte prefix to tells us about the upcoming data.

• Simple cases:
  • Null
  • NewObject: following the object data directly
• Extended cases:
  • NewDerivedObject: following a Type and the object data
  • InlineType: following a Type, used as the object itself
  • ExternalObject: following a fixed Int32 for the external ID
  • BackReference: following a fixed Int32 for the already encountered object ID

The most common cases by far (99%+) are Null, NewObject, and NewDerivedObject.
The important thing to note here is that all the common cases will profit from much faster read/write performance.

Surprisingly all the uncommon cases will be faster as well, since even though we have an additional 1 byte, there are fewer branches in total (considering that VarInt contains 4+ branches by itself already)

• Rework ReferenceFormatter using the scheme described above

Type Serialization (Done!)

• Type Codes (for framework types)
  In the rare case that Ceras has to embed a Type into the binary, it is written using its full name; which is perfectly fine for user-defined types.
  But for framework types (like List<>, Int32, ...) we could write something like a "builtin type-code" to save space.

AotGenerator (Done!)

• Instead of generating the content of the formatters using strings; the new AotGenerator should actually construct the expression tree that the "normal" Ceras uses (when not using VersionTolerance); and then convert that into a source-code-string. That way we get some huge advantages:

  • Improvements to DynamicFormatter will automatically be available in generated formatters!
  • Drastically reduced the potential for bugs, because now the generator doesn't have to essentially "rewrite" what the DynamicFormatter does
  • Performance features like merge-blitting are automatically implemented in aot code as well!

• Split the old AotGenerator.exe into a .dll and an .exe, so that usage in Unity is much easier. There could be a Unity-script that automatically listens for changes and recompiles the

• Add an attribute to generated formatters. That way when CerasAutoGenConfigAttribute is used, Ceras knows that it should ignore any old generated formatters while re-generating them.

Encoding (Done!)

• Improved String Encoding
  Currently we have to iterate over every string twice because we must know how many bytes it will require (using GetByteCount). That takes time. Another approach is to guess the byte-length, then write, then see if we have to relocate the string (in other words, do it all again).
  Every serializer I know of does one of those two things.
  I would prefer if Ceras would try to be more efficient by encoding strings in a more intelligent scheme. The idea is that we'd write up to 254 bytes and then, if there are still characters left, encode the remaining bytes in one big block.
  The only blocking issue here is that String.Create is only available in .net standard 2.1; and without it we'd pay with a performance hit at deserialization time (having to allocate a char array, then creating a new string from that). However the performance impact might be negligible (memcpy is much faster than the utf8 decoding step), the char array can be thread-local and recycled, and we can completely avoid the hit in netstandard2.1 later; whereas we'd have to live with the not-as-efficient encoding forever if we don't do this change now.

Config

• Ability to configure formatter per Member!

  • "Late initialization" to allow changes to the TypeConfig for as long as possible (until the first de/-serialization)
  • Ensure that declaring types of members using a custom formatter are in fact handled by DynamicFormatter

• config.IntegerEncoding
  Allow users to decide when they want to use fixed encoding vs variable encoding. For example if you want to use Ceras for networking you want to throw in as much compression as you can, every cpu cycle that goes towards sending less data is worth it. So you could opt to encode all int, short, long, ... with variable encoding, making Ceras use WriteUInt32 instead of WriteUInt32Fixed.
  Or, if your aim is to save data to disk (save-game, settings, level-data, game-database...) you want things to go fast, so you can always use fixed encoding, which is larger (always 2/4/8bytes) but much faster.

  • UseReinterpretFormatter is now superseded by IntegerEncoding

• PersistentName for Types and Members. Influences member order. Enables Ceras to work together with obfuscators.

  • Add this new setting to TypeConfig
  • Automatically set by [MemberConfig], [DataMember], or member.Name
  • Maybe have "config.OnGetPersistentName", so the user can do all sorts of trickery (maybe having encrypted type names even in the attributes, only decrypting them when Ceras needs them)

Version Tolerance

Right now (in v4) when an application wants to read any binary data with Ceras, it must already have the correct Types (classes and structs). In other words, the format must be known.

This is fine for very high-performance use cases, but sometimes you want to trade in a bit of performance for a bit more leeway in terms of compatibility.
For example having a server-client network scenario where the client is slightly outdated...

Or another scenario would be an application wanting to inspect or even modify data when it doesn't know the specific format.

Formats like Json, Xml, or the MsgPack embed additional information so they can be completely self-describing.
With the following improvements, Ceras format can be a self-describing as well!

That way Ceras could even handle simple cases (like changing an int field to a float or something) automatically, and provide an API for the user to handle more complicated cases.

• More information in embedded Schema

  • Type.Name: allow for types to change their name! Also makes Ceras compatible with obfuscators!
  • MemberType: each member also records its type, in addition to the name. Allows for members changing their type, and even automatic conversion.
  • Formatter: embed an ID for each used formatter (reinterpret, array, list, varint, dynamic, user, ...). That allows us to be robust against changes in IntegerEncoding, or warn the user when they're trying to read something but some formatter is missing! (maybe the old data was written using a user-created formatter)

• Ensure only Schema of types handled by DynamicFormatter/SchemaDynamicFormatter are actually written (but allow users from manually generating/writing a Schema of any type)

• Make Schema public and add a OnSchemaRead callback that is called when Ceras loaded a new Schema, and produced some mappings and conversions in order to load the old data. That way users can see how the format changed and what Ceras did to resolve the differences. Also provide a way to save/load a Schema to/from byte[].

• (maybe) Inspect / .ToJson()
  With all the new information in Schema, it should be possible to allow users to info it should be possible to even (one-way) convert it into a json-string.

• config.VersionTolerance.PrefixSize setting to let the user select the prefix size of members (currently fixed UInt32). It should be possible to select ushort, byte, and even varint. Epecially interesting for networking purposes.

• config.EmbedSchema setting that you can disable, in which case you're responsible for somehow storing the Schema manually. Could be useful for network scenarios.

Non breaking changes

• Type encoding should use size-limited strings to prevent an attacker from overloading the serializer that way.

• (Maybe) Special handling for very large structs (>64 bytes). We could have a ISerializeByRef interface implemented by DynamicFormatter, ReinterpretFormatter and ArrayFormatter.

• Ensure all lookups of private methods actually work in .NET Core as well (ex "GetUninitializedObject" which is private there)

• Try to automatically select a constructor in more cases. Maybe filter the ones we can't use / map, then use the one that takes the most arguments?

• When used in Unity: Catch and rethrow MissingMethodException and tell the user what the problem actually is (IL2CPP either removing a method, or not generating a generic instantiation for it). Explain how it can be fixed: Add link.xml for stripped methods. Call generic methods in their closed form beforehand. Maybe we could even generate some code for the user to copy-paste in the latter case.

• Support open and half-open MethodInfos. (comment)

• Change exception when no ctor is found to tell people about [CerasConstructor]

Feature: "Serialization Constructors", factory methods, pooling, ...

Serialization constructors are becoming more and more important. The latest feature requests need them indirectly as well.

"What even is a 'serialization constructor'? Do we really need it? I've never heard of it, sounds useless!"

(Click here to expand; unless you already know...)

 

Example 1

Pretty much nothing in the System.Linq.Expressions.* namespace has a public constructor.
So how do you use those types then?
There's the Expression class that has tons of static methods like Expression.Block(...) which instantiate all the expressions for you. So serializing/deserializing can't be done the normal way.
There are a few solutions:

• The "classic" solution: manually writing a formatter for every single type in that namespace. This will obviously work, but it will take ages to do...
• Saving the NodeType and content, then later using the static factory methods in Expression.* to re-creating the objects... (not so easy because objects depend on each other)
• Configure the types dynamically: include private fields for those objects, force "Overwrite ReadOnly Fields", and create objects from an uninitialized state. (This would be the easiest and most safe method to do when the API is done)

Example 2

var myList = new List<int>() { 1, 2, 3, 4, 5 };
var roc = new ReadOnlyCollection(myList);

In case you've never seen or worked with a ReadOnlyCollection: It's just a collection that is initialized once at the start and can't be changed anymore after that (unless the underlying IList<> changes, then those changes are reflected in the ReadOnlyCollection). It's basically just a super simple wrapper around any given IList<>. Should you try to modify it in some way, it will just throw a NotSupportedException. Even if you'd try to be sneaky like IList<int> castedToList = roc; it wouldn't work...

So we see that it's not always possible to just new() an object and then fill its fields and properties.
Sometimes we're dealing with objects that have a certain way they're intended to be used / constructed.

So what's a serialization constructor now!?!?

It's simply "the constructor that the serializer is supposed to use when deserializing". It's not a fixed concept, it's rather a convention that some serializers have established.
Actually it doesn't even have to be a normal constructor on the object itself. As we've seen in the Expression example the "serialization constructor" might even be a static method somewhere (aka a "factory method").

... so now that we know all that, we can now continue

Current workaround

In the current version (3.0.2) Ceras you can call CerasSerializer.AddFormatterConstructedType(typeof(MyThing)); and implement a custom IFormatter<MyThing> and handle the construction yourself in there.

However, the goal of Ceras is to make a serializer that doesn't suck! And the current workaround does indeed suck. So that definitely has to be improved 😄

Pooling

Object pooling is a closely related topic. Maybe your objects are expensive to create, or maybe you need to recycle objects to reduce GC pressure (hello Unity users 😜) ...
In any case, "Pooling" essentially asking the same question: "How does Ceras obtain or create an object while deserializing?"

Ceras currently supports pooling through the two callbacks in the config: Func<Type, object> ObjectFactoryMethod and Action<object> DiscardObjectMethod.

A typical implementation of a ObjectFactoryMethod has to check what type is requested (maybe using multiple if()s, or a dictionary, ...) so it comes with a little overhead.
So pooling could be improved and implementing serialization-constructors is the perfect opportunity.

API

So here's the preliminary API I came up with. I think it covers every conceivable situation.

// Use 'new()', this is exactly what Ceras currently does.
config.ObjectConstruction.UseDefaultConstructor<Person>();


// Use 'FormatterServices.GetUninitializedObject()'. 
// Probably won't be used often (if at all), because of the much
// better / more performant options below
config.ObjectConstruction.UseUninitializedObject<Person>(); 


// User provided custom delegate to create an "empty" instance.
// Probably rarely used as well.
config.ObjectConstruction.UseDelegate<Person>( () => new Person() );


// Use the given MethodInfo by trying to matching field and property
// names to the arguments of the method. If the matching fails for any
// reason you can provide a "ParameterMapping" class as well that
// tells Ceras what field or property in the serialized data each parameter
// gets its value from. This is probably the most useful method of all of them.
config.ObjectConstruction.UseFactoryMethod<T>(constructorOrMethodInfoReturningT); 
        

// Convenience method. A wrapper around the previous method.
// The parameter is an 'Expression<Func<T>>', so you can very easily
// specify a method without having to use reflection to obtain a 'MethodInfo'
config.ObjectConstruction.UseFactoryMethod<Block>(() => BlockCreator.Create(...));


// Convenience Method: Takes a Type and forwards all its methods to 'UseFactoryMethod'
config.ObjectConstruction.UseFactory(typeof(MyStaticPool));


// Similar to UseFactory, but takes a instance of an object and tries to
// use each instance-method as a factory. This creates a closure, or in
// other words: it binds to the given instance.
config.ObjectConstruction.UseFactory(myFactoryInstance);

Some notes:

• Performance will stay a top-priority! If at all possible, Ceras will always "unpack" any delegates and compile target methods directly into its dynamically generated code, completely removing any overhead.

• UseFactory is probably the most useful method because it allows you to very easily use your already existing pooling solution (however it might look like). Even if you don't have a class that directly matches the expected format, you can just create a class that forwards all method calls to your actual pool.

• This would replace the current approach, where you can only provide a callback like Func<Type, object> which always forces you to check the type. That costs some performance (only very little, but still) and leads to less readable code. This new API would solve these issues completely.

• You won't have to call AddFormatterConstructedType anymore (in case you're using it). The new API would obviously not need that anymore.

Bug: VersionToleranceMode.Standard causes InvalidOperationException on deserialization

If VersionTolerance is set on SerializerConfig, public and protected fields in base class causes System.InvalidOperationException.

public abstract class BaseCls
{
    public string Field1;
}
public class TestCls : BaseCls
{
}
void Driver()
{
    var sc = new SerializerConfig();
    sc.DefaultTargets = TargetMember.AllFields;
    sc.VersionTolerance.Mode = VersionToleranceMode.Standart;
    
    var ceras = new CerasSerializer(sc);
    TestCls tc = new TestCls();
    TestCls tcClone = ceras.Deserialize<TestCls>(ceras.Serialize(tc));
}

System.InvalidOperationException
Message: member not found

at Ceras.TypeConfig.GetMemberConfig(MemberInfo memberInfo)
at Ceras.CerasSerializer.REadScheme(Byte[] buffer, Int32& offset, Type type, Boolean isStatic)
...

Platform
Win7, net4.5, Version: 4.0.30.0

Bug: System.IndexOutOfRangeException: Index was outside the bounds of the array.

Describe the bug

My software tries to deserialize a previously serialized class. During deserialization, the exception System.IndexOutOfRangeException: Index was outside the bounds of the array. is raised.

Here's the backtrace:

 System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Ceras.SerializerBinary.ReadVarInt(Byte[] bytes, Int32& offset, Int32 bits)
   at Ceras.SerializerBinary.ReadUInt32Bias(Byte[] buffer, Int32& offset, Int32 bias)
   at Ceras.Formatters.ReferenceFormatter`1.Deserialize(Byte[] buffer, Int32& offset, T& value)
   at Ceras.CerasSerializer.Deserialize[T](T& value, Byte[] buffer, Int32& offset, Int32 expectedReadLength)
   at Ceras.CerasSerializer.Deserialize[T](Byte[] buffer)

My problem is that I don't know if the saved data has been corrupted or not, and in that case it's quite tricky to create a reproduce code because of quite many classes involved.

I could send over the serialized data file, but I don't know if it's much use without the involved classes.

Basically I'm trying to figure out if it's a bug caused by any change in my code, if it's a bug within Ceras or if the data has been corrupted. Is there any way to figure this out? Note that 200+ serialized files are loaded prior that exception following the same data structure. Also judging from the timestamps there doesn't seem to be a system crash which could have caused a corruption.

The serialized data file is only 2kb in size. How should I proceed with providing additional information?

How to reproduce the bug

• None yet, see above

Platform

• net4.7+
• Are you using Unity? no

[Feature Request] LZ4 compression

Hi,

I'm developing (as a hobby) a Tile-based 2D platformer game in Unity. At the moment I'm still creating some parts for a custom level editor. One of the more important issues in this project is data (de-)serialization:

I have a Level class which derives from ScriptableObject (in order to be able to edit certain values in the Unity Inspector). This class holds a Tile[,] array which gets serialized to a byte[] array / deserialized from it.

Currently I'm using neuecc's MessagePack serializer but if your benchmarks are still up to date I'd like to try out Ceras since I'm looking for maximum performance.

However, one thing Messagepack has is LZ4 compression. Since the byte[] array in my Level class needs to be serialized again using the default [SerializeField] attribute for the Unity serializer, reducing its size by applying LZ4 compression improves editor responsiveness immensely. Without compression I've even had the Unity editor freeze up/crash on me when the ScriptableObject of a large level is selected in the Project view (since apparently the Inspector deserializes the SO every frame to draw the fields, possibly applies changes and then serializes it again).

I saw that LZ4 is already on your list of planned features. Since I didn't find a schedule for these planned features my hope is that opening a feature request for this will help increase its priority ;)

My proposal would be to have an overload of your Serialize/Deserialize method with an optional parameter for the compression type. This way existing users can keep their syntax as is while adding something like "CompressionType.LZ4" as additional parameter applies the compression.

This would also allow changing it to "CompressionType.GZip" later (although GZip is not a priority for me since even with the smaller size resulting from it the compression would usually take longer than LZ4).

The default value could be "CompressionType.None" so that anyone who would need to could keep calling the same method overload and simply pass a variable with the appropriate CompressionType for their purposes.

If you need any more info let me know.

Cheers!

Sign Ceras library

The library cannot be currently referenced from a signed assembly:

System.IO.FileLoadException: 'Could not load file or assembly 'Ceras, Version=4.0.40.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from HRESULT: 0x80131044)'

HOWTO: Deserialize with Type parameter

Hi,
I need to do this:
object Deserialize(Type type, Stream source);
I try a lot of way but I don't find a solution.

A sample, from your sample:

` class TestCeras
{

    class Person { public string Name; public int Age; }
    static void Main(string[] args)
    {
        var p = new Person { Name = "riki", Age = 5 };
        var ceras = new CerasSerializer();
        var bytes = ceras.Serialize(p);

        Person res = (Person) Deserialize(typeof(Person), new MemoryStream(bytes));
    }

    private static object Deserialize(Type type, Stream source)
    {
        var ceras = new CerasSerializer();
        //var res = ceras.Deserialize<type>(source);
        return res;
    }

}`

Bug: Dictionary<int, int> Serialize fail

new CerasSerializer().Serialize(new Dictionary<int, int> { [1] = 1 });

Exception StackTrace (zh-tw):

System.Reflection.TargetInvocationException
  HResult=0x80131604
  Message=引動過程的目標傳回例外狀況。
  Source=mscorlib
  StackTrace: 
   於 System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   於 System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   於 System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   於 System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   於 System.Activator.CreateInstance(Type type, Object[] args)
   於 Ceras.Resolvers.CollectionFormatterResolver.GetFormatter(Type type)
   於 Ceras.CerasSerializer.GetSpecificFormatter(Type type, TypeMetaData meta)
   於 Ceras.Formatters.ReferenceFormatter`1.GetOrCreateEntry(Type type)
   於 Ceras.Formatters.ReferenceFormatter`1.Serialize(Byte[]& buffer, Int32& offset, T value)
   於 .................

內部例外狀況 1:
ArgumentException: 指定的類型不可以是泛型類型定義。
Arg_ParamName_Name

fail on Marshal.SizeOf(KeyValue<int,int>) in ReflectionHelper.IsBlittableType.
expcetion always happen when key and value are both ValueType.

Serialization of Actions, Funcs, delegates and events.

tl;dr: Static methods: 🆗 Everything else: WIP 🚧 (work in progress)

Can Ceras serialize my Action, Func, event field, ... ?

At the moment only references to static methods are supported.
That means:

Action myAction = SomeStaticClass.MyStaticMethod;

ceras.Serialize(myAction); // works fine

Static methods are a special case. It's trivial because the only thing one really needs to remember is the target method (and ceras already serializes MethodInfos).

What about serializing references to instance methods? Or lambda expressions?

Beyond the simple case things become pretty complicated pretty quickly.
So that's why I'm opening this issue:

• let people know about the limitations, and general things you should be aware of when trying to serialize delegates
• track the state of the "delegate serialization" feature in Ceras.

The main issue with delegates is that they drag in other stuff in a completely invisible way.
And not just that, often times the compiler will also add in completely unrelated variables to the hidden capturing class!

Example:

You'd think that the only thing that's getting captured is the 6.

• The 5 is already part of the generated code (because it is a static class and a static field).
• The 6 is a local variable, so it gets captured into a anonymous field.
• The 7 is part of the code itself, nothing to do here.

Now lets take a look at the actual code that's getting executed here:

Ok... so far so good.
But once we check out the compiler generated object (which holds those captured variables) we're in for a pretty big surprise.

Where does DelegateValueHolder suddenly come from?

DelegateValueHolder is a simple test class that is being used further down in my DelegatesTest() method, and it's completely unrelated to what we're doing.

So why is it being included in the generated class then? WTF?

The answer is: optimization. The compiler only generates one capture class for each method that is being used as a capture source.

So that means when serializing delegates of any kind, we can't even be sure that we're not accidentally capturing all sorts of crazy unrelated stuff. The compiler can (and will as we've seen) put whatever it wants into one big capture-object.

Now what does that mean for Ceras and delegate serialization?
It means that we cannot assume anything about how a delegate works or is constructed.
There are 4 things to know in order to construct (deserialize) an arbitrary delegate:

1. The type. That one is obvious, we need to know if our delegate is a Func<>, Action, MyCustomDelegateDefinition, System.EventHandler, or whatever else. All of them inherit from System.MulticastDelegate, which brings us to:
2. The invocation list. Since pretty much all delegates are MulticastDelegate, that means invoking one can actually cause multiple method calls (yes, that includes also includes any normal Action)! Anyone can call Delegate.Combine(a,b); and assign the result back to what looks like a simple direct method reference. So there's no way around that. Luckily myFunc.GetInvocationList(); gives us all targets.
3. The target object.
4. The target method.

The last two points also have their own intricacies since there are 3 different cases:

• Static: Target method is static. So the method is saved directly. The object is null. This is the most simple case, there are no possible side-effects and everything should always work perfectly fine.
• Instance: For instance methods, the method is saved directly, while the object becomes the object on which to call the method. This is where the first issues appear. If you save a delegate like this var my
• Lambda: This is where stuff gets complicated. The method is part of a hidden, compiler-generated class! And as we've seen it might include all sorts of references!

In the object-case Ceras is pretty much forced to include a reference to the object.
After all, how else could it construct the delegate when deserializing again?
So what would we do? Simply include the object in full? Most likely that target object has some fields and properties, which in turn reference other objects...
So that'd essentially drag in a huge graph of objects.
Obviously this is just asking for trouble.

In the lambda-case things are even more difficult.
Lambda functions, more often than not, capture multiple objects (not just one as in the object-case!).
And to make it even worse, some of the objects might even not be related to the delegate at all!
But wait! There's more!
Not only does the compiler give us some auto-generated object full of random methods and object references, it is of course also allowed to name all those hidden things however it likes.
Why this is a huge problem (with no clear or easy solution at all) is explained in an answer on StackOverflow; which basically says that any code change can throw off the namings (and thus make previously serialized data invalid). Even adding/removing/changing some other, completely unrelated, thing can (and will) break everything. (Link to the post)

Possible Solutions

So what can we do?

• Naive way
  Simply allowing every object to be dragged into the serialization graph is a sure-fire way to immediately get all sorts of bugs, and that is generously assuming that the serialization/deserialization itself won't already break in all sorts of places. So that one will obviously get us nowhere.

• Simply resolve by ID
  If we only consider delegates pointing to instance-methods (so no lambda-expressions!), we could just have a sort of resolver.
  The idea is really very simple and in fact it already works. If we mark the target object with the IExternalRootObject interface, then Ceras would simply use the given ID that the object provides. And at deserialization time Ceras gives us that ID again and we resolve the correct kind of object. I'll post an example of that soon (remind me if I don't 😄).
  Now while that works perfectly fine, it's not exactly obvious. So maybe some new implementation which does exactly the same but is just named differently...

• White-List?
  Maybe some sort of white-list on each delegate field could be used?
  When serializing and we're dealing with some crazy lambda expression, Ceras would use the white-list to check which of the fields in the hidden compiler-generated object it should include.

I'm not sure what's the best way to do this.
Maybe a good start would be to first collect all sorts of use-cases, so feel free to post here and tell me about what delegates you'd like to serialize, why, and how you'd expect Ceras to deal with references to the target object, or objects/variables.
Any ideas welcome :P

Trying to serialize Dictionary throws NullReferenceException

Hey,
I'm currently looking for a serializer to serialize and deserialize the data which gets generated by my Unity project. I previously had a look at neuecc's MessagePack but it has some limitations, mainly no support for circular references and the need to use attributes on all fields/properties. From some of the issues over there in which you seemed to have similar problems I found your Ceras serializer which looks promising.

I've tried out some simple things with it in a new empty project and found that circular references do indeed seem to work nicely (e.g. a "Node" class which has another Node as child).

One data type I need in my real project however is a ConcurrentDictionary<long, Node>. Trying to serialize/deserilize this kept throwing exceptions at me, even while simplifying the test project more and more. I finally arrived at the version below (SSCCE) which is just a simple Dictionary<int, string> but the exception persist...

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Ceras;

[ExecuteInEditMode]
[Serializable]
public class Main : MonoBehaviour
{
	#region Fields
	[SerializeField]
	private string name;
	[SerializeField]
	private int id;

	[NonSerialized]
	private Dictionary<int, string> simpleDict = new Dictionary<int, string>();
	[SerializeField]
	private byte[] serializedSimpleDict;

	[SerializeField]
	private bool doCreate;
	[SerializeField]
	private bool doSerialize;
	[SerializeField]
	private bool doPrint;
	[SerializeField]
	private bool doClear;
	[SerializeField]
	private bool doDeserialize;

	[NonSerialized]
	private CerasSerializer ceras;
	#endregion

	#region Properties
	
	#endregion

	#region Constructor
	
	#endregion

	#region Methods
	void Update()
	{
		if(doCreate)
		{
			simpleDict.Add(id, name);

			doCreate = false;
		}

		if(ceras == null)
		{
			SerializerConfig config = new SerializerConfig();
			config.DefaultTargets = TargetMember.AllFields;
			ceras = new CerasSerializer(config);

			Debug.Log("Created new CerasSerializer");
		}

		if(doSerialize)
		{
			int size = ceras.Serialize<Dictionary<int, string>>(simpleDict, ref serializedSimpleDict);
			Debug.Log("Serialized to " + size + " bytes");

			doSerialize = false;
		}

		if(doPrint)
		{
			Debug.Log(simpleDict.Count);
			foreach(KeyValuePair<int, string> kvp in simpleDict)
			{
				Debug.Log(kvp.Key + " = " + kvp.Value);
			}

			doPrint = false;
		}

		if(doClear)
		{
			simpleDict = new Dictionary<int, string>();

			doClear = false;
		}

		if(doDeserialize)
		{
			ceras.Deserialize<Dictionary<int, string>>(ref simpleDict, serializedSimpleDict);

			doDeserialize = false;
		}
	}
	#endregion
}

This is the error I'm getting:

NullReferenceException: Object reference not set to an instance of an object
Ceras.Formatters.ReferenceFormatter1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) (at Assets/Plugins/Ceras/Formatters/ReferenceFormatter.cs:155) (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure,byte[]&,int&,System.Collections.Generic.Dictionary2<int, string>)
Ceras.Formatters.DynamicObjectFormatter1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) (at Assets/Plugins/Ceras/Formatters/DynamicObjectFormatter.cs:229) (wrapper dynamic-method) System.Object.lambda_method(System.Runtime.CompilerServices.Closure,byte[]&,int&,System.Collections.Generic.Dictionary2<int, string>)
Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) (at Assets/Plugins/Ceras/Formatters/ReferenceFormatter.cs:155)
Ceras.CerasSerializer.Serialize[T] (T obj, System.Byte[]& buffer, System.Int32 offset) (at Assets/Plugins/Ceras/CerasSerializer.cs:338)
Main.Update () (at Assets/Scripts/Main.cs:66)

Steps to reproduce:

1. Create the above script in a Unity project
2. Attach it as a Component to a GameObject in the Scene
3. Enter some values, e.g. "Alpha" and 123
4. Toggle the "Do Create" bool (quick and dirty way to add a button for testing)
5. Check with "Do Print" that the item got added (see in Console)
6. Press "Do Serialize" and the above exception is thrown (remember to manually untoggle "Do Serialize" again!)

Using this setup:

• Unity 2018.3.3f1
• Scripting Runtime Version = ".NET 4.x Equivalent"
• Scripting Backend = " IL2CPP"
• Api Compatibility Level = ".NET 4.x"
• Latest Ceras commit (Feb 10, 2019, 5c10f30)

Am I missing something? Or are Dictionaries not serializable?

Primitive Types changing type after ser/des in Dictionary<string, object>

On .NET 4.7.2:

namespace NetFX
{
	class Program
	{
		static void Main(string[] args)
		{
			var data = new Dictionary<string, object>();
			data["test"] = 5;

			var s = new CerasSerializer();
			var desData = s.Deserialize<Dictionary<string, object>>(s.Serialize(data));

			Console.WriteLine(desData["test"].GetType());
			Console.ReadLine();
		}
	}
}

Output should be: System.Int32
Output actually is: System.Byte

Comparison with Odin serializer?

Hello! Firstly, thanks so much for Ceras; it has been much easier to use than other serializers for us. I was curious on its comparison with Odin serializer? We already use that for serializing for editing data in Unity, but we use Ceras for our network serialization. I was curious if you had any input between the two!

https://github.com/TeamSirenix/odin-serializer

Bug: VersionToleranceMode.Standard causes deserialization to fail

First of all, thank you for Ceras!

Describe the bug

When using VersionToleranceMode.Standard, the deserialization fails with

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

This only happens when deserializing the second Plugin. A workaround would be to construct a new instance of the serializer after each deserialization.

How to reproduce the bug

namespace Repro
{
    public class PluginLocation
    {
        public string PluginName { get; set; }
    }

    public class Plugin
    {
        public PluginLocation PluginLocation { get; set; }
    }

    public class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            var plugin = new Plugin();
            var plugin2 = new Plugin();

            var serializerConfig = new SerializerConfig();
            serializerConfig.VersionTolerance.Mode = VersionToleranceMode.Standard;
            var s = new CerasSerializer(serializerConfig);

            var data = s.Serialize(plugin);
            var data2 = s.Serialize(plugin2);

            s.Deserialize<Plugin>(data);
            s.Deserialize<Plugin>(data2); // This will fail
        }
    }
}

Platform

• net4.7.2

Fix Unity formatter addons

From #26:

I just had to remove a few "ref" keywords in the "Ceras.UnityAddon/Extension" class where "WriteFloat32FixedNoCheck" is called.

As a small side note regarding that Extension: It might be good to use the "#if UNITY_2017_2_OR_NEWER" / "#endif" preprocessor directives at the "#region 2017.2" / "#endregion" lines in order to prevent compiler errors with older Unity versions ;)

TODO:

• Fix compiler errors
• Add if checks to support older unity versions

Will probably be fixed together with the new release 👌 (in a few days)

Bug: Error when serializing structs

We are getting "Index was outside the bounds of the array" exception when serializing structs. I've tried many versions, also latest 4.0.36.
You can reproduce this with this code:

#r "nuget:Ceras/4.0.36"

using Ceras;

public struct Price
{
    public decimal Value { get; set; }
    public string Curreny { get; set; }
}
CerasBufferPool.Pool = new CerasDefaultBufferPool();
var p = new Price{ Value = 1.2m, Curreny = "TRY"};

var serializer = new Ceras.CerasSerializer(new SerializerConfig { PreserveReferences = true });

var s = serializer.Serialize(p);

Here is the stacktrace :

   at Ceras.SerializerBinary.FastCopy(Byte[] sourceArray, Int32 sourceOffset, Byte[] targetArray, Int32 targetOffset, Int32 n) in C:\projects\ceras\src\Ceras\SerializerBinary.cs:line 462
   at Ceras.CerasSerializer.Serialize[T](T obj) in C:\projects\ceras\src\Ceras\CerasSerializer.cs:line 359
   at async Program.<Initialize>(?) in :line 15
   at Program.<Main>()

If I try with 4.0.26 version it can't serialize and returns zero byte array. Also If I convert "Price" struct to a class, it can serialize and deserialize successfully.

Support for Optional Field Attribute

Ignore and Include attributes are great features, but for some cases, optional field attribute is required. Are you planning to support it for future releases?

Unity IL2CPP Guide outdated?

Hi again,

I've recently started implementing some new systems for my Unity game (custom sprite renderer/animator) and wanted to try building a first executable version. After receiving a blank screen in it I turned on "Development Build" and "Autoconnect Profiler" in the Build Settings and discovered a few exceptions being thrown related to Ceras.

So I started reading your guide for IL2CPP but I'm guessing some of the steps are a bit outdated?

In step 1 I found the setting in question under "config.Advanced.AotMode". So far so good.

However, in step 2 the [CerasAutoGenFormatter] attribute seems to have changed? If I add it to my Tile struct I get these errors:

Assets\Scripts\Level\Tile\Tile.cs(7,2): error CS0246: The type or namespace name 'CerasAutoGenFormatterAttribute' could not be found (are you missing a using directive or an assembly reference?)

Assets\Scripts\Level\Tile\Tile.cs(7,2): error CS0246: The type or namespace name 'CerasAutoGenFormatter' could not be found (are you missing a using directive or an assembly reference?)

Step 3 is a bit unclear: "Compile your assembly". Which assembly? Usually Unity compiles everything automatically so if there's anything extra needed in this case a more indepth step-by-step explanation would be helpful ;)
About the AotGenerator.exe: Is it included somewhere already or do I need to build this myself (is this what you mean by "Compile your assembly")?

In step 4 GeneratedFormatters.UseFormatters(config); doesn't seem to work. However, CerasUnityFormatters.ApplyToConfig(config); from here does work. Is this the new/renamed version of that method?
And is the Extension.cs file still needed? I think I remember you saying something about rebuilding some stuff to handle the Unity types automatically?

Including the link.xml from step 5 is easy enough.

Any help with this would be much appreciated!
Cheers :)

Serializing 'MemberInfo' does not work as intended

As the title says, MemberInfo and all classes that inherit from it (FieldInfo, MethodInfo, ...) won't get serialized correctly.

Ceras internally serializes those, but it does so using a different mechanism.
Which means that user-classes that have a field like public FieldInfo MyField; will have problems.

It shouldn't be too hard to fix it though! I'm working on it...

I open this issue just to inform everyone else who might have had problems with those types.

Support for read-only properties ?

Hello,

I wanted to check out the library real quick and scratched my head for a good while, and it turned out it didn't serialise read-only properties.
I did a dirty search and replace to add private setters everywhere to help the library out and it worked !

Would it be possible to modify these read-only properties through reflection by retrieving the backing field somehow ?

Serialization format

Hello Riki, your serializer looks very promising!

I'm wondering whether its physical format stable enough to be standardized?
Would be great to have a wire-compatible javascript version of the .NET serializer.
Existing formats like Json, Bson or MessagePack are quite limited (don't support circular references, etc).

Regards, Alex

Dictionary within objet[] within Dictionary within Dictionary causing exception

class Program
{
	static void Main(string[] args)
	{
		var data = new Dictionary<string, object>
		{
			["test"] = new Dictionary<string, object>
			{
				["test"] = new object[]
				{
					new Dictionary<string, object>
					{
						["test"] = 3
					}
				}
			}
		};

		var s = new CerasSerializer();

		//  throws here
		s.Deserialize<Dictionary<string, object>>(s.Serialize(data));
	}
}

System.Exception: 'Cannot deserialize type 'System.String' because it has no parameterless constructor (support for serialization-constructors will be added in the future)'

More Tests

Some features could definitely use some more tests:

• Version Tolerance

  • should get a few more tests on its own
  • maybe there's some way we can repeat every normal test again, but with version tolerance turned on
  • there are some example tests that check the basic functionality, but no edge-cases yet. There are a lot of comments in the code (SchemaFormatter.cs, TypeFormatter.cs, ReferenceFormatter.cs, ...) that explain what could happen and what should be tested...

• Type Binder (scan the binary, check if actually used correctly)

• Ensure that having delegates which are not marked as [Ignore] throws an exception (until #11 is done)

• Tests for all network related stuff. Like checksums, ensuring KnownTypes works correctly, ...

• Whatever else that can be tested! Your ideas count, post them here :)

Version tolerance, optimization and "tainting" of formatters.

This issue is supposed to tell you about upcoming changes to the VersionTolerance feature, and why they're happening. Those changes will be a "breaking change" (so the new Ceras version won't be able to load any data serialized using the old Ceras version).

In case you are not really interested in WHY this is happening, and you don't mind the breaking change, then you can just stop reading here since knowing that a breaking change is coming is probably enough for you.

All of the below points play together to produce an interesting problem (of which the solution requires some big changes). So for those of you who are not fully aware of how Ceras works internally I wrote these short introduction segments that tell you what you need to know to understand the problem:

Intro

So when writing an object with the VersionTolerance feature Ceras embeds type and field information so we later know how to read that data.
When some fields got added, removed or renamed, Ceras can use that embedded data to still read the data, even though the type is now different.

Optimization

In order to read and write user-objects faster, Ceras dynamically compiles new formatters (called a DynamicFormatter). So when you have a Person object that contains a string Name; and a int Health; Ceras will compile a formatter that takes a person object and writes (or reads) those two things.
That improves speed dramatically as there's no need for reflection, no need to go through any "list of fields" or anything like that. It's as if Ceras already had built-in support for your custom object type.

The important thing to know is that if your class has a reference to some other class, that will be handled well by simply creating another DynamicFormatter that handles this other type.

The actual problem

So since we want things to go as fast as possible we create a DynamicFormatter once per type and re-use that of course. No point in re-compiling the same formatter again and again, right?

Now imagine we want to serialize some Person object (any user object really).
Ceras will generate the DynamicFormatter for it since it is a unknown/custom/user-type.
We serialize the object using the dynamic formatter we just compiled and all is well.

Next we want to read some previously serialized data.
This previously serialized data is pretty old and from a time where some fields where missing from the object (compared to how our Person class looks today).
We read the schema of the type and see it's one that we already know.
But wait a second, the dynamic formatter we've compiled already can't be used, it turns out the schema in the data describes an object that looks different even though it has the same type name.
Well not a surprise because it is an older version of the type.

Uh oh, what now, would we need to throw out the dynamic formatter we already have?
We could just put all our dynamically created formatters into some sort of cache together with the schema the formatter was made for, that'd sorta solve it.

But it turns out that this isn't even our worst problem.
There are other formatters that might rely on this formatter already.
Maybe you have a House-class in your code that contains references to multiple Person objects.
The DynamicFormatter for House obviously needs to use the DynamicFormatter for Person.

Maybe you already noticed the problem here. If the House-Formatter already has a reference to the Person-Formatter, then that can't be changed anymore.
So now the House-Formatter is invalid as well! Because it is indirectly dealing with the changed/updated Person as well...

In the above example formatters become "tainted" by the schema they are using.
And all the other formatters that use a tainted formatter thus become tainted by the schema as well.
For dynamic formatters things get especially complicated as the classes they're dealing with often have multiple fields (references) to different objects. So one dynamic formatter might become tainted by its own schema, as well as by all the many other schemata that it indirectly references.
And then some formatters might refer to themselves as well...

Solution

So what can we do?
When a schema for some type changes (like when we're reading some old data and it tells us about the old schema that was used to originally write that data) we can notify the formatters that the global schema for a certain type has now changed, so they can react and update their inner workings accordingly.
Doing that isn't all that easy, as not every formatter is interested in every schema-change, and we can't really inform everyone all the time on every little schema-change; that would absolutely throw performance out the window.

Consequences

• The way schema-data will be embedded in Ceras' AutomaticEmbedded mode will drastically change, breaking binary-compatibility with older Ceras versions.
• Serialization and Deserialization when using VersionTolerance will generally become a bit faster.
• Serialization and Deserialization now supports reading old mixed-version data.

Bug: AotGenerator outputs broken code

Hi,

I'm trying to use Ceras for a TCP client in a Unity app running on a Microsoft HoloLens (UWP).
I'm following this guide and I have problems at step 3) when generating formatters with AotGenerator.exe.
I reduced my problems to a simple case:

Describe the bug

• What are you trying to do?
  Generating formatters for this class with AotGenerator.exe:

using Ceras.Formatters.AotGenerator;
using System.Collections.Generic;

[CerasAutoGenFormatter]
public class TestClass {
    public int l1;
    public int l2;
    public List<int> l;

    public TestClass() { }

    public TestClass(int l1, int l2, List<int> l) {
        this.l1 = l1;
        this.l2 = l2;
        this.l = l;
    }
}

• What do you think should happen?
  resulting cs file should be syntacticly correct and working.

• What is actually happening?

using Ceras;
using Ceras.Formatters;
namespace Ceras.GeneratedFormatters
{

    static class GeneratedFormatters
    {
        internal static void UseFormatters(SerializerConfig config)
        {
            config.ConfigType<TestClass>().CustomFormatter = new TestClassFormatter();
        }
    }

    internal class TestClassFormatter : IFormatter<TestClass>
    {
        IFormatter<System.Collections.Generic.List`1[[System.Int32, mscorlib, Version = 4.0.0.0, Culture = neutral, PublicKeyToken = b77a5c561934e089]]> _list`1Formatter;
 IFormatter<System.Int32> _int32Formatter;
        IFormatter<System.Int32> _int32Formatter;

        public void Serialize(ref byte[] buffer, ref int offset, TestClass value)
        {
            _list`1Formatter.Serialize(ref buffer, ref offset, value.l);
            _int32Formatter.Serialize(ref buffer, ref offset, value.l1);
            _int32Formatter.Serialize(ref buffer, ref offset, value.l2);
        }

        public void Deserialize(byte[] buffer, ref int offset, ref TestClass value)
        {
            _list`1Formatter.Deserialize(buffer, ref offset, ref value.l);
            _int32Formatter.Deserialize(buffer, ref offset, ref value.l1);
            _int32Formatter.Deserialize(buffer, ref offset, ref value.l2);
        }
    }

}

Errors relating to naming ambiguity caused by "_int32Formatter" and broken syntax from
"_list`1Formatter".

How to reproduce the bug

• Create new Unity 3D project
• Add Ceras according to here
• Create TestClass.cs in Assets folder (code see above)
• build project (for build settings see screenshot)
  
• run Ceras.AotGenerator.exe <UNITY_BUILD_DIRECTORY>\Test\Managed\Assembly-CSharp.dll AotGeneratedFormatters_Test.cs

Platform

• Which version of Ceras are you using? net4.5, net4.7+, or netstandard2.0?
  Ceras4.7 and AotGenerator from here

• Are you using Unity? If so, what version? (2018.1, 2018.2, ...)
  2019.1.9f1

• If you are using Unity, are you using IL2CPP?
  yes

I can fix the file manually for now, but it would be cool to get this working.

Best regards

Optimization: Merge formatters in DynamicObjectFormatter

Assume we have a class like class C { int a,b,c,d,e,f; }.

When Ceras uses its DynamicObjectFormatter to handle it, there will be 6 separate constants emitted all for the same Int32Formatter instance.

That could be optimized a bit by merging formatters of the same type.

GenerateSerializer and GenerateDeserializer are affected. Also SchemaDynamicFormatter

have demo for network?

now i use protobuf in post request and recive request (serialize object to outputStream and deserialize from inputStream):

ProtoBuf.Serializer.NonGeneric.Serialize(toStream, object);

ProtoBuf.Serializer.NonGeneric.Deserialize(type, fromStream);

if change to ceras,does performance is good then protobuf?

Bug: Trying to serialize System.Drawing.Color throws exception

If a class has any field with type System.Drawing.Color, CerasSerializer.Serialize throws Exception:

Expression of type 'System.Drawing.Color' cannot be used for parameter of type 'System.Object' of method 'Void SetValue(System.Object, System.Object)

I am using version 4.0 rc, .Net 4.5, to reproduce the bug, try to serialize the following class with following Config;

var sc = new SerializerConfig();
sc.DefaultTargets = TargetMember.AllFields;
sc.VersionTolerance = VersionTolerance.AutomaticEmbedded;

var ceras = new CerasSerializer(sc);
ceras.Serialize(new MyClass());

public class MyClass
{
        private System.Drawing.Color backColor;
        public System.Drawing.Color BackColor { get {return backColor;} set{ backColor = value;} }
}

DataContractSerializer compatibility?

hi,
any plan make DataContractSerializer compatibility?
like that AArnott/MessagePack-CSharp#22

Bug: Bitmap deserialization problem

When you serialize an object that has a bitmap type field and deserialize it back sequentially, it works great, but when you serialize it to a file and start a new application to deserialize it back, it throws System.ArgumentException, probably it uses some static information for deserialization, this is why it fails with a new application ( just a guess)

To reproduce the bug:

static void Test_Serialize()
{
	var element = new Element() { Id = 5, Name = "elem1" };
	element.pic = new Bitmap(@"C:\Users\eataseven\Documents\asd.png");

	SerializerConfig sc = new SerializerConfig();
	sc.DefaultTargets = TargetMember.AllFields;
	sc.Advanced.BitmapMode = BitmapMode.SaveAsBmp;
	sc.Advanced.RespectNonSerializedAttribute = false;
	sc.VersionTolerance.Mode = VersionToleranceMode.Standard;
	sc.VersionTolerance.VerifySizes = false;

	var ceras = new CerasSerializer(sc);
	var data = ceras.Serialize(element);
	File.WriteAllBytes("elem.bin", data);
}

static void Test_Deserialize()
{
	SerializerConfig sc = new SerializerConfig();
	sc.DefaultTargets = TargetMember.AllFields;
	sc.Advanced.BitmapMode = BitmapMode.SaveAsBmp;
	sc.Advanced.RespectNonSerializedAttribute = false;
	sc.VersionTolerance.Mode = VersionToleranceMode.Standard;
	sc.VersionTolerance.VerifySizes = false;

	var ceras = new CerasSerializer(sc);
	var element = ceras.Deserialize<Element>(File.ReadAllBytes("elem.bin"));
}

static void Main(string[] args)
{
	// works if run sequential
	Test_Serialize();
	Test_Deserialize();

	// does not work if 
	// comment out Test_Deserialize
	// run Test_Serialize only (elem.bin is generated)

	// comment out Test_Serialize, uncomment Test_Deserialize
	// run Test_Deserialize only 
        // throws exception
}

An unhandled exception of type 'System.ArgumentException' occurred in System.Drawing.dll
Additional information: Parameter is not valid.
at System.Drawing.Bitmap..ctor(Stream stream)
at Ceras.Formatters.BitmapFormatter.Deserialize(Byte[] buffer, Int32& offset, Bitmap& img)
at lambda_method(Closure , Byte[] , Int32& , Bitmap& )
at Ceras.Formatters.ReferenceFormatter1.Deserialize(Byte[] buffer, Int32& offset, T& value) at lambda_method(Closure , Byte[] , Int32& , Element& ) at Ceras.Helpers.SchemaDynamicFormatter1.Deserialize(Byte[] buffer, Int32& offset, T& value)
at lambda_method(Closure , Byte[] , Int32& , Element& )
at Ceras.Formatters.ReferenceFormatter`1.Deserialize(Byte[] buffer, Int32& offset, T& value)
at Ceras.CerasSerializer.Deserialize[T](T& value, Byte[] buffer, Int32& offset, Int32 expectedReadLength)
at Ceras.CerasSerializer.Deserialize[T](Byte[] buffer)
at CerasSerializerTest.Program.Test_Deserialize() in C:\Users\eataseven\Documents\Visual Studio 2010\Projects\GMHTestSuite\CerasSerializerTest\Program.cs:line 668
at CerasSerializerTest.Program.Main(String[] args) in C:\Users\eataseven\Documents\Visual Studio 2010\Projects\GMHTestSuite\CerasSerializerTest\Program.cs:line 677
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Platform:
Ceras 4.0.26 - net4.5 - Win7

Better performance

I performed a benchmark against MessagePack.

    public class TradeProp
    {
        public string Ticker { get; set; }
        public bool IsValid { get; set; }
        public DateTime DateAndTime { get; set; }
        public float Price { get; set; }
        public uint Size { get; set; }
        public EExchange Exchange { get; set; }
        public ETradeCondition Condition { get; set; }
        public bool HasQuotes { get; set; }
        public QuoteProp Nbbo { get; set; }
    }

    public class QuoteProp
    {
        public string Ticker { get; set; }
        public DateTime DateAndTime { get; set; }
        public float Bid { get; set; }
        public uint BidSize { get; set; }
        public EExchange BidExchange { get; set; }
        public float Ask { get; set; }
        public uint AskSize { get; set; }
        public EExchange AskExchange { get; set; }
    }

Collection : 142,725 trades
Iteration : 10

-----------------------     MESSAGE PACK - TypelessContractless
 - size (byte) => 27,573,212
 - serialize (ms) => avg: 124 | min: 89 | max: 188
 - deserialize (ms) => avg: 219 | min: 158 | max: 471

-----------------------     MESSAGE PACK (LZ4) - TypelessContractless
 - size (byte) => 2,188,703
 - serialize (ms) => avg: 101 | min: 95 | max: 112
 - deserialize (ms) => avg: 192 | min: 168 | max: 253

-----------------------     CERAS (prop) - Without Attributes
 - size (byte) => 5,870,497
 - serialize (ms) => avg: 189 | min: 177 | max: 220
 - deserialize (ms) => avg: 263 | min: 238 | max: 337

-----------------------     CERAS (field) - Without Attributes
 - size (byte) => 5,870,497
 - serialize (ms) => avg: 167 | min: 154 | max: 206
 - deserialize (ms) => avg: 216 | min: 202 | max: 225

Collection : 1,340,724 trades
Iteration : 10

-----------------------     MESSAGE PACK - TypelessContractless
 - size (byte) => 259,002,992
 - serialize (ms) => avg: 869 | min: 822 | max: 1,172
 - deserialize (ms) => avg: 2,084 | min: 1,713 | max: 2,268

-----------------------     MESSAGE PACK (LZ4) - TypelessContractless
 - size (byte) => 20,851,685
 - serialize (ms) => avg: 890 | min: 881 | max: 907
 - deserialize (ms) => avg: 2,074 | min: 1,869 | max: 2,592

-----------------------     CERAS (prop) - Without Attributes
 - size (byte) => 59,315,386
 - serialize (ms) => avg: 1,931 | min: 1,900 | max: 2,031
 - deserialize (ms) => avg: 3,038 | min: 2,442 | max: 3,736

-----------------------     CERAS (field) - Without Attributes
 - size (byte) => 59,315,386
 - serialize (ms) => avg: 1,688 | min: 1,650 | max: 1,760
 - deserialize (ms) => avg: 2,655 | min: 2,041 | max: 3,575

Cannot resolve types in referenced assemblies

While deserialization, if the reference assembly is not resolved yet, Ceras throws a System.Exception.
Additional information: Cannot find type X after searching in all user provided assemblies and all loaded assemblies...

Maybe it is a normal behaviour of deserializer, but not sure. I came through with 3 different (not so cool) approach

1. I generated a dummy object with this type before deserialization and it worked.
2. I traversed the dependency hierarchy using a graph traversal logic and load all assemblies manually
   with Assembly.Load
3. I tried to load all .dll files in the executable folder with Assembly.Load (in a try-catch scope)

Do I miss something?

Not serializing types properly

The serializer is serializing all of the fields on one of my types to null. I'm sure I'm missing something basic here, but I would love some help debugging this.

serialize/deserialize with property

Hi,

With 2 object.

• TradeFields: only public Fields
• TradeProp: only public Properties

Serialize and desiralize only one object.

With fields its perfect, but with prop I obtains only 1 byte serialized.
without exception

    public class TradeProp
    {
        public string Ticker { get; set; }
        public bool IsValid { get; set; }
        public DateTime DateAndTime { get; set; }

        public float Price { get; set; }
        public uint Size { get; set; }
        public EExchange Exchange { get; set; }
        public ETradeCondition Condition { get; set; }

        public bool HasQuotes { get; set; }

        public TradeProp() { }
    }

    public class TradeFields
    {
        public string Ticker;
        public bool IsValid;
        public DateTime DateAndTime;

        public float Price;
        public uint Size;
        public EExchange Exchange;
        public ETradeCondition Condition;

        public bool HasQuotes;

    }

performance improvement

hi
maybe you want follow that
AArnott/MessagePack-CSharp#27

How is Ceras with Obfuscation?

Currently I am using protobuf.net for my classes, I typically use the Protomember attribute for versioning my classes/fields, which works well with obfuscation.

How does Ceras's Version-Tolerance stand up to obfuscation?

Convert existing tests in "LiveTesting.cs" to actual tests

LiveTesting.cs has a ton of tests and examples, all of them use Debug.Assert.
We should convert them to actual tests in the test project.

Problems with inherited properties with initializers?

So to explain the bad title:
On Ceras 3.0.2 let's say I have these classes:

public abstract class Person
{
    public List<Person> Friends { get; private set; } = new List<Person>();
}

public class Adult : Person
{
    public byte[] Serialize() => new CerasSerializer().Serialize<object>(this);
}

and later on my program modifies the Friends list. Unfortunately these don't seem to get serialized. I'm only getting an empty list on deserialization. Maybe this is related #9 (comment) ? So I tried this: (lazily construct the collection)

public abstract class Person
{
    private List<Person> _friends; 
    public List<Person> Friends
    {
        get => _friends ?? (_friends = new List<Person>());
        private set => _friends = value;
    }
}

public class Adult : Person
{
    public byte[] Serialize() => new CerasSerializer().Serialize<object>(this);
}

It works that way, and stuff gets demonstrably serialized. But weird thing is that this works when it's not an inherited property:

public class Person
{
    public List<Person> Friends { get; private set; } = new List<Person>();

    public byte[] Serialize() => new CerasSerializer().Serialize<object>(this);
}

I'd like to conduct further tests but I'm on deadline and I'm fine with this workaround for now.

Bug: AOT generated formatters are not used in AOT mode

Remember: The more faster I can reproduce your problem, the faster I can fix it :P

Describe the bug

• What are you trying to do?
  Use generated formatters for classes in AOT mode, e.g.:

[CerasAutoGenFormatter]
public class TestAot
{
	public int anInt;
}

• What do you think should happen?
  The generated formatter should get used automatically when trying to serialize the TestAot class, and I should be able to serialize an instance of it in AOT mode. I'd also be fine with needing to manually call UseFormatters on the generated class if that would make sense.

• What is actually happening?
  When I try to serialize aTestAot object, I get: InvalidOperationException: No formatter for the Type 'TestAot' was found. Ceras is trying to fall back to the DynamicFormatter, but that formatter will never work in on AoT compiled platforms. Use the code generator tool to automatically generate a formatter for this type.

As far as I can tell, the generated script's UseFormatters(SerializerConfig config) function is never called. If I change the generated class to be public and call it manually, I get a different error when trying to serialize:

ArgumentNullException: Value cannot be null.
Parameter name: method
System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) (at <d7ac571ca2d04b2f981d0d886fa067cf>:0)
System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method) (at <d7ac571ca2d04b2f981d0d886fa067cf>:0)
Ceras.Formatters.ReferenceFormatter`1[T].CreateSpecificSerializerDispatcher_Aot (System.Type type, Ceras.Formatters.IFormatter specificFormatter) (at <7a851d90e8b74dd689c0d63616010e91>:0)
Ceras.Formatters.ReferenceFormatter`1[T].GetOrCreateEntry (System.Type type) (at <7a851d90e8b74dd689c0d63616010e91>:0)
Ceras.Formatters.ReferenceFormatter`1[T].Serialize (System.Byte[]& buffer, System.Int32& offset, T value) (at <7a851d90e8b74dd689c0d63616010e91>:0)
Ceras.CerasSerializer.Serialize[T] (T obj, System.Byte[]& buffer, System.Int32 offset) (at <7a851d90e8b74dd689c0d63616010e91>:0)
Ceras.CerasSerializer.Serialize[T] (T obj) (at <7a851d90e8b74dd689c0d63616010e91>:0)

How to reproduce the bug
Use the above test class and generate an AOT formatter using the AotGenerator tool. Here is my test code:

var config = new Ceras.SerializerConfig();
config.Advanced.AotMode = Ceras.AotMode.Enabled;

// This is what I mean by calling this function manually.
// You need to modify the generated code for it to be accessible.
// Ceras.GeneratedFormatters.GeneratedFormatters.UseFormatters(config);

var serializer = new Ceras.CerasSerializer(config);
var o = new TestAot { anInt = 1 };
var bytes = serializer.Serialize(o);
o = serializer.Deserialize<TestAot>(bytes);

Platform

• Which version of Ceras are you using? net4.5, net4.7+, or netstandard2.0?
  netstandard2.0

• Are you using Unity? If so, what version? (2018.1, 2018.2, ...)
  2018.3.14f1

• If you are using Unity, are you using IL2CPP?
  No, I'm doing this in the editor

how reuse CerasSerializer to Improve performance?

I know that CerasSerializer is not thread safe.
If I apply to SOA, every request is made to create a Serializer, I feel that there is a lot of performance waste.
How can I avoid doing duplicate things every request?
For example, dynamically generate Lambda for DynamicFormatter<T>, get specific IFormatter from CerasSerializer.GetFormatter<T>, etc...

Is it possible to reuse the same CreasSerializer instance?
Use lock?.....I feel worse...
Use pool?.....

Any suggestions?

Allow serialization of an invocation list with more than one target?

Hello! I was curious on why Ceras can currently only serialize delegates with an invocation length of one. Would it be difficult or impossible to allow multiple?

Also, it would be great to put the name of the delegate (like you do in ThrowStatic and ThrowInstance) so you know what delegate it is!

TypeBinder example

There's an ITypeBinder mentioned in the readme, the example it gives there should be actually shown in the tutorial code.

todo: write a short tutorial that shows how to do it, explains how it works and why it makes sense to have that (granted, the last two things are already present in the readme)

Non-generic implementers of ICollection<> throw exception on serialization

Hi all!

I recently updated to 3.0.2 and now GetGenericTypeDefinition() is called before checking that if it's really a generic type or not in CollectionFormatterResolver.

I could narrow down the issue where I have a non-generic class let's say MultiLangText : IDictionary<string, string> which in turn also inherit from ICollection<KeyValuePair<string, string>>. At

Ceras/src/Ceras/Resolvers/StandardResolvers.cs

Line 128 in ea17d0f

if (type.GetGenericTypeDefinition() == typeof(List<>))

type seems to be assumed to be generic and that throws an exception during serialization.

here's a related stacktrace

at System.RuntimeType.GetGenericTypeDefinition()
   at Ceras.Resolvers.CollectionFormatterResolver.GetFormatter(Type type)
   at Ceras.CerasSerializer.GetSpecificFormatter(Type type, TypeMetaData meta)
   at Ceras.Formatters.ReferenceFormatter`1.GetOrCreateEntry(Type type)
   at Ceras.Formatters.ReferenceFormatter`1.Serialize(Byte[]& buffer, Int32& offset, T value)
   at lambda_method(Closure , Byte[]& , Int32& , ContentModelRoot )
   at Ceras.Formatters.ReferenceFormatter`1.Serialize(Byte[]& buffer, Int32& offset, T value)
   at Ceras.CerasSerializer.Serialize[T](T obj, Byte[]& buffer, Int32 offset)
   at Ceras.CerasSerializer.Serialize[T](T obj)

Writing a custom formatter for my problematic types successfully worked around the issue but that should be temporary.

Cheers!

Bug: AOT generator creates duplicate field names when target class has fields of same type

Remember: The more faster I can reproduce your problem, the faster I can fix it :P

Describe the bug

• What are you trying to do?
  Generate formatters for classes that have multiple fields of the same type, e.g.

[CerasAutoGenFormatter]
public class TestWithSameTypeFields
{
	public int anInt;
	public int anotherInt;
}

which generates:

internal class TestWithSameTypeFieldsFormatter : IFormatter<TestWithSameTypeFields>
{
	IFormatter<int> _intFormatter;
	IFormatter<int> _intFormatter;

	public void Serialize(ref byte[] buffer, ref int offset, TestWithSameTypeFields value)
	{
		_intFormatter.Serialize(ref buffer, ref offset, value.anInt);
		_intFormatter.Serialize(ref buffer, ref offset, value.anotherInt);
	}

	public void Deserialize(byte[] buffer, ref int offset, ref TestWithSameTypeFields value)
	{
		_intFormatter.Deserialize(buffer, ref offset, ref value.anInt);
		_intFormatter.Deserialize(buffer, ref offset, ref value.anotherInt);
	}
}

which doesn't compile because it has two fields called _intFormatter.

• What do you think should happen?
  In the generated class, there should be one formatter field per type, not per target class field

How to reproduce the bug
Use the example class and generate a formatter for it with the AotGenerator tool.

Platform

• Which version of Ceras are you using? net4.5, net4.7+, or netstandard2.0?
  netstandard 2.0

• Are you using Unity? If so, what version? (2018.1, 2018.2, ...)
  2018.3.14f1

• If you are using Unity, are you using IL2CPP?
  Not yet

support Exception serialize/deserialize?

I want serialize/deserialize Exception and derived types.
The initial idea is implement ExceptionFormatter, like below.

public sealed class ExceptionFormatter<T> : IFormatter<T> where T : Exception {
   public void Serialize(ref byte[] buffer, ref int offset, T value) {
      var binaryFormatter = new BinaryFormatter();
      ... append byte[] generated by binaryFormatter to the buffer...
   }
   public void Deserialize(byte[] buffer, ref int offset, ref T value) {
      var binaryFormatter = new BinaryFormatter();
      ... use the binaryFormatter to read the buffer to deserialize value...
   }
}

BinaryFormatter seems too fat.
Did I miss the reference Exception?
Any suggestions?

thanks.