akkadotnet / hyperion Goto Github PK
View Code? Open in Web Editor NEWPolymorphic serialization for .NET
License: Apache License 2.0
Polymorphic serialization for .NET
License: Apache License 2.0
We would like to use Hyperion in our project and I see that the dev branch has a .NET Core compatible version (0.9.5). Would it be possible to push it on nuget ?
Even if it is not fully tested, it would be useful to be able to use Hyperion without the hassle of cloning the dev branch and publishing to a local nuget server.
dotnet xunit
is out and allows passing -teamcity
as an argument to force TeamCity formatting. This wasn't available in dotnet test
so there's been the chance that we'd get false positives in our Mono Builds in TeamCity.
dotnet xunit
installation notes:
https://xunit.github.io/docs/getting-started-dotnet-core.html
While working on NBench tests I've seen few places, where objects serialization ends up with stack overflow, namely:
LinkedList<>
serialization/deserialization. This will be tricky as invalid behavior is visible only when nbench tests will be compiled and run under Release configuration. On Debug it's working fine. (Edit: fixed by #6 ).Dictionary<byte, char>
(upon deserialization value 'z' was deserialized to '\0' instead, the track of a bug followed back to IlCompiler) (Edit: fixed by #4 ).Setting serializer to use preserveObjectReferences
option haven't changed anything there.
Regarding LinkedList<>
probably the best option will be to implement custom serializer - conceptually this data structure is simple, but internal implementation (used by the serializer) is complex an may cause problems.
Should push to the same MyGet feed that is used for the Akka.NET nightly builds: http://getakka.net/docs/akka-developers/nightly-builds
cc @heynickc
Minor, but can we have a nuget icon.
Do u want a PR? I find https://thenounproject.com a good source
I went with https://thenounproject.com/term/beetle/861510/ for https://github.com/SimonCropp/NServiceBus.Hyperion
This should be a separate test class - idea is to have a project resource (binary file) with some well known data structure that is has been serialized with Hyperion in a specific version (starting from v0.9.2) and deserialize it with current /dev to make sure that all fields have been deserialized correctly and no exception has occurred. This should be a typical generic class containing several fields including collections.
This doesn't need to mean, that starting from now we're going to maintain backward compatibility, but that we want to make those changes explicit.
/cc #28
Because there is not SERIALIZATION
build constant anymore.
There is also no tests related with ISerializable: https://github.com/akkadotnet/Hyperion/blob/dev/Hyperion.Tests/ISerializableTests.cs
I created a PR with failing unit test exposing the problem: #65
Starting test execution, please wait...
[xUnit.net 00:00:00.5834487] Discovering: Hyperion.Tests
[xUnit.net 00:00:00.8800229] Discovered: Hyperion.Tests
[xUnit.net 00:00:01.5499118] Starting: Hyperion.Tests
[xUnit.net 00:00:01.9438699] Hyperion.Tests.ISerializableTests.SkipNonSerializedAttributeIfNoSerilizedAttribute [SKIP]
[xUnit.net 00:00:01.9442333] Not implemented yet
[xUnit.net 00:00:01.9531298] Hyperion.Tests.ISerializableTests.ShouldNotThrowIfNoConstructorWithSerializationInfo [SKIP]
[xUnit.net 00:00:01.9533130] Not implemented yet
Skipped Hyperion.Tests.ISerializableTests.SkipNonSerializedAttributeIfNoSerilizedAttribute
Skipped Hyperion.Tests.ISerializableTests.ShouldNotThrowIfNoConstructorWithSerializationInfo
[xUnit.net 00:00:02.4549861] Hyperion.Tests.JsonObjectTests.CanSerializeSimpleJObject [FAIL]
[xUnit.net 00:00:02.4565330] System.NullReferenceException : Object reference not set to an instance of an object.
[xUnit.net 00:00:02.4572192] Stack Trace:
[xUnit.net 00:00:02.4578435] at Hyperion.SerializerFactories.DictionarySerializerFactory.<>c__DisplayClass3_0.<BuildSerializer>b__1(Stream stream, Object obj, Serializer
Session session)
[xUnit.net 00:00:02.4590671] at Hyperion.ValueSerializers.ObjectSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
[xUnit.net 00:00:02.4591752] at Hyperion.Serializer.Serialize(Object obj, Stream stream)
[xUnit.net 00:00:02.4592541] JsonObjectTests.cs(25,0): at Hyperion.Tests.JsonObjectTests.CanSerializeSimpleJObject()
Failed Hyperion.Tests.JsonObjectTests.CanSerializeSimpleJObject
Error Message:
System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at Hyperion.SerializerFactories.DictionarySerializerFactory.<>c__DisplayClass3_0.<BuildSerializer>b__1(Stream stream, Object obj, SerializerSession session)
at Hyperion.ValueSerializers.ObjectSerializer.WriteValue(Stream stream, Object value, SerializerSession session)
at Hyperion.Serializer.Serialize(Object obj, Stream stream)
at Hyperion.Tests.JsonObjectTests.CanSerializeSimpleJObject() in C:\repos\Hyperion\Hyperion.Tests\JsonObjectTests.cs:line 25
[xUnit.net 00:00:03.0441936] Hyperion.Tests.CollectionTests.CanSerializeMultiDimentionalArray [SKIP]
[xUnit.net 00:00:03.0444410] add support for multi dimentional arrays
Skipped Hyperion.Tests.CollectionTests.CanSerializeMultiDimentionalArray
[xUnit.net 00:00:03.0831862] Finished: Hyperion.Tests
Total tests: 138. Passed: 134. Failed: 1. Skipped: 3.
Test execution time: 3.5277 Seconds
Test Run Failed.
Bug found while working on akkadotnet/akka.net#2421 .
I think, that fix for the situation is to harden the constrains for EnumerableSerializerFactory
- it works in assumption that the serialized/deserialized type has Count
property + either Add
or AddRange
method. However it doesn't check if those methods have a correct signatures. Current implementation require:
void Add(T item)
(modifying collection).void AddRange(IEnumerable<T> items)
(also modifies existing collection).I think, that the solution should check if those methods return Void
type and have only one input parameter.
I've found and issue with Hyperion related to stream reading.
I did not analyze carefully which line of code is failing (it seems, it serious issue related to StreamEx class). E.g. next code:
public static int ReadInt32(this Stream self, DeserializerSession session)
{
byte[] buffer = session.GetBuffer(4);
self.Read(buffer, 0, 4);
return BitConverter.ToInt32(buffer, 0);
}
self can return less than four bytes and it is normal situation (e.g. for network streams, or compressed streams), as result int value will be incorrect and other data will be garbage.
You need to replace read method to something like
public override int Read(byte[] buffer, int offset, int count)
{
var total = 0;
while (count > 0)
{
var cnt = _origStream.Read(buffer, offset, count);
offset += cnt;
count -= cnt;
total += cnt;
}
return total;
}
But only for places where you need to read specific count of bytes
I'm unable to round-trip serialization of an Expression<Func<T, bool>>.
I can see the CanSerializeLambdaExpression
test within the code base that is similar but my following test case fails, the Method property (of type MethodInfo) inside the MethodCallExpression is null upon de-serialization.
Hopefully someone can help me resolve this. It might take me a while to navigate the codebase and track down the solution myself. Happy to help with a pull request if someone has an idea though.
[Fact]
public void CanSerializeStronglyTypedLambdaExpression() {
Expression<Func<Something, bool>> expr = s => new[] { "Bar", "Car" }.Contains(s.Foo);
var serializer = new Serializer(new SerializerOptions(preserveObjectReferences: true));
using (var ms = new MemoryStream())
{
serializer.Serialize(expr, ms);
ms.Seek(0, SeekOrigin.Begin);
var deserialized = serializer.Deserialize<Expression<Func<Something, bool>>>(ms);
Assert.NotNull(((MethodCallExpression)deserialized.Body).Method);
Assert.True(expr.Parameters[0] == ((MemberExpression)(((MethodCallExpression)deserialized.Body).Arguments[1])).Expression);
Assert.True(deserialized.Compile()(new Something { Foo = "Bar" }));
}
}
public class Something {
public string Foo { get; set; }
}
When trying to send a Dictionary from a dotnet core 2.1 cluster node to a framework 4.7.2 cluster node, using Hyperion, it fails to deserialize on the receiving side.
It seems the framework 4.7.2. node tries to deserialize the dictionary referencing the assembly "System.Private.CoreLib", which only exists in dotnet core.
The exception reads:
Cause: System.IO.FileNotFoundException: Could not load file or assembly 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
A solution that generates this exception can be found here:
https://github.com/NielsHoogeveen/HyperionTest
In the present shape, the serializer will not be able to deserialize structs having some private readonly fields in them. What does that mean. It means that following C# 6 struct will throw an exception upon deserialization:
public struct MyStruct
{
public int Value1 { get; } // this will fail - hidden readonly field
public readonly string Value2; // this will fail too
public MyStruct(int value1, string value2)
{
Value1 = value1;
Value2 = value2;
}
}
So far current compiler tried to set private readonly fields limitations by using reflection. For structs this won't work. Using FieldInfo.SetValue
or FieldInfo.SetValueDirect
will throw exception here. However we may try to utilize a special case here - C# compiler ensures that all readonly fields must be initialized in a struct constructor.
@Scooletz : maybe you have some ideas over here. I've seen your proposition on direct mapping from byte chunks to structs, but this must also work with reference types in them.
CultureInfo implementation of property-getters uses a private field that is null after deserializing, creating exceptions when accessing any property.
Test to confirm issue(put in CustomObjectTests.cs
):
[Fact]
public void CanSerializeCultureInfo()
{
var expected = CultureInfo.CurrentCulture;
Serialize(expected);
Reset();
var actual = Deserialize<CultureInfo>();
Assert.Equal(expected, actual);
}
Hi,
is it possible to extend the IgnoreAttribute
to use it with properties?
In the implementation its only allowed for fields:
[AttributeUsage(AttributeTargets.Field, AllowMultiple = false, Inherited = true)]
I have been getting random exceptions when deserializing objects from NetworkStream. It seems that Hyperion is not waiting for the whole message but trying to deserialize a message that is half received
It works fine if the client and the server are running on the same machine but when they are running on different machines I am getting exceptions like null reference exceptions or it is trying to look for types with half written assemblies for example if I have JMM.Util.Msg it is throwing exceptions saying somthing like cannot find type "JMM.Ut". I was able to fix the problem by using BinaryWriter/BinaryReader to write an int indicating the msg length followed by a byte[] obtained from Hyperion serializer of the message.
This is my deserialization code:
var msg = wireSerRead.Deserialize(ns);
This is my serialization code:
wireSerWrite.Serialize(msg, ms);
var msArr = ms.ToArray();
await ns.WriteAsync(msArr, 0, msArr.Length);
ns is of type NetworkStream
ms is of type MemoryStream
my serializers are instantiated like so:
Serializer wireSerRead = new Serializer(new SerializerOptions(false, false));
Serializer wireSerWrite = new Serializer(new SerializerOptions(false, false));
I created a subclass of System.Exception
with its own property that I wanted to serialise. When including all constructors (even methods for ISerializable
). This was packed in a faulted Task
object. After deserialising (using Akka) all Exception properties were fine, but properties I defined on the class were receiving default values (boolean sent with true
was received as false
).
After looking at ExceptionSerializerFactory
, I realised that all objects inheriting from System.Exception
fall into that serialiser but it only serialises properties known to the System.Exception
class (such as class name, message, inner exception, ..).
I would expect that this serialiser would either take into account that the exception may have other properties or at least offer a way to have the exception be sent over another serializer which transports this information.
when Uri
is serialized with preserveObjectReferences and versionTolerance enabled, deserialization will fail with InvalidCastException
.
It happens like this. During serialization,
Uri
itselfUri
contains the url itself in several fields as stringstring
valuesession.TryGetObjectId
will find the previous object, unfortunatelly it won't be string
, but Uri
InvalidCastException
Reason is, that SerializerSession
is using default equality comparer for tracked objects and string
and Uri
are equal, when having the same value.
A couple of issues with the nightlies, which haven't been published correctly for some time now:
BUILD_NUMBER
environment variable for versioning, even though this is specified in the build scriptNugetPublish
stage of the build script. We've been able to publish packages correctly for this build script when doing production releases, so it's something off with the "nightly" configuration.My payload contains data like this in an FSharp Record. The error goes away if the values if the values are different. Only occurs when preserve object refs
is true.
Section = Some "sd"
SubDivision = Some "sd"
This is the exception.
$exception {"Index was out of range. Must be non-negative and less than the size of the collection.\r\nParameter name: index"} System.ArgumentOutOfRangeException
Here is the line that fails.
public class DeserializerSession
{
////
public object GetDeserializedObject(int id)
{
return _objectById[id]; // <- here
}
////
Stack Trace
> Hyperion.dll!Hyperion.DeserializerSession.GetDeserializedObject(int id = 14) Line 66 C#
Hyperion.dll!Hyperion.ValueSerializers.ObjectReferenceSerializer.ReadValue(System.IO.Stream stream = {System.IO.MemoryStream}, Hyperion.DeserializerSession session = {Hyperion.DeserializerSession}) Line 34 C#
Hyperion.dll!Hyperion.Extensions.StreamEx.ReadObject(System.IO.Stream stream = {System.IO.MemoryStream}, Hyperion.DeserializerSession session = {Hyperion.DeserializerSession}) Line 172 C#
When working with Distributed Data, I came to the point when we need to serialize generic data types in efficient way - both in terms of performance and payload size. Protobuf is a no-go here: it's not able to store generic data. JSON is inefficient. Hyperion can deal with generics efficiently, but there's a catch.
Right now we are storing type information as a string. As someone pointed out this may be a security issue (we have the same for json.net). Regardless, there is another problem with storing types this way, and it's related to generics. This is something I need to verify, but the deal is:
Potentially we can make use of known types property of Hyperion to safe space (I'm not sure, but AFAIK type name is changed to int identifier). However known types won't be able to utilize this feature if we have generic type, because we need to have type written as a single value. Therefore having known type of Dictionary<,>
won't help us anywhere if we have to serialize Dictionary<string, int>
/Dictionary<string,string>
etc. as their type signatures don't match the one of known type.
We could deal with it this way:
Instead of setting known type as a single key, in case of generics we could serialize it as a sequence of keys, i.e. 3-key sequence representing Dictionary<,>
/string
/int
which would be then composed into Dictionary<string,int>
. Each key could be cached separately.
This is only an idea and needs further investigation.
These should all be one folder:
Hyperion.PerfTest
Hyperion.PerformanceTests
Hyperion.Tests.Performance
I've pushed an example solution to https://github.com/goodisontoffee/HyperionContainerSerialisation
I am sending a message containing an IEnumerable<IContainer<Nullable>> and with serialize-messages = on set in my HOCON for Hyperion the receipt of the message fails in my test with a message that suggests the presence of IContainer is being lost.
System.ArgumentException: Object of type 'HyperionContainerSerialisation.IContainer1[System.Nullable
1[System.DateTime]][]' cannot be converted to type 'System.Collections.Generic.IEnumerable1[System.Nullable
1[System.DateTime]]'.
If the type in the collection is a class that contains instances of IContainer it deserialises without issue.
I've noticed that there are a lot of unnecessary checks:
public static bool IsHyperionPrimitive(this Type type)
{
return type == Int32Type ||
type == Int64Type ||
type == Int16Type ||
type == UInt32Type ||
type == UInt64Type ||
type == UInt16Type ||
type == ByteType ||
type == SByteType ||
type == DateTimeType ||
type == BoolType ||
type == StringType ||
type == GuidType ||
type == FloatType ||
type == DoubleType ||
type == DecimalType ||
type == CharType;
}
But we could replace most of these checks with simple type.IsPrimitive
check, like this:
public static bool IsHyperionPrimitive(this Type type)
{
return type.IsPrimitive
|| type == StringType
|| type == GuidType
|| type == DateTimeType
|| type == DecimalType;
}
Also this answer might be useful - https://stackoverflow.com/a/16589255
I think, that the serializer is great subject for a property-based tests. As the serializer can actually generated and compile code in the runtime, the number of possible edge cases and tweak will be hard to catch.
I'm using Akka.Serialization.Hyperion 1.1.3.32-beta,
On receiver side i never receive message when message contains Maybe<MyCustomClass>
due deserialization exception.
Is this problem may be related to Hyperion or i need to register custom serializer or something i'm missing?
Stack trace:
System.InvalidCastException: Unable to cast object of type 'Common.Types.Maybe1[Domain.Models.Vending.Devices.States.DoorSensorState]' to type 'Domain.Models.Vending.Devices.States.DoorSensorState'.\r\n at lambda_method(Closure , Stream , DeserializerSession )\r\n at lambda_method(Closure , Stream , DeserializerSession )\r\n at Hyperion.SerializerFactories.EnumerableSerializerFactory.<>c__DisplayClass5_0.<BuildSerializer>b__1(Stream stream, DeserializerSession session)\r\n at lambda_method(Closure , Stream , DeserializerSession )\r\n at Hyperion.Serializer.Deserialize[T](Stream stream)\r\n at Akka.Serialization.HyperionSerializer.FromBinary(Byte[] bytes, Type type)\r\n at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)\r\n at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, SerializedMessage message, IActorRef senderOption)\r\n at Akka.Remote.EndpointReader.<Reading>b__11_1(InboundPayload inbound)\r\n at lambda_method(Closure , Object , Action
1 , Action1 , Action
1 )\r\n at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture4.Handle(T value)\r\n at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction
1 partialAction)\r\n at Akka.Actor.UntypedActor.Receive(Object message)\r\n at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)",
Need to add API diffing tests to protect against accidental binary incompatibility changes, similar to what we do here inside Akka.NET: https://github.com/akkadotnet/akka.net/tree/dev/src/core/Akka.API.Tests
Looks like I accidentally deleted this from the solution while I was cleaning up the project structure:
https://github.com/akkadotnet/Hyperion/blob/v0.9.6/Hyperion.PerformanceTests/SpeedAndSizeTests.cs
These were helpful! We should add them back. CC @Horusiath
Need to ensure that tests run on the following environments:
All of the above should target the same version of the .NET Standard, which will likely be 2.0 as part of the Akka.NET 1.5 release.
Complex objects can't be serialized if System.Uri is part of the properties. Can't convert string into Uri exception occurs while deserializing.
I've today updated my application to Akka.Net library version 1.3 from 1.2 and found that it wouldn't work. At startup when Akka system is being created it throws following exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.MissingMethodException: Method not found: 'Void Hyperion.SerializerOptions..ctor(Boolean, Boolean, System.Collections.Generic.IEnumerable`1<Hyperion.Surrogate>, System.Collections.Generic.IEnumerable`1<Hyperion.SerializerFactories.ValueSerializerFactory>, System.Collections.Generic.IEnumerable`1<System.Type>)'.
at Akka.Serialization.HyperionSerializer..ctor(ExtendedActorSystem system, HyperionSerializerSettings settings)
at Akka.Serialization.HyperionSerializer..ctor(ExtendedActorSystem system)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.Activator.CreateInstance(Type type, Object[] args)
at Akka.Serialization.Serialization..ctor(ExtendedActorSystem system)
at Akka.Actor.Internal.ActorSystemImpl..ctor(String name, Config config)
at Akka.Actor.ActorSystem.Create(String name)
at TestAkka.Program.Main(String[] args) ...
I tried the same Akka stack on the new application to eliminate other causes and found that result the same.
Here is the list of used packages (except System.*
)
<package id="Akka" version="1.3.0" targetFramework="net46" />
<package id="Akka.Persistence" version="1.3.0" targetFramework="net46" />
<package id="Akka.Serialization.Hyperion" version="1.3.0-beta46" targetFramework="net46" />
<package id="DotNetty.Buffers" version="0.4.6" targetFramework="net46" />
<package id="DotNetty.Codecs" version="0.4.6" targetFramework="net46" />
<package id="DotNetty.Common" version="0.4.6" targetFramework="net46" />
<package id="DotNetty.Handlers" version="0.4.6" targetFramework="net46" />
<package id="DotNetty.Transport" version="0.4.6" targetFramework="net46" />
<package id="Google.Protobuf" version="3.3.0" targetFramework="net46" />
<package id="Hyperion" version="0.9.5" targetFramework="net46" />
<package id="Microsoft.Extensions.DependencyInjection.Abstractions" version="1.1.0" targetFramework="net46" />
<package id="Microsoft.Extensions.Logging" version="1.1.1" targetFramework="net46" />
<package id="Microsoft.Extensions.Logging.Abstractions" version="1.1.1" targetFramework="net46" />
<package id="Microsoft.NETCore.Platforms" version="1.1.0" targetFramework="net46" />
<package id="Microsoft.Win32.Primitives" version="4.3.0" targetFramework="net46" />
<package id="NETStandard.Library" version="1.6.1" targetFramework="net46" />
<package id="Newtonsoft.Json" version="9.0.1" targetFramework="net46" />
And corresponding configuration
<configSections>
<section name="akka" type="Akka.Configuration.Hocon.AkkaConfigurationSection, Akka" />
</configSections>
<akka>
<hocon>
<![CDATA[
akka {
actor {
serializers {
wire = "Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion"
}
serialization-bindings {
"System.Object" = wire
}
}
}
]]>
</hocon>
</akka>
The test code is straight forward
class Program
{
public static readonly string DEFAULT_SYSTEM_NAME = "test";
static void Main(string[] args)
{
ActorSystem.Create(DEFAULT_SYSTEM_NAME);
}
}
This code works fine with previous version of the Hyperion 0.9.2
<package id="Akka.Serialization.Hyperion" version="1.2.3.43-beta" targetFramework="net46" />
<package id="Hyperion" version="0.9.2" targetFramework="net46" />
System.InvalidCastException: 'Unable to cast object of type 'NobodySurrogate' to type 'Akka.Actor.IActorRef'.'
Using Hyperion 0.9.2
It tries to cast Akka.Actor.ActorPath.Surrogate
type to ActorPath
System.InvalidCastException: Unable to cast object of type 'Surrogate' to type 'Akka.Actor.ActorPath'.
at lambda_method(Closure , Stream , DeserializerSession )
at Hyperion.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
I can't reproduce this bug, outside of AtLeastOnceDelivery
specs
When JObject is used for messaging then Akka fails to route such messages due to serialization problem.
In my particular case I have a strong typed message class with IDictionary<string, object>
property and Akka.Serialization.Hyperion
configured as default serializer. At some moment this dictionary may contain JObject. Such messages destroy actor life-cycle due to serialization problem.
I made a test project to illustrate such behavior
JObjectSerializationTest.zip
System.NullReferenceException: Object reference not set to an instance of an object.
at Hyperion.SerializerFactories.DictionarySerializerFactory.<>c__DisplayClass3_0.<BuildSerializer>b__1(Stream stream, Object obj, SerializerSession session)
at Akka.Serialization.HyperionSerializer.ToBinary(Object obj)
at JObjectSerializationTest.UnitTest.<>c__DisplayClass1_0.<ShouldBeSerializedByHyperion>b__0(Serializer serializer, Object entity)
Used packages versions
<package id="Akka" version="1.3.0" targetFramework="net46" />
<package id="Akka.Serialization.Hyperion" version="1.2.3.43-beta" targetFramework="net46" />
<package id="Hyperion" version="0.9.2" targetFramework="net46" />
<package id="Newtonsoft.Json" version="9.0.1" targetFramework="net46" />
<package id="NUnit" version="3.7.1" targetFramework="net46" />
<package id="System.Collections.Immutable" version="1.3.1" targetFramework="net46" />
As Roger mentioned in akkadotnet/akka.net#2514 (comment) , current state of version tolerance is unfinished - it means that we're serializing field infos as part of the payload manifest, however we don't utilize this knowledge on the deserialization phase. We probably should use that information there to restore fields state.
I am testing the Akka Cluster update process.
I have two nodes with the only difference that they have different assembly versions (with the same code).
One "old" node has version 0.0.0-local
assemblies and the new node has 0.0.1-local
ones.
I am running everything under net-core under ubuntu (Akka.Net 1.3.0). Also, I've made custom Hyperion build with net-core support and Akka.Serialization.Hyperion with core support so cluster is running well.
So after updating assembly version numbers on on of the nodes I receive this error messages on the "old" node:
File name: 'ClusterKit.API.Client, Version=0.0.1.0, Culture=neutral, PublicKeyToken=null'
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type, ObjectHandleOnStack keepalive)
at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName)
at System.RuntimeType.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Boolean throwOnError)
at Hyperion.Extensions.TypeEx.<>c.<GetTypeFromManifestName>b__27_0(ByteArrayKey b)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Hyperion.Extensions.TypeEx.GetTypeFromManifestVersion(Stream stream, DeserializerSession session)
at Hyperion.Serializer.GetDeserializerByManifest(Stream stream, DeserializerSession session)
at Hyperion.Serializer.Deserialize[T](Stream stream)
at Akka.Serialization.HyperionSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.Serialization.MessageContainerSerializer.FromBinary(Byte[] bytes, Type type)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption)
at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Akka.Actor.ActorCell.HandleFailed(Failed f)
at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)
The main problem with futher debugging is that I could not manage to load Hyperion solution properly under VS2017 (is there any instruction to correctly load Hyperion.FSharpTestTypes?). Meanwhile, build scripts are working well.
Any advice of fixing issue?
Thanks.
At this moment we have two compilation chains:
Moreover when we need to construct a serializer for a custom data structure (either class or struct), we always do that through DefaultCodeGenerator, which uses only Expression-based compilation chain.
The idea here is to build a code generator using IlCompiler<>
as well.
While working on #82 I've found out that SerializerOptions.PreserveObjectReferences
actually never worked for user-defined objects. Example:
var serializer = new Serializer(new SerializerOptions(preserveObjectReferences: true));
var serializerSession = new SerializerSession(serializer);
var deserializerSession = new DeserializerSession(serializer);
using (var stream = new MemoryStream())
{
var o = new MyClass {First = "hello", Second = 123};
serializer.Serialize(o, stream, serializerSession); // serialize object 1st time
stream.Position = 0;
serializer.Serialize(o, stream, serializerSession); // serialize it again using the same session
}
First serialization passes, but second one throws ArgumentException: An item with the same key has already been added.. From what I've seen, generated object serializer simply doesn't look it object has been saved previously before trying to serialize it again.
At this point it's not quite clear, how much of the work is missing in order to being able to use serializer on projects using .NET Core. We need to investigate that and make it happen.
Been working on fixing the build issues with #57, and the problem is that our build script for Hyperion is messed up. TL;DR; it's a weird one-off at the moment and doesn't use any of the convention-based stuff that Akka.NET and all of our other projects depend on.
Going to recommend that we move it onto the pb-lib
template since that "just works"
Right now we can use SerializerSession
/DeserializerSession
for object caching when preserve object references flag is set. This way we can send only object IDs in subsequent deserialization instead of passing full payload every time.
However this option has pretty limited configuration. What we could do is:
[PreserveReferences]
to be attached to class itself. So instead of trying to preserve everything we would apply some rules (since we may expect some types of object to be more frequently reused within a single session while other shouldn't be even bothered to preserve).Type
or singleton messages could be cached permanently, IActorRef
could live probably in long-living cache, while some other user messages may change much more frequently, therefore be cached in more frequently cleaned cache layer.When trying to serialize a large object, setting preserveObjectReferences to true causes the serialization to hang indefinitely.
I've only noticed this behavior with large objects.
I guess its perfectly fine to turn off that option, but thought I'd report it anyway as it seems strange.
Reproducible in both Wire and Hyperion and on .NET Core 2.1 and .NET Framework 4.6
static void Main(string[] args)
{
Hyperion.Serializer s = new Hyperion.Serializer(new Hyperion.SerializerOptions(preserveObjectReferences:true));
var obj = new List<Dictionary<string, object>>();
for (var i = 0; i < 1000000;i++)
{
var item = new Dictionary<string, object>();
item["x"] = i;
item["y"] = DateTime.Now;
obj.Add(item);
}
var dict = new Dictionary<string, object>();
dict["key"] = obj;
using (var ms = new System.IO.MemoryStream())
{
s.Serialize(dict, ms);
Console.WriteLine("Serialized Size: {0}", ms.Length);
}
}
Hi there,
thanks for fixing bug I've previously reported about strings.
I'd like to point out another bug I've found after installed version 0.8.2.
This is the unit test code:
using DeepEqual.Syntax;
using Mercury.Cache.UnitTest.Data;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace Mercury.Cache.UnitTest
{
[TestClass]
public class serialization
{
[TestMethod]
public void ComplexObject()
{
List<PerfStats> stats = new List<PerfStats>();
ComplexObject objectToCache = new ComplexObject();
Stopwatch stopwatch = new Stopwatch();
// This is not provided by default behaviour. I should serialize/deserialize even if
// I don't know anything about Type.
WireFormatter.RegisterType(typeof(ComplexObject));
foreach (Formatter formatter in Enum.GetValues(typeof(Formatter)))
{
PerfStats itemStats = new PerfStats();
stopwatch.Restart();
try
{
byte[] serialized = WireFormatter.SerializeAny(objectToCache);
ComplexObject cachedObject = WireFormatter.DeserializeAny(serialized);
stopwatch.Stop();
itemStats.Formatter = formatter;
itemStats.ElapsedTime = stopwatch.ElapsedMilliseconds;
stats.Add(itemStats);
cachedObject.ShouldDeepEqual(objectToCache);
}
catch (DeepEqualException ex)
{
itemStats.Exception = ex;
}
catch (Exception ex)
{
Console.WriteLine("Formatter " + formatter.ToString().ToUpperInvariant() + " implementation got problems!!!");
Console.WriteLine("Exception: " + ex.ToString());
}
}
IEnumerable<PerfStats> stats1 = stats.OrderBy(x => x.ElapsedTime).ToList();
Console.WriteLine("---[ Best performers for Elapsed Time ]---");
foreach (PerfStats item in stats1)
{
Console.WriteLine("Formatter: " + item.Formatter);
if (item.Exception == null)
{
Console.WriteLine("Elapsed time: " + item.ElapsedTime);
}
else
{
Console.WriteLine("EXCEPTION: " + item.Exception.ToString());
}
Console.WriteLine("------------------------------------------------------------------");
}
}
class PerfStats
{
public Formatter Formatter { get; set; }
public long ElapsedTime { get; set; }
public Exception Exception { get; set; }
}
}
}
[Serializable]
[System.Diagnostics.DebuggerStepThrough]
public class ComplexObject
{
public ComplexObject()
{
Thing1 = true;
Thing2 = int.MaxValue;
Thing3 = 'q';
Thing4 = "asdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasasdfasdfasas";
Thing7 = uint.MaxValue;
}
public bool Thing1 { get; set; }
public int Thing2 { get; set; }
public char Thing3 { get; set; }
public string Thing4 { get; set; }
public uint Thing7 { get; set; }
}
[System.Diagnostics.DebuggerStepThrough]
public class WireFormatter
{
static Wire.Serializer staticFormatter;
static List<Type> knowTypes = new List<Type>();
public static void RegisterType(Type type)
{
if (!knowTypes.Contains(type))
knowTypes.Add(type);
staticFormatter = new Wire.Serializer(CreateOptions());
}
static WireFormatter()
{
staticFormatter = new Wire.Serializer(CreateOptions());
}
static Wire.SerializerOptions CreateOptions()
{
return new Wire.SerializerOptions(versionTolerance: true, knownTypes: knowTypes, preserveObjectReferences: true);
}
public WireFormatter() { }
/// <summary>
/// Serializza un oggetto in un array di bytes.
/// </summary>
/// <param name="obj"></param>
/// <returns></returns>
[System.Diagnostics.DebuggerStepThrough]
static public byte[] SerializeAny(object obj)
{
// SANITIZE
if (obj == null)
return null;
using (MemoryStream stream = new MemoryStream())
{
// Scrive il contenuto
staticFormatter.Serialize(obj, stream);
return stream.ToArray();
}
}
/// <summary>
/// Trasforma un array di bytes nell'oggetto originario.
/// </summary>
/// <param name="obj"></param>
/// <returns></returns>
[System.Diagnostics.DebuggerStepThrough]
static public object DeserializeAny(byte[] obj)
{
// SANITIZE
if (obj == null)
return null;
using (MemoryStream stream = new MemoryStream(obj))
{
return staticFormatter.Deserialize(stream);
}
}
public byte[] Serialize<T>(T value)
{
// SANITIZE
if (value == null)
return null;
Wire.Serializer formatter = new Wire.Serializer();
using (MemoryStream stream = new MemoryStream())
{
// Scrive il contenuto
formatter.Serialize(value, stream);
return stream.ToArray();
}
}
public T Deserialize<T>(byte[] bytes)
{
// SANITIZE
if (bytes == null)
return default(T);
Wire.Serializer formatter = new Wire.Serializer();
using (MemoryStream stream = new MemoryStream(bytes))
{
return formatter.Deserialize<T>(stream);
}
}
}
Thanks for your great efforts and fantastic work you've done since so far!
NullReferenceExcetions throws on ISerializable deserialization
public class Person : ISerializable
{
public Person(string firstName, string lastName)
{
FirstName = firstName;
LastName = lastName;
}
public string FirstName { get; set; }
public string LastName { get; set; }
public void GetObjectData(SerializationInfo info, StreamingContext streamingContext)
{
info.AddValue("FirstName", FirstName);
info.AddValue("LastName", LastName);
}
}
var expected = new Person("Scott", "Hanselman");
Serialize(expected);
Reset();
var actual = Deserialize<Person>();
Assert.Equal(expected.FirstName, actual.FirstName);
Assert.Equal(expected.LastName, actual.LastName);
Error Message:
System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at Hyperion.SerializerFactories.ISerializableSerializerFactory.<>c__DisplayClass2_0.<BuildSerializer>b__0(Stream stream, DeserializerSession session)
at Hyperion.ValueSerializers.ObjectSerializer.ReadValue(Stream stream, DeserializerSession session)
at Hyperion.Serializer.Deserialize[T](Stream stream)
at Hyperion.Tests.TestBase.Deserialize[T]() in C:\Projects\Hyperion\Hyperion.Tests\TestBase.cs:line 44
at Hyperion.Tests.ISerializableTests.CanSerializeClassesWithISerializable() in C:\Projects\Hyperion\Hyperion.Tests\ISerializableTests.cs:line 78
Current version in Nuget does not have a strong name. As result, there are problems when this library is used in strongly named applications
Hi guys,
the job done with Wire/Hyperion is impressive!
But it's not very useful if developers like me cannot quickly become familiar with the technology.
The developers community need to have a documentation, API reference, wiki, samples, developers blog (who've decided to call it Hyperion and why? :) very interesting!)... but currently we have only a very brief readme file!
In my particular interest:
Currently we're using https://github.com/AqlaSolutions/AqlaSerializer (based on protobuf-net, but provides full support for the object references serialization) for our serialization needs, it satisfies all the requirements, but we're looking for something more reliable, clean and rapidly developing.
Regards!
I have Akka.Serialization.Hyperion 1.3.2-beta54 and Hyperion 0.9.6.
On that versions I'm getting an error: Generic IDictionary<TKey,TValue> are not yet supported .
When I can expect the support will be provided?
Related asynkron/Wire#137
Hyperion does not seem to deserialize ObjectDisposedException
. Got many problems because it was wrapped in some messages containing the Exception but could reproduce it in the end with this simple lines:
var message = new ObjectDisposedException("oh no");
var serializer = system.Serialization.FindSerializerFor(message);
var binary = serializer.ToBinary(message);
var msg = serializer.FromBinary(binary, null);
Error is:
An exception occurred, System.MissingMethodException: No parameterless constructor defined for this object.
The error feels misleading as most immutable Akka messages do not have a parameterless constructor. Also it does not mention the class where it misses the constructor. I wrapped the Akka.Serialization.HyperionSerializer
with my own Akka serializer but I could also track the outer-most parent class. That was in my case Akka.Actor.Status.Failure
so it took my a while to go down to ObjectDisposedException
.
Versions:
A temporary workaround to replace those instances with null
would be appreciated.
Has two test failures:
[21:29:10] [docker]
[21:29:10] [docker] Test run for /checkout/Hyperion.Tests/bin/Release/netcoreapp1.0/Hyperion.Tests.dll(.NETCoreApp,Version=v1.0)
[21:29:11] [docker] Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
[21:29:11] [docker] Copyright (c) Microsoft Corporation. All rights reserved.
[21:29:11] [docker]
[21:29:11] [docker] Starting test execution, please wait...
[21:29:13] [docker] [xUnit.net 00:00:01.0619402] Discovering: Hyperion.Tests
[21:29:13] [docker] [xUnit.net 00:00:01.3994812] Discovered: Hyperion.Tests
[21:29:14] [docker] [xUnit.net 00:00:01.5054871] Starting: Hyperion.Tests
[21:29:14] [docker] [xUnit.net 00:00:02.0633171] Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(val1: True, val2: False) [FAIL]
[21:29:14] [docker] [xUnit.net 00:00:02.0670384] Assert.Equal() Failure
[21:29:14] [docker] [xUnit.net 00:00:02.0671854] Expected: False
[21:29:14] [docker] [xUnit.net 00:00:02.0672427] Actual: True
[21:29:14] [docker] [xUnit.net 00:00:02.0690800] Stack Trace:
[21:29:14] [docker] [xUnit.net 00:00:02.0782522] /checkout/Hyperion.Tests/CustomObjectTests.cs(87,0): at Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(Boolean val1, Boolean val2)
[21:29:14] [docker] [xUnit.net 00:00:02.0818967] Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(val1: False, val2: True) [FAIL]
[21:29:14] [docker] [xUnit.net 00:00:02.0820607] Assert.Equal() Failure
[21:29:14] [docker] [xUnit.net 00:00:02.0821049] Expected: True
[21:29:14] [docker] [xUnit.net 00:00:02.0825918] Actual: False
[21:29:14] [docker] [xUnit.net 00:00:02.0826474] Stack Trace:
[21:29:14] [docker] [xUnit.net 00:00:02.0830364] /checkout/Hyperion.Tests/CustomObjectTests.cs(87,0): at Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(Boolean val1, Boolean val2)
[21:29:14] [docker] Failed Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(val1: True, val2: False)
[21:29:14] [docker] Error Message:
[21:29:14] [docker] Assert.Equal() Failure
[21:29:14] [docker] Expected: False
[21:29:14] [docker] Actual: True
[21:29:14] [docker] Stack Trace:
[21:29:14] [docker] at Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(Boolean val1, Boolean val2) in /checkout/Hyperion.Tests/CustomObjectTests.cs:line 87
[21:29:14] [docker] Failed Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(val1: False, val2: True)
[21:29:14] [docker] Error Message:
[21:29:14] [docker] Assert.Equal() Failure
[21:29:14] [docker] Expected: True
[21:29:14] [docker] Actual: False
[21:29:14] [docker] Stack Trace:
[21:29:14] [docker] at Hyperion.Tests.CustomObjectTests.CanSerializeDefaultCtorArguments(Boolean val1, Boolean val2) in /checkout/Hyperion.Tests/CustomObjectTests.cs:line 87
[21:29:15] [docker] [xUnit.net 00:00:03.0937879] Hyperion.Tests.CollectionTests.CanSerializeMultiDimentionalArray [SKIP]
[21:29:15] [docker] [xUnit.net 00:00:03.0939669] add support for multi dimentional arrays
[21:29:15] [docker] Skipped Hyperion.Tests.CollectionTests.CanSerializeMultiDimentionalArray
[21:29:15] [docker] [xUnit.net 00:00:03.4139414] Finished: Hyperion.Tests
[21:29:16] [docker]
[21:29:16] [docker] Total tests: 119. Passed: 116. Failed: 2. Skipped: 1.
[21:29:16] [docker] Test Run Failed.
[21:29:16] [docker] Test execution time: 5.1388 Seconds
But still has a green status. Looks like it's "forgetting" that it failed or that the failure status is overridden. Probably a result of running the build twice. cc @heynickc
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.