amzn / ion-object-mapper-dotnet

Ion Object Mapper for .NET

This is the Ion Object Mapper for .NET, which is a convenience library built to assist developers when dealing with data that's in the format of Amazon Ion. It is used to easily transform C# classes and objects into Ion format and vice-versa.

Requirements

.NET

The Ion Object Mapper targets .NET Core 3.1. Please see the link below for more information on compatibility:

• .NET Core

Getting Started

See the Cookbook for a simple usage guide.

Spec

See the Spec for a detailed specification of the Ion Object Mapper.

Contributing

If you are interested in contributing to the Ion Object Mapper, please take a look at CONTRIBUTING.

Development

Setup

Assuming that you have .NET Core SDK version 3.1 or later installed from the Microsoft .NET downloads site, use the below command to clone the repository.

$ git clone https://github.com/amzn/ion-object-mapper-dotnet.git
$ cd ion-object-mapper-dotnet

Changes can now be made in the repository.

The project currently uses default dotnet CLI tools, you can build the project simply by:

$ dotnet build

Running Tests

You can run the unit tests by:

$ dotnet test Amazon.IonObjectMapper.Test

You can also run the performance tests by:

$ dotnet test Amazon.IonObjectMapper.PerformanceTest

Documentation

DocFX is used for documentation. Please see the link below for more detail to install DocFX

• DocFX installation

You can generate the docstring HTML locally by running the following in the root directory of this repository:

$ docfx docs/docfx.json --serve

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

ion-object-mapper-dotnet's Issues

Add support to identify target type from Ion annotation for deserialization

The object mapper will write type information as Ion annotation if the class or property to be serialized has C# Attribute IonAnnotateType. For deserialization, the object mapper should be able to identify the type annotation in the Ion value stream and map the annotation to corresponding C# type if that type also has C# Attribute IonAnnotateType.

For example, when we serialize the Car instance with dynamic property Engine.

public class Car
{
    public string Make { get; set; }
    public string Model { get; set; }
    public int Year { get; set; }
    public dynamic Engine { get; set; }
}

[IonAnnotateType]
public class Engine
{
    public int Cylinders { get; set; }
    public DateTime ManufactureDate { get; set; }
}

Car car = new Car
{
    Make = "make",
    Model = "model",
    Year = 2021,
    Engine = new Engine
    {
        Cylinders = 4,
        ManufactureDate = DateTime.Now
    }
};

It should produce Ion value like following:

{
  Make: "make",
  Model: "model",
  Year: 2021,
  Engine: 'MyApp.Engine'::{
    Cylinders: 4,
    ManufactureDate: 2021-12-08T14:32:59-00:00
  }
}

And for deserialization, it should be able to map the annotation MyApp.Engine to corresponding C# type MyApp.Engine instead of using object type that gets from Car class through reflection.

Handle readonly property backing field logic during deserialization instead of serialization

The ordinary deserialization logic does not work for readonly properties. To get around this we have been serializing the backing fields for readonly properties so we can still set the underlying value of the property during deserialization. However this results in those backing fields being inserted (as an extra column) into QLDB whenever the driver does an insert with an object with a readonly property.

ie.

Let's change the readonly property / backing field logic so that it is handled exclusively during deserialization instead of serialization (so the backing field logic is completely hidden from the end users).

Implement the MaxDepth option

Implement MaxDepth which determines how far down on object tree to recurse (for the sake of avoiding stack overflow).

Error when deserializing Ion blob to nullable `Guid` type

When the property has a nullable Guid type, i.e. Guid?, the object mapper will throw exception when trying to deserialize the Ion blob to Guid type.
Example class:

public class AnnotateGuidsClass 
{
    public Guid? NullableGuid { get; set; }
}

Exception:

System.ArgumentException: Object of type 'System.Byte[]' cannot be converted to type 'System.Nullable`1[System.Guid]'.

Implement the PropertyNameCaseInsensitive option

Option to determine whether or not property names are case insensitive.

Logging capabilities

When serialization or deserialization encounters an (expected) issue, we should be able to log them to our users to be resolved. In particular, this would be a necessity in non-fatal serialization issues, where we cannot throw a serialization/deserialization exception.

I think relying on Apache's Common.Logging would be a reasonable choice for this, since it does not enforce a specific logging implementation on our consumers, similar to Java's SLF4J.

This came up in my work on #29, where the ability to log would allow us to report this field naming conflict as a non-fatal warning to the user, instead of either a) silently ignoring the field or b) throwing some sort of name conflict error.

Update README to add more instructions, FAQs etc.

We need to update the README.md to add more instructions, FAQs etc about how to use the IonObjectMapper properly, in addition to existing COOKBOOK.md and SPEC.md.

Old version of Amazon.IonDotnet referenced

An old version of Amazon.IonDotnet is referenced in the Amazon.IonObjectMapper.csproj file. This causes issues when implementing the IIonSerializer because the nuget package references version 1.2.2.

This is the code I am trying to write:

    public class EnumSerializerAsInt<T>
        : IIonSerializer
        where T : System.Enum
    {
        void Serialize(IIonWriter writer, object item)
        {
            writer.WriteInt((int)item);
        }

        object Deserialize(IIonReader reader)
        {
            var valueObject = Enum
                .ToObject(typeof(T), reader.IntValue());
            return valueObject;
        }
    }

    public class EnumSerializerAsString<T>
        : IIonSerializer
        where T : System.Enum
    {
        void Serialize(IIonWriter writer, object item)
        {
            writer.WriteString(item.ToString());
        }

        object Deserialize(IIonReader reader)
        {
            if (!Enum.TryParse(typeof(T), reader.StringValue(), out object valueObject))
                throw new Exception($"Unable to parse string value into {typeof(T).Name}: {reader.StringValue()}");

            return valueObject;
        }
    }

And the error I get is The type 'IIonWriter' is defined in an assembly that is not referenced. You must add a reference to assembly 'Amazon.IonDotnet, Version=1.1.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'. (CS0012) ([Project name redacted])

But I believe the solution should be to update the reference to IonDotNet from version 1.1.0 to 1.2.2 in this file:

ion-object-mapper-dotnet/Amazon.IonObjectMapper/Amazon.IonObjectMapper.csproj

Line 21 in ebe5714

<PackageReference Include="Amazon.IonDotnet" Version="1.1.0" />

Implement the IgnoreReadOnlyProperties option

Option to determine whether or not to ignore read only properties during serialization/deserialization.

Support Latest Amazon.IonDotnet?

It seems I'm unable to use this library with Amazon.IonDotnet 1.2.3 and later versions of AWSSDK.QLDBSession because of package downgrade issues.

Is there any obstacle to updating this package so that it works with these other updated packages?

This is the particular error you'll encounter with later versions:

The type 'IIonReader' is defined in an assembly that is not referenced. You must add a reference to assembly 'Amazon.IonDotnet, Version=1.1.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'.

Implement the IonConstructor attribute

By default, C# objects are created using the Activator.CreateInstance(Type) method which calls the default constructor. If you want to supply a constructor yourself, you can markup a constructor with the IonConstructor Attribute. Additionally, the parameters to the constructor must be specified using the IonPropertyName Attribute again:

public class Wheel
{
    private string specification;
    
    [IonConstructor]
    public Wheel([IonPropertyName("specification")] string specification)
    {
         this.specification = specification;
    }
}

Unsupported C# types

The ObjectMapper doesn't support serialize/deserialize char C# type currently. The property or field that has char type will be serialized to an empty Ion struct, and will be deserialized as the default char value which is '\0' (U+0000).

Performance Test

We need to have a plan for performance testing in Ion Object Mapper, so we'll have a baseline for the serde operation speed and know how to make it faster.

Handle scenario where an [IonPropertyName] annotated property name conflicts with a field name in the same class

The property should get precedence over the field during serialization and deserialization.

Error when inserting object when class implements interface

The error System.InvalidOperationException: This operation is only valid on generic types occurs whenever we try to insert an object which implements interfaces. Example:

interface IInterface
{
    
}
class Car: IInterface
{
    public string Make { get; set; }  
    public string Model { get; set; }
    public int Year { get; set; }
}

The exception may come from the GetGenericTypeDefinition() API call in

ion-object-mapper-dotnet/Amazon.IonObjectMapper/IonSerializer.cs

Line 185 in 554acb5

Type genericDictionaryType = item.GetType().GetInterfaces().FirstOrDefault(t => typeof(IDictionary<,>).IsAssignableFrom(t.GetGenericTypeDefinition()));

without checking the type is generic type first, see https://docs.microsoft.com/en-us/dotnet/api/System.Type.GetGenericTypeDefinition?view=netcore-3.1#exceptions

The `IonAnnotateType` attribute should also work on properties

Currently the IonAnnotateType attribute only works for classes, not for properties. For the following class:

public class Car
{
    [IonAnnotateType(Name = "my.custom.engine.type")]
    public Engine Engine { get; set; }
}

[IonAnnotateType(Name = "Engine")]
public class Engine {}

This Ion should be produced:

{
  engine: 'Amazon.IonObjectMapper.Test.my.custom.engine.type'::{
  }
}

instead of

{
  engine: 'Amazon.IonObjectMapper.Test.Engine'::{
  }
}

Note that the IonAnnotateType for properties should take precedence over the IonAnnotateType for classes.

Implement the IonPropertyGetter & IonPropertySetter attributes

By default, C# methods are ignored, however, provided the “get” signature is a no-argument method and the “set” signature is a one-argument void method, methods can be use to get and set properties. The Ion property name must be specified and will not be inferred from the method name.

public class Car
{
     private string color;
    
    [IonPropertyGetter("color")]
    public string GetColor() 
    {
        return "#FF0000";
    }
    
    [IonPropertySetter("color")]
    public void SetColor(string input) 
    {
        this.color = input;
    }
}

`IgnoreDefaults` option has no effect on value types

The IgnoreDefaults option will only ignore fields and properties that has reference types which defaults to null value. The option has no effect on value types like int, float, bool etc.

Example class:

public class IgnoreDefaultsClass
{
    public int A { get; set; }
    public float B { get; set; }
    public bool C { get; set; }
}

serialize object new IgnoreDefaultsClass() will produce following Ion when IgnoreDefaults is set to true:

{
  a: 0,
  b: 'numeric.float32'::0e0,
  c: false
}

Disable inheritance for `IonAnnotateType` and `IonDoNotAnnotateType`

By default, attributes will be inherited by classes that are derived from the classes to which the attribute is applied, see https://docs.microsoft.com/en-us/dotnet/standard/attributes/writing-custom-attributes#inherited-property. So the IonAnnotateType and IonDoNotAnnotateType attributes will be inherited by derived classes and the ExcludeDescendants option will have no effect.

We should disable inheritance for the two attributes and write custom logic to look up the inheritance chain to find the first parent class that has the two attributes and and doesn't set the ExcludeDescendants option to decide if we should annotate the type.

The `Format` property in `IonSerializationOptions` has no effect

The ObjectMapper will always output Ion text dump regardless of the Format property value in the options.

Steps to reproduce:

public class Person
{
    public string FirstName { get; set; }
    public int Age { get; set; }
}

IonSerializer ionSerializer = new IonSerializer(new IonSerializationOptions
{
    Format = IonSerializationFormat.BINARY
});

Person person = new Person
{
    FirstName = "John",
    Age = 13
};

MemoryStream stream = (MemoryStream) ionSerializer.Serialize(person);
Console.WriteLine(BitConverter.ToString(stream.ToArray()));

which will print:

7B-66-69-72-73-74-4E-61-6D-65-3A-22-4A-6F-68-6E-22-2C-61-67-65-3A-31-33-7D

That is an Ion text dump instead of an Ion binary dump. The above is the UTF-8 encoding of string {firstName:"John",age:13}.

Expected behavior:
It should print Ion binary dump which should be:

E0-01-00-EA-EE-95-81-83-DE-91-87-BE-8E-89-66-69-72-73-74-4E-61-6D-65-83-61-67-65-D9-8A-84-4A-6F-68-6E-8B-21-0D

Handle Ion clob serde behavior properly

Current IonClobSerilaizer implementation:

public override string Deserialize(IIonReader reader)
{
    byte[] clob = new byte[reader.GetLobByteSize()];
    reader.GetBytes(clob);
    return Encoding.UTF8.GetString(clob);
}

public override void Serialize(IIonWriter writer, string item)
{
    writer.WriteClob(Encoding.UTF8.GetBytes(item));
}

Problems in current implementation:

1. The Ion clob specification allows arbitrary interpretation of the raw binary data but current implementation only allows UTF-8 encoding, which seems weird because string in C# actually uses UTF-16 encoding.
2. The C# string type will map to Ion string by default. So there is no use case for the Serialize(IIonWriter writer, string item) method to serialize a C# string to Ion clob, only the Deserialize method will be used when deserializing Ion clob data.

We should consider:

1. Allow encoding format to be passed in for Ion clob serde
2. What's the proper mapping between Ion clob type and C# type, probably the clob should map to C# byte array byte[] instead of the current string type.

Implement IonSerializers option for custom serializers

IonSerializers: Dictionary<Type, IonSerializer>

A Dictionary of IonSerializers which specifies, for a key Type, a custom Ion serializer for that type.

We will provide extension points to allow for arbitrarily fine-grained control over the serialization process. There are a few ways to specify this. To the IonSerializer, one can pass in options including a Dictionary of Type onto IonSerializer. This mapping will take precedence over the default mapping.

new IonSerializer(new IonSerializationOptions
{
     IonSerializers = new Dictionary<Type, IonSerializer>()
    {
        {typeof(string), new MyCustomStringIonSerializer()},
        {typeof(DateTime), new MyCustomDateTimeIonSerializer()}
    },
});