jamescourtney / flatsharp Goto Github PK

Tracking issue to investigate unions. Any solution must fit nicely in with the C# type system and must be amenable to attribute-based declaration. Further, we must be able to support disparate sets of elements that can't all be associated via inheritance, etc.

Breaking changes:

• Compiler no longer supports the "NoCustomType" metadata on unions.
• Compiler no longer suports the "ObsoleteDefaultConstructor" metadata. It has been replaced with "fs_defaultCtor", which provides more options.
• For memory vectors, compiler generates nullable Memory<byte>? instead of previous behavior of non-nullable. This is more in-line with how flatbuffers treats vectors as optional fields.
• ITypeModel.MustAlwaysSerialize has been removed.
• ITypeModel.GetNotEqualToDefaultValueExpression has been added.
• FlatBufferUnion and derivatives no longer supply a Clone method.
• OnInitialized partial methods for generated types now accepts a FlatSharpCompileContext parameter instead of no parameter.
• Generated code requires C# 8 compiler, C# 9 for full nullable support

Features/Other changes

• ~10-15% performance uplift for both serialize and parse workflows
• file_identifier and root_type support
• new fbs annotations
• FlatSharp code and core libs have full nullable annotations
• Support for fixed-size vectors in structs: field:[int:8]
• Support for init-only properties
• FlatSharp tests now treat warnings as errors when compiling generated code
• Deserialized FlatSharp objects now implement IFlatBufferDeserializedObject
• Non-Generic Serialize/Parse methods exposed.
• Types are no longer required to expose parameterless constructors (can use FlatSharp constructor instead).

I want to preface this by saying I'm nearly certain I've done something wrong---my normal wheelhouse is C++, not C#.

I have a Winforms project targeting .NET Framework 4.7.2 in Visual Studio Professional 2019 and I've installed FlatSharp.Runtime + FlatSharp.Compiler, both at version 4.1.0 with NuGet.

Building the project results in the following errors:

Rebuild started...
1>------ Rebuild All started: Project: SYCLCommsTester, Configuration: Debug Any CPU ------
1>  Compiling 
1>  C:\Users\<...>\source\repos\SYCLCommsTester\packages\FlatSharp.Compiler.4.1.0\tools\net47\FlatSharp.Compiler.exe C:\Users\<...>\source\repos\SYCLCommsTester\SYCLCommsTester\Flatbuffers\SYCL.fbs
1>C:\Users\<...>\source\repos\SYCLCommsTester\SYCLCommsTester\obj\Debug\SYCL.fbs.generated.cs(67,18,67,23): error CS1001: Identifier expected
1>C:\Users\<...>\source\repos\SYCLCommsTester\SYCLCommsTester\obj\Debug\SYCL.fbs.generated.cs(67,18,67,23): error CS1002: ; expected
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

For comparison, I tried using flatc directly, which appeared to produce valid code (although I don't currently have the FlatBuffers C# code installed in the project to check).

Further, I'm pretty sure the schema is valid as the same one works without incident in a C++ project.

Intellisense seems to think there might be a missing using directive or assembly reference, and sure enough, FlatSharp.Compiler isn't listed in visual studio's list of references for the project... though given my lack of experience with C# I can't say whether it's supposed to be there or not.

As an aside, it wasn't really clear to me from the readme which packages I'm meant to install, so while writing this issue I ended up installing FlatSharp (also v 4.1.0) itself as well to see if that fixed things (it didn't) 😅

Attached below are relevant files.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    <ProjectGuid>{13F04FF5-53E7-4D70-A600-3CE96960CCEF}</ProjectGuid>
    <OutputType>WinExe</OutputType>
    <RootNamespace>SYCLCommsTester</RootNamespace>
    <AssemblyName>SYCLCommsTester</AssemblyName>
    <TargetFrameworkVersion>v4.7.2</TargetFrameworkVersion>
    <FileAlignment>512</FileAlignment>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <Deterministic>true</Deterministic>
    <NuGetPackageImportStamp>
    </NuGetPackageImportStamp>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <PlatformTarget>AnyCPU</PlatformTarget>
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
    <Optimize>false</Optimize>
    <OutputPath>bin\Debug\</OutputPath>
    <DefineConstants>DEBUG;TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <PlatformTarget>AnyCPU</PlatformTarget>
    <DebugType>pdbonly</DebugType>
    <Optimize>true</Optimize>
    <OutputPath>bin\Release\</OutputPath>
    <DefineConstants>TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <ImportGroup Label="ExtensionTargets">
    <Import Project="flatbuffers.targets" />
    <Import Project="..\packages\FlatSharp.Compiler.4.1.0\build\net47\FlatSharp.Compiler.targets" Condition="Exists('..\packages\FlatSharp.Compiler.4.1.0\build\net47\FlatSharp.Compiler.targets')" />
  </ImportGroup>
  <ItemGroup>
    <Reference Include="FlatSharp.Runtime, Version=4.1.0.0, Culture=neutral, PublicKeyToken=06041f83a2658c5b, processorArchitecture=MSIL">
      <HintPath>..\packages\FlatSharp.Runtime.4.1.0\lib\netstandard2.0\FlatSharp.Runtime.dll</HintPath>
    </Reference>
    <Reference Include="System">
      <HintPath>..\..\..\..\..\..\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.dll</HintPath>
    </Reference>
    <Reference Include="System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51, processorArchitecture=MSIL">
      <HintPath>..\packages\System.Buffers.4.4.0\lib\netstandard2.0\System.Buffers.dll</HintPath>
    </Reference>
    <Reference Include="System.Core" />
    <Reference Include="System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51, processorArchitecture=MSIL">
      <HintPath>..\packages\System.Memory.4.5.3\lib\netstandard2.0\System.Memory.dll</HintPath>
    </Reference>
    <Reference Include="System.Numerics" />
    <Reference Include="System.Numerics.Vectors, Version=4.1.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
      <HintPath>..\packages\System.Numerics.Vectors.4.4.0\lib\net46\System.Numerics.Vectors.dll</HintPath>
    </Reference>
    <Reference Include="System.Runtime.CompilerServices.Unsafe, Version=4.0.4.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
      <HintPath>..\packages\System.Runtime.CompilerServices.Unsafe.4.5.2\lib\netstandard2.0\System.Runtime.CompilerServices.Unsafe.dll</HintPath>
    </Reference>
    <Reference Include="System.Xml.Linq" />
    <Reference Include="System.Data.DataSetExtensions" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Deployment" />
    <Reference Include="System.Drawing" />
    <Reference Include="System.Net.Http" />
    <Reference Include="System.Windows.Forms" />
    <Reference Include="System.Xml" />
  </ItemGroup>
  <ItemGroup>
    <FlatSharpSchema Include="Flatbuffers\SYCL.fbs" />
    <Compile Include="Form1.cs">
      <SubType>Form</SubType>
    </Compile>
    <Compile Include="Form1.Designer.cs">
      <DependentUpon>Form1.cs</DependentUpon>
    </Compile>
    <Compile Include="Program.cs" />
    <Compile Include="Properties\AssemblyInfo.cs" />
    <EmbeddedResource Include="Form1.resx">
      <DependentUpon>Form1.cs</DependentUpon>
    </EmbeddedResource>
    <EmbeddedResource Include="Properties\Resources.resx">
      <Generator>ResXFileCodeGenerator</Generator>
      <LastGenOutput>Resources.Designer.cs</LastGenOutput>
      <SubType>Designer</SubType>
    </EmbeddedResource>
    <Compile Include="Properties\Resources.Designer.cs">
      <AutoGen>True</AutoGen>
      <DependentUpon>Resources.resx</DependentUpon>
    </Compile>
    <None Include="packages.config" />
    <None Include="Properties\Settings.settings">
      <Generator>SettingsSingleFileGenerator</Generator>
      <LastGenOutput>Settings.Designer.cs</LastGenOutput>
    </None>
    <Compile Include="Properties\Settings.Designer.cs">
      <AutoGen>True</AutoGen>
      <DependentUpon>Settings.settings</DependentUpon>
      <DesignTimeSharedInput>True</DesignTimeSharedInput>
    </Compile>
  </ItemGroup>
  <ItemGroup>
    <None Include="App.config" />
  </ItemGroup>
  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
  <PropertyGroup>
    <PreBuildEvent>
    </PreBuildEvent>
  </PropertyGroup>
  <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
    <PropertyGroup>
      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
    </PropertyGroup>
    <Error Condition="!Exists('..\packages\FlatSharp.Compiler.4.1.0\build\net47\FlatSharp.Compiler.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\FlatSharp.Compiler.4.1.0\build\net47\FlatSharp.Compiler.targets'))" />
  </Target>
</Project>

namespace SYCL;

enum MessageType : byte { None = -2, Nack, Ack }

enum EventType : byte {
	evInvalid = -1,
	evReady,
	evLockCovers,
	evCanUnlock,
	evCanUnlockReceived,
	evEnableUnlockCovers,
	evButtonUnlock,
	evPrepareUnlockComplete,
	evUnlockAllowed,
	evUnlockAllowedReceived,
	evCoverOpen,
	evCoverClosed,
	evTimer,
	evAbort,
	evPeerAbort,
	evPeerAbortReceived,
	evPeerPresent,
	evPeerCoversUnlocked,
	evPeerCoversUnlockedReceived,
	evPeerLockCovers,
	evPeerLockCoversReceived,
	evPeerCoversLocked,
	evPeerCoversLockedReceived,
	evResumeComplete
}

table Message {
	type:MessageType = None;
	event:EventType = evInvalid;
}

root_type Message;

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by the FlatSharp FBS to C# compiler (source hash: 4.1.0.rpjvg+iPJ95nZOpRm7z5r/ZNW2oz0cV66tShV6dLRQE=)
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.CompilerServices;
using FlatSharp;
using FlatSharp.Attributes;
namespace SYCL
{
    [FlatBufferEnum(typeof(System.SByte))]
    [System.Runtime.CompilerServices.CompilerGenerated]
    public enum MessageType : System.SByte
    {
        None = (sbyte)(-2),
        Nack = (sbyte)(-1),
        Ack = (sbyte)(0),
    }
    
    [FlatBufferEnum(typeof(System.SByte))]
    [System.Runtime.CompilerServices.CompilerGenerated]
    public enum EventType : System.SByte
    {
        evInvalid = (sbyte)(-1),
        evReady = (sbyte)(0),
        evLockCovers = (sbyte)(1),
        evCanUnlock = (sbyte)(2),
        evCanUnlockReceived = (sbyte)(3),
        evEnableUnlockCovers = (sbyte)(4),
        evButtonUnlock = (sbyte)(5),
        evPrepareUnlockComplete = (sbyte)(6),
        evUnlockAllowed = (sbyte)(7),
        evUnlockAllowedReceived = (sbyte)(8),
        evCoverOpen = (sbyte)(9),
        evCoverClosed = (sbyte)(10),
        evTimer = (sbyte)(11),
        evAbort = (sbyte)(12),
        evPeerAbort = (sbyte)(13),
        evPeerAbortReceived = (sbyte)(14),
        evPeerPresent = (sbyte)(15),
        evPeerCoversUnlocked = (sbyte)(16),
        evPeerCoversUnlockedReceived = (sbyte)(17),
        evPeerLockCovers = (sbyte)(18),
        evPeerLockCoversReceived = (sbyte)(19),
        evPeerCoversLocked = (sbyte)(20),
        evPeerCoversLockedReceived = (sbyte)(21),
        evResumeComplete = (sbyte)(22),
    }
    
    [FlatBufferTable]
    [System.Runtime.CompilerServices.CompilerGenerated]
    public partial class Message : object
    {
        partial void OnInitialized();
         public Message() { this.OnInitialized(); }
        public Message(Message source)
        {
            this.type = source.type;
            this.event = source.event;
            this.OnInitialized();
        }
        
        [FlatBufferItem(0, DefaultValue = global::SYCL.MessageType.None)]
        public virtual global::SYCL.MessageType type { get;  set; } = global::SYCL.MessageType.None;
        [FlatBufferItem(1, DefaultValue = global::SYCL.EventType.evInvalid)]
        public virtual global::SYCL.EventType event { get;  set; } = global::SYCL.EventType.evInvalid;
    }
}

// <auto-generated>
//  automatically generated by the FlatBuffers compiler, do not modify
// </auto-generated>

namespace SYCL
{

using global::System;
using global::System.Collections.Generic;
using global::FlatBuffers;

public enum MessageType : sbyte
{
  None = -2,
  Nack = -1,
  Ack = 0,
};

public enum EventType : sbyte
{
  evInvalid = -1,
  evReady = 0,
  evLockCovers = 1,
  evCanUnlock = 2,
  evCanUnlockReceived = 3,
  evEnableUnlockCovers = 4,
  evButtonUnlock = 5,
  evPrepareUnlockComplete = 6,
  evUnlockAllowed = 7,
  evUnlockAllowedReceived = 8,
  evCoverOpen = 9,
  evCoverClosed = 10,
  evTimer = 11,
  evAbort = 12,
  evPeerAbort = 13,
  evPeerAbortReceived = 14,
  evPeerPresent = 15,
  evPeerCoversUnlocked = 16,
  evPeerCoversUnlockedReceived = 17,
  evPeerLockCovers = 18,
  evPeerLockCoversReceived = 19,
  evPeerCoversLocked = 20,
  evPeerCoversLockedReceived = 21,
  evResumeComplete = 22,
};

public struct Message : IFlatbufferObject
{
  private Table __p;
  public ByteBuffer ByteBuffer { get { return __p.bb; } }
  public static void ValidateVersion() { FlatBufferConstants.FLATBUFFERS_1_12_0(); }
  public static Message GetRootAsMessage(ByteBuffer _bb) { return GetRootAsMessage(_bb, new Message()); }
  public static Message GetRootAsMessage(ByteBuffer _bb, Message obj) { return (obj.__assign(_bb.GetInt(_bb.Position) + _bb.Position, _bb)); }
  public void __init(int _i, ByteBuffer _bb) { __p = new Table(_i, _bb); }
  public Message __assign(int _i, ByteBuffer _bb) { __init(_i, _bb); return this; }

  public SYCL.MessageType Type { get { int o = __p.__offset(4); return o != 0 ? (SYCL.MessageType)__p.bb.GetSbyte(o + __p.bb_pos) : SYCL.MessageType.None; } }
  public SYCL.EventType Event { get { int o = __p.__offset(6); return o != 0 ? (SYCL.EventType)__p.bb.GetSbyte(o + __p.bb_pos) : SYCL.EventType.evInvalid; } }

  public static Offset<SYCL.Message> CreateMessage(FlatBufferBuilder builder,
      SYCL.MessageType type = SYCL.MessageType.None,
      SYCL.EventType event = SYCL.EventType.evInvalid) {
    builder.StartTable(2);
    Message.AddEvent(builder, event);
    Message.AddType(builder, type);
    return Message.EndMessage(builder);
  }

  public static void StartMessage(FlatBufferBuilder builder) { builder.StartTable(2); }
  public static void AddType(FlatBufferBuilder builder, SYCL.MessageType type) { builder.AddSbyte(0, (sbyte)type, -2); }
  public static void AddEvent(FlatBufferBuilder builder, SYCL.EventType event) { builder.AddSbyte(1, (sbyte)event, -1); }
  public static Offset<SYCL.Message> EndMessage(FlatBufferBuilder builder) {
    int o = builder.EndTable();
    return new Offset<SYCL.Message>(o);
  }
  public static void FinishMessageBuffer(FlatBufferBuilder builder, Offset<SYCL.Message> offset) { builder.Finish(offset.Value); }
  public static void FinishSizePrefixedMessageBuffer(FlatBufferBuilder builder, Offset<SYCL.Message> offset) { builder.FinishSizePrefixed(offset.Value); }
};


}

Implement buffer validation as per: https://github.com/dvidelabs/flatcc/blob/master/doc/binary-format.md#verification

Oracle-based testing is not good enough. This issue tracks adding comprehensive output verification for struct serialization scenarios.

Produce some real documentation for various custom compiler attributes, etc that goes beyond the scope of samples.

Include:

• Flatbuffers supported/unsupported features
• Index of all Flatsharp FBS custom attributes
• Some words about different deserialization options
• Extensibility notes

I did some prototyping in #42, but am not familiar with visitors & do not have a vision for what the code generation would look like. This requires #40

I do not intend to continue working on #42 because I'm not sure what the solution would look like. Hopefully this helps.

FlatSharp looks like a very nice library for FB in C#, however when trying to test FS vs "plain" FB, I immediately ran into a showstopper with my existing and widely used schema; no support for key/sorted dictionaries in schema. I was unable to generate anything so I don't have an idea (yet) on what it would take to support it. Is supporting key something that is considered or planned?

I will remove (key) for testing probably, but this will break native usage which relies on binary search for lookup. Perhaps a manual sort will suffice?

Change:

for (int i = 0; i < vector.Count; ++i)
{
   sum += vector[i];
}

return sum;

to

int sum1, sum2, sum3, sum4;
for (int i = 0; i < vector.Count - 4; i += 4)
{
   sum1 += vector[i];
   sum2 += vector[i + 1];
   sum3 += vector[i + 2];
   sum4 += vector[i + 3];
}
return sum1 + sum2 + sum3 + sum4;

This change:

• reduces the number of comparisons and jumps when dealing with vectors
• does not interfere with bounds check eliding logic in the JIT
• allows the processor to run out of order instructions (as sum1 does not depend on sum2, etc)

Hello,

it seems that OnInitialized is getting invoked before things are deserialized with a greedy table like this:

table Character (fs_serializer, fs_nonVirtual) 
{
    // ...
}

Generated subclass constructor:

public tableReader_261a33b842a84271a44efba629b74851(TInputBuffer buffer, int offset) 
: base(__CtorContext)
{
    checked
    {
        buffer.InitializeVTable(offset, out var __vtableLocation, out var __vtableMaxIndex);
        base.Shortname = ReadIndex0Value(buffer, offset, __vtableLocation, __vtableMaxIndex);
        //...
    }
}

Generated base class constructor:

protected Character(FlatBufferDeserializationContext context)
{
    checked
    {
        this.OnInitialized(context);
    }
}

Since this is a protected base constructor, it should be safe to move the OnInitialized(context) call into the subclass constructor after everything is deserialized?

The problem with parsing is that it relies on InputBuffer class.
This would need something like this for reading Spans.

public class SpanInputBuffer : InputBuffer
{
    private Span<byte> _span;
    public SpanInputBuffer(Span<byte> span) 
    {
       _span = span;
    }
   
    public override byte ReadByte(int offset) => _span[offset];
    [...]
}

However as we all know spans are stack-only, so they can not be field members.
A possible solution would be to read from buffers with unsafe code instead (e.g. from IntPtr or byte*).
Kudos to @nxrighthere for the fix suggestion.

Investigate what it would take to add proper enum support safely. The biggest risk is that it's easy for a developer to change

public enum MyEnum

to

public enum MyEnum : long

which causes a binary break in the FlatBuffer protocol, which wouldn't be at all obvious to the engineer. One way to do this would be to annotate the enum properties with a "RawDataType" attribute that we could check against the backing type of the enum, and scream if they don't match.

IList<T> is implemented by many different classes:

• T[]
• List<T>
• FlatBufferVector<T>
• ReadOnlyCollection<T>
• Custom implementations

Some of these, such as T[] are special and can observe large performance gains when using the type directly, but suffer the virtual method penalty when exposed through the IList<T> interface.

Devirtualization would look like:

public void ListOp<T>(IList<T> list)
{
     if (list is T[] array)
     {
         {array-loop}
     }
     else if (list is List<T> realList)
     {
        {list-loop}
     }
     else
     {
         {ilist-loop}
     }
}

This issue tracks the work necessary to investigate:

• Which well known IList<T> implementations have substantial performance gains when the underlying type is used directly? Probably T[] and List<T>, but anything else?
• What is the penalty for devirtualization via the is operator? How expensive are these comparisons?
• Is there a length threshold at which this becomes (un)profitable?
• What is the code size impact? ie, does the increase in binary size come with an increase in performance?

Customers should be able to specify their own type model containers or facades when invoking the Flatsharp compiler. This will (probably) need to take place in the form of a plugin that the compiler can load and invoke to request the type model container it should use.

FlatSharp should interoperate cleanly with other tools, and the current set of attributes pollutes the global namespace (ie, what if another library used "Precompiledserializer" or whatnot?)

Proposal here is to namespace all flatsharp specific attributes with "fs_" this is unobtrusive and allows us to stay out of the way of others.

Set up some CI pipeline for pull requests!

I have my tables setup like this.

Which is based on this template i made (which does parse correctly)

When the file is deserialized, it will print out the exception. In turn it causes the json serializer to break due to "Error getting value from Item2, operation not valid to current state of object"

The items appear to be null when previewed.

Which appears to give issues with json's deserializer. I am unsure if there's a particular setting i might need, or is this a potential issue with the library?

Investigate support for nint in serializers. Does this help performance at all?

Hello, my C#/NuGet knowledge is a little weak, so I apologize if this is a dumb question. I'm using Visual Studio for macOS and I installed the NuGet package for the compiler. I'd like to setup a script to execute it from the command line to generate the C# code from my .fbs file. I assume I have to invoke dotnet [something-or-other]. I found this documentation for .NET Core tools, but running dotnet tool list does not show that the FlatSharp compiler is installed, although I do see it in the [home]/.nuget/packages/flatsharp.compiler/3.0.0/tools/netcoreapp2.1 folder. How can I install the compiler package and invoke it from the command-line without referencing that specific path?

Thank you very much.

Antlr parser grammar needs a semicolon token to recognize these statements.

I didn't see any warnings or errors, which is disappointing. I had no idea why no code was being generated.

Make compiler errors more visible and actionable to the user. Detect features that are unsupported and report them as errors instead of failing mysteriously.

The compiler doesn't seem to generate the code if the items used from the Include are in a different namespace.

__
e.g.

A.fbs:

Include "B.fbs"
namespace my.namespace.stuff;

table Bar {
  MyFoo:Foo
}

B.fbs

namespace my.namespace.otherstuff; 

table Foo {
}

Compiler unable to resolve Foo in A.fbs

We encounter an issue when using FlatSharp in Unity.
It throw compilation error.
FlatSharp compilation error: (16,68): error CS0433: The type 'Span<T>' exists in both 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089', Context = "Span<byte> target"

Any hints how to fix this? Thanks.

Flatsharp compiler should emit code with Nullable annotations enabled.

First of all, thanks for a really nice library! We're currently in the process of replacing our proprietary serializer with FlatBuffers and this library is really helpful.

I'm curious if there is any simple way to convert a scalar type in FlatBuffers to a common C# CLR type. This was something which our previous solution handled quite nicely.

In my scenario I want to be able to automatically parse DateTime and DateTimeOffset from a long (a unix timestamp in milliseconds).

Basically, what I want to do is have a custom converter from long to DateTime. So when the parser runs across a long which has the target CLR-type of DateTime or DateTimeOffset, I want to make a quick conversion from a UNIX timestamp, and when the serializer runs across a DateTime, I simply want to output a UNIX timestamp.

I've tried to implement an ITypeModel called DateTimeTypeModel but that interface is really cumbersome to work with. The RuntimeTypeModel has an internal constructor, so you can't inherit it, but it would help easing implementation.

Any advice on how to do this?

I tried to create a new Asp.Net Core (Mvc) Formatter using FlatSharp but unfortunately the Read*() / Write*() methods that need to be implemented (see TextOutputFormatter / TextInputFormatter) to do so are all non-generic ones so gluing FlatSharp and Asp.Net Core formatters together is a bit.. difficult right now (or maybe I am just blind at first glance / attempt).

Is there any non-generic .Serialize() / Parse() already available with FlatSharp that I didn't see, is it maybe planned or not? Just trying to get an idea if that's something planned / doable / I can maybe help a bit with.

Thanks!
-Jörg

Would be fantastic to be able to generate F# code as well as C#. In particular, F# supports discriminated unions natively so the clunky FlatBufferUnion could be removed.

.NET Core 3.1 and 5.0 support SIMD intrinsics.

• Are any of these useful to FlatSharp? Some candidates:
  • GetMaxSize for tables and vectors.
  • VTable equality checks (does Span.SequenceEqual already do this internally?)
• If any of them seem useful, can we use them without introducing unsafe code?

Hi there, I've provided an example below of a case that works and one that fails. For the case the fails can you please provide a solution.

using System;
using System.Collections.Generic;
using FlatSharp;
using FlatSharp.Attributes;

namespace demo_flatsharp
{
    class Program
    {
        static void Main(string[] args)
        {
            // This works:
            var myDemo = new MyDemo()
            {
                MyList = new List<MyData>() { new MyData { Secret = 0 } }
            };
            byte[] myBytes = new byte[100];
            FlatBufferSerializer.Default.Serialize(myDemo, myBytes.AsSpan());

            // This fails:
            IList<MyData> myData = new List<MyData>() { new MyData { Secret = 0 } };
            myBytes = new byte[100];
            FlatBufferSerializer.Default.Serialize(myData, myBytes.AsSpan());

            // 'Unable to compile. This represents a bug in FlatSharp type model validation. Errors = (10,104): error CS0246: The type or namespace name 'demo_flatsharp' could not be found (are you missing a using directive or an assembly reference?)
            //(14, 106): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(21,72): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(28,57): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(35,155): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(53,122): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(85,61): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(96,28): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(157,111): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(172,78): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(102,85): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)
            //(128,50): error CS0115: 'Serializer.tableReader_4b979bc24fe74c6686a70bb5377edfad.Secret': no suitable method found to override
            //(87,45): error CS0246: The type or namespace name 'demo_flatsharp' could not be found(are you missing a using directive or an assembly reference?)'
        }
    }

    [FlatBufferTable]
    public class MyDemo
    {
        [FlatBufferItem(0)]
        public virtual IList<MyData> MyList { get; set; }
    }

    [FlatBufferTable]
    public class MyData
    {
        [FlatBufferItem(0)]
        public virtual int Secret { get; set; }
    }
}

Yo,

There are no new commits for months, has development been stopped?

Cheers!

When using build-time generation, the first build fails as the newly generated .cs files are not added to the current compilation task.
Second build runs fine, which is the workaround I am currently using in my automation.

Hi there, are nullables supported if not can support be added.

I get the following exception:

FlatSharp.TypeModel.InvalidFlatBufferDefinitionException: Unable to create runtime type model for Nullable1`

    [FlatBufferTable]
    public class MyData
    {
        [FlatBufferItem(0)]
        public virtual int? Secret { get; set; }
    }

opened by mistake

Hi there, FlatSharp seems to be pretty neat. I've worked with protobuf-net in Unity before and I'm looking for a similar library for Flatbuffers, so I thought I'll check it out.

Support for Unity is mentioned in the readme, so maybe you can help me with this issue I'm having. So far I've imported FlatSharp and FlatSharp.Runtime into my Unity Project including all of the dependencies. That means, I also include System.Memory.dll which creates some issues as we'll see later.

When FlatSharp generates a Serializer at runtime it's throwing a FlatSharpCompilationException. Those are the 3 CompilerErrors that it reports:

(17,54): error CS0433: The type 'Span<T>' exists in both 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
UnityEngine.Debug:LogError(Object)
FlatBufferTestClass:Start() (at Assets/FlatBufferTestClass.cs:21)

(38,105): error CS0433: The type 'Span<T>' exists in both 'System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
UnityEngine.Debug:LogError(Object)
FlatBufferTestClass:Start() (at Assets/FlatBufferTestClass.cs:21)

(13,59): error CS0535: 'GeneratedSerializer' does not implement interface member 'IGeneratedSerializer<Test>.Write(SpanWriter, Span<byte>, Test, int, SerializationContext)'
UnityEngine.Debug:LogError(Object)
FlatBufferTestClass:Start() (at Assets/FlatBufferTestClass.cs:21)

So Span is already defined in the Unity-provided mscorlib. A google search brought up some information on why that might be the case:
dotnet/runtime#26144
dotnet/corefx#28034

The base issue seems clear, but I'm not sure what to make of that information, so any help would be appreciated.

Btw, I'm on Unity 2019.3.14.f1, .Net Standard 2.0, Visual Studio 2017 (v15.9.12)

Namaste!
Today I've upgraded FlatSharp from 4.1.0 to 4.2.2 and got a problem during solution building with a few such error messages:

CaseCreated.fbs.generated.cs(106,56,106,64): error CS8400: Feature 'not pattern' is not available in C# 8.0. Please use language version 9.0 or greater.

It turns out compilers code contains such strings:

if (index1Value is not null)
{
  runningSum += GetMaxSizeOf_c96a83b5ae264b13b4665aede73aac9f(index1Value);
}

Notice is not null. Seams like it's a feature of C# 9.0 which is not supported in .NET Core 3.1. I don't plan to move to .NET 5 for the moment. Can you return back old good != null? Or I can return it back just tell me which files to start to look for this code?

Currently including the FlatSharp.Compiler NuGet package in my ASP.NET Core 3.1 and I am getting incompatible framework error when trying to build. It tells me: The framework 'Microsoft.NETCore.App', version '2.1.0' was not found. When I do not include the Compiler, the project builds successfully. Is there a way to easily bump the version requirements or is there some work that needs to be done to make it compatible?

The one thing we haven't been able to get this fantastic project to do for us, is to populate all the values when serializing even when they're defaults. I believe the stock C# fbs compiler implements the force_defaults feature, but we'd much rather use FlatSharp.

Are we just missing how to enable that behavior? I looked through the samples and the code a little and haven't found anything like that -- options only really seem to exist on the Deserialization front (even though they're sometimes referred to as SerializerOptions).

Hi,

first of all thank you for making this awesome library.

I've tried to run the schema below but the compiler complained that "Structs may not have serializers" because of the sf_serializer attribute:

struct EnchantmentAttribute (fs_serializer, fs_nonVirtual) {
EnchantmentLevel:int;
}

Does this mean that structs are only intended to be used with the runtime FlatSharp.dll?

I have the following struct in my fbs schema file

struct Hash {
    sha256: [uint8:32];
}

When trying to compile this I get the following error:

Syntax error FBS file: Token=':', Msg='mismatched input ':' expecting ']'' Line='4:18'

According to the flatbuffer guide this is a valid schema. Is this a bug in the FlatSharp compiler?

Howdy,

This is probably a total noob question, but I'm not able to get a serializer instantiated:

            Upstream up = new Upstream();
            up.Structure = new FSeed();
            up.Structure.Data = new ConfigEntry[1];
            var e0 = new ConfigEntry();
            up.Structure.Data[0] = e0;
            e0.Tag = "stuff";
            e0.Type = "string";
            e0.Used = true;
            
            Console.WriteLine("test serializing up");
            
            var testBuffer = new byte[1000];
            
            var serializer = new FlatBufferSerializer(new FlatBufferSerializerOptions(FlatBufferDeserializationOption.Default));
            
            var testLength = serializer.Serialize(up, testBuffer);
            
            Upstream up2 = serializer.Parse<Upstream>(testBuffer);

I keep on getting this exception:

System.TypeInitializationException: The type initializer for 'FlatSharp.RoslynSerializerGenerator' threw an exception. ---> System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
   at System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence)
   at System.Reflection.Assembly.LoadFile(String path)
   at FlatSharp.RoslynSerializerGenerator..cctor()
   --- End of inner exception stack trace ---
   at FlatSharp.RoslynSerializerGenerator..ctor(FlatBufferSerializerOptions options, TypeModelContainer typeModelContainer)
   at FlatSharp.FlatBufferSerializer.GetOrCreateSerializer[TRoot]()
   at FlatSharp.FlatBufferSerializer.Serialize[T,TSpanWriter](T item, Span`1 destination, TSpanWriter writer)
   at FlatSharp.FlatBufferSerializer.Serialize[T](T item, Span`1 destination)
   at LibToFuzz.FuzzMe.Main(String[] args) in C:\Users\PeterFuzz\go\src\github.com\fuzzbuzz\platform\sfz\sfz-build-csharp\LibToFuzz\LibToFuzz\Program.cs:line 104

I'm bewildered. Nothing inside the exception object resembles a filename.

Hi,

on the starting page the example of MonsterTable has attributes like:
[DefaultValue(100s)]
but correct format is:
[DefaultValue((short)100)]

next
[FlatBufferItem(4, IsDeprecated = true)]
does not compile, because of the "IsDeprecated" parameter.

and there is
public virtual Float Z { get; set; }
with "F"loat instead of "f"loat.

Summary

Flatsharp recently added a feature to allow custom IDs on tables. This allowed users to specify tables with out-of-order IDs:

table MyTable {
  string:string (id:2);
  int:int (id: 1);
  byte:byte (id:0);
}

Flatsharp had a longstanding internal bug that this revealed due to storing the IDs as they are discovered. In situations like this, it is possible for the higher-numbered ID values to be truncated from the buffer. This bug dates back years, but is -- luckily -- relatively infrequent and only occurs when a table has a large number of items.

The fix for this will be available shortly in 4.2.4.

@Eshva -- this may be applicable to you.

James, thanks for this implementation. I like the fact that it's based on Memory<T> and Span<T>, this is something fresh for me in the serializers area.

Can you please provide data size comparison with those serialization libraries in the readme?

Oracle-based testing is not good enough. This tracks adding comprehensive tests for writing table data and verifying output on a per-byte basis.

I created an empty table and the compiler warn me that
FlatSharp.Compiler.targets(28, 5): Message='Can't create table type model from type Package.LoginPayload because it does not have any non-static, non-deprecated [FlatBufferItem] properties.', Scope=$.

Empty table is not supported by FlatSharp or Flatbuffer?

Add nullable annotations to FlatSharp itself -- not just the generated code.