rixed / dessser Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 2.0 2.08 MB

(de)serializer generator

License: GNU General Public License v3.0

Makefile 1.89% OCaml 90.70% M4 0.72% Nix 0.07% C++ 5.21% C 1.34% Roff 0.06%

dessser's People

Contributors

Stargazers

Watchers

Forkers

darlentar axiles

dessser's Issues

More consistent behavior of assert_

There should be no need for a "debug" flag in des/ser, given asserts are supposed to be NOP when NDEBUG.

OCaml name validation must consider case (`In` should not be renamed into `In_`)

dessserc: An option to skip generating declaration files

Compilation error: windows cannot find it's type aliases

c:\Users\grixe\Sources\ramen-admin\src>c:\Qt\Tools\mingw810_64\bin\mingw32-make.exe
c:/Qt/Tools/mingw810_64/bin/mingw32-make -f Makefile.Release
mingw32-make[1]: Entering directory 'c:/Users/grixe/Sources/ramen-admin/src'
g++ -c -fno-keep-inline-dllexport -DSODIUM_STATIC -O2 -std=gnu++1z -Wall -Wextra -Wextra -fexceptions -mthreads -DUNICODE -D_UNICODE -DWIN32 -DMINGW_HAS_SECURE_API=1 -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_CHARTS_LIB -DQT_PRINTSUPPORT_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_TESTLIB_LIB -DQT_CORE_LIB -DQT_TESTCASE_BUILDDIR="\"C:/Users/grixe/Sources/ramen-admin/src\"" -I. -I. -I/usr/local/include -I..\..\dessser\src -I..\..\..\libsodium-win64\include -IC:\Qt\5.15.2\mingw81_64\include -IC:\Qt\5.15.2\mingw81_64\include\QtCharts -IC:\Qt\5.15.2\mingw81_64\include\QtPrintSupport -IC:\Qt\5.15.2\mingw81_64\include\QtWidgets -IC:\Qt\5.15.2\mingw81_64\include\QtGui -IC:\Qt\5.15.2\mingw81_64\include\QtANGLE -IC:\Qt\5.15.2\mingw81_64\include\QtNetwork -IC:\Qt\5.15.2\mingw81_64\include\QtTest -IC:\Qt\5.15.2\mingw81_64\include\QtCore -Irelease -I/include -IC:\Qt\5.15.2\mingw81_64\mkspecs\win32-g++  -o release\desssergen\team_name.o desssergen\team_name.cc
In file included from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/minwindef.h:163,
                 from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/windef.h:8,
                 from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/windows.h:69,
                 from ..\..\dessser\src/dessser/Ip.h:16,
                 from ..\..\dessser\src/dessser/runtime.h:25,
                 from desssergen\team_name.cc:18:
C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/winnt.h:334:11: error: 'CHAR' does not name a type; did you mean 'WCHAR'?
   typedef CHAR *PCHAR,*LPCH,*PCH;
           ^~~~
           WCHAR

OCaml backend: lists should be... lists.

Create an eval function

Very slow JSON parsing for some invalid input

With that JSON (out of ramen alerting configuration):

{"via": {"Kafka": { "options": [("topic.message.timeout.ms","5000")], "topic": "alert-events", "partition": 0, "text": "yada" }}}

Note: it should be ... "options": [["topic.message.timeout.ms","5000"]],..., it is apparently confused by the parentheses.

SER/DES state parameter is really not that useful and should be done away with

The only 2 encoders using it are Csv and SExpr, which use it for static configuration.
Let's just have a global config for those and call it a day?
Alternatively, if we want to avoid global vars at all cost, we could have functors?

Random type and expression generator

To use with quickcheck, testing that:

evey expressions that type-check can be compiled in all backends
every type can be printed and also parsed back into the same type
every expressions can be printed and parsed back (requires a thorough overhaul of expression printing)

uuid_of_uint128 is too slow

JSON encoding for sum type with no parameter should be a mere string

Make it possible to define some types in advanceso that compound types can be identified with external types

Case at hand:
RamenIp.t is the type used by all IP operations but dessser defines a new V4 of u32 | v6 of u128 type, that is structurally equivalent but still a distinct type.
We'd like to be able to tell dessser that there already exist a type definition for those types, under the name RamenIp.t.

C++ backend: an option to choose which type of pointers to use

S-Expr SER

Another type error involving external types

Fatal error: exception Type Error: In expression
    (apply (ext-identifier test_ext of-ringbuf) (identifier "with_nullbit_done2_fst_126"))
expression
    (identifier "with_nullbit_done2_fst_126")
should be a (Ptr; (Ptr; Size)[[]]) but is a Ptr
Raised at file "src/DessserExpressions.ml", line 1068, characters 10-60
Called from file "src/batList.mlv", line 1023, characters 6-11
Called from file "src/DessserTypeCheck.ml" (inlined), line 111, characters 25-44
Called from file "src/DessserTypeCheck.ml", line 192, characters 12-33
Called from file "src/DessserExpressions.ml", line 1241, characters 15-29
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1241, characters 15-29
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1244, characters 15-30
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1244, characters 15-30
Called from file "src/DessserExpressions.ml", line 1231, characters 15-30
Called from file "src/DessserTypeCheck.ml" (inlined), line 15, characters 2-1023
Called from file "src/DessserCompilationUnit.ml", line 123, characters 13-33
Called from file "src/DessserHeapValue.ml", line 331, characters 29-56
Called from file "src/dessserc.ml", line 123, characters 27-59
Called from file "list.ml", line 117, characters 24-34
Called from file "src/dessserc.ml", line 156, characters 17-65
Called from file "src/dessserc.ml", line 708, characters 13-17

OCaml backend should use slices for strings

json_skip function present twice in generated code

Tail should return a nullable

regression: JSon serializer do not include field name on external types?

Detect code duplication

One of the most challenging issue when using dessser is to forget to bind largish chunks of code to a stage-1 variable, leading to very large generated code, which is not always easy to spot when looking at the DIL code, since it is large and repetitive by nature.

There could be a function we call explicitly before transpiling to the backend, that tries to detect that this occurred and warn.
Ie. detection of common subtrees in large tree.

Make an abstraction for the environment

Can't use "this" and "this subtype" in the same type definition

Fatal error: exception Unknown type "t". Only known types are: subtype
Raised at file "src/DessserTypes.ml", line 1083, characters 4-26
Called from file "src/DessserTypes.ml", line 1166, characters 15-26
Called from file "src/batArray.mlv", line 483, characters 12-49
Called from file "src/DessserTools.ml", line 134, characters 6-26
Called from file "list.ml", line 221, characters 17-20
Called from file "src/DessserTypes.ml", line 1122, characters 10-21
Called from file "src/DessserTypes.ml", line 1107, characters 20-33
Called from file "src/DessserTypes.ml", line 1118, characters 55-63
Called from file "array.ml", line 103, characters 21-40
Called from file "src/DessserTypes.ml", line 1118, characters 25-69
Called from file "src/DessserTypes.ml", line 1145, characters 12-20
Called from file "list.ml", line 106, characters 12-15
Called from file "src/dessserc.ml", line 90, characters 2-32
Called from file "src/dessserc.ml", line 708, characters 13-17

Option to not include code that deal with dynamic fieldmask

For the many cases where no fieldmask are needed.
Generated code should be noticeably faster.

CSV parser option to parse lists as clickhouse does

Arrays are serialized in CSV as follows: first, the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double-quotes.

and the array serialization that's referred to is:

Arrays are written as a list of comma-separated values in square brackets.

-- https://clickhouse.tech/docs/en/interfaces/formats/

Note that what CH calls array we call list.

Add aggregation hints to record/tuple items

Some introspection for generated types

For instance, a const betraying the type, then a function providing the label name for enums (or erring out for non enum likes), etc.
The idea is that it should be possible to write static code that dynamically adapts to any type.
Required in C++ backend first but the API would benefit from being the same for other backends.

Have a transform function to desser so values can be updated without full meterialization

As was used by rebinary.

Add optional units to record/tuple items

Add a "pointer-of-{input,output}-file" expression

So we can read/write directly from/into a file

loops should have a procedural interface

functions are hard to optimize

deadloop in read_while_loop when deserializing rowbinary

Named types that are not used recursively should not be pointy

Have some codec for JSON

If only with a specific field order for the decoder

DIL type notation should accept a name hint for types

For instance, [ A | B ] AS a_or_b should make backend code generator use that name for that type.
This, of course, does not make a_or_b an external type ($a_or_b is invalid).

Could also come handy for improving support for recursive types.

There should be no need to reorder record fields in dessser

...that's purely a RaQL thing.

Do not depend on Unix specific C libs

such as arpa/inet.h, charconv

Random value generator for a given type

Can be used to test actual serializer/deserializers by checking a given value can go through ser/deser without loss, for every backend and format.

The simplest is to generate values as S-Expressions that can then be unserialized using the S-expression DESserializer.

printer displays addresses for pointers instead of dereferencing?

Seen on ramen's TargetConfig: "TargetConfig [ 0x55555da310, ... ]"

OCaml backend looks ok: "TargetConfig [|{ program=... }|]" only because it's hand coded. It's not using the automatic conversion to string.

A way to setget global variables of any maybe_nullable type

Also required by rebinary for the time offset

Add optional doc to types

Have desser function provide each individual ser/des functions the path within vtyp

OsX: C++ compilation error

 [48 1! / 111] >check_heapvalue>src/DessserQCheck.ml:1125                      /var/folders/x6/4gb9ck5d4kb6s97zqqh8vvmh0000gn/T/dessser_converter_b0b81a.cc:150:69: error: no matching constructor for initialization of
      '::dessser::gen::test::t3c0e2f6ceb2032915f19660622878e72'
  ...id_20113 { .v_96a026eaf0977b4366de8e19d25b7792_ejgvx = drec_fst_4864, .v_96a026eaf0977b4366de8e19d25b7792_kngke = drec_fst_4867 };
     ^        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Propose a `shared_ptr<T _const_>` as possible pointer type

OCaml backend name simplification should not abbreviate one branch more than another at the same depth

of_json function about 4 times larger than expected

Add default values to types and codecs

So that we can save space on the wire (for the encodings that allow it).
Also, makes it nicer for manually entered values (such as in JSON)

Actually, not all encoding formats allow this.
RamenRingBuf, for instance, has no way to skip a value, Every non null values are expected to appear in a specific order. For skipping values we would need to adapt the nullbit mask to have one bit per nullable or default-able value. Actually, that would be a presence mask, and nullable values would just have an implicit default to null. Alternatively, we could have an implicit default value for any type, and have one bit per field.

Type error when generating code for ringbuf or JSON encoder

Testing dessserc
Testing I/O encoding ringbuf
...on tests/test_shrink_bug.type
Fatal error: exception Type Error: In expression
    (apply (myself "(THIS; Ptr)") (if (eq (unsafe-nth (u8 0) (identifier "repeat_n_326")) (i32 0)) (identifier "dlist2_snd_329") (identifier "dlist2_snd_329")))
expression
    (if (eq (unsafe-nth (u8 0) (identifier "repeat_n_326")) (i32 0)) (identifier "dlist2_snd_329") (identifier "dlist2_snd_329"))
should be a Ptr but is a (Ptr; (Ptr; Size)[[]])
Raised at file "src/DessserExpressions.ml", line 1062, characters 10-60
Called from file "src/batList.mlv", line 1023, characters 6-11
Called from file "src/DessserTypeCheck.ml" (inlined), line 111, characters 25-44
Called from file "src/DessserTypeCheck.ml", line 191, characters 12-33
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1267, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/batList.mlv", line 239, characters 23-28
Called from file "src/DessserExpressions.ml", line 1218, characters 15-40
Called from file "src/DessserExpressions.ml", line 1262, characters 15-29
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/batList.mlv", line 239, characters 23-28
Called from file "src/DessserExpressions.ml", line 1218, characters 15-40
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1266, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1225, characters 15-30
Called from file "src/DessserTypeCheck.ml" (inlined), line 15, characters 2-1023
Called from file "src/DessserCompilationUnit.ml", line 123, characters 13-33
Called from file "src/dessserc.ml", line 123, characters 27-59
Called from file "list.ml", line 117, characters 24-34
Called from file "src/dessserc.ml", line 156, characters 17-65
Called from file "src/dessserc.ml", line 708, characters 13-17
make: *** [Makefile:456: dessserc-check] Error 2

Clearly this is because those 2 encoding formats uses a custom pointer instead of just Ptr.
Some function, somewhere, is not abstract enough and assume a Ptr.

rixed / dessser Goto Github PK

dessser's People

Contributors

Stargazers

Watchers

Forkers

dessser's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs