rixed / dessser Goto Github PK
View Code? Open in Web Editor NEW(de)serializer generator
License: GNU General Public License v3.0
(de)serializer generator
License: GNU General Public License v3.0
There should be no need for a "debug" flag in des/ser, given asserts are supposed to be NOP when NDEBUG.
c:\Users\grixe\Sources\ramen-admin\src>c:\Qt\Tools\mingw810_64\bin\mingw32-make.exe
c:/Qt/Tools/mingw810_64/bin/mingw32-make -f Makefile.Release
mingw32-make[1]: Entering directory 'c:/Users/grixe/Sources/ramen-admin/src'
g++ -c -fno-keep-inline-dllexport -DSODIUM_STATIC -O2 -std=gnu++1z -Wall -Wextra -Wextra -fexceptions -mthreads -DUNICODE -D_UNICODE -DWIN32 -DMINGW_HAS_SECURE_API=1 -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_CHARTS_LIB -DQT_PRINTSUPPORT_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_TESTLIB_LIB -DQT_CORE_LIB -DQT_TESTCASE_BUILDDIR="\"C:/Users/grixe/Sources/ramen-admin/src\"" -I. -I. -I/usr/local/include -I..\..\dessser\src -I..\..\..\libsodium-win64\include -IC:\Qt\5.15.2\mingw81_64\include -IC:\Qt\5.15.2\mingw81_64\include\QtCharts -IC:\Qt\5.15.2\mingw81_64\include\QtPrintSupport -IC:\Qt\5.15.2\mingw81_64\include\QtWidgets -IC:\Qt\5.15.2\mingw81_64\include\QtGui -IC:\Qt\5.15.2\mingw81_64\include\QtANGLE -IC:\Qt\5.15.2\mingw81_64\include\QtNetwork -IC:\Qt\5.15.2\mingw81_64\include\QtTest -IC:\Qt\5.15.2\mingw81_64\include\QtCore -Irelease -I/include -IC:\Qt\5.15.2\mingw81_64\mkspecs\win32-g++ -o release\desssergen\team_name.o desssergen\team_name.cc
In file included from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/minwindef.h:163,
from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/windef.h:8,
from C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/windows.h:69,
from ..\..\dessser\src/dessser/Ip.h:16,
from ..\..\dessser\src/dessser/runtime.h:25,
from desssergen\team_name.cc:18:
C:/Qt/Tools/mingw810_64/x86_64-w64-mingw32/include/winnt.h:334:11: error: 'CHAR' does not name a type; did you mean 'WCHAR'?
typedef CHAR *PCHAR,*LPCH,*PCH;
^~~~
WCHAR
With that JSON (out of ramen alerting configuration):
{"via": {"Kafka": { "options": [("topic.message.timeout.ms","5000")], "topic": "alert-events", "partition": 0, "text": "yada" }}}
Note: it should be ... "options": [["topic.message.timeout.ms","5000"]],...
, it is apparently confused by the parentheses.
The only 2 encoders using it are Csv and SExpr, which use it for static configuration.
Let's just have a global config for those and call it a day?
Alternatively, if we want to avoid global vars at all cost, we could have functors?
To use with quickcheck, testing that:
Case at hand:
RamenIp.t is the type used by all IP operations but dessser defines a new V4 of u32 | v6 of u128
type, that is structurally equivalent but still a distinct type.
We'd like to be able to tell dessser that there already exist a type definition for those types, under the name RamenIp.t
.
Fatal error: exception Type Error: In expression
(apply (ext-identifier test_ext of-ringbuf) (identifier "with_nullbit_done2_fst_126"))
expression
(identifier "with_nullbit_done2_fst_126")
should be a (Ptr; (Ptr; Size)[[]]) but is a Ptr
Raised at file "src/DessserExpressions.ml", line 1068, characters 10-60
Called from file "src/batList.mlv", line 1023, characters 6-11
Called from file "src/DessserTypeCheck.ml" (inlined), line 111, characters 25-44
Called from file "src/DessserTypeCheck.ml", line 192, characters 12-33
Called from file "src/DessserExpressions.ml", line 1241, characters 15-29
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1241, characters 15-29
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1244, characters 15-30
Called from file "src/DessserExpressions.ml", line 1252, characters 15-30
Called from file "src/DessserExpressions.ml", line 1244, characters 15-30
Called from file "src/DessserExpressions.ml", line 1231, characters 15-30
Called from file "src/DessserTypeCheck.ml" (inlined), line 15, characters 2-1023
Called from file "src/DessserCompilationUnit.ml", line 123, characters 13-33
Called from file "src/DessserHeapValue.ml", line 331, characters 29-56
Called from file "src/dessserc.ml", line 123, characters 27-59
Called from file "list.ml", line 117, characters 24-34
Called from file "src/dessserc.ml", line 156, characters 17-65
Called from file "src/dessserc.ml", line 708, characters 13-17
One of the most challenging issue when using dessser is to forget to bind largish chunks of code to a stage-1 variable, leading to very large generated code, which is not always easy to spot when looking at the DIL code, since it is large and repetitive by nature.
There could be a function we call explicitly before transpiling to the backend, that tries to detect that this occurred and warn.
Ie. detection of common subtrees in large tree.
Fatal error: exception Unknown type "t". Only known types are: subtype
Raised at file "src/DessserTypes.ml", line 1083, characters 4-26
Called from file "src/DessserTypes.ml", line 1166, characters 15-26
Called from file "src/batArray.mlv", line 483, characters 12-49
Called from file "src/DessserTools.ml", line 134, characters 6-26
Called from file "list.ml", line 221, characters 17-20
Called from file "src/DessserTypes.ml", line 1122, characters 10-21
Called from file "src/DessserTypes.ml", line 1107, characters 20-33
Called from file "src/DessserTypes.ml", line 1118, characters 55-63
Called from file "array.ml", line 103, characters 21-40
Called from file "src/DessserTypes.ml", line 1118, characters 25-69
Called from file "src/DessserTypes.ml", line 1145, characters 12-20
Called from file "list.ml", line 106, characters 12-15
Called from file "src/dessserc.ml", line 90, characters 2-32
Called from file "src/dessserc.ml", line 708, characters 13-17
For the many cases where no fieldmask are needed.
Generated code should be noticeably faster.
Arrays are serialized in CSV as follows: first, the array is serialized to a string as in TabSeparated format, and then the resulting string is output to CSV in double-quotes.
and the array serialization that's referred to is:
Arrays are written as a list of comma-separated values in square brackets.
-- https://clickhouse.tech/docs/en/interfaces/formats/
Note that what CH calls array we call list.
For instance, a const betraying the type, then a function providing the label name for enums (or erring out for non enum likes), etc.
The idea is that it should be possible to write static code that dynamically adapts to any type.
Required in C++ backend first but the API would benefit from being the same for other backends.
As was used by rebinary.
So we can read/write directly from/into a file
functions are hard to optimize
If only with a specific field order for the decoder
For instance, [ A | B ] AS a_or_b
should make backend code generator use that name for that type.
This, of course, does not make a_or_b
an external type ($a_or_b
is invalid).
Could also come handy for improving support for recursive types.
...that's purely a RaQL thing.
such as arpa/inet.h, charconv
Can be used to test actual serializer/deserializers by checking a given value can go through ser/deser without loss, for every backend and format.
The simplest is to generate values as S-Expressions that can then be unserialized using the S-expression DESserializer.
Seen on ramen's TargetConfig: "TargetConfig [ 0x55555da310, ... ]"
OCaml backend looks ok: "TargetConfig [|{ program=... }|]" only because it's hand coded. It's not using the automatic conversion to string.
Also required by rebinary for the time offset
[48 1! / 111] >check_heapvalue>src/DessserQCheck.ml:1125 /var/folders/x6/4gb9ck5d4kb6s97zqqh8vvmh0000gn/T/dessser_converter_b0b81a.cc:150:69: error: no matching constructor for initialization of
'::dessser::gen::test::t3c0e2f6ceb2032915f19660622878e72'
...id_20113 { .v_96a026eaf0977b4366de8e19d25b7792_ejgvx = drec_fst_4864, .v_96a026eaf0977b4366de8e19d25b7792_kngke = drec_fst_4867 };
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So that we can save space on the wire (for the encodings that allow it).
Also, makes it nicer for manually entered values (such as in JSON)
Actually, not all encoding formats allow this.
RamenRingBuf, for instance, has no way to skip a value, Every non null values are expected to appear in a specific order. For skipping values we would need to adapt the nullbit mask to have one bit per nullable or default-able value. Actually, that would be a presence mask, and nullable values would just have an implicit default to null. Alternatively, we could have an implicit default value for any type, and have one bit per field.
Testing dessserc
Testing I/O encoding ringbuf
...on tests/test_shrink_bug.type
Fatal error: exception Type Error: In expression
(apply (myself "(THIS; Ptr)") (if (eq (unsafe-nth (u8 0) (identifier "repeat_n_326")) (i32 0)) (identifier "dlist2_snd_329") (identifier "dlist2_snd_329")))
expression
(if (eq (unsafe-nth (u8 0) (identifier "repeat_n_326")) (i32 0)) (identifier "dlist2_snd_329") (identifier "dlist2_snd_329"))
should be a Ptr but is a (Ptr; (Ptr; Size)[[]])
Raised at file "src/DessserExpressions.ml", line 1062, characters 10-60
Called from file "src/batList.mlv", line 1023, characters 6-11
Called from file "src/DessserTypeCheck.ml" (inlined), line 111, characters 25-44
Called from file "src/DessserTypeCheck.ml", line 191, characters 12-33
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1267, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/batList.mlv", line 239, characters 23-28
Called from file "src/DessserExpressions.ml", line 1218, characters 15-40
Called from file "src/DessserExpressions.ml", line 1262, characters 15-29
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/batList.mlv", line 239, characters 23-28
Called from file "src/DessserExpressions.ml", line 1218, characters 15-40
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1266, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1246, characters 15-30
Called from file "src/DessserExpressions.ml", line 1238, characters 15-30
Called from file "src/DessserExpressions.ml", line 1235, characters 15-29
Called from file "src/DessserExpressions.ml", line 1225, characters 15-30
Called from file "src/DessserTypeCheck.ml" (inlined), line 15, characters 2-1023
Called from file "src/DessserCompilationUnit.ml", line 123, characters 13-33
Called from file "src/dessserc.ml", line 123, characters 27-59
Called from file "list.ml", line 117, characters 24-34
Called from file "src/dessserc.ml", line 156, characters 17-65
Called from file "src/dessserc.ml", line 708, characters 13-17
make: *** [Makefile:456: dessserc-check] Error 2
Clearly this is because those 2 encoding formats uses a custom pointer instead of just Ptr.
Some function, somewhere, is not abstract enough and assume a Ptr.
Unless basic types for which a hash function already exists.
As for the operator<<
function, also define a hash function for ext_t variant.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.