Experiments with JS style Binary Typed Array in Janet about janet

[bakpakin]

This is really awesome!!! This would be awesome for scientific computing, machine learning, data science, and a whole bunch of other things in Janet!

As for marshaling, we probably need an extension to the C API for abstract types to allow for custom marshalling and unmarshalling code, like how get and put were added for abstract types.

[bakpakin]

Another question is "should we make this part of the core library?" I'm leaning towards yes for now, but I think that would depend on large this gets. If it is included, we should probably include an option to disable it at compile time like the peg module and the asm functions. Making it a native module instead I think would also be OK.

[jfcap]

I think making this a "core" feature of Janet could have numerous advantages.
I've already included a JANET_NO_TYPED_ARRAY flag to disable it at compile time.
My intention was to implementent only the bare minimal code in the corelib
(not much that's already there + marshal code ).
Letting all other "potentially cool features" as extern extensions.
I completely agree with you that janet core should remain as light as possible.

[jfcap]

Some feedback about the global design of the implementation are welcome.

[bakpakin]

This looks fine to me in general. Are you planning to add some other operators/methods for typed arrays? A slice method would be very useful. Open a pull request when you think its ready.

[jfcap]

Work in progress with typed arrays

Marshal/unmarshal is implemented.

janet:34:> (def a (tarray/new :float64 10))
<ta/float64 0x000005CA1750>
janet:67:> (def b (unmarshal (marshal a))
janet:98:(> )
<ta/float64 0x000005CA3B40>
janet:100:> (tarray/properties b)
{ :buffer <ta/buffer 0x000005CA3BD0>
  :size 10
  :stride 1
  :type :float64
  :byte-offset 0
  :type-size 8}
janet:122:> (tarray/properties a)
{ :buffer <ta/buffer 0x000005CA17E0>
  :size 10
  :stride 1
  :type :float64
  :byte-offset 0
  :type-size 8}

At this point , I need some feedback on the implentation choices.

For this the first step was to give "generic" marshalling capabilities
to abstract type (AT).
This require some "introspection capabilities" to AT.

janet:0:> (abstract/properties :ta/float64)
{ :size 40
  :tag 1.69717e+09
  :name :ta/float64
  :salt 1010
  :marshal true}

I do this by adding the possibility to register AT in the vm.

typedef struct {
  const JanetAbstractType *at;
  size_t size;         
  const uint32_t salt;
  uint32_t tag; 
  void (* marshal)(void *p,JanetMarshalContext *ctx);
  void (* unmarshal)(void *p,JanetMarshalContext *ctx);
} JanetAbstractTypeInfo;

JANET_API void janet_register_abstract_type(const JanetAbstractTypeInfo * info);

This allow to extend AT whitout breaking light "standard" janet AT.
I use the janet_vm_table to register these AT.
It seems to work, but i'm not completely sure about this choice.

If you can take a look at the code, it will help.

https://github.com/jfcap/janet.git (branch typed-array)

[bakpakin]

Lots of progress!

I've taken a look at all of the changes and a couple of things I've noticed about the design:

This allow to extend AT whithout breaking light "standard" janet AT.

Wouldn't it be easier to just extend JanetAbstractType to have some of the extra fields in JanetAbstractTypeInfo, and drop JanetAbstractTypeInfo altogether? I think simplicity should be prioritized over backwards compatibility at this point - we are still pre 1.0. We could just add the marshal and unmarshal elements to JanetAstractType, and existing abstract type definitions would just need two extra NULL pointers in the initializers - not that much work, and a pretty clear error message if someone tries to compile an old native module.

As for the interface:

janet_register_abstract_type(const JanetAbstractTypeInfo *info);

becomes

janet_register_abstract_type(const JanetAbstractType *atype)

---

The tag/salt/name system seems too complicated, and only needed because types and type_info are separate. Two alternatives:

• Require the programmer to manually set tag to a unique 32 bit integer for each type. No hashing, no salt, simple. Similar to what Java does for it's Serializable interface and serialVersionUID. The overhead could be 4 bytes per abstract.
• Prefix marshalled abstracts by the name of their type. Like "ta/buffer"|length|data. Multiple abstracts of the same type would be efficient, since marshal caches strings, subsequent references to "ta/buffer" will be pretty short, often only 2 or 3 bytes. I think this will be the easiest, and requires types to have only one unique identifier, their name.

These changes would simplify both the interface and the code, although I may be overlooking something.

[jfcap]

Here a stripped down version with AT name as identifier for marshaling.
https://github.com/jfcap/janet.git (branch typed-array)

To be done :

• tarray --> @[] array slice method
• tarray/buffer --> @"" buffer method ?
• a test suite

what else ?

[bakpakin]

Looks pretty good so far. Just looking at it, in the unmarshalling functions (janet_unmarshal_uint and so on) should check for the end of source with the MARSH_EOS macro. Unmarshalling should be resistant to bad input, although I'm not sure how safe the current unmarshal function is on bad input (safe in theory, not adequately tested in practice).

[jfcap]

Unmarshalling should be resistant to bad input,

Is it really possible "to protect janet" from bad (forged) marshalled data ?

[bakpakin]

It’s more that Janet shouldn’t segfault on bad data or construct functions that will cause segfaults later, it should throw an error.

[bakpakin]

There’s nothing wronged with “forged” data, as long as the data follows the protocol. Data that doesn’t follow the protocol should not be able to crash Janet.

[jfcap]

What bothers me, is for example in the unmarshalling code for tarray views :

static void ta_view_unmarshal(void *p, JanetMarshalContext *ctx) {
    JanetTArrayView *view = (JanetTArrayView *)p;
    size_t offset;
    Janet buffer;
    janet_unmarshal_size(ctx, &(view->size));
    janet_unmarshal_size(ctx, &(view->stride));
    janet_unmarshal_uint(ctx, &(view->type));
    janet_unmarshal_size(ctx, &offset);
    janet_unmarshal_janet(ctx, &buffer);
    view->buffer = (JanetTArrayBuffer *)janet_unwrap_abstract(buffer);
    view->data = view->buffer->data + offset;
}

to be "forged marshalled data safe" I've to put some guards around view->data = view->buffer->data + offset;. It's not a big deal here, and I will do it.

Nevertheless I must admit that i'm not verry confortable with the idea that "marshalled data" could be assumed safe for the vm.
Peraphs i'm missing something here, but is marshalling/unmarshalling vm byte code not always inherently unsafe ?

[bakpakin]

Nope, Janet can verify all bytecode.

[bakpakin]

By that I mean Janet bytecode shouldn’t cause a segfault - of course, malicious bytecode could do other more terrible things.

[jfcap]

Ok, This is what I meant.