jamie-pate / jstruct

jstruct

C Library and code generator that declaratively transforms data between JSON and C.

The jstruct preprocessor generates C code that transforms data between C structures and JSON strings. It reads annotated comments from the structure declarations in your header, and creates c data structures that allow it to efficiently and automatically import/export JSON data.

After calling jstruct_import() the caller must call array_list_free(result.allocated) once they are finished with the imported struct or suffer memory leakage. The imported data may contain pointers to the json_object Those pointers shouldn't be accessed once all references have been removed (EG: with json_object_put())

Development Roadmap

• M1 Export structs containing primitive types and arrays to same. (char*, int, bool, uint64_t etc)
• M2 Mechanical parsing of annotated header files to produce augmented struct declarations and data tables used by the export process.
• M3 Export structs containing other structures and arrays (struct* and struct[])
• M4 Import structs containing primitive types and arrays
• M5 Import nested structs and arrays of structs (feature complete)
• M6 ~~Import foreign (un-annotated) structs in a binary compatible way. ~~
  • Annotate a whole struct with a single json object literal
  • Guarantee that json_struct_value = foreign_struct_value is always valid (generate assertions checking that all property sizes and offsets match)
• Planned features
  • Automatically free pointers in nested structs which were allocated by jstruct_array_malloc
  • Hybrid export/import. Handle c structs with json_object members automatically

Sample

main_jstruct.jstruct.h (generates main_jstruct.h and main_jstruct.init.h)

//@json
struct my_json_data {
    /*
    @schema {
        "title": "ID",
        "description": "unique object id",
        "type": "int"
    }
    */
    uint64_t id;

    /* don't include in json */
    //@private
    int _id;

    bool active;

    /* add the ability to null this field even though it's not a pointer */
    //@nullable
    double ratio;
    char *name;

    /* automatically parsed as an array atm */
    char **tags;
}

//@json
struct my_json_container {
    struct my_json_data main_data;

    /* static arrays are automatic */
    struct my_json_data array_data[5];
    /* as are struct arrays atm*/
    struct my_json_data *alloc_array_data;
}

// After M6

struct foreign_struct {
#ifndef DONT_USE_UINT64_T_FOR_SOME_REASON
    uint64_t id;
#else
    unsigned long long id;
#endif
    int _id;
    enum jstruct_error err;

    bool active;
    double ratio_double;
    char *name;
    unsigned long long ull;
    char **tags;  
};
/* @json {
    "_id": "@private",
    "ratio_double": {
        "@nullable": true,
        "@name": "ratio"
    },
    "@name": "other_name"
} */
struct my_json_foreign_struct {
    // place all members of a struct 'inline' inside this struct 
    // @inline
    struct foreign_struct fs;
};

main.c

#include "main_jstruct.h"
// Initializers must be in a separate file annd included only in one c file
#include "main_jstruct.init.h"

#include <json-c/json_object.h>

int main() {
    char *data_tags[] = {"main", "data", "sample"};
    struct my_json_container container={
        .main_data={
            .id=1,
            ._id=2,
            .ratio_double=3.5,
            .name="main_data",
            .tags=data_tags
        },
        .array_data={
            {
                .id=3,
                .ratio__null__=true
            },{.id=5},{.id=6}
        }
    }
    /* malloc macro (automatically sets container.array_data__length__ = 2) */
    jstruct_array_malloc(container, alloc_array_data, struct my_json_data, 2)

    struct json_object *obj = jstruct_export(&container, my_json_container);
    if (obj) {
        printf("%s", json_object_to_json_string(obj));
    }


    // After M6
    struct foreign_struct foreign_data = {
        .id=1,
        ._id=2,
        .ratio_double=3.5,
        .name="foreign_data",
        .tags=data_tags
    };
    struct my_json_foreign_struct jforeign_data = foreign_data;

    struct json_object *fobj = jstruct_export(&jforeign_data, my_json_foreign_struct);
}

Build and Install

From git

Required Packages: autoconf,libtool,libjson-c-dev v0.10+, python-pycparser v2.11+

• install check: http://check.sourceforge.net/web/install.html (needed for autoreconf) or git clone https://github.com/libcheck/check.git && cd check and follow the instructions in README
• git clone repo
• libtoolize
• autoreconf --install
• ./configure
• make && sudo make install

From release tarball

Requires libjson-c-dev with headers at ${INCLUDE_DIR}/json-c/* >= v0.10 (probably)

• tar -zxf jstruct.tar.gz
• cd jstruct
• ./configure && make && sudo make install

Tests

C check Tests

Requires check http://check.sourceforge.net/web/install.html

• make check - checks the functionality of the runtime library

Python unittest2 Tests

Requires python-unittest2 package

• tests/test_all.py - checks the functionality of the jstruct annotation parser

jstruct's Issues

Details on how to use example

Could you give some details on how to make the example work? It seems the user must use jstruct_parse.py to generate a header. I tried to do that (using the contents of the example in the README) and get an error.

./jstruct_parse.py ../example/main_jstruct.jstruct.h
Traceback (most recent call last):
  File "./jstruct_parse.py", line 191, in <module>
    args.define
  File "./jstruct_parse.py", line 124, in parse_and_generate
    ast, text = parse_jstruct(filename, include_paths=include_paths, defines=defines)
  File "./jstruct_parse.py", line 64, in parse_jstruct
    raise plyparser.ParseError(message)
pycparser.plyparser.ParseError: ../example/main_jstruct.jstruct.h:10:5: before: uint64_t

Use python2 in python scripts

On Arch Linux, the default python version is 3. This causes build errors and can be corrected by specifying the python version in the shebang. Would you change

parse/jstruct_parse.py
tests/test_all.py

from #!/usr/bin/env python to #!/usr/bin/env python2

make check error

Should I expect master to be stable? When running make check from master, I get the following error:

gcc -DHAVE_CONFIG_H -I. -I..  -I../  -pthread -g -O2 -Wall -Werror -Wpedantic -pedantic -std=c99 -std=c11 -MT check_jstruct-check_jstruct.o -MD -MP -MF .deps/check_jstruct-check_jstruct.Tpo -c -o check_jstruct-check_jstruct.o `test -f 'check_jstruct.c' || echo './'`check_jstruct.c
check_jstruct.c: In function ‘test_data’:
check_jstruct.c:104:31: error: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘uint64_t {aka long unsigned int}’ [-Werror=format=]
  fprintf(stdout, "TEST_ID: %llu:%d %s\n", data.id, data._id, json_object_to_json_string(obj));
                               ^
cc1: all warnings being treated as errors
make[2]: *** [Makefile:625: check_jstruct-check_jstruct.o] Error 1
make[2]: Leaving directory '/home/fritz/src/jstruct/tests'
make[1]: *** [Makefile:918: check-am] Error 2
make[1]: Leaving directory '/home/fritz/src/jstruct/tests'
make: *** [Makefile:409: check-recursive] Error 1

Still doesn't deal with #ifdef and #endif and #else properly

EG:

#define SIZE 3

//@json
struct jstruct_annotated_struct {
    int some_field;
#define CANT_EXPAND_SOME_ARRAY
#ifndef CANT_EXPAND_SOME_ARRAY
   unsigned int some_array[SIZE];
#endif
// ideally #else would work here too.
#ifdef CANT_EXPAND_SOME_ARRAY
    unsigned int some_array_0;
    unsigned int some_array_1;
    unsigned int some_array_2;
#endif
}```

fails to parse.

json_ctor_decl(array) asserts that the array length > 0

0 Length arrays should be valid.

Support 'short' and 'unsigned short'

These types are somewhat common.

Prevent infinite loop while annotating and an unknown type is encountered.

            try:
                types = try_get_types(dereference)

        if is_array and arraydecl is None:
            types['member'] = types['json']
            types['json'] = 'json_type_array'
        return (types, arraydecl)
    except ExpansionError as err:
        # try again in case it's an array
        if not is_array:
            dereference -= 1
            is_array = True
            if initial_err is None:
                initial_err = err
                initial_tb = sys.exc_info()[2]

-                raise err

Generate `<basename>.init.c` instead of `<basename>.init.h` by default

Annotation for foreign structs

Add @foreign annotation for structs defined in read only foreign headers.

Eg:

//@json
/* @foreign struct tm {
    tm_isdst: 'private' 
}
*/

Not 100% sure on the syntax yet

Problem compiling

bvt@bvt-VirtualBox:~/jstruct-master$ make
make all-recursive
make[1]: Entering directory '/home/bvt/jstruct-master'
Making all in jstruct
make[2]: Entering directory '/home/bvt/jstruct-master/jstruct'
depbase=echo jstruct.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||';
/bin/bash ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -Wall -Werror -Wpedantic -pedantic -std=c99 -std=c11 -MT jstruct.lo -MD -MP -MF $depbase.Tpo -c -o jstruct.lo jstruct.c &&
mv -f $depbase.Tpo $depbase.Plo
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -Wall -Werror -Wpedantic -pedantic -std=c99 -std=c11 -MT jstruct.lo -MD -MP -MF .deps/jstruct.Tpo -c jstruct.c -fPIC -DPIC -o .libs/jstruct.o
In file included from jstruct.c:1:0:
jstruct.h:13:10: error: #include expects "FILENAME" or
#include JSON_OBJECT_H
^
jstruct.h:49:9: error: unknown type name ‘json_type’
json_type json;
^
jstruct.h:52:9: error: unknown type name ‘json_type’
json_type member;
^
In file included from error.h:29:0,
from jstruct.h:78,
from jstruct.c:1:
result.h:6:32: fatal error: json-c/json_object.h: No such file or directory
compilation terminated.
Makefile:450: recipe for target 'jstruct.lo' failed
make[2]: *** [jstruct.lo] Error 1
make[2]: Leaving directory '/home/bvt/jstruct-master/jstruct'
Makefile:410: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/bvt/jstruct-master'
Makefile:342: recipe for target 'all' failed
make: *** [all] Error 2

@name annotation causes offset() etc to fail.

Hande export/import of nested json_object * pointers

Handle export/import of nested json_object * pointers automatically. EG: jstruct_result currently contains a @private json_object *inner_result property which is appended to the exported json object manually. Unless a json_object * is marked with @private it should automatically be added into the export and should automatically get a reference to place in the .allocated arraylist of the jstruct_result and the imported struct.

Add inline annotation

Add @inline annotation. Should merge child structs with the outer struct

struct my_inner {
    int inner_id;
};
struct my_container {
    int id;
    //@inline
    struct my_inner inner;
};

would import/export the following json:

{
    id: 1;
    inner_id: 2;
}

Allow json-c/json_object.h header to be located in a different place.

Add defines that can be overridden:

• JSON_OBJECT_H : <json-c/json_object.h>
• ARRAYLIST_H: <json-c/araylist.h>
• JSTRUCT_H: <jstruct/jstruct.h>

Support for `unsigned long` `unsigned long long` etc

• long
• unsigned long
• unsigned long long
• unsigned int

Track pointers allocated by jstruct_array_malloc somehow

Automatically track pointers in nested structs which were allocated by jstruct_array_malloc, and have a mechanism to free them with a single fn call. This may require an extra field added to generated struct definitions that contains an arraylist of allocated pointers.

support `#else` directive