Reading Map objects from JSON array about json_events

[aeb-dev]

The use case that drew my attention to json_events is, that I download a large JSON array where I want to process each object in the array one after the other. Loading the whole JSON into memory for parsing is a no-go. I'm new to Dart/Flutter (coming mainly from C#, C++, Python, Ruby) so I was little bit surprised there was no streaming JSON parser (I'm so used to JSON.Net, I'm probably a little bit spoiled :-) So: json_events to the rescue! (Thanks for providing this!)

You're welcome! Hearing such things make the effort worth it.

But using the JsonArrayTraverser / JsonObjectTraverser feels a little "bulky". Wouldn't it be nice, if you could simply get Map<string, dynamic> objects from a JSON-Stream with an array? Then json_serializable can be used to get the objects from it.

I think this against the idea of json_events. Currently I don't accumulate anything because we can never be sure of the size. So if we were to return Map<> this means you are allocating whatever is coming through the stream which still can be GBs of data.

You can still accumulate yourself if you prefer, but I doubt this is what you want

class MyClass with JsonObjectTraverser {
  late int x;
  late List<MyClass> arr;
  late List<int> pArr;
  late List<List<int>> pNestedArray;
  String? text;

  Map<String, dynamic> jsonMap = {};

  @override
  Future<void> readJson(String key) async {
    switch (key) {
      case "x":
        jsonMap[key] = await this.readPropertyJsonContinue<int>();
        // rest of the code ...
    }
  }
}

[e-tobi]

You're right any sub-object may be large. In my case I know this isn't the case. I'm now using a JsonArrayToMapConverter so I can simply do: jsonEvenStream.transform(JsonArrayToMapConverter()) and get a Stream of (small) Map objects. these can then be mapped easily to real objects using json_serializable. Suites my needs. But I think this is a very common use case: Loading a list of data from a REST API or file with a large array but relative small elements.

[aeb-dev]

I believe the solution here is not providing a new converter but providing an easier way for people so they won't have to write too much code. I have been thinking about this for a while but I could not find a good way.

One way of doing this might be providing map instead of the readJson function. This map contains key, setter, an optional creator if the type is custom object.

class MyClass with JsonObjectTraverser {
  late int x;
  late List<MyClass> arr;

  Map<String, (void Function(dynamic) setter, dynamic Function()? creator)> fields => {
    "x": ((v) => x = v as int?, null),
    "arr": ((v) => arr.add(v as MyClass), MyClass.new),
  }
}

Don't mind the ugliness of the code 😄 We can improve that as well

[e-tobi]

Uhh... I started with Dart a week ago - this does not look easier to me. I still prefer my solution :-)

final response = await http.Client().send(
      http.Request("GET", Uri.parse("https://server.com/products")));

var productsStream = response.stream
    .transform(const Utf8Decoder())
    .transform(const JsonEventDecoder())
    .flatten()
    .transform(JsonObjectArrayDecoder())
    .map((m) => Product.fromJson(map));

await for (final product in productsStream) {
    doFancyStuffWithProduct(product);
}

It's designed to only deal with JSON arrays, but that's what I needed.