Deserialize sealed classes generated by oneof about krotodc

[mscheong01]

Since krotoDC generated class objects can be converted to and from protobuf-java class objects, you can deserialize the json to protobuf-java class object and simply call .toDataClass() (which is extension function generated by krotoDC).
If this does not work for your usecase, please let me know 😄

[anjanbk]

I see, it seems like that only works if the serialization also happens from the protobuf-java class (unless I'm missing something else). E.g. as follows:

    val example = TestWrapper(
        data = TestWrapper.Data.TypeA("foobar")
    )

    val javaJson = gson.toJson(example.toProto())
    val ktJson = gson.toJson(example)

    // This does not work
    println(
        gson.fromJson(
            ktJson,
            TestWrapperProtobufJava::class.java).toDataClass())

    // This works
    println(
        gson.fromJson(
            javaJson,
            TestWrapperProtobufJava::class.java).toDataClass()
    )

The problem with that is that serializing the protobuf-java class (i.e. javaJson) ends up looking as follows:

{
    "dataCase_": 1,
    "data_": "foobar",
    "memoizedIsInitialized": 1,
    "unknownFields": {
        "fields": {}
    },
    "memoizedSize": -1,
    "memoizedHashCode": 0
}

grpc-kotlin would also generate classes that serialized this way, and it was one of the main reasons we moved to KrotoDC. We are storing this data in a database, and the expectation is that the data stored looks like the JSON representation of the message.

The use-case I'm trying to accomplish is defining a protobuf message that looks like TestWrapper above), and being able to serialize an instance of it to a Json String that looks like what the Kotlin data class serializes to:

{
    "data": {
        "typeA": "foobar"
    }
}

...and then being able to deserialize that back into the Kotlin data class (either directly or indirectly).

Is there a supported way to do that?

[mscheong01]

The serialized Protobuf JSON you provided seems to be a direct JSON serialization of the Protobuf Java class, leading to the exposure of hidden fields such as memoizedHashCode, etc., which I believe is the root of your problem.

Have you considered using the serialization methods that Protobuf-Java provides? This approach serializes Protobufs into their corresponding official JSON format(only including proto defined fields), as detailed in this guide: https://www.baeldung.com/java-convert-json-protobuf.

The primary reason I haven't provided a JSON deserialization method for KrotoDC data classes is that doing so would result in a slightly different JSON format compared to the original Protobufs, which seems like a poor design choice. However, adding a utility function to serialize/deserialize from the original Protobuf JSON formats could be a beneficial enhancement to this library.

[mscheong01]

I've added toJson, fromJson generated functions that provide Json conversion and released it as 1.1.0 🙂

[anjanbk]

Just tried it out — there seems to be a bug in the code generator if a message has a long name 😅. Our system has a fairly deep namespace tree, and one of our messages has a somewhat large name, and it triggered the bug.

Try the following:

syntax = "proto3";

option java_multiple_files = true;
option java_package = "com.example.application.system.types";
option java_outer_classname = "VeryLongMessageNameProto";

package com.example.application.system.types;

message VeryVeryLongTestProtocolBuffersMessageName {
  repeated string type = 1;
}

The fromJson function applies a newline after apply, which doesn't compile

public fun KrotodcVeryVeryLongTestProtocolBuffersMessageName.Companion.fromJson(json: String):
    KrotodcVeryVeryLongTestProtocolBuffersMessageName =
    com.example.application.system.types.VeryVeryLongTestProtocolBuffersMessageName.newBuilder().apply
    { JsonFormat.parser().ignoringUnknownFields().merge(json, this@apply) }.build().toDataClass();

Instead, generating the following should work:

public fun KrotodcVeryVeryLongTestProtocolBuffersMessageName.Companion.fromJson(json: String):
    KrotodcVeryVeryLongTestProtocolBuffersMessageName =
    com.example.application.system.types.VeryVeryLongTestProtocolBuffersMessageName.newBuilder()
        .apply{ JsonFormat.parser().ignoringUnknownFields().merge(json, this@apply) }.build().toDataClass();

[mscheong01]

Yes, I see... Kotlin poet inserts a newline if the provided code is too long at any whitespace and for cases where the newline is inserted between apply and { the code does not compile 😢 Inserting a newline before .apply will probably fix this issue for good (I would rather not remove the whitespace between apply and { since it is convention to have it)

[mscheong01]

@anjanbk I've fixed this in #27 and released it as 1.1.1

[anjanbk]

That fixed the bug, thanks.

Since Kotlin doesn't let you directly enforce the presence of a companion object's method through a generic type constraint, how would you recommend using the fromJson and toJson that you added in a generic serializer/deserializer?

[mscheong01]

That doesn't seem possible ATM 😞
If we want to get this supported, we would have to generate conversion functions as class methods instead of extension functions.
Even then, since fromJson is a static function, we'll have to find another way for deserialization.

What would you rate the priority of this feature for your work?
do you have usecases where you use KrotoDC classes as other class' field types?
otherwise, you could just directly call the extension functions.

[anjanbk]

An alternative could be to not use sealed classes for oneofs, but that's a bigger change.

The general use-case we have is to define all of our data types in Protobufs, and store those types in our databases as JSONB blobs. Our Kotlin system serializes/deserializes this data when writing/reading that data in the DB. As a result, there needs to be a generic serializer that's able to handle any type as long as that type is defined correctly (e.g. gson.fromJson(jsonString, MyType::class.java). Gson (and most JSON serializers) require custom type adapters to handle sealed classes.

For now, we've defined a type adapter that uses reflection to handle this (something kind of like this: https://gist.github.com/krishnabhargav/7b1832eeb86aa213ba5bb239153977ea)

Would love for there to be a better way to do it though, as this is a common pattern.

[mscheong01]

I don't have an immediate solution in mind right now 🤔, but it's great to hear you've found a workaround! I'll definitely keep this issue in mind and look into a more permanent solution in the future. For now, I'm going to close this issue and open a new one focused on this specific feature. Also, a big thanks for using krotoDC and sharing your feedback! 🙇