Kafka MaskField transform problem about scylla-cdc-source-connector

[Gor027]

Hi, thanks for raising this issue, however, it seems to be a more Kafka-related one rather than an issue in the source connector.

The source connector captures the row-level changes and produces Kafka messages after executing an INSERT, UPDATE or DELETE operation. So, the content of the Kafka message describes the pre-image and post-image entries, and the changed columns with their values appear in nested structures before and after:

e.g. Kafka message after an Insert operation

{
  "source": {
    "version": "1.0.4-SNAPSHOT",
    "connector": "scylla",
    "name": "testing",
    "ts_ms": 1675094304126,
    "snapshot": {
      "string": "false"
    },
    "db": "test",
    "keyspace_name": "test",
    "table_name": "t",
    "ts_us": 1675094304126125
  },
  "before": null,
  "after": {
    "testing.test.t.After": {
      "pk": {
        "int": 14
      },
      "v": {
        "testing.test.t.v.Cell": {
          "value": {
            "int": 15
          }
        }
      }
    }
  },
  "op": {
    "string": "c"
  },
  "ts_ms": {
    "long": 1675099424937
  },
  "transaction": null
}

Unfortunately, the Kafka single message transformations only support top-level fields for now, and very recently an improvement proposal to add support for nested structures was created. I could not find a way to apply transformations to single fields inside a nested structure, but a simple workaround I can suggest is to use the Replace transformation to mask out the after or before struct entirely (the MaskField transformation cannot be used on structs). Perhaps, you can use the Flatten transformation to have only top-level fields and then mask out the necessary fields. The latter (if it works, cause I haven't tested it yet) seems to be a smarter workaround imho.

[Gor027]

@avelanarius should we close this issue, now?