Feature: add JSON parsing to dump.transformation attribute about greenmask

[wwoytenko]

@joao-zanutto
That's a good point. I suspect it can be simply solved, but I should ensure that it can be completed seamlessly.
Thank you!

[wwoytenko]

I've just checked. The Json format file is already working

{
  "common": {
    "pg_bin_path": "/usr/lib/postgresql/16/bin",
    "tmp_dir": "/tmp"
  },
  "storage": {
    "type": "s3",
    "s3": {
      "endpoint": "http://playground-storage:9000",
      "bucket": "adventureworks",
      "region": "us-east-1",
      "access_key_id": "Q3AM3UQ867SPQQA43P2F",
      "secret_access_key": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
    }
  },
  "validate": null,
  "dump": {
    "pg_dump_options": {
      "dbname": "host=playground-db user=postgres password=example dbname=original",
      "jobs": 10
    },
    "transformation": [
      {
        "schema": "humanresources",
        "name": "employee",
        "transformers": [
          {
            "name": "NoiseDate",
            "params": {
              "ratio": "10 year 9 mon 1 day",
              "column": "birthdate"
            }
          }
        ]
      }
    ]
  },
  "restore": {
    "pg_restore_options": {
      "jobs": 10,
      "dbname": "host=playground-db user=postgres password=example dbname=transformed"
    }
  }
}

greenmask --config config.json validate

But I suspect it might not work correctly with transformer parameters. I will check

[wwoytenko]

It is not as simple. Currently, it raises an error. I will check the implementation later

export DUMP_TRANSFORMATION='[ { "schema": "humanresources", "name": "employee", "transformers": [ { "name": "NoiseDate", "params": { "ratio": "10 year 9 mon 1 day", "column": "birthdate" } } ] } ]'
greenmask --config config.yml validate

{"level":"fatal","error":"5 error(s) decoding:\n\n* 'dump.transformation[0]' expected a map, got 'string'\n* 'dump.transformation[1]' expected a map, got 'string'\n* 'dump.transformation[2]' expected a map, got 'string'\n* 'dump.transformation[3]' expected a map, got 'string'\n* 'dump.transformation[4]' expected a map, got 'string'","time":"2024-04-08T16:38:50Z"}

[wwoytenko]

Looks like it can't unmarshal the structures, only the scalar types

[joao-zanutto]

@wwoytenko I found possible solution in viper repo: spf13/viper#339 (comment)

I tried to stab the issue in the #61 PR, but it seems that it's not able to unmarshal the internal structures recursively. I'm still not really familiar with the mapstructure package so I'm trying to figure out a way to make it call itself recursively.

If you manage to find a solution to this, please let me know. (this is the last modification we need to use Greenmask in production)

joao-personal@joao:~/greenmask$ export DUMP_TRANSFORMATION='[{"schema":"humanresources","name":"employee","transformers":[{"name":"NoiseDate","params":{"ratio":"10year9mon1day","column":"birthdate"}}]}]'
joao-personal@joao:~/greenmask$ STORAGE_DIRECTORY_PATH=. PGHOST=localhost PGDATABASE=original PGPASSWORD=example PGPORT=54316 ./greenmask validate
2024-04-08T13:03:13-07:00 FTL error="unable to build runtime context: cannot validate and build table config: unable to init transformer: error parsing \"ratio\" parameter: unable to scan parameter via Driver: bad interval format"

UPDATE: @wwoytenko I just realized I had an issue with my transformation, if you take a close look, the ratio parameter is missing spaces between the interval arguments (year, month, day). Fixing this issue in the input and re-running made it work

[wwoytenko]

I am closing it. It is impossible due to the limitation described here