Comments (6)
@joao-zanutto
That's a good point. I suspect it can be simply solved, but I should ensure that it can be completed seamlessly.
Thank you!
from greenmask.
I've just checked. The Json format file is already working
{
"common": {
"pg_bin_path": "/usr/lib/postgresql/16/bin",
"tmp_dir": "/tmp"
},
"storage": {
"type": "s3",
"s3": {
"endpoint": "http://playground-storage:9000",
"bucket": "adventureworks",
"region": "us-east-1",
"access_key_id": "Q3AM3UQ867SPQQA43P2F",
"secret_access_key": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
}
},
"validate": null,
"dump": {
"pg_dump_options": {
"dbname": "host=playground-db user=postgres password=example dbname=original",
"jobs": 10
},
"transformation": [
{
"schema": "humanresources",
"name": "employee",
"transformers": [
{
"name": "NoiseDate",
"params": {
"ratio": "10 year 9 mon 1 day",
"column": "birthdate"
}
}
]
}
]
},
"restore": {
"pg_restore_options": {
"jobs": 10,
"dbname": "host=playground-db user=postgres password=example dbname=transformed"
}
}
}
greenmask --config config.json validate
But I suspect it might not work correctly with transformer parameters. I will check
from greenmask.
It is not as simple. Currently, it raises an error. I will check the implementation later
export DUMP_TRANSFORMATION='[ { "schema": "humanresources", "name": "employee", "transformers": [ { "name": "NoiseDate", "params": { "ratio": "10 year 9 mon 1 day", "column": "birthdate" } } ] } ]'
greenmask --config config.yml validate
{"level":"fatal","error":"5 error(s) decoding:\n\n* 'dump.transformation[0]' expected a map, got 'string'\n* 'dump.transformation[1]' expected a map, got 'string'\n* 'dump.transformation[2]' expected a map, got 'string'\n* 'dump.transformation[3]' expected a map, got 'string'\n* 'dump.transformation[4]' expected a map, got 'string'","time":"2024-04-08T16:38:50Z"}
from greenmask.
Looks like it can't unmarshal the structures, only the scalar types
from greenmask.
@wwoytenko I found possible solution in viper repo: spf13/viper#339 (comment)
I tried to stab the issue in the #61 PR, but it seems that it's not able to unmarshal the internal structures recursively. I'm still not really familiar with the mapstructure
package so I'm trying to figure out a way to make it call itself recursively.
If you manage to find a solution to this, please let me know. (this is the last modification we need to use Greenmask in production)
joao-personal@joao:~/greenmask$ export DUMP_TRANSFORMATION='[{"schema":"humanresources","name":"employee","transformers":[{"name":"NoiseDate","params":{"ratio":"10year9mon1day","column":"birthdate"}}]}]'
joao-personal@joao:~/greenmask$ STORAGE_DIRECTORY_PATH=. PGHOST=localhost PGDATABASE=original PGPASSWORD=example PGPORT=54316 ./greenmask validate
2024-04-08T13:03:13-07:00 FTL error="unable to build runtime context: cannot validate and build table config: unable to init transformer: error parsing \"ratio\" parameter: unable to scan parameter via Driver: bad interval format"
UPDATE: @wwoytenko I just realized I had an issue with my transformation, if you take a close look, the ratio parameter is missing spaces between the interval arguments (year, month, day). Fixing this issue in the input and re-running made it work
from greenmask.
I am closing it. It is impossible due to the limitation described here
from greenmask.
Related Issues (20)
- Feat: RandomMacAddress transformer implementation
- permission denied for large object during dump action HOT 10
- fix: Enrich dynamic parameter validation warning
- feat: Set min and max values not required for int values
- feat: Implement LargeObjects inclusive and exclusive list
- feat: Noise* transformers - allow empty min or max params
- Feature request: transformer "timestamp with time zone" HOT 3
- fix: validate --table option wrong shcema and table parsing
- Feat: Implement database subset. Stage1
- feat: Database subset. Stage2 HOT 1
- Delete old dumps except for the last X HOT 1
- Data Generator Request: RandomCity HOT 4
- feat: Subset stage 3: Circular dependencies subset
- backlog 2024 Q3
- feat: Transformers collection and python library
- Container image HOT 5
- feat: PostgreSQL 17 support
- Unable to use `on-conflict-do-nothing` flag due to other missing flags that cannot be supplied HOT 6
- feat: Subset stage 4: Multi cycles in the same SCC
- feat: Subset stage 5: Virtual foreign keys
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from greenmask.