Scratch Data is a wrapper that lets you stream data into and out of your analytics database. It takes arbitrary JSON as input and lets you perform analytical queries.

Clone the repo:

$ git clone [email protected]:scratchdata/scratchdata.git
$ cd scratchdata

Start the service:

$ go run . 

With no configuration, this will automatically set up a local DuckDB database ready for reading and writing.

Create a config.yaml file with all of your settings and run:

$ go run . config.yaml

$ curl -X POST "http://localhost:8080/api/data/insert/events?api_key=local" \
    --data '{"user": "alice", "event": "click"}'

The "events" table and columns are automatically created.

curl -G "http://localhost:8080/api/data/query" \
     --data-urlencode "api_key=local" \
     --data-urlencode "query=select * from events" 

You can share data as CSV or JSON by creating "share links".

$ curl -X POST "http://localhost:8080/api/data/query/share?api_key=local" \
    --data '{"query": "select * from events", "duration": 120}'

This will produce a query ID that expires in 120 seconds. From there, send the following link to users:

http://localhost:8080/share/<query_id>/data.csv
http://localhost:8080/share/<query_id>/data.json

You can set up multiple databases and copy data between them. You can run a SQL query against your source database and Scratch will automatically create a table and insert data into a destination.

$ curl -X POST "http://localhost:8080/api/data/copy?api_key=local" \
    --data '{"query": "select * from events", "destination_id": 3, "destination_table": "events"}'

To see the full list of options, look at: https://docs.scratchdata.com