Simple config auth about scratchdb

[NortySpock]

Sweet, your documentation fixes in the README set me straight. That looks like it worked! Off to go try out playing with the configuration and loading some data in! Thank you!

$ curl -X POST "http://localhost:8989/api/data/insert/events?api_key=local"     --data '{"user": "alice", "event": "click"}'
ok 

$ curl -G "http://localhost:8989/api/data/query" \ \
     --data-urlencode "api_key=local" \
     --data-urlencode "query=select * from events" 
[
	{"__row_id":1778898631140184064,"event":"click","user":"alice"}
]

[NortySpock]

For the hobbyist interested in trying out this tool, what config options would I need to set to either meet the authentication requirements, or disable them, or run the program in a different way in order to be able to try out this tool?

In terms of my skill level, I write SQL all day, but I've never managed to set up an HTTPS server -- only http.

[breadchris]

Hey @NortySpock, thanks for checking out this project! This has been something we have been meaning to fix (hence why the issue exists). I can talk with @poundifdef tomorrow about how we want to fix this and get a fix up.

[poundifdef]

If you just run go run . then you should be able to follow the steps in the readme and most everything should work. The API portion of the project will work using a default local config on a local DuckDB instance. If you modify config.yaml (to connect to an external database) then that'll work as well. It does not use HTTPS locally.

Thing that does not, work locally is the UI because currently requires google auth. @breadchris is generously working on a fix for this, so stay tuned!

Can you share more about what you're looking to do? Happy to give more specific help too.

[NortySpock]

If you just run go run . then you should be able to follow the steps in the readme and most everything should work.

I had had another process using port 8080, so I had wanted to use a different port (hence my starting with the config file rather than using the defaults). Turning that process off and running ./scratchdata, it spins up just fine.

Running along with your quickstart, curl -X POST "http://localhost:8080/api/data/insert/events&api_key=local" --data '{"user": "alice", "event": "click"}'

returns

Unauthorized to curl, with the logs showing

2024/04/12 07:42:55 /home/runner/work/scratchdata/scratchdata/pkg/storage/database/gorm/gorm.go:304 record not found
[1.451ms] [rows:0] SELECT * FROM `api_keys` WHERE hashed_api_key = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" AND `api_keys`.`deleted_at` IS NULL ORDER BY `api_keys`.`id` LIMIT 1

Possibly I am using a slightly different version of curl? I tried --json but that was an unrecognized command line option. I'm sure I'm doing something wrong, and I have not had a lot of spare time to troubleshoot :)

$ curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Can you share more about what you're looking to do? Happy to give more specific help too.

Just some simple data exploration -- home server metrics, log entries, garbage collection cycle statistics, some smart-home events, maybe go on a novice spelunking tour of some stock market data. I'm running pretty low-end hardware at home, and I became infatuated with the Benthos and VictoriaMetrics tools, which make it a snap to stream data in via just converting the data to json and firing it at an API endpoint. But VictoriaMetrics is not a general purpose columnar database, it's tightly focused around metrics. Scratchdata + DuckDB sounded like a really nice pairing to let me automate the data ingestion without having to think about table schema much ahead of time. Once I get this working, I was going to use dbt to create views to slice, dice, and remix the data. It's just that SQLite is a little slower at bulk analytics than I have patience for :)

Edit: just re-tested again, having re-downloaded the release for v1.0.14 ... same error. Tried deleting the data folder and starting "from nothing", same error.

Edit2: the storage/database/gorm/gorm.go:304 record not found error makes me think perhaps scratchdata is trying to fetch a record out of the database... If the database had just been created from a first-time start, it might have nothing in that table...

[poundifdef]

I figured it out. There is a typo in our curl command in the docs which I'm fixing now. To insert data, you want to use ? instead of & for the query separator.

-/api/data/insert/events&api_key=local" 
+/api/data/insert/events?api_key=local" 

For changing the port: you can run ./scratchdata your_custom_config.yaml. So if you copy the default config from the repo, you can change the port from 8080 to something else in the api.port setting.

I also see that the --json parameter was only added to curl in 2022, so that may not be present everywhere. Using --data should work and I'll update the docs there as well.