Playground quickstart: 4 issues found getting sandbox/start_playground.sh running (Podman, sfw, stale dataset checksum, node_id dedup bug)

mikesmullin

## Summary Followed the [Playground guide](docs/user_guide/playground.mdx) on a fresh clone (`sandbox/start_playground.sh`, default `csr_readonly` mode, `panama` dataset) and hit four distinct blockers along the way. Filing this as one report since they were found together, but happy to split into separate issues if preferred. Two are environment-specific (Podman, `sfw`), two are real bugs reproducible on any machine. Environment: Arch Linux (rolling), kernel 6.18.38-3-lts, x86_64, Podman 6.0.1 (aliased as `docker`), Python 3.12.12 managed via `uv`. --- ### 1. `Dockerfile` images aren't resolvable under Podman without a `registries.conf` `sandbox/start_playground.sh` → `docker build` fails on the second stage: ``` Error: creating build container: short-name "postgres@sha256:4f736ae..." did not resolve to an alias and no containers-registries.conf(5) was found ``` Both `FROM` lines in the `Dockerfile` use short image names pinned by digest (`rust:1.96.0-bookworm@sha256:...`, `postgres:17-bookworm@sha256:...`). Docker resolves unqualified names against `docker.io` implicitly; Podman requires a `registries.conf` (`unqualified-search-registries`) to do the same, and machines with no such file (nothing in `/etc/containers/` or `~/.config/containers/`) fail outright. Workaround: add `~/.config/containers/registries.conf` with `unqualified-search-registries = ["docker.io"]`. Possible fix: fully-qualify both `ARG`s in the `Dockerfile` (`docker.io/library/rust:...`, `docker.io/library/postgres:...`). No-op for Docker (docker.io is already its default registry), and removes the Podman dependency on host-level registry config. --- ### 2. `sandbox/start_playground.sh` / `run_benchmarks.sh` hard-require `sfw` with no usable fallback Per `AGENTS.md`, `sfw` is a host-provided package-manager safety wrapper, not something the repo ships. Both scripts exit immediately if it's not on `PATH`: ``` Error: sfw is required before the playground can install Python dependencies. Install sfw, or provision sandbox/playground/.venv from requirements.txt ahead of time. ``` The suggested fallback ("provision the venv ahead of time") isn't actually reachable — the `command -v sfw` check happens unconditionally before the script ever inspects whether `sandbox/playground/.venv` already exists and is fully populated. So there's currently no way to run the playground at all on a workstation without `sfw` installed, even if you're willing to provision the venv by hand first. Suggested fix: only require `sfw` when the script is actually about to invoke pip to install/change something — i.e., skip the `sfw` requirement (and the `run_venv_pip install` calls) if the venv already exists and its installed packages already satisfy `requirements.txt`. --- ### 3. Pinned checksum for the `panama` dataset is stale `sandbox/common/run_benchmarks.py` pins: ```python expected_sha256="a2e37e8b878c12fb8f946d4e85026a4ae9026dc866b1aa925730bc1b50e52914" ``` for `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip`. As the URL implies, ICIJ regenerates this "latest" export periodically (ours came with `GENERATED_ON_20260729.txt` inside, i.e. regenerated 2026-07-29), so the pin goes stale over time and every fresh playground run fails with: ``` RuntimeError: Panama Papers / ICIJ Offshore Leaks archive checksum mismatch: expected a2e37e8b..., got 34475194b6a8c2d683fddc55cca02f88f08f0a538521fb13a324975221624380. ``` Verified the new download is a well-formed, uncorrupted zip (not a MITM/corruption issue) before treating this as a legitimate upstream update. Current sha256 as of 2026-07-31: `34475194b6a8c2d683fddc55cca02f88f08f0a538521fb13a324975221624380`. Given `LATEST` in the filename, this will keep recurring. Might be worth a periodic re-pin job, or documenting the expected cadence/how to re-pin in the playground guide. --- ### 4. `transform_panama` doesn't dedupe `node_id` across ICIJ's category CSVs, breaking `COPY` into `panama.nodes` With the current dataset snapshot, loading fails: ``` ERROR: duplicate key value violates unique constraint "nodes_pkey" DETAIL: Key (node_id)=(51122) already exists. CONTEXT: COPY nodes, line 785427 ``` Root cause: ICIJ's export legitimately reuses the same `node_id` across category files when a person/entity plays more than one role (e.g. `51122` = "Peng, Wan-Hsiung" appears in both `nodes-officers.csv` and `nodes-intermediaries.csv`, same person). We counted 1,150 such cross-file collisions in the current snapshot. `transform_panama` (in `sandbox/common/run_benchmarks.py`) iterates all `nodes-*.csv` files and writes every row into a combined `nodes.csv` keyed by `node_id text PRIMARY KEY`, without checking for IDs already emitted by an earlier file — so the second occurrence always breaks the load. Fix applied locally (happy to send as a PR): skip a `node_id` if it's already been seen from an earlier file, so the first occurrence (in whatever file ordering `node_files` sorts to) wins: ```python for row in csv.DictReader(handle): node_id = first_value(row, "node_id", "id", "_id") if not node_id or node_id in node_ids: continue node_ids.add(node_id) ... ``` --- ### 5. (Environment-specific, informational) `venv` creation breaks under `uv`-managed Python shims Not a pgGraph bug, but noting it in case others hit it: both scripts resolve the interpreter via `command -v python3.NN`, which on a machine with Python managed by `uv` (`~/.local/bin/python3.12` → shim → real `uv`-installed interpreter) returns the shim path. Creating a venv through the shim (rather than the interpreter's real, fully-resolved path) writes a `pyvenv.cfg` with `home` pointing at the shim's directory (`~/.local/bin`), which has no `lib/python3.12` beside it — so the new venv's own interpreter can't find its standard library and crashes before `ensurepip` even runs: ``` Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding ModuleNotFoundError: No module named 'encodings' ``` Fix applied locally in both `start_playground.sh` and `run_benchmarks.sh`: resolve the interpreter with `readlink -f` before using it to create the venv. Harmless for normal (non-shimmed) system Pythons, and fixes it for `uv`/`pyenv`/`asdf`-style setups. Also happy to include in a PR. --- Happy to open PRs for #1, #4, and #5 (all low-risk, mechanical fixes) if that's useful — let me know which you'd want, or if you'd rather handle them yourselves.

mikesmullin

## Update: 3 more issues found once the playground was actually running After getting past the original 4 blockers, we hit three more problems while using the playground itself (clicking around the Streamlit UI). Filing here rather than a new issue since it's the same investigation thread. ### 6. `main()` re-verifies/rebuilds the graph on every single script rerun `sandbox/playground/app.py`'s `initialize_graph()` (which calls `bootstrap.ensure_graph_loaded()`) was called unconditionally at the top of `main()`, with no caching. Since Streamlit reruns the entire script on *every* widget interaction, this meant the "Preparing Panama graph..." verification (registration check + potential `graph.build()`) ran again on every click, and the metric strip visibly flashed back to 0/0 each time before repopulating. Fix: decorate `initialize_graph()` with `@st.cache_resource(show_spinner=False)` (same pattern already used for `runtime()`), so it only runs once per server process. ### 7. Clicking "Run SQL" could re-trigger `graph.build()` and wipe the projection Separately, the "Run SQL" button handler called `ensure_graph_loaded(connection, config)` again directly (`app.py`, in the `run_clicked` branch) — a second, uncached call on top of #6. If `_ensure_active_graph()`'s status check ever decided a rebuild was needed at that point, it would invoke `graph.build()`, which takes **~52 seconds** for the Panama dataset (2,016,523 nodes / 6,678,534 directed edges, confirmed via `graph.build()`'s own `build_time_ms` output). This surfaced as: ``` psycopg.errors.QueryCanceled: canceling statement due to statement timeout ``` thrown from inside `bootstrap._ensure_active_graph` → `graph.build()`, and afterward `graph.status()` showed `node_count: 0, edge_count: 0` — the cancellation wiped the projection instead of preserving the previously-good build. Root cause of *why* the build got canceled: `sandbox/playground/execution.py`'s `run_statements()` sets `statement_timeout` via `SELECT set_config('statement_timeout', %s, false)` — the `false` makes it session-scoped, not transaction-scoped. Since `client.py` deliberately reuses one long-lived autocommit connection for the whole server process, that 30-second cap (`PlaygroundConfig.statement_timeout_ms`, default 30000) set by the *first* query a user ever ran leaked forward into every later statement on that connection, including any subsequent `graph.build()` call triggered by #7 — which reliably needs more than 30s and so was reliably killed. Fixes: - Removed the redundant `ensure_graph_loaded(connection, config)` call from the Run SQL handler — the graph is already guaranteed loaded once via #6's cached `initialize_graph()` before the SQL editor is even rendered. - Added a `try/finally` in `run_statements()` that resets `statement_timeout` back to `0` after every execution, so it can no longer leak into later statements on the reused connection. Verified end-to-end against the real `bootstrap.ensure_graph_loaded` / `execution.run_statements` code and a live Postgres connection: one `ensure_graph_loaded()` call takes ~52s and builds the graph; three subsequent simulated "Run SQL" executions on the same connection each completed in 44-47ms with the projection staying intact throughout (previously it would have been destroyed by the second or third click). ### 8. Unpinned `pandas`/`pyarrow`/`numpy`/`narwhals` cause a silent segfault when rendering results `sandbox/playground/requirements.txt` pins `streamlit` and `psycopg` directly, but not their transitive deps needed for dataframe rendering. `pip install` resolved those to whatever was newest at install time — `pandas==3.0.5`, `pyarrow==25.0.0`, `narwhals==2.24.0` (Streamlit's dataframe-compatibility shim) — and that combination segfaults the Python process (no traceback, just process death) the instant `st.dataframe(...)` tries to render a real query result. Reproduced both via the live server (crashed after clicking a sidebar example + Run SQL) and via Streamlit's own headless `AppTest` harness against the unmodified `app.py`. Fix: pin the dataframe stack to a known-stable combination in `requirements.txt`: ``` pandas==2.2.3 pyarrow==17.0.0 numpy==1.26.4 narwhals==1.9.4 ``` Verified via `AppTest` replaying the exact failing sequence (click a sidebar query example, then Run SQL) — clean run, 3 result tables rendered, no crash, metrics stable. --- Happy to send all of this (findings #1-8) as one or more PRs if useful — let us know how you'd like it split up.

evokoa-admin

Thanks for letting us know, we will investigate and update again!