Barty-Bart/ornith-1.5-35b-tests

Ornith 1.5 35B-A3B on a Mac Studio — every prompt and every raw output, including the builds that don't run.

★ 3Forks 1HTMLGitHub ↗Compare

README

Ornith 1.5 35B-A3B — local coding and vision tests

Watch the full test

Watch the full test on YouTube

Every prompt and every raw output from testing Ornith 1.5 35B-A3B on one Mac Studio. Nothing here is edited, summarised or cleaned up — the HTML files are exactly what the model wrote, including the broken ones.

The headline build is the 4-bit, 21.6 GB, which runs on a 32 GB Mac. An 8-bit column is included as a control.

The four variants

Each task was built four times, --no-session every time, so each run starts from zero context and the only variable is the label.

File Quant Thinking Prompt
4bit-nothink.html 4-bit off prompt.md
4bit-thinking.html 4-bit on prompt.md
4bit-thinking-upgraded.html 4-bit on prompt-upgraded.md
8bit-thinking.html 8-bit on prompt.md

The three basic runs share one byte-identical prompt. The upgraded prompt is a different file: it adds three forced steps — write a plan, build it, then prove it runs.

The thinking-off runs exist because the earlier Qwen and DeepSeek baselines were run with thinking off, and comparing against them any other way would be unfair.

Results — 7 of 12 builds work

Task 4bit-nothink 4bit-thinking 4bit-upgraded 8bit-thinking
Tower defence works fails works works
Spreadsheet works works works works
FPS (three.js) fails fails fails fails

All four spreadsheets pass a real formula battery — operator precedence, parentheses, left-associative subtraction, cell references, SUM, AVERAGE. 8 of 8 each.

"Works" means it was loaded in a browser, the button that starts it was clicked, the state moved and the console was clean. Not that it parsed.

Every failure is a transcription slip

Not one is a design failure. All twelve files contain a plausible, well-structured version of what was asked for. What breaks is a character, or an ordering mistake:

Build Cause
Tower 4bit-thinking between: true at init inverts the guard in startWave() — the button is dead forever, and nothing is logged
FPS 4bit-nothink function loop() declared twice
FPS 4bit-thinking new THREE CapsuleGeometry — a missing dot
FPS 4bit-thinking-upgraded <!-- comment --> inside a JS module
FPS 8bit-thinking obstacles used nine lines before it is declared

The FPS folder has errors.md with the exact lines, plus 4bit-thinking-PATCHED.html — that file with the single missing dot restored and nothing else changed. It is a complete, working game.

Two things that did not help

Thinking. The tower failed with it on and passed with it off.

The upgraded prompt. It asks the model to prove the build runs. The model has no browser, so on the FPS task "prove it runs" became "write a QA file asserting it runs" — while the file had a syntax error in it. A self-check is worth exactly what the checker can observe.

8-bit did not fix the FPS either, and its failure is the most instructive on the whole project: it parses perfectly, renders a complete HUD, sizes its canvas correctly, and the console is clean. Only the 3D scene is missing. Nothing short of executing the page finds it.

Build times

Task 4bit-nothink 4bit-thinking 4bit-upgraded 8bit-thinking
Tower defence 1m 36s 2m 27s 16m 43s 1m 46s
Spreadsheet 16m 31s 18m 50s 21m 23s 7m 29s
FPS (three.js) 2m 36s 1m 46s 25m 29s 2m 03s

The 8-bit spreadsheet finished in under half the 4-bit's time, which is worth a second look before assuming 4-bit is the fast one.

Vision

01-invoices — three invoices of increasing density, every line item extracted and checked. 129 of 132 values correct. Open 01-invoices/results.html.

Layout

00-setup/           the rig, the model, why not the official MLX build
01-invoices/        invoice images, ground truth, rendered results
02-tower-defence/   2 prompts, 4 outputs
03-spreadsheet/     2 prompts, 4 outputs
04-fps-threejs/     2 prompts, 4 outputs, the failure write-up, 1 patched file

Open any .html straight from disk. The three.js files fetch three.js from a CDN, so those need a network connection; everything else is self-contained.

Contributors

Barty-Bart

Issues