Take a photo of your fridge, get recipes you can actually cook with what's in it.

FridgeAI detects the ingredients in a fridge photo with a local vision model, lets you correct the list, matches it against a recipe catalogue by ingredient coverage, and enriches the results with flavour-pairing intelligence from Epicure — substitutions for the ingredients you're missing and a ranked "buy one more thing" suggestion list.

flowchart LR
    A[Snap / pick a photo] --> B[Upload]
    B --> C[Ollama vision model\ndetects ingredients]
    C --> D[Review & correct\nthe detected list]
    D --> E[Coverage-ranked\nrecipe matching]
    E --> F[Epicure substitutions\n+ suggestions]
    F --> G[Scan history]

A more detailed version of this flow — including the intermediate conversation states — is diagrammed in docs/architecture/fridgeAI_processing_flow.svg.

• Features
• Architecture
  • Tech stack
  • Request/processing flow
  • Conversation state machine
  • Data model
  • API reference
  • Security
  • Repository layout
• Running it locally
  • Prerequisites
  • 1. Clone and install
  • 2. Start the local Supabase stack
  • 3. Configure the backend
  • 4. Run migrations and seed data
  • 5. Start Ollama
  • 6. Run the backend
  • 7. Configure and run the frontend
  • 8. Try it end to end
• Configuration reference
• Testing
• Troubleshooting

• Photo capture and upload — native camera screen (expo-camera) or photo library picker, client-side downscale/compress (expo-image-manipulator) before upload.
• Local, private vision inference — ingredient detection runs against an Ollama vision model on your own machine; no photo ever leaves your infrastructure for detection.
• Editable ingredient review — detected ingredients are resolved against a 131-ingredient seeded dictionary via fuzzy matching (Postgres pg_trgm); the user can remove wrong detections and search-add missing ones before matching.
• Coverage-ranked recipe matching — recipes are scored by the fraction of their ingredients present in the user's confirmed list and returned best-match first, each with its own missing-ingredient list.
• Flavour-aware substitutions — for every missing ingredient in a match, FridgeAI asks Epicure's neighbors MCP tool for flavour-close ingredients and offers ones already in the user's list as a substitute.
• "Add one ingredient" suggestions — ranks ingredients the user doesn't have by how many additional recipes they'd unlock, using Epicure's pairing_score to pick a flavour-compatible candidate.
• Scan history — every conversation (scan) is persisted per user and browsable later, complete with its original photo (served via a short-lived signed URL) and its results.
• Session auth — email/password accounts via better-auth, with Expo's secure-storage cookie plugin on the client and a global session guard on the server.
• Hardened by default — every route is authenticated unless explicitly marked @Public(), requests are rate-limited, bodies are strictly validated and size-capped, uploaded files are content-sniffed (not trusted by extension), and Postgres Row-Level Security backs every table.

1. Create — POST /api/v1/conversations creates a row in AWAITING_PHOTO owned by the caller.
2. Upload — POST /api/v1/conversations/:id/photo (multipart, capped at MAX_UPLOAD_BYTES) content-sniffs the file's magic bytes (never trusts the extension or Content-Type header), stores it in Supabase Storage under the user's own prefix, flips the conversation to PROCESSING, and fires the vision pipeline without blocking the response (202 Accepted).
3. Detect — a bounded promise queue (VISION_MAX_CONCURRENCY concurrent, VISION_MAX_QUEUE deep) sends the image to Ollama with a strict JSON output schema, resolves each returned label against the ingredient dictionary (exact term → trigram fuzzy match ≥ INGREDIENT_FUZZY_THRESHOLD), and moves the conversation to AWAITING_REVIEW. Labels that don't resolve to anything in the dictionary are kept as unmatchedLabels instead of being silently dropped. Any vision failure moves the conversation to FAILED with an errorCode instead of hanging.
4. Review — the client shows the detected chips; the user can remove wrong ones and search-add correct ones (GET /api/v1/ingredients?search=, trigram-ranked).
5. Match — PATCH /api/v1/conversations/:id/ingredients writes the confirmed list, scores every recipe by matched_ingredients / total_ingredients (a single coverage CTE against recipe_ingredients), keeps matches at or above RECIPE_MIN_COVERAGE, and returns the top RECIPE_MATCH_LIMIT sorted by coverage.
6. Enrich — for each match's missing ingredients, EpicureService asks the Epicure MCP neighbors tool for flavour-close ingredients and keeps the ones already in the user's confirmed list as a substitution. GET /api/v1/conversations/:id/suggestions separately ranks ingredients the user could buy by how many additional recipes they'd unlock, using Epicure's pairing_score to break ties on flavour compatibility. Every Epicure call is cached in epicure_cache (EPICURE_CACHE_TTL_DAYS) and capped per-conversation (EPICURE_MAX_CALLS_PER_CONVERSATION); setting EPICURE_ENABLED=false makes both endpoints return empty arrays instead of failing.
7. History — GET /api/v1/conversations lists the caller's own past scans, newest first, cursor-paginated.

stateDiagram-v2
    [*] --> AWAITING_PHOTO
    AWAITING_PHOTO --> PROCESSING
    PROCESSING --> AWAITING_REVIEW
    AWAITING_REVIEW --> REVIEW_ACCEPTED
    AWAITING_REVIEW --> PROCESSING: retake photo
    REVIEW_ACCEPTED --> AWAITING_MATCHING
    AWAITING_MATCHING --> COMPLETED
    AWAITING_PHOTO --> FAILED
    PROCESSING --> FAILED
    AWAITING_REVIEW --> FAILED
    REVIEW_ACCEPTED --> FAILED
    AWAITING_MATCHING --> FAILED
    FAILED --> PROCESSING: retry
    COMPLETED --> [*]

Every transition is checked server-side against this table (backend/src/conversations/conversation-status.ts) before it's written; an invalid transition is rejected with 409 Conflict rather than silently applied.

Nine application tables (plus better-auth's own user/session/account/verification), all under Row-Level Security:

Migrations are plain, reviewed .sql files under backend/drizzle/, applied with drizzle-kit migrate. better-auth's own tables are generated separately by its CLI (auth migrate) — see db:migrate.

All product routes are mounted under /api/v1 and require a valid better-auth session cookie unless noted public. better-auth's own routes live under /api/auth/* (sign-up, sign-in, sign-out, get-session, ...) and are documented by the library itself. Live Swagger docs for the whole API are served at GET /api/v1/docs in non-production environments.

A request for another user's conversation returns 404, never 403 — existence of another user's data is not disclosed.

• Auth by default. A global SessionGuard (APP_GUARD) rejects every request without a valid better-auth session unless the route carries @Public(). Only GET /api/v1/health is public.
• Ownership checks, not just auth. Every conversation lookup filters by user_id in addition to id; a mismatched owner gets 404 Not Found.
• Row-Level Security. All nine application tables have RLS enabled in Postgres itself — a defense layer independent of the application code, in case a future code path (or a direct DB client using the anon/authenticated key) forgets a WHERE user_id = ... clause.
• Strict input validation. ValidationPipe runs globally with whitelist: true and forbidNonWhitelisted: true — any unexpected field in a request body is rejected outright, not silently dropped.
• Uploads are content-sniffed. image-signature.ts checks the file's actual magic bytes against JPEG/PNG/WEBP/HEIC signatures; a .txt renamed to .jpg is rejected with 415, and bodies over MAX_UPLOAD_BYTES are rejected with 413 before ever reaching the vision pipeline.
• Private storage, signed access. The Supabase Storage bucket is private; photos are only reachable through short-lived signed URLs (SIGNED_URL_TTL_SECONDS) minted per-request for the owning user.
• Rate limiting. A global throttle (THROTTLE_LIMIT requests / THROTTLE_TTL_MS) plus a tighter per-route limit on photo upload guard against abuse and runaway Ollama/Epicure spend.
• Hardened HTTP surface. helmet() is applied globally, x-powered-by is disabled, CORS is closed by default (CORS_ORIGINS opt-in only), and every response carries an x-request-id for correlating logs.
• No plaintext credentials. Password hashing, session tokens, and cookie handling are entirely owned by better-auth; the application code never touches a password.
• Third-party calls can't fail the product. Every Epicure MCP call is wrapped so a timeout, transport error, or malformed response degrades to an empty result ([]) instead of a 5xx — the core scan → review → match flow is fully independent of Epicure's availability.

fridgeAI/
├── backend/                   NestJS API
│   ├── src/
│   │   ├── auth/               better-auth ESM bridge, session guard, decorators
│   │   ├── common/              request-id middleware, exception filter, image signature sniffing, promise queue
│   │   ├── config/               Zod env schema
│   │   ├── conversations/        the scan lifecycle: controller, service, processing pipeline, status machine
│   │   ├── db/                    Drizzle schema + pg pool module
│   │   ├── epicure/                MCP client, cache repository, service (substitutions/suggestions)
│   │   ├── health/                  liveness endpoint
│   │   ├── ingredients/              dictionary search + fuzzy resolver
│   │   ├── recipes/                   recipe detail endpoint
│   │   ├── storage/                    Supabase Storage wrapper (save/download/sign/remove)
│   │   ├── seed/                        seed data (JSON) + seed script
│   │   └── vision/                       provider abstraction + Ollama HTTP implementation
│   ├── drizzle/                 reviewed SQL migrations
│   └── test/                    Jest unit specs (co-located `*.spec.ts`) + e2e specs against a real test DB
├── frontend/                   Expo Router app
│   ├── app/                     screens: index (home), camera, processing, ingredient-review, recipe/matches,
│   │                            recipe/[recipeId], history, login, register
│   ├── api/                     typed fetch client, better-auth-aware request wrapper, per-resource functions
│   ├── components/              ThemedView/Text/Pressable, RecipeCard, IngredientChip, Spinner, Empty/ErrorState, ...
│   ├── providers/                Theme + TanStack Query providers
│   └── lib/                       better-auth Expo client
├── supabase/                   local Supabase CLI project config (`supabase start`/`stop`)
├── docs/architecture/          processing-flow diagram
└── .github/workflows/ci.yml    lint + build + unit + e2e (against a real local Supabase stack) on every push/PR

• Node.js 24+ and npm 11+ (this is an npm workspaces monorepo — root, backend/, frontend/).
• Docker running (the Supabase CLI runs Postgres/Storage/Auth/Studio as containers).
• Supabase CLI — already a root devDependency, invoked via npx supabase / the root npm scripts, no separate install needed.
• Ollama installed on the host (not containerized — Docker Desktop on macOS has no GPU passthrough).
• Expo Go on a physical device, or an iOS Simulator / Android emulator, if you want to run the frontend on a device rather than the web preview.

git clone [email protected]:QuickerMaths/fridgeAI.git
cd fridgeAI
npm install

This installs both workspaces (backend, frontend) from the single root lockfile.

npm run db:up

This runs supabase start and prints a block of local credentials. Copy the service_role key — you'll need it in the next step. Default local ports: API 54321, Postgres 54322, Studio 54323.

Stop it later with npm run db:down.

cp backend/.env.example backend/.env

Then edit backend/.env:

• SUPABASE_SERVICE_ROLE_KEY — paste the service_role key printed by npm run db:up.
• BETTER_AUTH_SECRET — any random string ≥ 32 characters, e.g. openssl rand -base64 32.
• Everything else has a sane local default (DATABASE_URL and SUPABASE_URL already point at the local stack's default ports). See the configuration reference for what each variable does.

If you'd rather point at a hosted Supabase project instead of the local stack, skip npm run db:up and paste that project's connection string / service-role key here instead — no code changes needed either way.

npm run db:migrate   # applies better-auth's tables, then the Drizzle migrations
npm run seed          # loads the 131-ingredient dictionary and 16 seed recipes (idempotent)

ollama serve
ollama pull qwen2.5vl:7b

OLLAMA_MODEL in backend/.env can point at a different vision-capable model (e.g. llava:13b) if qwen2.5vl:7b is too slow for your hardware — no code change needed, the JSON-schema output contract is model-independent.

npm run dev:backend

Starts NestJS in watch mode on http://localhost:3000. Verify it's up:

curl http://localhost:3000/api/v1/health

Swagger docs are at http://localhost:3000/api/v1/docs.

cp frontend/.env.example frontend/.env

Set EXPO_PUBLIC_API_URL in frontend/.env. On a physical device this must be your machine's LAN IP, not localhost (the phone can't resolve your laptop's localhost), e.g. EXPO_PUBLIC_API_URL=http://192.168.1.20:3000. For the web preview or a simulator on the same machine, http://localhost:3000 works.

If you changed the LAN IP, also add it to CORS_ORIGINS in backend/.env if you're testing the Expo web target in a browser (native clients aren't subject to CORS).

npm run dev:frontend

This runs expo start. Press i for the iOS simulator, a for Android, w for web, or scan the QR code with Expo Go on a physical device.

Register an account, land on the home screen, tap the scan button, take (or pick) a photo of a fridge, wait for the processing spinner, review/correct the detected ingredients, tap "Find recipes", open a match, and check "View scan history" afterwards to see it saved.

All backend configuration is validated at boot with Zod (backend/src/config/env.schema.ts); a missing or malformed value fails startup immediately with the offending key named, rather than failing later at request time.

Frontend: EXPO_PUBLIC_API_URL (see step 7) is the only required variable.

npm run lint:backend         # ESLint, backend
npm run build:backend        # nest build (also a typecheck)
npm run test:backend         # Jest unit tests (co-located *.spec.ts)
npm run test:e2e:backend     # Jest e2e against a real, migrated `fridgeai_test` Postgres database
npm run lint:frontend        # ESLint (eslint-config-expo), frontend
npm run typecheck:frontend   # tsc --noEmit, frontend

test:e2e:backend creates/migrates a dedicated fridgeai_test database and a fridge-photos-test storage bucket against your running local Supabase stack the first time it's run (pretest:e2e), stubs out VisionProvider and the Epicure MCP client so no test call ever reaches Ollama or the public internet, and truncates the conversation-scoped tables between tests. It intentionally runs single-worker (--runInBand): two Jest workers issuing concurrent TRUNCATE ... RESTART IDENTITY CASCADE against the same database reliably deadlock in Postgres.

.github/workflows/ci.yml runs this entire matrix on every push/PR against a real Supabase stack spun up in the CI runner (supabase start), with EPICURE_ENABLED=false so CI never depends on the public Epicure endpoint.

• DATABASE_URL is required (or similar) on boot — a required env var is missing/invalid; the error names the exact key. Check backend/.env against backend/.env.example.
• ECONNREFUSED to Ollama, scan ends in FAILED — ollama serve isn't running, or OLLAMA_MODEL isn't pulled (ollama pull qwen2.5vl:7b). The app surfaces a "Try again" action; it never hangs. Detection is fully local — nothing is sent to Epicure or any other external service during this step.
• Physical device can't reach the backend — EXPO_PUBLIC_API_URL must be your machine's LAN IP, not localhost; make sure the phone and computer are on the same network and the backend's CORS_ORIGINS (if you're on Expo web) includes that origin.
• Epicure substitutions/suggestions are empty — either EPICURE_ENABLED=false, or the public endpoint is unreachable/rate-limiting; both degrade gracefully by design (see Security). The core scan → review → match flow is unaffected either way.
• 401 on every request — the session cookie didn't round-trip; confirm you're signed in (authClient.useSession() client-side) and that BETTER_AUTH_URL/CORS_ORIGINS match the origin the request is actually coming from.