tbuckley / evals-pwa

evals-pwa

A Promptfoo-inspired evaluation framework for AI prompts as a static PWA. There is no cloud backend, so all your data stays on your device (except for any prompts, which run directly against the Gemini/etc backends). It's free (ignoring your costs to run prompts), private, and requires no installation.

Try it out at evals-pwa.pages.dev, hosted on Cloudflare Pages.

Usage

See src/routes/documentation/+page.md

Building

This project uses:

• Svelte+SvelteKit as a framework
• shadcn-svelte for UI components
• Tailwind for styling
• zod for validation
• mdsvex for markdown rendering

evals-pwa's Issues

Handle long lines in output

whitespace-pre-wrap is an issue

Include config description in dropdown

Pass vars to javascript functions, llm-rubric

Let users cancel a run

Should referenced paths be relative to the file mentioning them?

For example, if tests/foo.yaml has a var file:///images/bar.png, is that images/bar.png or tests/images/bar.png?

Switch to nunjucks

More compatibility with the ecosystem -- promptfoo and ninja-based Python tools like LLM and Prompt Poet.

If an image file does not exist, show the full path not just the folder

Test references prompts/foo.png
Actual path is tests/foo.png

Error: "prompts not found" (or similar)

Support markdown formatting in responses

On mobile, sidebar links should close sidebar

Auto-create runs folder

Allow prompts/tests to be in separate files

Support model options

Remember whether to show vars across sessions

Show a nice error if yaml doesn't parse

Switch to static adapter

Show results in real time

Use pre-wrap for cell formatting

Show variables in table

Allow json for config (or external tests/vars) in addition to yaml

Host on Cloudflare Pages

Filter table to failures

Switch from svelte-headless-table to custom solution

Too many issues, eg column resizing breaking.

Show overall success rate

Also allow importing .js files as text

Allow variables inside assertion args

Notify when evals are complete

Catch errors in JS assertions

Consider how to allow multiple configs

Add llm-rubric assertion

Vars: { prompt: string, provider: string | Provider }

Optimize few-shot examples

Let user choose how many examples
Can take a greedy approach to start? Pick the single example with best impact, then the next one...
Estimate the cost per run

Allow users to resize columns

Show errors during run to the user

Reset scroll position when you navigate

Add about page

Let users generate a config from a description

Handle errors if config.yaml does not already exist

Provide defaults for a test case via defaultTest

Add table of contents to documentation

Support images / files

Show cost of query, total cost for run

Support relative file paths

Add cache

Show images in variable columns

Let users view and edit all API keys, even if not required for the config