Documentation: https://lab.pollack.ai/docs/agent-judge/tutorial | API Reference

The agent says it is done. Should you merge?

Part of that question already has an answer you know how to write down: it compiles, the tests pass, the file is there. Part of it does not.

JUnit handles assertions where we know how to write the oracle. Agent Judge extends that testing discipline to criteria whose oracle requires richer evidence or judgment.

Eleven executable Maven modules for Agent Judge, sequenced by the evaluation problem rather than by the API. Released Agent Judge 0.17.0; no API key or model calls. First builds download Maven dependencies; prepared builds can run offline.

There are two paths through this repository.

The case study has its own module numbering, local to itself, and its own README with exact run commands. It complements the progressive tutorial rather than replacing it.

• Java 21+
• Maven 3.8+ (or the included ./mvnw)

git clone https://github.com/markpollack/agent-judge-tutorial.git
cd agent-judge-tutorial
./mvnw install                # prepare dependencies and run regression tests
./mvnw exec:java -pl module-01-oracle-boundary

To prepare and verify every demo's dependencies, including nested fixture builds, run the integration suite online once (JBang required). Then repeat with both JBang and Maven offline:

./integration-testing/scripts/run-integration-tests.sh
./integration-testing/scripts/run-integration-tests.sh --offline

For an individual prepared demo, export MAVEN_ARGS="${MAVEN_ARGS:-} -o" also keeps its nested Maven builds offline.

The website tutorial carries the teaching explanations; this repository supplies runnable examples and their expected results.

Run every module from the repository root: the demos and tests resolve test-workspace relative to where Maven was launched.

Modules 01 through 06, in directory order. About 18 seconds of runtime, and 10 to 15 minutes of talking.

./mvnw exec:java -pl module-01-oracle-boundary      # where JUnit stops
./mvnw exec:java -pl module-02-build-and-tests      # the oracles you already trust
./mvnw exec:java -pl module-03-coverage-evidence    # a measurement is not a decision
./mvnw exec:java -pl module-04-custom-judge         # write one: task, criterion, evidence
./mvnw exec:java -pl module-05-derived-judge        # several facts, one finding
./mvnw exec:java -pl module-06-definition-of-done   # would you merge this?

01, 02, 03 and 06 carry the narrative. 04 and 05 are the craft beats, showing how a judge is written before 06 puts five of them in one table. Skip them if the discussion runs long; the arc still closes without them.

It ends here, which is the whole point of running it:

  CRITERION          ORACLE     STATUS
  ------------------ ---------- ------
  build              known      PASS
  tests              known      PASS
  coverage           known      FAIL
  package-structure  derived    PASS
  architectural-fit  judgment   FAIL

  done: false

Modules 02, 03 and 06 run real Maven builds. Run each once before presenting so the dependencies are warm.

No module on this path needs an API key or a model endpoint. First builds download Maven dependencies; with those dependencies prepared, the path can run offline as described above.

Plus judge-junit - a small bridge turning an Agent Judge outcome into a useful JUnit assertion failure.

The mistake is treating every eval as an LLM-as-a-judge problem. Start by asking what kind of oracle you actually have.

Most criteria that feel like they need a model do not. The sequence reveals this through code rather than teaching it up front.

Use the least interpretive instrument that can reliably answer the question. Agent Judge does not replace JUnit, AssertJ, ArchUnit, Checkstyle, JaCoCo, JApiCmp, or the compiler. Module 01 keeps four criteria in plain assertTrue on purpose.

Compose requirements. Aggregate estimates of the same uncertain property. A definition of done is a conjunction, not a vote - 3 of 5 = 0.60 lets "the tests pass" offset "it does not compile". Module 06 is the first rule; module 08 is the second.

test-workspace/ is a small Maven project an agent modified. Its HelloController is the codebase's idiom; its ReportController is what the agent produced. Everything in the tutorial judges that change.

Modules 01 and judge-junit carry JUnit tests. Everything else is a main.

./mvnw test

A credential-free suite that runs each module and asserts its required output.

cd integration-testing
./scripts/run-integration-tests.sh            # everything
./scripts/run-integration-tests.sh --demo     # the live path only
jbang RunIntegrationTest.java module-01-oracle-boundary

Optional Claude validation runs only when AGENT_JUDGE_TUTORIAL_AI_VALIDATE=true. Candidate verification can select an isolated repository and pre-release artifact with AGENT_JUDGE_TUTORIAL_MAVEN_REPO and AGENT_JUDGE_TUTORIAL_AGENT_JUDGE_VERSION; the committed examples target 0.17.0.

• Agent Judge - the library
• Documentation
• Agent Experiment - the runner that uses these juries