ALCI Assessor

An AI Literacy assessment chatbot built with the Embabel Agent Framework.

Conducts the AI Literacy Collaboration Index (ALCI) assessment through a conversational interface -- CLI or web chat. Scans repositories for observable evidence, administers the ALCI instrument, assesses literacy level across three disciplines, and generates recommendations.

Quick Start

# CLI mode (Anthropic only -- one API key to get started)
docker run -it \
  -v /path/to/project:/repo \
  -v ./assessments:/assessments \
  -e ANTHROPIC_API_KEY=your-key \
  alci-assessor

# CLI mode (dual provider -- cost optimised)
docker run -it \
  -v /path/to/project:/repo \
  -v ./assessments:/assessments \
  -e ANTHROPIC_API_KEY=your-anthropic-key \
  -e OPENAI_API_KEY=your-openai-key \
  -e SPRING_PROFILES_ACTIVE=openai \
  alci-assessor

# Web mode
docker run -p 8080:8080 \
  -v /path/to/project:/repo \
  -v ./assessments:/assessments \
  -e ANTHROPIC_API_KEY=your-key \
  -e SPRING_PROFILES_ACTIVE=web \
  alci-assessor

Open http://localhost:8080 after the container starts. The chat UI loads automatically and begins the assessment session.

Running Locally (without Docker)

Prerequisites

JDK 21 (Eclipse Temurin recommended)
Maven 3.9+
An Anthropic API key

Wrapper Script (Quickest Way)

export ANTHROPIC_API_KEY=your-key

./run.sh                    # CLI mode
./run.sh web                # Web mode → http://localhost:8080
./run.sh openai             # CLI with dual-provider (needs OPENAI_API_KEY)
./run.sh web openai         # Web with dual-provider
./run.sh --help             # Show all options

The script validates prerequisites, builds the JAR, and starts the application.

CLI Mode

# Build the application
mvn package -DskipTests

# Run in CLI mode (interactive terminal)
ANTHROPIC_API_KEY=your-key java -jar target/alci-assessor-0.1.0-SNAPSHOT.jar

# Assess a local project
ANTHROPIC_API_KEY=your-key java -jar target/alci-assessor-0.1.0-SNAPSHOT.jar \
  --assessor.output-dir=./assessments

The Spring Shell prompt appears. Type assess --team "My Team" --repo /path/to/project to start an assessment.

Web Mode

# Run with the web profile — chat UI on http://localhost:8080
ANTHROPIC_API_KEY=your-key \
  SPRING_PROFILES_ACTIVE=web \
  java -jar target/alci-assessor-0.1.0-SNAPSHOT.jar

Open http://localhost:8080 in your browser.

Dual Provider (Cost Optimised)

ANTHROPIC_API_KEY=your-anthropic-key \
  OPENAI_API_KEY=your-openai-key \
  SPRING_PROFILES_ACTIVE=openai \
  java -jar target/alci-assessor-0.1.0-SNAPSHOT.jar

Running from Source (Development)

# Run directly with Maven (no JAR build needed)
ANTHROPIC_API_KEY=your-key mvn spring-boot:run

# Run in web mode from source
ANTHROPIC_API_KEY=your-key mvn spring-boot:run -Dspring-boot.run.profiles=web

Build

# Compile and run tests
mvn verify

# Build the fat JAR (skip tests for speed)
mvn package -DskipTests

# Build the Docker image
docker build -t alci-assessor .

Configuration

Variable	Required	Default	Description
`ANTHROPIC_API_KEY`	Yes	--	Anthropic API key for reasoning-tier LLM calls
`OPENAI_API_KEY`	No	--	OpenAI API key for balanced/fast calls
`SPRING_PROFILES_ACTIVE`	No	(shell)	`web`, `openai`, or `web,openai`
`ASSESSOR_OUTPUT_DIR`	No	`./assessments`	Directory for assessment output

LLM Provider Modes

Anthropic only (default) -- one API key, all roles use Claude:

docker run -it -e ANTHROPIC_API_KEY=your-key alci-assessor

Dual provider (cost optimised) -- Claude for reasoning, OpenAI for balanced/fast phases. Roughly 40-60% cheaper per assessment:

docker run -it \
  -e ANTHROPIC_API_KEY=your-anthropic-key \
  -e OPENAI_API_KEY=your-openai-key \
  -e SPRING_PROFILES_ACTIVE=openai \
  alci-assessor

Custom role overrides -- set model roles individually:

docker run -it \
  -e ANTHROPIC_API_KEY=your-key \
  -e EMBABEL_LLM_ROLE_BALANCED=openai \
  -e EMBABEL_LLM_ROLE_FAST=openai-mini \
  alci-assessor

Model Routing

Role	Anthropic Only	Dual Provider	Used By
`reasoning`	Claude Sonnet 4	Claude Sonnet 4	Level assessment, recommendations
`balanced`	Claude Sonnet 4	GPT-4o	Document assembly, trajectory
`fast`	Claude Sonnet 4	GPT-4o-mini	Question generation

Of 10 agent actions, only 2 use the reasoning tier. 4 actions use no LLM at all (scanning, user I/O, questionnaire scoring, file writing).

Architecture

The assessment runs as a 10-action GOAP agent (ALCIAssessorAgent). All user I/O flows through UserInteractionPort, which has two implementations:

ShellInteraction -- active without the web profile; reads/writes the terminal
WebInteraction -- active under the web profile; bridges the blocking agent thread to the browser via LinkedBlockingQueue pairs and SSE

Three phases:

Evidence -- repository scan + clarifying questions + ALCI Part A placement + Part B/C deep dive
Judgment -- LLM level assessment + recommendations + trajectory comparison
Delivery -- document assembly + markdown write to /assessments/

Enforcement Mechanism Map

ENFORCEMENT LOOPS
=================

PR Loop (every push/PR)
  deterministic:
    lint-markdown.yml -----> markdownlint-cli2 "**/*.md"
    kotlin-tests.yml ------> mvn -B verify (tests + JaCoCo coverage check)
    kotlin-tests.yml ------> OWASP dependency-check (advisory)
    kotlin-tests.yml ------> docker build -t alci-assessor:test .
  agent:
    code-reviewer ---------> CUPID + LP review (Conventional Comments)

Weekly Loop (Dependabot + GC rules + mutation testing)
  deterministic:
    dependabot.yml --------> maven + github-actions updates
    mutation-testing.yml --> mvn pitest:mutationCoverage (score tracking)
  agent:
    GC1 doc-freshness -----> README, CHANGELOG, HARNESS staleness check
    GC2 convention-drift --> CLAUDE.md vs actual code patterns
    GC3 stale-agents ------> AGENTS.md entry age check
    GC4 snapshot-staleness -> observability/snapshots age check

Quarterly Loop (manual)
  assessment ------------> /assess (ALCI self-assessment)
  harness-audit ---------> constraint drift check
  reflection-review -----> REFLECTION_LOG.md promotion
  mutation-trends -------> compare quarterly PIT scores
  cost-review -----------> MODEL_ROUTING.md vs actual spend

Harness Status

Metric	Count
Constraints	8
Enforced (deterministic + agent)	8
Unverified	0
GC rules	5

See HARNESS.md for the full constraint catalogue.

Project Structure

Path	Purpose
`CLAUDE.md`	Project conventions for AI coding tools
`AGENTS.md`	Compound learning memory
`HARNESS.md`	Enforcement constraint catalogue
`MODEL_ROUTING.md`	Model tier assignments for development agents
`REFLECTION_LOG.md`	Post-task reflections
`.claude/skills/`	2 skills (literate-programming, cupid-code-review)
`.claude/agents/`	4 agents (orchestrator, implementer, reviewer, integration)
`.claude/commands/`	1 command (/reflect)
`specs/001-alci-assessment/`	Spec and implementation plan
`observability/snapshots/`	Health snapshot directory

License

Apache License 2.0

russmiles/ai-literacy-assessor

README