Reliability-constrained, multi-objective optimization for LLM and agent programs.

• The thesis
• Related work
• Install
• Quick start
• Example output
• Status
• Contributing
• License

A good AI program is not the one with the highest score. It's the one that satisfies its behavioral contracts while achieving the best defensible trade-off between quality, robustness, cost, and efficiency.

Concretely, that means:

• Contracts, not just scores — hard/soft behavioral requirements (groundedness, schema validity, tool scope) that gate candidates, separate from the objectives you're optimizing. A candidate that violates a contract is excluded from consideration entirely, no matter how well it scores elsewhere.
• Pareto frontiers, not a weighted-sum score — collapsing accuracy/cost/robustness into one number conceals the trade-off. This framework returns the non-dominated candidate set and lets you pick a profile (quality_first, balanced, low_cost, high_reliability).
• Component attribution, not just an aggregate score — for programs built as a Pipeline (a named sequence of steps), controlled zero-ablation swaps one named step for a passthrough and measures the score delta, answering why one architecture outperforms another, not just that it does.
• Automated stress-testing — candidates are scored on perturbed variants of your trainset (reordered clauses, injected distractors, and your own domain-specific perturbers), not just the nominal examples.

reliopt is a meta-layer, not a replacement for your agent framework: it wraps arbitrary Python callables today, with adapters for popular frameworks planned as the ecosystem grows.

See research/RELATED_WORK.md for how this relates to DSPy, Promptfoo, Inspect, and MO-CAPO.

pip install reliopt

from reliopt import Program, Contract, Objective

program = Program(my_rag_agent)

program.contracts(
    Contract.groundedness(minimum=0.90),
    Contract.schema_valid(),
)

program.objectives(
    Objective.accuracy(),
    Objective.cost(minimise=True),
    Objective.latency(minimise=True),
)

result = program.compile(trainset=data)

print(result.render_table())
best = result.select("balanced")
print(result.evidence_card(best.id).render())

See examples/rag_demo.py for a runnable end-to-end walkthrough.

Output of python examples/rag_demo.py, run against the fake, deterministic RAG program that ships with the repo (zero API calls/cost):

6 candidate programs tested
5 rejected for contract violations
Pareto-optimal candidates: 1

Candidate   accuracy    cost        latency     Status
candidate_0 0.667       0.000       0.000       dominated/rejected
candidate_1 1.000       0.000       0.000       dominated/rejected
candidate_2 1.000       0.000       0.000       dominated/rejected
candidate_3 1.000       0.000       0.000       dominated/rejected
candidate_4 1.000       0.000       0.000       dominated/rejected
candidate_5 1.000       0.000       0.000       Pareto-optimal

Evidence card for 'balanced' pick:
Candidate candidate_5
  accuracy             1.0000
  cost                 0.0000
  latency              0.0000
  contract: groundedness         ✅

Early, actively-developed v0.1. The core loop (Program adapter → Contracts → Objectives → Perturbation engine → Pareto compiler → Evidence Card) works end-to-end today; several pieces are intentionally thin placeholders with clearly marked seams — see ARCHITECTURE.md for what's real vs. stubbed, and CHANGELOG.md for what's shipped.

New to the project? Start with an issue labeled good first issue.

See CONTRIBUTING.md. Issues and PRs welcome — good first issue labels are being seeded.

MIT