Evaluation suites

Evaluation suites are first-class objects in Orkena. They are the gate between a draft and production.

What a suite contains#

suite: evs_refund_regression
datasets:
  - refund-standard-v2
  - refund-edge-cases-v1
judge:
  rubric: rubric_refund_v3
  pass_threshold: 4.2

• Dataset — input/output pairs plus judge criteria
• Judge rubric — the scoring criteria (typically LLM-as-judge with a defined rubric)
• Pass threshold — the score a version must meet

Promotion gate#

A version reaches production only through gate.eval:

1. A draft is submitted for promotion.
2. The regression suite runs against the candidate version.
3. Score ≥ threshold → publishable.
4. Below threshold → the promotion is blocked, recorded, and the draft returns to the author.

Self-proposed rules#

When the runtime proposes a new operating rule from consolidated memory, the same path applies: the rule stays inert until it passes the suite AND a human declines to veto it. The veto window, the reviewers, and the eval result are all ledger events.

Monitoring#

Run suites on a schedule or on demand. Every result is a ledger event; a history of scores per version is visible in the UI and exportable.