BENCHMARK
We publish our own overhead.
Governance is not free, and vendors who imply otherwise are not measuring. Two numbers, both true: on a real three-node agent chain with an LLM in the loop, Orkena lands within four percent of LangGraph end to end (3.67s vs 3.53s). On null-load stress tests with no LLM to hide behind, the governed substrate is 12–61× slower — and that work is what buys per-tenant isolation, evidence-grade audit and durable execution.
What we measured
Orkena against LangGraph on identical graphs, with persistence and
audit enabled on both sides. LangGraph runs with
langgraph-checkpoint-postgres and a hand-rolled audit
log — the fair comparison, not raw in-memory dispatch. The earlier
"100–165× slower" headline compared Orkena's full production path
against a substrate that persisted nothing; that number is retired
to benchmarks/raw/ with a README explaining why.
The honest result is two numbers. On a workload where the model dominates wall-clock — which is most of them — the substrate underneath does not matter, and it doesn't. On null-load stress tests where the substrate is the workload, governance costs real time, and we publish that number too.
The workloads
| Task | Shape | Both sides |
|---|---|---|
| governed_chain | Linear chains — 5, 10, 20 sequential governed nodes, no LLM, no external I/O | persistence + audit enabled |
| fanout_governed | 4-branch parallel fan-out with merge — no LLM, no external I/O | durable fan-out path (checkpointer vs outbox → leased steps → fan-in) |
| llm_e2e | Real chain — 3 sequential LLM calls (qwen3-coder via a custom OpenAI-compatible endpoint) | both lanes governed, real provider |
For llm_e2e, "engine" is the same graph run with a mock
model adapter (no network): engine share of wall-clock is the
headline — the engine must be a rounding error on LLM-dominated
latency. For the null-load tasks, the engine is measured by
subtracting a direct-ORM persistence probe (same row writes, no
engine loop) from the total.
The method
Reproducible steps, in order:
- Provision the environment (below) identically for both runtimes.
- Deploy the same graph on each: Orkena with policy evaluation, in-transaction ledger commit and guardrails enabled; LangGraph with the Postgres checkpointer and audit log.
- Run each task 7 times (
--reps 7), warmup excluded; record median and p95 of end-to-end latency. - Isolate the engine: mock model adapter for
llm_e2e; direct-ORM persistence probe (median of 3 probe reps) for the null-load tasks. - Verify the ledger after every run — a regulator can rerun
verify.pyoffline.
Environment: Windows 11 workstation, Python 3.12.13, Node v24.11.1,
Postgres 16 on localhost:5433 (role
meridian_app, RLS enforced), LangGraph 1.2.2 with
langgraph-checkpoint-postgres 3.1.2, psycopg 3.3.4,
SQLAlchemy 2.0.51. Model provider: custom OpenAI-compatible endpoint
serving qwen3-coder.
Raw results
Recorded run, 2026-08-10, git 04047b6 — reproduced in full from benchmarks/RESULTS.md:
Null-load linear chains
| Task | Orkena total median | Orkena total p95 | Orkena engine | LangGraph total median | LangGraph total p95 | LangGraph engine | M/L median |
|---|---|---|---|---|---|---|---|
| governed_chain_5 — 5 sequential nodes | 187.52 ms | 203.72 ms | 90.99 ms | 15.22 ms | 15.82 ms | 5.07 ms | 12.3× |
| governed_chain_10 — 10 sequential nodes | 353.85 ms | 374.82 ms | 161.53 ms | 22.84 ms | 23.71 ms | 7.39 ms | 15.5× |
| governed_chain_20 — 20 sequential nodes | 571.39 ms | 576.14 ms | 236.82 ms | 29.74 ms | 35.09 ms | 9.36 ms | 19.2× |
Null-load durable fan-out
| Task | Orkena median | Orkena p95 | LangGraph median | LangGraph p95 | M/L median |
|---|---|---|---|---|---|
| fanout_governed — 4 branches + merge, durable both sides | 828.86 ms | 889.65 ms | 13.53 ms | 15.67 ms | 61.3× |
Real chain — 3 sequential LLM calls
| Task | Orkena total | Orkena engine | Orkena engine share | LangGraph total | LangGraph engine | LangGraph engine share | M/L total |
|---|---|---|---|---|---|---|---|
| llm_e2e — real chain_3 (env-config model) | 3,674.73 ms | 144.43 ms | 3.9% | 3,534.06 ms | 13.72 ms | 0.4% | 1.0× |
Analysis
On the LLM-bound workload, Orkena lands within four percent of LangGraph end to end — 3,674.73 ms vs 3,534.06 ms, a 1.0× ratio. Engine share of wall-clock is 3.9% on Orkena and 0.4% on LangGraph; both sit below the 5% threshold at which the substrate becomes visible. In production workloads where the model has to think, the runtime underneath does not dominate.
On null-load tests, the gap is real and published: 12.3–19.2× slower on linear chains and 61.3× slower on durable fan-out. That gap comes from work the platform does on every step:
- A tenant-isolated Postgres session per run — RLS enforced by the database, not remembered by application code
- A SHA-256-chained ledger append per event, verified after every bench run
- CEL-evaluated declarative mapping expressions — transform nodes are documents, not Python lambdas
- A worker-crash-safe durable executor on fan-out — outbox, leased steps, fan-in rendezvous
LangGraph makes you build every one of these yourself. We ship them. If your workload needs governance, RLS and evidence-grade audit, this is what they cost. If it doesn't, LangGraph is faster — and this page will keep saying so.
Caveats
- Sample size: 7 reps per task, warmup excluded. These are medians, not confidence intervals.
- One machine: a Windows 11 workstation with local Postgres — no cloud instance classes, no network-to-provider variance exercised.
- One LLM workload shape: a 3-node sequential chain against one provider (custom OpenAI-compatible endpoint serving qwen3-coder). Different models, providers or fan-out-with-LLM shapes will shift the end-to-end ratio.
- Null-load is a stress test, not a workload: no real production traffic is a chain of 20 null nodes. The 12–61× figures bound the worst case, not the typical one.
Rerun in your environment before quoting to your team. The harness is the same script that produced this page.
How to reproduce
The benchmark harness, workload definitions and raw output live in
the benchmarks/ directory of the Orkena repository —
the same script (uv run python benchmarks/run_bench.py)
that generated the tables above, writing JSON to
results/.
Pre-launch, the repository is shared with design partners under NDA rather than public. The raw output for this run is reproduced in full above; public clone instructions will be published here when the repository opens.
Changelog
| Date | Change |
|---|---|
| 2026-08-16 | Initial public methodology; raw results published for the 2026-08-10 recorded run |
What we'd change if we could
The Cedar-native policy evaluator (BE-17, cedarpy over Cedar's Rust engine) landed on 2026-08-16 — after the recorded run above. Its acceptance-run latency (p50 110 µs, p99 352 µs per evaluation, pre-parsed policy set) is expected to cut the policy-path overhead further on the null-load tasks. We'll republish this page with a new recorded run — the tables above will be replaced, not amended.