LedgerRate Benchmark Specification

Version 1.0 · July 2026 · Status: Draft Methodology licensed CC-BY 4.0. Results data proprietary.

1. Scope & Design Principles

LedgerRate measures the cost per successful outcome (CPO) of AI models on real business workflows. It does not measure quality ceilings; it measures what a defined unit of work costs when done to an acceptable standard.

1. Two-level architecture. Primitives are a frozen, industry-agnostic capability taxonomy — the durable standard. Workloads are concrete, versioned instantiations of a primitive within a business function. Primitive-level results are comparable across editions; workload versions are swappable without touching the standard.
2. Functions, not industries. Coverage expands by business function (finance ops, customer support, sales ops, HR, legal ops) because functions are cross-industry and map to budget line items.
3. Deterministic-first grading. ≥5 of 7 primitives graded by deterministic checks. Judged tasks carry deterministic sub-checks and mandatory human spot-checks.
4. Independence constraints. All runs on personal infrastructure, public model APIs, fully synthetic data. Raw logs published with every edition.

1.1 The grid

v1 instantiates all 7 primitives in finance ops (§4). The primitive taxonomy is frozen for all v1.x releases.

2. Metric Definitions

2.1 Per cell (workload × model × mode)

Grading costs (judge tokens, execution infrastructure) are excluded from CPO and reported separately.

2.2 Derived indices

• Agentic multiplier — agentic CPO ÷ single-shot CPO, per workload.
• Qualifying model — pass rate ≥ 0.80 on that workload. The quality bar for rate-card inclusion.
• Cheapest qualifying model — lowest CPO among qualifying models. The routing answer.
• Frontier premium — flagship-model CPO ÷ cheapest-qualifying CPO.
• CPO trendline — per workload version, across editions.

3. Run Protocol

• n = 15 independent runs per cell — one attempt per distinct dataset item (15 invoices, 15 batches, 15 questions…). Pass rate therefore reflects coverage across item difficulty, not resampling of one item.
• Modes: single-shot (one request, no tools) and agentic (tool loop, max 25 steps; exceeding the cap = fail). execute.* workloads run agentic-only.
• Sampling: temperature 0 where supported; provider defaults otherwise, documented per model.
• Edition window: all runs within 72 hours. Pricing frozen at window open from official price pages; frozen table published with the edition.
• Retries: none. Model output failures count as fails. Infrastructure errors (429/5xx) are excluded and rerun — infra failures are not model failures.
• Prompts: identical across models per workload, published verbatim. No model-specific tuning; documented exceptions only where a provider requires a structural difference (e.g., tool schema format).

4. Workload Specifications (v1 — Finance Ops)

Every workload defines: dataset, input, required output, pass criteria, grading, rate-card unit. All datasets are generated synthetically from seeded generators; generator code hash published, seeds held out (§7).

4.1 extract.invoice.v1 — Extract

• Dataset: 15 synthetic invoice PDFs; varied layouts; planted edge cases (multi-page, credit note, foreign currency, discount lines).
• Output: JSON per published schema — vendor, invoice number, dates, currency, line items, subtotal, tax, total.
• Pass: all required fields exact-match after normalization (dates ISO, amounts decimal, whitespace/case-insensitive strings). Field-level accuracy reported as diagnostic.
• Rate-card unit: Process 1,000 invoices.

4.2 triage.ap-inbox.v1 — Classify

• Dataset: 15 batches × 20 synthetic AP-inbox emails. Labels: invoice, statement, dunning notice, PO response, other.
• Pass: batch accuracy ≥ 95% (≥19/20).
• Rate-card unit: Sort 1,000 AP emails.

4.3 reconcile.ledger.v1 — Reconcile

• Dataset: synthetic bank feed (200 transactions) vs. GL extract with 12 planted discrepancies (missing entry, duplicate, amount mismatch, date drift).
• Output: discrepancy list with type per item.
• Pass: recall = 100% on planted discrepancies AND ≤1 false positive.
• Rate-card unit: Reconcile a month of transactions.

4.4 generate.variance-commentary.v1 — Generate

• Dataset: budget-vs-actual tables (12 line items) + context notes.
• Output: commentary per template: top-3 drivers, magnitude, cause, ≤300 words.
• Pass (hybrid): (a) deterministic — every figure cited in text matches source; correct top-3 drivers by absolute variance; (b) judge rubric score ≥ 4/5 on clarity and cause-attribution.
• Rate-card unit: Draft 50 variance commentaries.

4.5 synthesize.board-brief.v1 — Synthesize

• Dataset: synthetic board pack (5 documents, ~40 pages) with a hidden checklist of 20 planted key facts.
• Output: 1-page executive brief.
• Pass: ≥90% of checklist facts present (judge-matched, human spot-checked) AND zero fabricated figures (deterministic — every number in the brief must exist in the pack) AND within length limit.
• Rate-card unit: Turn a board pack into an exec brief.

4.6 query.nl2sql.v1 — Query

• Dataset: synthetic finance database (GL, AP, AR, dimension tables); 15 natural-language questions of graded difficulty.
• Pass: generated SQL executes and returns a result set equal to the reference result (order-insensitive unless ordering is asked).
• Rate-card unit: Answer 100 ad-hoc finance questions.

4.7 execute.month-end.v1 — Execute (agentic only)

• Environment: sandbox exposing tools — SQL over the §4.6 database, file read, journal-entry writer.
• Task: mini month-end close: identify required accruals from source docs, reconcile one account, propose journal entries, produce a close summary.
• Pass (end-state): proposed JEs match reference set (accounts, amounts within $0.01, direction) AND resulting trial balance balances. Summary is informational, not graded.
• Rate-card unit: Run a mini month-end close.

5. Judged-Task Protocol

Applies to generate.* and synthesize.*:

1. Judge model fixed per edition and disclosed. Judge sees anonymized outputs (no model identity).
2. Rubrics published verbatim. Deterministic sub-checks always run first; a deterministic fail is a fail regardless of judge score.
3. Human spot-check on ≥20% of judged runs, sampled randomly. Judge–human disagreement >10% triggers full manual regrade of that cell and a methodology note.
4. If a model under evaluation is the judge's family, this is disclosed; sensitivity check run with a second judge on a sample.

6. Rate Card Construction

The consumer-facing artifact. One line per workload:

Job · volume · rate (cheapest qualifying model) · flagship rate Process 1,000 invoices — $X.XX (Model A) · $Y.YY (Model B flagship)

• Rate = CPO × volume, cheapest qualifying model (§2.2). The quality bar (pass ≥ 0.80) is printed on the card.
• Flagship rate shown alongside — the gap is the story.
• Standard footer on card and every chart: brand · methodology v1.0 · edition date.
• Rates are measurements under this methodology, not quotes or guarantees; disclaimer accompanies every card.

7. Versioning & Gaming Policy

• Workload versioning: any change to dataset design, prompts, or pass criteria bumps the workload version (.v1 → .v2). Scores are never compared across workload versions; trendlines break and restart, annotated.
• Primitive taxonomy: frozen for v1.x. Additions only in v2.0 with public rationale.
• Held-out variation: datasets regenerate each edition from seeded generators with fresh seeds. Generator code hash published; seeds withheld until the following edition, then released with raw logs.
• Gaming: vendors optimizing to published instances gain nothing against regenerated data. A formal gaming policy publishes before any certification engagement.
• Errata: errors corrected loudly — correction note on the edition page, re-published tables, changelog entry.

8. Expansion Roadmap

Unfilled cells are future workloads; each new function column is an edition milestone and a content moment. The standard (§1–3, §5–7) never changes when a column is added.

Changelog: v1.0 — initial specification.