Data & reporting · finance workload

query.nl2sql.v1

Answer 15 natural-language finance questions of graded difficulty by writing SQL against a synthetic finance database (GL, AP, AR, dimension tables).

No published results for this workload yet — results appear here with the next edition.

Rate-card unit

Answer 100 ad-hoc finance questions — rates on the rate card are the cost per successful outcome scaled to this unit.

Pass criteria

Generated SQL executes and returns a result set equal to the reference result (order-insensitive unless ordering is asked for). Grading is deterministic; a model output either passes or fails, and infrastructure errors are excluded and rerun rather than counted against the model.

Prompt (verbatim)

Every model receives exactly this prompt with the item's fields substituted for the {placeholders} — no model-specific tuning.

You are querying a SQLite finance database with this schema:

{schema}

Write a single SQLite SELECT query answering the question below.
Respond with ONLY the SQL, no explanation.

Question: {question}

Dataset

Datasets are generated synthetically from seeded generators; the seed is frozen per edition and held out until the run window closes. Generator source hash (SHA-256): 5c4794bca1ba10bf39043edc2c5a9e13a99b1aa18cf889c5d714353a6c569acd. See the methodology §7 for the verifiability protocol and any edition page for the raw graded logs.