Most AI pilots in finance fail the same way: the demo impresses, the tool gets rolled out, and three months later nobody can say whether it's saving money. The fix isn't better AI — it's a better evaluation. This is the playbook we'd suggest to any finance team, and it's the same logic our benchmark runs on.
1. Pick one job, and define it as a unit of work
Not "use AI in AP." One job: code an invoice, sort an inbox email, draft a variance comment. The unit of work must be countable — you'll be dividing dollars by it later. If you can't finish the sentence "we pay $___ per ___ today," narrow the job until you can.
A good first candidate has three properties: it's high-volume (so small per-unit savings matter), it has a checkable right answer (so grading isn't a debate), and a mistake is recoverable (so the pilot can't hurt you). Invoice data extraction and inbox triage qualify. "Negotiate with this vendor" does not.
2. Write the pass criteria before you run anything
Decide, in writing, what counts as an acceptable result — before you see any AI output. After you've seen output, you'll unconsciously grade on a curve.
Good pass criteria are binary and mechanical: "all seven fields match the source document exactly," "the email is in the correct category," "every discrepancy is found with at most one false alarm." Resist partial credit. A 90%-correct invoice extraction still requires a human to find the wrong 10%, which usually costs as much as doing it from scratch — that's why our benchmark grades pass/fail and reports field-level accuracy only as a diagnostic.
3. Build a test set the vendor hasn't seen
Take 30–50 real (anonymized) examples of the job, including your ugly cases: the multi-page invoice, the credit note, the email that's technically a statement but mentions an invoice. Hold this set back — don't share it with a vendor, don't paste it into a tool during setup. An evaluation on examples the system was tuned on measures memory, not capability.
Grade a human baseline on the same set first. You need two numbers from your current process: pass rate and fully-loaded cost per unit. Without the baseline, an AI pass rate of 85% is uninterpretable — your team might be at 92%, or at 78%.
4. Measure cost per successful outcome, not cost per unit
This is the single most common evaluation mistake. The per-attempt price of an AI call is almost irrelevant; what matters is the expected cost of one acceptable result:
CPO = cost per attempt ÷ pass rate
A tool that costs $0.02 per attempt at a 60% pass rate delivers acceptable results at $0.033 each — plus the human cost of catching and redoing the 40%. A tool at $0.08 with a 98% pass rate costs $0.082 per outcome with almost no cleanup. The "cheap" option is often the expensive one, and per-token or per-seat pricing hides this completely.
Include the review cost honestly. If your control design requires a human to check every output (see the controls guide), the AI hasn't eliminated the labor — it has changed the job from doing to reviewing, which is faster but not free.
5. Run it, and let the failures teach you
Run the full held-out set through each candidate — same inputs, same instructions, no per-vendor tuning beyond what you'd realistically maintain in production. Log everything: inputs, outputs, cost, time. (Our benchmark publishes these logs verbatim for every result; ask your vendor for the equivalent.)
Then read every failure. Failures cluster, and the clusters tell you whether the fix is cheap or structural: a model that fails on foreign-currency invoices can be handled with a routing rule; a model that fails unpredictably cannot be handled at all, because you can't route around randomness — you can only review 100% of its output.
6. Decide with a table, not a feeling
At the end you should be able to fill in one row per option, including "current process":
| Option | Pass rate | Cost/attempt | CPO | Review cost | Failure pattern |
|---|
If the winning row isn't obviously better than the baseline row — meaningfully cheaper per outcome, or the same cost with better speed or auditability — the honest conclusion is "not yet." That's a fine outcome for a two-week pilot. Model prices have been falling and capabilities rising with every generation; a "not yet" from six months ago deserves a re-run, which is exactly why we publish editions monthly rather than once.
What this costs you
The evaluation above is roughly: one analyst-week to assemble the test set and baseline, a few days of vendor runs, and a day of grading and write-up. Against the cost of a mis-bought annual contract — or an ungoverned rollout that your auditors discover before you do — it is the cheapest control you will ever implement.