State-of-the-art on public Financial Benchmarks at $0.13 per Question
Hitesh Jain, Divy Vasal · 2026-06-03
Coral Bricks is built for domain-specific agents — the kind that need to reason across many datasets, apply industry-specific conventions, and stay economical to deploy widely. Financial analysis is one of the hardest workloads to do this on, so we picked it as the proof point. We built AlphaCumen — a financial agent — on Coral Bricks, and it achieved state-of-the-art accuracy on three public financial benchmarks at the lowest cost we've seen.
82.6% on Vals AI Finance Agent v2. 90% on Vals AI Finance Agent v1.1. 89.3% on FinanceBench. $0.13 per question.
If you're building domain-specific agents and worried about token costs, read on.
AlphaCumen
AlphaCumen is a multi-agent harness orchestrating 7 agents, 69 skills encoding financial conventions, four datasets: SEC, news, stocks, and options — using Kimi K2.6.
We measured accuracy on 3 finance benchmarks: Vals AI Finance Agent v2 & v1.1 test multi-filing, multi-metric, analyst-judgment-heavy work. And FinanceBench tests clean single-filing extraction.
We ran on the public slices each benchmark exposes for reproducibility; assuming those slices are representative, the 24–26 point lead over the top frontier model is wide enough that the ranking should carry to the full sets, and the 95% confidence intervals reported throughout make the residual uncertainty explicit. We're providing the overall results here; specifics on any row are available on request.
Vals AI Finance Agent v2
Description
27 public questions; 239 atom-level claims total. Newly released hard benchmark. Scoring is atom-level: each answer is evaluated against a structured rubric of 7–13 claims it has to make. Tasks span quantitative retrieval, market research, and financial projections — the same skills an entry-level financial analyst is expected to bring on day one. (Dataset)
Leaderboard
The Vals AI v2 leaderboard scores every entry through a unified agent harness orchestrating 1 agent, 6 generic tools, and 3 datasets — using the model under test. Frontier models cluster at 44–58%. Same model, different harness: Kimi K2.6 jumps from 44.87% on this harness to 82.6% on AlphaCumen's — a 38-point gain.
AlphaCumen: 82.6% overall ± 5% (95% CI; 59.3% strict pass).
Breakdowns
82.6% overall — the rubric-claim pass rate Vals AI publishes on the leaderboard. The remaining split three ways:
- Convention-input mismatches. Multi-step financial models — EV/EBITDA, DCF, LBO — where both AlphaCumen and the rubric do the math precisely, but pick different conventions on input choices like share-count basis, price source, or ratio definition. Both numbers are correct under the convention each assumes; the 1–2% gap is the convention choice, not arithmetic error.
- Narrative phrasing variance. Atom matches the underlying disclosure but uses paraphrased wording the rubric doesn't accept verbatim — common on narrative-heavy questions (AI risk framings, management-rationale callouts).
- Coverage gaps. 9 atoms require data outside our current ingestion.
Vals AI Finance Agent v1.1
Description
50 public questions. Easier predecessor of v2. Each answer is scored against a structured checklist of 4–5 claims it has to make (and a few it can't contradict). (Dataset)
Leaderboard
The Vals AI v1.1 leaderboard scores every entry through a unified agent harness orchestrating 1 agent, 4 generic tools, and 2 datasets — using the model under test. Frontier models cluster at 57–64%. Same model, different harness: Kimi K2.6 jumps from 57.1% on this harness to 90% on AlphaCumen's — a 33-point gain.
AlphaCumen: 90% ± 8% (95% CI).
Breakdowns
Defensible-answer rate: 100%. Accuracy under each question's intended convention, with every disagreement recorded and reviewable. The gap from raw match — 90% → 100% — is the convention layer doing its job.
45 of 50 (90%) pass the published checklist on the first try. The remaining 5 split two ways:
- Convention overrides (2/50). AlphaCumen pulled the right document and did the math right, but applied a different finance convention than the answer key's unstated one. We cross-checked both readings against widely cited analyst sources online and the conventions our answer used are equally defensible — so we override the rubric verdict and count both toward the 100% defensible rate.
- Benchmark-reference bucket (3/50). The published answer key itself is verifiably wrong against SEC filings — deal conflations, ambiguous question wording, stale trend claims contradicted by the issuer's actual disclosures.
Patronus AI FinanceBench
Description
150 questions. Single-filing extraction: each question is scoped to one annual or quarterly report and has a single human-verified gold answer. (Dataset)
Leaderboard
FinanceBench has no live agent leaderboard — the natural measure is whether the model's answer matches the gold key.
AlphaCumen: 89.3% ± 5% (95% CI). Raw gold-key match. After convention reviews, the defensible-answer rate is 99.3%.
Breakdowns
134/150 (89.3%) match the gold key directly. The remaining 16 split three ways:
- Convention disagreements (12). Defensible under either convention — quick ratio, inventory turnover (average vs ending), working capital, segment counting, effective-tax sign, ROA net-income line, "best performing top line" ambiguity, capital-intensive subjective framing.
- Benchmark-key errors (3). Gold answers that are internally inconsistent, mathematically broken, or contradicted by the issuer's actual disclosures.
- Genuine miss (1). Ambiguous period-comparison framing — AlphaCumen chose one defensible reading, rubric expected the other.
Cost
Per-query token and cost stats, measured over the 50 Vals AI v1.1 questions on Kimi K2.6 in production.
Token usage
| Metric |
avg |
p95 |
% of category |
| Turns / query |
23 |
39 |
— |
| Total tokens / query |
619,767 |
1,160,187 |
— |
| Input tokens |
601,481 |
— |
— |
| — New input |
102,644 |
187,571 |
17.1% |
| — Cached input |
498,837 |
963,028 |
82.9% |
| Output tokens |
18,286 |
— |
— |
| — Reasoning |
5,560 |
10,160 |
30.4% |
| — Tool-call |
7,319 |
20,198 |
40.0% |
| — Non-reasoning |
5,407 |
15,093 |
29.6% |
At least ~10.2× lower cost
API costs only, per-query. Kimi K2.6 actual; Opus projected over Kimi K2.6's exact token usage.
| Stack |
$ / query |
vs Opus |
| Opus 4.7 |
$1.348 |
1.0× |
| Kimi K2.6 |
$0.205 |
6.6× |
| Kimi K2.6 on Coral |
$0.133 |
10.2× |
Two drivers:
- Open, affordable base model. Kimi K2.6 lists at a fraction of Opus 4.7 per token.
- Built for the shape of agent workloads — multi-call, cache-heavy traffic. Coral turns that pattern into bill savings a metered API doesn't pass through.
Best accuracy and lowest cost in the same system.
What moved the numbers
Our fixes landed in four categories:
1. Finance rules in code, not prose. Inventory turnover, working capital, basis-point baselines — no single definition analysts agree on. Each rule lives in tested code (~a dozen dedicated computation tools), not in a natural-language prompt the model can paraphrase its way around. The other half is time — also flagged as the #1 failure mode (~63% of errors) by Daloopa in their recent finance-agent benchmark. Fiscal-year references resolve to each issuer's own calendar, not the default calendar year.
The deeper finding: conventions don't even agree within a benchmark. FinanceBench wants ending inventory on one question and average on another — same dataset, same metric, opposite "correct" answers. 12 of our 16 FinanceBench reviews, 2 of our 5 Vals AI v1.1 reviews, and 7 of our 11 Vals AI v2 sub-rubric misses are convention preferences, not factual errors. Each finance convention should be a per-customer setting, not a hardcoded default buried in the prompt.
2. Multi-filing and orchestration. A common real question is "did the company beat the forecast it gave investors?" — pull the forecast from one filing and the actual results from a later one, then compare line by line. AlphaCumen's planner routes this to the specialist that owns the filing-pair tool — one call returns both filings together, so the model can't mismatch quarters. The specialist then renders the comparison into a structured form with a row for every line the company originally guided on, so the model can't skip the non-headline items like stock-based compensation, capital spending, or share count, which often matter as much as the headline revenue or earnings number. Without the form, the model defaults to its training on financial news — where revenue and EPS are "the answer" — and skips the rest.
3. Data coverage. Roughly 10/50 Vals AI v1.1 questions originally failed with a data-coverage gap — the right answer lived in a form type our index didn't carry, or in a filing past our default asof clamp. Most of those now pass strict after we expanded ingestion to include proxy statements (board votes, exec comp), prospectuses (new-share details), and registration statements (capital raises) — filing types the typical finance dataset under-indexes — plus foreign-private-issuer monthly revenue reports and 10-K/A amended-annual filings.
Per-question asof was the single biggest lever in this category — making the asof horizon a per-question setting rather than a global clamp let the planner pull post-asof filings when the question explicitly references later quarters, fixing several failures on its own.
4. Retrieval ranking. Some answers live in a single cell of a table — and pure keyword search ranks that cell low because the surrounding narrative paragraphs repeat the metric name dozens of times while the table cell mentions it once. Embeddings don't save you either: a full sentence in the narrative is a closer semantic match to the query than a bare label-and-number. The fix is two-part — first route the query to the section of the filing that actually carries the granular numbers (the disclosure notes, not the management discussion), then run a structured-table extractor that records each cell as a (metric, period, value) triple — so the model can ask for "gross margin in Q3 2024" and get exactly that cell, instead of paragraphs that mention gross margin. Fixed an entire class of failure.
Key takeaways
- Domain-specific agents beat generic agents. Every Vals AI leaderboard entry already has web search, file retrieval and code execution; still stuck in the 57–64% range. The gap to 90% is closed by finance conventions encoded in tested code and a class of domain edge cases a generic stack can't reach.
- State-of-the-art accuracy at a fraction of closed-model cost. AlphaCumen runs Kimi K2.6 on Coral Bricks at $0.13 per question — at least 10.2× lower cost than Opus 4.7 on the same workload, with higher accuracy.
If you're building domain-specific agents and worried about token costs, talk to us.
Try AlphaCumen in the browser at coralbricks.ai/alphacumen or via GitHub. Architectural deep dive in our companion arXiv paper (coming soon).