Just the Retrieval Tools? Isolating What Closes the Gap on Vals AI Finance Agent v2
Hitesh Jain, Divy Vasal · 2026-06-10
In our last post AlphaCumen scored 82.6% on Vals AI Finance Agent v2 — a 38-point gain over Kimi K2.6 on Vals AI's reference agent harness (44.87%). A fair next question: how much of the 38-point gap is retrieval, and how much is reasoning?
Recap
Vals AI Finance Agent v2 is the public finance-agent benchmark — 27 multi-step finance research questions, graded claim-by-claim across 239 atoms. Its reference harness uses general-purpose tools: Tavily web search, EDGAR, and HTML parsing.
AlphaCumen is our finance agent harness — Kimi K2.6 driving a specialist swarm over a pre-built finance corpus, with 6 retrieval indices, 6 specialist subagents, 69 skills, and python execution.
How much of the gap is retrieval vs reasoning?
To isolate retrieval, we ran three configurations — same model (Kimi K2.6, the same one AlphaCumen uses), same data, same judge:
|
LLM harness |
Retrieval |
Skills + computation |
| A. Baseline Eval |
Vals AI |
Tavily / EDGAR / HTML parse |
calculator + price_history |
| B. Test |
Vals AI |
6 indices |
calculator only |
| C. Maximum accuracy achieved |
AlphaCumen |
6 indices |
69 skills + dedicated tools |
A is the leaderboard reference. C is AlphaCumen, our stack. B is the new measurement: keep the Vals harness loop, system prompt, termination contract, and judge identical to A — swap only the raw SEC and stock corpora for pre-processed indices used by AlphaCumen.
Retrieval alone lifts atom-pass only 5% from 44.87% to 49.8% (at 4× the Vals reference budget). 50.2% of the claims never produce an answer at all, exactly the rows the skill stack was built for.
The rest of the post is the how: what we built, what broke on the first sweep, what the row-level split says about where the skill stack earns its keep.
What changes we made to Vals AI harness?
We forked Vals AI's public reference harness (SHA 22a5ed49) verbatim and authored six pure-retrieval adapters into our backends:
| Adapter |
Backend |
bm25_sec |
OpenSearch sec_filings_chunked on coral-bm25 |
ann_sec |
OpenSearch sec_filings_ann (BGE-M3 1024-dim HNSW) |
bm25_gdelt |
OpenSearch gdelt_events_v2 |
bm25_scraped_articles |
OpenSearch scraped_articles_bge_m3 |
get_full_text |
OpenSearch get-by-id on sec_filings_chunked |
equity_bars |
DuckDB over S3 parquet (Daily OHLCV) |
The system prompt (asof = March 1, 2026), submit_final_result termination contract, turn / time limits, and the before_query nudge for text-only responses all stayed untouched from the Vals reference. The only delta vs the leaderboard config is the six retrieval adapters.
The first sweep — default 50-turn budget
We launched the full public benchmark with the Vals defaults: --max-turns 50, --max-time 1800, parallelism 3. Result:
89 / 239 atoms (37.2%)
The headline number hid the actual story: 132 of 239 atoms went unanswered because the corresponding rows produced no candidate answer at all and auto-graded 0/N. Kimi didn't converge — it stalled in reasoning prose and ran the budget out before reaching submit_final_result. 84 of those atoms came from rows that hit the 50-turn cap mid-stride; the other 48 from rows that hit the 30-minute wall inside a single LLM call where Kimi emitted the full 16K-token output budget without a single tool call. The 89 atoms that did pass belonged to the 14 rows that converged — and those rows converged fast: average 10 turns (range 5–20). The hard rows didn't just take longer; they didn't converge at all. The Vals reference budget assumed a model that wouldn't ramble.
Bumping the budget to 200 turns
We bumped budgets 4× (--max-turns 50 → 200, --max-time 1800 → 7200) and re-ran only the 13 silent rows.
Cleared 30 of 239 atoms — Kimi could now drift further before the budget cut it off at 4x more turns per row(52 vs 10). Overall atom-pass moved from 37.2% to 49.8%.
70 atoms still didn't converge even at 4× budget. All heavy multi-step compute. Kimi stalled across multiple turns, each filling its max-tokens budget of 16k tokens.
The lesson: bumping budget lifts accuracy but doesn't fix the hard rows. The hard rows need a structural fix, not a bigger budget.
What actually fixes the hard rows — AlphaCumen's skill stack
AlphaCumen routes each question to multiple specialists in parallel, each with access to precise skills that return structured data instead of the prose Kimi would otherwise generate. The finance conventions (lease treatment, share-count basis, formula choice) are encoded inside the skills themselves. The model can't drift when the answer arrives as a structured result.
The same 70 atoms that timed out under config B (Vals AI harness + AlphaCumen retrieval) all complete cleanly under config C (full AlphaCumen) at the original 30-minute Vals budget. The cleanest cut: 169 of 239 atoms get a candidate answer under B at 4× budget; all 239 atoms get one under C at 1× budget. The skill stack doesn't just improve answer quality — it makes convergence reliable on the atoms where bare-LLM stacks silently abandon.
Takeaways
- Retrieval is the small lever; skills + computation is the large one. Even a domain-specific retrieval upgrade closes only a sliver of the gap on this benchmark. What moves the needle is structured tools that encode the conventions of the domain.
- Generic agent harnesses + frontier LLMs stall on hard domain work. Multi-step finance reasoning isn't a retrieval problem or a model-size problem — it's a structure problem. Without specialist tools, the LLM drifts in prose and never reaches a final answer.
- Budget is not a substitute for structure. Throwing more turns and wall-time at stalling agents is expensive, partial, and leaves the hardest questions unanswered. Structural cures scale; budget doesn't.
AlphaCumen is built on Coral Bricks' inference platform. If you're building agents for a hard domain and want the same runtime underneath, talk to us.
Try AlphaCumen at coralbricks.ai/alphacumen.