FinAgentBench

Key findings

1. Execution is commoditized. Given a precise method spec and raw data, nearly every model — including sub-cent-per-run ones — perfectly replicates standard quantitative studies (momentum portfolios, event studies, an option-overlay backtest, an 11GB full-market study). See T1–T4.
2. The frontier appears at paper scale. Replicating a full machine-learning asset-pricing study end-to-end (T6: train four model classes, annual refits, out-of-sample portfolio evaluation) splits the field roughly in half. Failures are precision failures — every model produces plausible magnitudes; only some hold twenty-plus spec details simultaneously.
3. Reading long financial text yields real, measurable judgment. On analyst articles published after training cutoffs, flagship models reach Spearman ICs around 0.4–0.5 vs ≈0.2 for small models; with the text withheld, every model collapses to ≈0. See B2.
4. On historical data, apparent skill is often memorization. Some models predict historical event outcomes better with the news withheld — they recall two decades of price history from (ticker, date) alone. The honest metric is ΔIC = IC(text) − IC(withheld). See B1.
5. Models rank better than they call direction. Several models show significantly positive ICs with below-50% hit rates: a systematic long bias. Use these signals cross-sectionally, not for market timing.
6. Reliability, quantified. Re-running identical tasks: ~2–3% of runs silently produce wrong answers and ~6% die on provider infrastructure — consistent with what production-agent practitioners report as their top challenge.
7. Published ML alpha decays. Re-running a canonical ML asset-pricing design on 2015–2021 (after the original out-of-sample period), decile long-short Sharpe falls from 2+ to ≈0.2–0.3 and linear signals flip negative; the nonlinear-beats-linear ranking survives.

Methods in brief

• One harness, one platform. Every model runs the same two-tool agent loop (bash sandbox + answer submission, step-capped) through one inference platform; provider-specific API quirks are normalized at the gateway layer and documented.
• Script-graded, private ground truth. Track A answers are graded by tolerance bands (loose ±5%, strict ±1%) against reference implementations run on private data extracts — published numbers can't be recalled from training.
• Controls everywhere. Every text-prediction track has a withheld-text control run per model, bounding memorization; Track B2 additionally uses only articles published after model training cutoffs, with per-model cutoff verification.
• Replicates. Reliability numbers come from repeated identical runs (up to 5 seeds); leaderboard metrics are being extended with multi-seed error bars.

Track A — Execution: replicate from spec

The agent gets a task sheet (method spelled out, results withheld), read-only data, a bash sandbox, and a 40-step budget. Grading is scripted: each submitted statistic vs ground truth (loose ±5% / strict ±1%). Samples are private extracts, so published numbers can't be recalled from training.

T1 · 20-stock momentum

T2 · News event study

T3 · Covered-call option backtest

T4 · Full-market momentum (11GB)

T6 · Replicate an ML asset-pricing study (RFS-style, 4 models trained)

Track B — Judgment: predict returns from text

B1 · News headlines

1,008 stratified news events on 20 mega-caps (2004–2024) from an institutional news-event dataset. Model sees headline + ticker + date + trailing stats; predicts 5-day beta-adjusted return. Control: headline withheld (memorization baseline).

B2 · Analyst articles (post-cutoff)

Full-text investment research articles published Apr–Aug 2026 — after the training cutoff of the models under test. Model predicts 20-day market-adjusted return. Control: article withheld.

B4 · Earnings press releases (post-cutoff)

8-K item-2.02 earnings press releases filed with the SEC Feb–Jul 2026 (after most models' training cutoffs), fetched from EDGAR. Model reads the release and predicts the 5-day post-filing market-adjusted return. Control: text withheld.

B6 · Earnings-call transcripts (post-cutoff)

Full earnings-call transcripts (Feb–Jul 2026). Model reads management remarks + Q&A and predicts the 5-day post-call market-adjusted return. Control: transcript withheld.

B5 · 13F smart-money adds (post-cutoff, small n)

Top new/increased positions from Q1-2026 13F filings of 11 prominent funds (filed mid-May 2026). Model sees who bought, size, and portfolio weight; predicts the 20-day post-filing market-adjusted return. Control: same stock/date without the 13F context. n≈38 — indicative only.

B7 · FOMC statements & minutes (macro)

42 Federal Reserve releases (2024-01..2026-07, 8 post-cutoff). Model reads the release and predicts SPY's close-to-close move over the next 5 trading days (TLT also collected). Control: text withheld — the model knows only that a Fed release happened that day.

B3 · SEC filing MD&A

Management's Discussion & Analysis sections from 10-K/20-F filings of 33 AI-infrastructure companies (2017–2024); predict 20-day post-filing market-adjusted drift. Control: text withheld.

Judgment across model generations

Running each family's successive releases on the same post-cutoff article sample asks: is financial judgment improving generation over generation? Broadly yes — most families climb — though unevenly, and several legacy serving pools were no longer available to test (oldest generations excluded where marked on the platform).

Anthropic Claude Opus

OpenAI GPT

Moonshot Kimi

DeepSeek

Cost-optimal routing

Because every model runs the same tasks in the same harness, the results double as a routing table: the cheapest model that solves each difficulty tier perfectly. Routine replication does not need a flagship.

Agent efficacy — from ranking skill to P&L

Three falsifiable checks on the strongest signal (B2). 1) Tradability: weekly long/short tercile portfolios from each model's predictions, 20-day overlapping holds, 10bps one-way costs — all configurations finished positive over the (short, 3-month) sample. 2) Stability: re-running the same model on the same articles gives prediction rank-correlation ≈ 0.9 and near-identical ICs (Qwen3.8-Max 0.451→0.468, Sonnet-5 0.389→0.437 across seeds). 3) Pre-registration: predictions for brand-new articles are committed to this repository before outcomes exist — see the locked file; check back a month later.

67 trading days, Apr–Aug 2026, AI-heavy universe, overlapping cohorts — treat annualized figures as directional, not expected returns.

Which financial text carries alpha?

The same models, the same harness, four post-cutoff text sources — very different outcomes. Analyst opinion articles (B2) support strong cross-sectional ranking (best models IC ≈ 0.4–0.6): opinions diffuse slowly. Earnings press releases (B4) show near-zero or negative IC for most models: hard numbers are priced within minutes, so reading them the next day adds nothing — models that naively map "good quarter → buy" get systematically caught by post-announcement reversals. 13F disclosures (B5) show positive signal on a small sample. The lesson: text selection dominates model choice.

Side finding — ML asset-pricing alpha after publication

Building T6's ground truth required re-running a scoped version of a canonical machine-learning asset-pricing study (94 firm characteristics, expanding-window refits) on 2015–2021 — entirely after the original paper's out-of-sample period. The equal-weighted decile long-short portfolios that earned Sharpe ratios above 2 in the original sample largely vanish:

Nonlinear models still beat linear ones — the paper's qualitative ranking survives — but linear signals flip negative and the economic magnitude is a fraction of the published era. (Scope deviations: characteristics only, equal-weighted deciles, training history starts 2004.)

Honest caveats