← Full results & methodology · 中文版 · Working paper (PDF)
A plain-language tour of FinAgentBench — three days, ~$750 of computing credits, one identical test harness, and some results that surprised us.
When people ask "can AI do finance research?", they're really asking about two very different skills. The first is doing the work: take a research recipe and a pile of raw market data, write the code, and get the right numbers out — what a good research assistant does. The second is judgment: read something — a news story, an analyst's essay, an earnings report — and correctly guess what happens to the stock next. That's the skill that makes money, and it's much rarer in humans too.
We tested both, separately, on data an AI cannot have memorized. That last part turns out to be the whole game — more on that below.
We wrote five replication tasks of increasing difficulty — from "compute a momentum strategy on 20 stocks" up to "replicate the core results of a famous academic machine-learning paper, training four models along the way." Each AI got the recipe, the raw data, a terminal, and a step budget. A script — not a human, not another AI — graded every number they submitted against ground truth we computed ourselves.
On the first four tasks, essentially every model aced it — including a small open-source model that did a perfect replication for a tenth of a cent. Only the fifth task, the academic-paper-scale one, split the field: 9 of 22 models got everything right; the rest trained the machine-learning models correctly but fumbled details of the portfolio accounting.
The catch is reliability. Re-running the same task five times, most models were perfect five out of five — but a few occasionally produced a wrong number with full confidence, and about 6% of runs failed for boring infrastructure reasons (provider API quirks). If you employ an AI research assistant, you still need to check its work — which is exactly what surveys of companies deploying AI agents report.
Here's the result that should change how you read every "AI predicts stocks" headline. We showed models 1,008 historical news events (2004–2024) and asked them to predict each stock's next five days. Then we ran the exact same test with the news hidden — the model saw only the ticker and the date.
The best model predicted the past better with the news hidden. Given just "Apple, March 2, 2013", it ranked what happened next with startling accuracy — because those years of market history are in its training data. It isn't reading the news; it's remembering the answer key. The same thing happened, even more strongly, with Federal Reserve announcements: one model "predicted" the market's reaction to past Fed meetings with 90% accuracy — with the Fed statement completely hidden.
The fair test: nearly 300 investment-research essays published in April–August 2026 — after the training cutoff of the models being tested. No answer key to remember. Models read each essay and predicted the stock's next 20 trading days relative to the market.
Three things stood out. First, hide the essay and every model collapses to coin-flipping — so whatever signal exists comes from actually reading. Second, there's a real ladder: small cheap models show modest but genuine skill, big frontier models show clearly more. Third, the newest frontier generation is a step change, not a gradual improvement — one family's four previous versions all scored in a tight band, and its newest model doubled the score.
We ran the same experiment on seven kinds of financial text. The pattern is economically sensible and very consistent:
| Text the AI reads | Predictive value | Why |
|---|---|---|
| Analysts' opinion essays | Real signal | Interpretations spread slowly through markets |
| Famous funds' quarterly holdings disclosures | Positive (small sample) | "Smart money" trades keep drifting after disclosure |
| Earnings press releases | None | Hard numbers are priced in within minutes |
| Earnings call transcripts | None | Same — by the next day it's all in the price |
| Fed statements | None beyond memory | Macro news is priced instantly |
| Single news headlines | Barely any | One sentence is too thin |
In short: AI adds value where information diffuses slowly — opinions, interpretations, disclosures with a lag. It adds nothing where the market has already reacted. Worse: on earnings text the models extrapolate the announcement's direction while the market mean-reverts over the following weeks — a naive AI reader would systematically buy short-term tops.
We turned the essay-reading predictions into simulated weekly portfolios — buy the stocks each AI was most bullish on, short the ones it was most bearish on, hold a month, subtract trading costs. Over the three months of data, all ten configurations we tried came out positive, several strongly so. The predictions also replicate: run the same model twice and its predictions correlate about 0.9. Two more details worth knowing: the models' self-reported confidence is informative (their high-confidence half is roughly twice as accurate as their low-confidence half), and their predicted magnitudes — not just directions — are approximately correctly scaled.
Everything above — five replication exams, seven text experiments across ~1,800 labeled events, 23 models, reliability re-runs — consumed about $600 of computing credits and two days, orchestrated end-to-end by an AI agent (which also wrote this page, and got its own errors graded along the way). A comparable human-powered study — the kind with 25 authors, 306 survey respondents, and a year of calendar time — costs a six-figure budget. Both kinds of study matter. But when a question about AI capability comes up, the cost of just running the experiment has fallen roughly a thousand-fold. That changes who gets to find things out.