Most AI Forecasting Results Are Fiction

Leakage is the first way a forecasting score becomes fiction. There's a second, and it's worse because it doesn't require anything to go wrong.

Suppose your pipeline is genuinely leak-free. You try a prompt variant, and the score improves a little. You try a different ensemble size, a new aggregation coefficient, a tweaked retrieval strategy. Fifty variants later, you report the best one.

That number is not a measurement of forecasting skill. It's a measurement of how hard you searched. Statisticians call this the garden of forking paths; quants call it backtest overfitting; everyone agrees it invalidates the result, and almost everyone does it anyway, because the alternative (spending your test set sparingly, like it's expensive) requires discipline that evaporates under deadline pressure.

And here's what's changed: LLMs made trying fifty variants free. You don't even have to do it yourself anymore. An AI coding agent will happily generate and evaluate a thousand pipeline variants overnight, unattended, and greet you in the morning with the best score of the batch. Evaluation multiplicity used to be a temptation. Now it's industrial.

Two failure modes, one common structure: in both cases, the score is real but the meaning of the score has been destroyed, and nothing in the score itself tells you. A leaked fact and a cherry-picked variant both look exactly like skill.

Inside the machinery, the forecasting itself is a faithful mechanization of the superforecasting playbook, the process Tetlock's research found separates good forecasters from pundits. Take a question like “Will SpaceX reach orbit with Starship before 2027?” Supermike's pipeline anchors on reference classes and base rates first (how often do programs at this stage hit dates like this?), decomposes the question into estimable parts, reasons about the specific case using only evidence admitted by the as-of facade, then fans out an ensemble of deliberately diverse estimators (base-rate-heavy, inside-view-heavy, market-anchored, and extrapolative) whose disagreement is the raw material. The estimates are pooled in log-odds space, extremized and recalibrated using coefficients learned on a dedicated calibration split (never the test set, as noted above), and the ensemble's spread becomes an honest confidence band.

One model call is a sample. A forecast is what you get when a pipeline turns many decorrelated samples into a calibrated number.

And what comes back over the API is not a naked probability. Every response carries (numbers below illustrative):

A probability without receipts is an opinion with a decimal point. The envelope is the product as much as the number is.

Everything above still describes retrospective evaluation (useful for development, and always “suspect until leakage-audited” in our own house style). So here is the stance the whole system builds toward:

The only number we'll ever ask you to trust is the live one.

Every night, Supermike harvests genuinely open questions (questions whose answers do not exist yet), forecasts them, and registers each forecast immutably at the moment it's formed. When a question resolves, the forecast gets scored. Nothing in this loop can be tuned to the answers, because there are no answers to tune to. No retroactive edits, no quiet deletions of embarrassing calls: the registry only appends.

The live loop is running now, and the track record is accumulating. We'll publish it on a public dashboard, including every forecast, every resolution, and every score, flattering or not.

One more commitment about how we'll report it. On questions where a liquid market or crowd forecast exists, we will not claim “we beat the market,” because cherry-picked victories over a consensus are noise, and everyone in this field should stop publishing them. The honest, decision-relevant question is whether Supermike carries additional information: does blending Supermike with the consensus beat the consensus alone? That's the test we'll run and report, and we'll publish the result whichever way it comes out.

In the spirit of the whole exercise:

We are not claiming Supermike beats human superforecasters (we don't have the live sample size yet to claim anything, and we won't pretend otherwise). Parts of the system are honestly unfinished: a learned orchestration layer is designed but deliberately deferred until the corpus of scored forecasts is large enough to train it honestly; forecast memory (“have I forecast something like this before, and how did it resolve?”) is built but not yet wired into the forecast path. A Supermike forecast is also materially more expensive than a chat completion (it's an ensemble plus supervisory passes), and we'll publish the cost-versus-accuracy frontier rather than hiding it. And retrospective evaluation has one problem no retrieval discipline can fix: a model whose training data postdates a question may simply know the outcome. It's one more reason the live number is the only real one.