Calibration record
How much to trust a number before you use it. This page shows how our forecasts scored against what actually happened, family by family, once their questions resolved. Pastcasts and live forecasts are scored separately. The exceptions are the families scored against a third-party archive of GitHub events we have stopped trusting; their scores are withheld.
3 more pastcast scores are withheld because the Wikipedia pages we counted for them turned out to be redirects. 9 more pastcast scores are withheld because every day in the history we used to set their questions was counted at least twice.
Read from the record at 24 October 2026, 17:30 UTC.
The launch gate
Four statistical criteria, all required, evaluated on point-in-time backtests ("pastcasts") of the forecaster version that is live. Passing the gate is what lets us market a family; failing it is published just the same. A verdict covers one horizon and one domain, both named on the card: when the horizon selected below has had no gate run on it, this says so rather than showing another horizon's verdict.
| Criterion | Value | Rule | |
|---|---|---|---|
| Resolved forecasts | n = 892 | ≥ 150 | met |
| Brier skill over 50%-baseline | 0.108 | ≥ 0.100 | met |
| Beats always-50% (upper 95% bound of the Brier difference) | -0.013 | < 0 | met |
| Bias, calibration-in-the-large | +0.65 | |z| < 2 | met |
| Bias, conditional (Spiegelhalter) | +0.33 | |Z| < 2 | met |
Reliability
When we say 30%, does it happen about 30% of the time? Each point is one probability bin: what we said on the horizontal axis, what happened on the vertical. Perfect calibration sits on the diagonal. Point size shows how many forecasts are in the bin.
By question family
Brier score is the mean squared error of a probability: 0 is perfect. Two baselines: skill is against always saying 50% (Brier 0.25); vs base rate is against always saying the family's own observed frequency, a harder bar and the one that matters when a family's outcomes are lopsided. Families with fewer than 30 resolved forecasts are shown but not judged.
Maintenance forecasts are new. Their past scores are withheld, because their outcomes were counted from a third-party archive of GitHub events we have stopped trusting. We withheld them instead of publishing numbers we cannot stand behind.
Pastcast pool forecaster m5-v5, scheme forecast_referenced · up to 7 days (the gate's horizon)
| Family | n | Base rate | Mean forecast | Brier | Skill | vs base rate | Bias z |
|---|---|---|---|---|---|---|---|
| GDELT news tone (daily) | 864 | 47% | 46% | 0.223 | 0.109 | 0.106 | +0.56 |
| Hacker News points (daily)n < 30 | 28 | 57% | 52% | 0.232 | 0.071 | 0.052 | +0.60 |
| All families | 892 | 47% | 46% | 0.223 | 0.108 | 0.106 | +0.65 |
Live track forecasts issued in real time, all versions
| Family | n | Base rate | Mean forecast | Brier | Skill | vs base rate | Bias z |
|---|---|---|---|---|---|---|---|
| Prediction-market questionsn < 30 | 25 | 36% | 32% | 0.129 | 0.485 | 0.442 | +0.65 |
| Prediction-market questions (Manifold)n < 30 | 2 | 100% | 65% | 0.195 | 0.218 | — | +1.26 |
| All familiesn < 30 | 27 | 41% | 34% | 0.134 | 0.466 | 0.447 | +1.00 |
Against prediction markets
A separate test from the launch gate above, which is about our own question families. Here we forecast questions that a prediction market was already pricing, and score ourselves against the crowd's price at the same moment. The model was shown that price when it forecast.
This market comparison is from an earlier gate run, dated 2026-09-06, on forecaster m5-v5. The newest gate run has no market comparison.
Manifold Markets. On 315 resolved Manifold Markets questions, the crowd's Brier score at the same moment was 0.172; ours was 0.186. On questions a liquid market already prices, we do not beat the market.
Polymarket. On 16 resolved Polymarket questions, the crowd's Brier score at the same moment was 0.037; ours was 0.050. Too few to conclude anything; it is here because we said we would publish it either way.
manifold (earlier wording, retired). On 383 resolved manifold (earlier wording, retired) questions, the crowd's Brier score at the same moment was 0.155; ours was 0.199. On questions a liquid market already prices, we do not beat the market.
Our forecasts are for the questions markets don't cover. Brier score: mean squared error of a probability, 0 is perfect. Forecasts issued with the market price visible to the model, scored on the market's price at the forecast's own cutoff.
Every gate evaluation we published
Each as it was recorded, except the rows marked withheld, which keep their date and verdict and lose their statistics. Sample size restarts whenever the question scheme or the forecaster version changes, because pooling regimes was measured to flatter the numbers.
The maintenance families have no published score. Their outcomes were counted from a third-party archive of GitHub events we have since stopped trusting, so we withhold the scores rather than publish numbers we cannot stand behind.
- Oct 8, 2026PASSEDm5-v5 · n=892 · skill +0.11 · bias z +0.65 · Spiegelhalter +0.33
- Sep 6, 2026PASSEDm5-v5 · n=900 · skill +0.11 · bias z +0.50 · Spiegelhalter +0.15
- Sep 6, 2026WITHHELDm5-v5 · Withheld: scored against a GitHub event archive we have stopped trusting.
- Sep 6, 2026WITHHELDm5-v5 · Withheld: scored against a GitHub event archive we have stopped trusting.
- Sep 6, 2026PASSEDm5-v5 · n=900 · skill +0.11 · bias z +0.50 · Spiegelhalter +0.15
- Sep 6, 2026PASSEDm5-v5 · n=900 · skill +0.11 · bias z +0.50 · Spiegelhalter +0.15
- Sep 6, 2026WITHHELDm5-v5 · Withheld: scored against a GitHub event archive we have stopped trusting.
- Sep 6, 2026PASSEDm5-v5 · n=900 · skill +0.11 · bias z +0.50 · Spiegelhalter +0.15
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.54 · Spiegelhalter +0.10
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.33 · Spiegelhalter +0.49
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.33 · Spiegelhalter +0.49
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.33 · Spiegelhalter +0.49
- Sep 3, 2026NOT YETm5-v5 · n=396 · skill +0.30 · bias z -0.97 · Spiegelhalter -2.03
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.33 · Spiegelhalter +0.49
- Sep 3, 2026PASSEDm5-v5 · n=899 · skill +0.11 · bias z +0.33 · Spiegelhalter +0.49
- Sep 3, 2026PASSEDm5-v5 · n=324 · skill +0.29 · bias z -0.33 · Spiegelhalter -1.80
- Sep 3, 2026PASSEDm5-v5 · n=1223 · skill +0.16 · bias z +0.13 · Spiegelhalter -0.65
- Sep 2, 2026PASSEDm5-v5 · n=898 · skill +0.11 · bias z +0.30 · Spiegelhalter +0.50
- Sep 2, 2026PASSEDm5-v5 · n=394 · skill +0.11 · bias z -0.28 · Spiegelhalter -0.28
- Sep 2, 2026NOT YETm5-v5 · n=0
- Sep 2, 2026NOT YETm5-v3 · n=409 · skill -0.05 · bias z -5.31 · Spiegelhalter +4.74
- Sep 2, 2026NOT YETn=1783 · skill +0.03 · bias z -4.51 · Spiegelhalter +5.16
- Sep 1, 2026NOT YETn=1187 · skill +0.07 · bias z -5.32 · Spiegelhalter +0.22
- Sep 1, 2026NOT YETn=585 · skill +0.03 · bias z -5.08 · Spiegelhalter +2.07
- Aug 31, 2026NOT YETn=3041 · skill +0.10 · bias z +2.31 · Spiegelhalter +8.82
- Aug 31, 2026NOT YETn=2441 · skill +0.19 · bias z +11.30 · Spiegelhalter +2.34
How to read this
Pastcast. A forecast issued as if on a past date, with retrieval, thresholds and the model's own knowledge clamped to what was available then, then scored on the real outcome. It is how a young forecaster earns a sample size before it has lived long enough to accumulate one. Every pastcast is stored with the same append-only discipline as a live forecast.
Live track. Forecasts issued in real time. The log's head is anchored to a public repository once a day, so a forecast is checkable from outside from the first receipt after it was issued. Forecasts that resolved before 2 September 2026 pre-date that log, so their timestamps are ours alone and nobody outside could check them at the time. They are scored here on the same terms. Small today; it grows every week and will eventually replace the pastcast pool as the record that matters.
The criteria. At least 150 resolved forecasts; a Brier skill score of at least 0.10 over the always-50% baseline; that skill statistically significant under a block bootstrap (blocks by metric and date, so clustered questions do not count as independent evidence); and two bias tests, calibration-in-the-large and Spiegelhalter's conditional test, both inside |z| < 2.
Two baselines. Always-50% is the classical Brier baseline and the one the gate uses. Always-base-rate ("climatology") is stricter: it credits nothing for knowing how often a family resolves YES, only for telling questions apart. We compute it on the pool's own observed base rate, which flatters the baseline and understates our skill, the conservative direction. A family that beats 50% but not its base rate has learned the base rate and little else.
Horizon. Days from a forecast's as-of date (its issue date, for a live forecast) to resolution, bucketed. Skill at seven days says nothing about skill at ninety, so the selector above scopes every table on this page; buckets with no resolved forecasts are greyed. The page opens on the horizon the launch gate judges, because the archive also holds pastcasts at other horizons and pooling them was measured to flatter the headline; "all" pools them on purpose.
Against prediction markets. Questions that a market was already pricing when we forecast them, scored against the crowd's price at that same moment. The model sees that price in its context, so this measures whether we add anything to it. On liquid markets we do not, and we say so; the record is here because we said we would publish it either way. It is not part of the launch gate, which judges our own question families.
What this does not claim. Calibration on one family does not transfer to another, and a passing gate on backtests is not a passing gate on the live track. That is why both are shown, side by side, and why the live numbers carry a flag until they are large enough to mean something.