Home · Beta · Good Judgment Project Superforecasters
Beta · Human forecasters · cited; not independently recomputed
Good Judgment Project Superforecasters
- Source class
- Human forecasters
- Metric
- Mean Brier score
- Reported value
- ≈ 0.25 (vs. 0.37 control group)
- Measured
- 2014-12-31
Context
Across the IARPA Aggregative Contingent Estimation forecasting tournament (2011–2014); superforecasters were the top-2% of forecasters identified by year-1 accuracy and trained in probabilistic reasoning.
Citation
Mellers, B., Stone, E., Atanasov, P., Rohrbaugh, N., Metz, S. E., Ungar, L., Bishop, M. M., Horowitz, M., Merkle, E., & Tetlock, P. (2015). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Journal of Experimental Psychology: Applied, 21(1), 1–14.
https://doi.org/10.1037/xap0000040
What Phase 1 launch will add
Calibration Ledger has not independently recomputed the value above. Phase 1 launch (target Q3 2027, gated on prerequisites) will add for this source class:
- Independent recomputation from the original outcome data, under data-licensing agreement
- Time-windowed breakdown (rolling 3-month, 12-month, lifetime)
- Cross-domain calibration (does this source calibrate uniformly across topical verticals?)
- Append-only timestamp anchoring of every score so retroactive revisions are visible
- Per-source citation page with full Murphy decomposition (Reliability − Resolution + Uncertainty)
All other findings
- Metaculus community-prediction aggregate (Forecaster aggregator platform)
- Manifold Markets — platform calibration (Prediction market)
- GPT-4 (OpenAI) — pre-RLHF vs post-RLHF calibration (AI models)
- Sell-side equity analysts — earnings forecast accuracy (Analyst firms)
- Open Science Collaboration — psychological science replication rate (Scientific papers)
- Anthropic — Claude / language model self-knowledge (AI models)
- Camerer et al. — social science experiment replication (Nature/Science 2010-2015) (Scientific papers)
- Federal Reserve Survey of Professional Forecasters — GDP / inflation accuracy (Analyst firms)
- Hausfather et al. — climate model projections vs. observed warming (Scientific papers)
Related
- All beta findings — at-a-glance + JSON + BibTeX exports
- Methodology v1.1 — full Brier + Murphy + append-only framework
- Operator track record — methodology applied to Paulo de Vries’s own dated forecasts
- Source classes — what each of the 6 source classes will score at Phase 1
- Roadmap — milestone status + Q3 2027 launch gate + kill criterion
Last verified: 2026-04-28. Cited; Calibration Ledger has not independently recomputed this finding. Independent recomputation in Phase 1 (Q3 2027). Operator: Paulo de Vries. Contact: contact@calibrationledger.com.