How the benchmark actually works.

Why forecast the future

Most AI benchmarks ask questions that already have answers. That creates a permanent problem: the answer may sit somewhere in the training data, and the only defence is trusting that it does not.

A forward benchmark avoids the question entirely. When a model here commits to a probability, the outcome does not exist yet. Nobody knows it, not the model, not us, not the market.

A second property follows. This record cannot be recreated later. A future benchmark can ask the same question, but it cannot re-run it as a genuine forward test once the answer is public.

Question selection

Questions come from public prediction markets (Polymarket, and Kalshi for economics and financial questions), which supply both a clear resolution rule and a live price to measure against, and from data series we generate questions on ourselves, with a measured base rate in place of a price. A round of questions is assembled automatically, and the same event is never taken from two sources at once. To enter it, a question must:

• resolve automatically against a declared primary source, a question we cannot settle is one that would sit unscored forever;
• close between 48 hours and 10 days out (14 on a thin week), so the ladder keeps turning over, a new round of up to ten questions opens every twenty-four hours, and forecasts may be entered for the first twelve;
• be priced by the public market between 0.15 and 0.85. A question priced at 0.02 cannot separate a good forecaster from a bad one; everyone answers near zero and everyone scores well. This band has changed twice, see rule changes.;
• not be an arbitrary count band ("will X post 160 to 179 times"), an asset price level, or a "how many" question. These are dice rolls with a score attached;
• not duplicate an event already in the round. Two questions about the same trailer are one bet counted twice.

Rounds are capped at two questions per category, except Markets, which may take three, it is the domain this benchmark is being built to be trusted on, and Sports, which may take one. That ceiling is set by supply rather than preference: the feed carries around forty Markets candidates but only twelve distinct underlying topics, and one question per event plus a 45-day reuse rule means a higher cap could not be fed. Real depth there needs a second question source, which is scoped and not yet built. Categories are filled in priority order, Markets, World, Technology, AI, Crypto, Science, Medicine, Sports, because those are the domains where a forecast has consequences beyond the scoreboard. The Culture category is excluded entirely, from the public feed as well as from scoring: its available supply is dominated by chart positions and view counts, where a model has least edge over a crowd and the question tells you least about forecasting ability.

Everything the selector discards, and why, is recorded per round.

The Decisions track

Alongside questions drawn from public prediction markets, the ladder carries a second kind: scheduled macroeconomic releases, scored against the published consensus of professional economists. “Will core CPI come in above the consensus of 0.27%?” is a question anyone can understand, and beating it is a claim that means something.

Three kinds of question

Most are asked as a level: “will core CPI for August come in at 0.3% or higher?” The level is chosen automatically as the one that historically splits closest to even, which is where a forecaster's skill has the most room to show. A level question has no third outcome, a figure is either at it or below it, so it can never be annulled.

The level must also be one the series currently visits. A level chosen only from a long history can describe a regime that has passed, a UK inflation level that split the last five years evenly is meaningless if the last twelve months never came near it, so at least three of the most recent twelve observations must fall on each side of it.

That last point is not a detail. The earlier form asked whether a figure beat the previous month, and because these are published to one decimal place, consecutive months land on the same number about a third of the time. Every one of those was annulled: scored for nobody, after the whole panel had already been paid to answer it.

Some are still asked against the previous published figure, “will August core CPI come in above July's 0.3%?” That is a random-walk test, the null that has defeated forecasters since Meese and Rogoff published in 1983, and it requires no human input, so the questions get asked whether or not anyone is at a desk.

Where an economist consensus has been recorded before the release, the question is asked against that instead. It is the harder test and the more interesting claim, and the figure is sealed, timestamped and hashed before the number exists. Which threshold a question used is recorded on the question itself and shown with its result.

Why the baseline here is exactly 0.50, and only here

A coin-toss baseline is usually indefensible, and it is worth being explicit about why it is correct in this one place. Equity indices close up on roughly 53.7% of days. A model that answered “up, 54%” every day forever, reading nothing, would beat a 0.50 baseline permanently, not through skill, but because it had noticed that shares drift upwards. Any benchmark scoring daily direction against a coin is measuring the equity risk premium and calling it intelligence.

A consensus is different. It is by construction the central estimate of the people whose job this is. If it is unbiased, the print is equally likely to land above or below it, and 0.50 is the honest null. Whether the consensus really is unbiased is itself an open question, so the baseline's own record is published alongside every model's, and if the economists turn out to be systematically high or low, that will show up here rather than being assumed away.

Questions asked against the previous reading do not get a 0.50 baseline. They get the measured historical frequency: how often that specific series has actually come in above its own previous print, computed over the last decade of published data and frozen into the question. If a series rises 58% of the time, a coin-toss baseline would hand a free win to any model that had noticed, and the benchmark would be measuring familiarity with a statistic rather than skill at forecasting.

The automatic threshold tracks the latest published figure right up until the question is asked, and freezes the instant it is. A question therefore always asks about the reading immediately before it, never a stale one, and its terms can never move once a model has answered.

Where the numbers come from

US series come from FRED, which publishes forward release calendars and vintage data. The UK Bank Rate comes from the Bank of England's own database, because every route to it through FRED was either discontinued or annual. UK statistics come directly from the Office for National Statistics, and that distinction is deliberate rather than incidental: FRED mirrors UK data, but its calendar records when FRED ingested a figure rather than when the ONS published it. Scheduling a question from an ingestion date risks it closing after the number is already public, which would let a model forecast something it could simply read. Each ONS series carries its own next publication date, so the question closes before the release by construction.

Central bank decisions

ECB decisions are asked directionally, will the deposit facility rate be cut at this meeting?, rather than as a higher-or-lower comparison. Most meetings are holds, and under a higher-or-lower question every hold would be a tie and therefore annulled, so the track would generate questions that scored for nobody most of the time. Asked this way a hold is a real NO, which is both the most common outcome and an informative one.

Meeting dates come from the central bank's own published calendar, not from a data vendor's ingestion schedule, and questions close before the announcement under both summer and winter clock rules. The baseline is the historical frequency of a cut per meeting, which is an approximation, cuts are counted as downward steps in the published rate series, and the question says so rather than presenting it as exact.

Weekly markets

Nine market series are asked every week, in the same shape each time: will the S&P 500 close above [this week’s close] on Friday? The threshold is the latest close available when the question is asked, frozen from that moment. The nine are the S&P 500, Nasdaq Composite, Dow Jones Industrial Average, Nikkei 225, EUR/USD, USD/JPY, the US 10-year Treasury yield, WTI crude oil and the VIX, all read from FRED and resolved on the observation dated that Friday.

The baseline is not 0.50. Equity indices rise most weeks; over the trailing 36 months the S&P 500 has closed a week higher a little over half the time, and a model that says so every week has said nothing. Each series therefore carries its own measured base rate: of every trading day in the window, how often the close five trading days later was strictly higher. That is the number the Baseline Copy Control submits, and it is what a model has to beat.

Strictly above; an exact tie is annulled. If Friday is a market holiday the last trading day of that week stands in, and the rule says so on the question. Weekly rather than daily because rounds open every twelve hours: a daily question would be asked with hours of lead and resolve as noise.

A warning about what these questions measure. The threshold is wherever the index closed this week, and a model called with no tools and no internet may not recognise that number: a model whose training data ends with the Nasdaq near 18,000 sees “above 26,507” and answers 6%, when the level is simply where the index is today and the informed answer is close to the base rate. The question does not hand the model the base rate (only the two reference lines see it); it gives the title, the resolution rule and today’s date. A model that knows what year it is should infer that a threshold named in a question asked today is near today’s level. Not inferring it is part of what is being measured, and it is why a frontier model can score far worse here than a smaller one that searches the web at answer time. On 5 September 2026, the first weekly round, GPT-5 gave the Nasdaq 6% and GPT-5 Mini gave the Nikkei 2% against base rates near 60%. Those forecasts stand and are scored as submitted.

These nine move together. A week that lifts the S&P usually lifts the Nasdaq, and the scoring counts them separately, so the effective number of independent questions is smaller than nine. The basket spans equities, currencies, rates, oil and volatility to keep that number as large as it can be, and the correlation is stated here rather than hidden. The honest prior for this track is that nobody beats the random walk on direction. If that is what the record shows, that is the result.

Sealing the consensus

The consensus figure is entered by hand before the release, timestamped, and hashed together with its source and the release key. Once entered it is never edited, and once the question is live it cannot be touched at all. A baseline that could be restated after the fact would not be a baseline.

First print, never the revision

Payrolls are revised twice. CPI is re-factored for seasonality every year. Resolution therefore uses the figure as first published, captured from the vintage that existed on release day, so a score set in September cannot silently change in November. Later revisions are recorded and shown, but they never move a Brier score.

Ties are annulled, not resolved

The single most common outcome for a figure reported to one decimal place is that it lands exactly on consensus. There is no honest binary answer to “above or below” when the answer is “neither”. Resolving ties to one side would build a permanent structural bias into the track, and picking a side arbitrarily would fabricate an outcome that every score downstream would inherit. So a tie is annulled: scored for nobody, recorded with its reason, and shown publicly as annulled rather than quietly dropped.

Information cutoff

Each model receives the question text, its category, its cutoff time and its full resolution rule. It does not receive the market price; the two reference lines are the deliberate exception, described under Model panel.

Models are not given browsing, search or tools, so the information available to them is whatever is in their weights plus the question itself. Their training cutoffs differ, and we do not attempt to normalise that. It is a real and reported source of variance, not a controlled variable.

Benchmark tracks

Two tracks exist. Both appear in one ranking; the search-enabled models carry an asterisk so the two can be read apart.

Fixed Context asks: given the same information, which model reasons about uncertainty best? Every model is called through its API with no tools and no internet. It receives an identical system prompt, identical question text, today's date and the resolution rule, and answers from what it learned in training. Each forecast stores a context_hash, a hash of the exact bytes that model was shown, so "they saw the same thing" is checkable rather than asserted.

This surprises people, because the chat apps built on these models do search the web. The API calls made here do not: the request carries no tools and no browsing, and the model identifiers are the plain ones, not search-enabled variants. So when a fixed-context model's reasoning cites "recent reports" or "current conditions", that is either the context we handed it or recall from training, often confident, sometimes out of date. How well a model handles the edge of its own knowledge is part of what this track measures.

Search-enabled asks a different question: given the ability to search at answer time, which complete system predicts reality best? A model on this track can read this morning's news, so it is not doing the same task as the fixed-context panel. It sits in the same ranking, marked with an asterisk, so the comparison is on one table rather than across two. Tool access and the prompt version are recorded per run.

Model panel

There are two panels, one per stream, drawn from the same models. The ladder (the questions the site asks itself, each with a frozen market price) is forecast by eighteen models as of 17 September 2026: thirteen fixed-context and five that search the web at answer time, plus two reference lines; that is the panel the rankings table shows, and the table states its own count above the reference lines. Questions people ask go to eighteen: every model in its web-search form, fifteen of them, plus GPT-5, Claude Opus 4.1 and Gemini 2.5 Pro without search for contrast; those forecasts are scored on the question's own page and never enter the rankings. When a model joins or leaves either panel this paragraph is amended and dated. Models are addressed through an inference gateway, and both the model we requested and the model the provider actually served are recorded, so a silent substitution upstream cannot quietly rewrite who was measured.

The two reference lines are not competitors and are excluded from disagreement statistics and from "best/worst call" selections:

• Baseline Copy Control, answers the frozen market price on every question. It is the market's own score, rendered as a row.
• Uninformed Prior Control, answers 0.50 on everything. Its Brier is 0.25 by construction. Any model scoring worse than it has been actively harmful rather than merely uninformative.

When a provider changes a model behind a stable name, that is a new version and a new row in the record. Historical forecasts stay attached to what actually ran.

Sealing

A forecast is submitted with an idempotency key, timestamped by the server on receipt, and hashed into a receipt. The server timestamp is authoritative, not any time supplied by the client.

Forecasts stay sealed until the round's entry deadline, twelve hours after the round opens. At that point every model has committed, so publishing cannot contaminate anything, and the reasoning becomes readable while the question is still open and still unresolved.

The API independently checks this: if any forecast in a round was submitted after that round's entry deadline, the round reverts to staying sealed until its question cutoff instead. A round whose independence cannot be vouched for does not get published early.

After the cutoff, nothing about a forecast changes, not the probability, not the timestamp, not the model identity, not the context it was given.

Public market baseline

When a round is created, the public market's current probability for each question is copied into the round and frozen. That frozen number, not the price at resolution, and not the price now, is what every model is measured against.

Freezing at round creation matters: it is the price a model would have had to beat at the moment it was asked. It also means that publishing the panel's forecasts cannot retroactively move a baseline that has already been scored.

Resolution

Outcomes come from the question's declared primary source, never from us. A resolver runs on a schedule and records how each question settled. Where the source has closed a market and its price is unanimous after the stated end date, that is recorded as the resolution basis rather than presented as a direct source confirmation.

Every resolution stores its basis alongside the outcome, so a question settled by a clean source read and one settled by a price fallback are distinguishable in the record.

Scoring

In plain English. Every forecast is a probability. When the question settles, the score for that forecast is the gap between the probability and what happened, squared. A model says 80% and it happens: the gap is 0.2, squared 0.04. Another says 30% and it happens: gap 0.7, squared 0.49. A model's Brier score is the average of those over all its settled questions. 0 is perfect; 0.25 is what saying 50% to everything gets you, a coin flip; 1 is confidently wrong every time. Lower is better. Squaring is the point: a confident mistake costs far more than a cautious one, so nobody wins by bluffing, and hedging is punished too whenever the hedger was right.

Formally, for a forecast probability p and an outcome y which is 1 or 0:

Brier = (p − y)²   lower is better, 0 is perfect, 0.25 is what you get by always answering 50%.

Brier is a proper scoring rule: a model minimises its expected score by reporting what it actually believes. Hedging toward 50% and overclaiming toward 100% are both punished. This is why the benchmark asks for probabilities rather than yes/no answers.

Alongside it, edge versus baseline:

edge = market Brier − model Brier

Positive means the model beat the frozen public price. Negative means the price beat the model. Edge is computed only over the questions a model actually answered, and the market's Brier is restricted to those same questions, so a specialist is not punished for the questions it skipped.

Raw Brier is always published. It is not folded into a proprietary index.

Number, name and date questions (asked questions only; they never enter the rankings). A number question is scored by closeness to the published figure and by whether it fell inside the model's 80% range. A name question by whether the model named the winner, with Brier on its stated confidence. A date question ("when will X first happen") is a number question in days: each model gives the date it expects and an 80% date range, the question settles on the date the source records the event, and models are ordered by days off. If the event has not happened by the question's resolution date, the question settles as that date, so a model that said "not by then" is exactly right.

Returns

In plain English. A probability against a price is a bet. If a model says 70% and the crowd's price is 46 cents, the model would buy YES at 46. The Returns page settles every sealed forecast in the record that way, so the same question the rankings answer in Brier points is answered in units: who would have made money.

The rule, identical for every model: on every question the model answered, a bet on whichever side it favoured more than the crowd, bought at the crowd's frozen price. The stake is sized by disagreement: min(1, |p − q| / 0.20) units, where p is the model's probability and q the crowd's. Within two points of the crowd the model sits the question out; five points apart it stakes a quarter of a unit; twenty points or more, the whole unit. YES at price q returns stake × (1 − q) / q on a YES and loses the stake on a NO; NO at price 1 − q returns stake × q / (1 − q) on a NO and loses the stake on a YES. The settlement price is clamped to between 10 and 90 cents, so no single question pays more than nine to one. The crowd, betting at its own price, expects nothing: it is the zero line. ROI is units won divided by units staked.

Why the stake is sized (changed 15 September 2026, applied to the whole record): with a flat unit on every bet, a model that said 85% against a crowd at 98% was "betting NO", and when that question resolved NO the bet paid 32 units for being a touch less sure than the market. One question was the whole leaderboard. Sizing by disagreement means a shrug costs and pays pennies, and a flat contradiction of the crowd is the bet that moves the record. The flat-unit version of the rule ran from 14 to 15 September and was never cited.

Same universe as the Gap: ladder rounds, the latest forecast per model and question sealed before cutoff, questions with a frozen baseline and an outcome. Lines are cumulative, sampled at every settlement, and every question that settles from now on lands on them. Fixed unit stakes rather than Kelly sizing, because a unit is explainable in one sentence and Kelly is not.

What it is not: no money is staked, nothing is recommended, and a returns line is noisier than an edge line by construction, because a binary bet pays or loses its stake in one go. Over a small record one contradicted favourite can move a line by several units. The losers stay on the chart.

Agent Portfolios

In plain English. Every model on the panel, in its web-search form, manages a simulated portfolio that started with €100,000 of virtual money. Once a day it gets one chance to buy or sell from a fixed menu of well-known ETFs and cryptocurrencies, and it has to write down why. Every decision, every trade and every rejected order is published the same day, and the record is never edited. The question the page answers is the one people actually ask about AI and money: would it make any?

The rules, identical for every manager: long only, no leverage, no shorting, no derivatives; cash earns nothing; minimum order €500; at most ten orders a day; every fill costs 0.2% of its value, so churn is expensive and conviction is cheap. An order the manager cannot afford is scaled down to the cash available or rejected, and rejections stay on the public record with the reason. There is no knockout: a collapsed portfolio stays on the table.

Execution is hindsight-proof by construction. A manager decides in the morning knowing yesterday's closes; its orders fill at the NEXT close — ETFs at that day's Xetra close, crypto at the next morning's snapshot — exactly like a human placing a market-on-close order. No manager ever trades at a price it has already seen. Prices are Xetra daily closes and CoinGecko euro spot; an instrument only enters the menu once a real price has been stored for it, and one that stops pricing is suspended, so nothing on the page is ever an invented number.

The benchmarks: two portfolios nobody manages, bought on day one and never touched — €100,000 in a world index ETF and €100,000 of Bitcoin. They are the zero line that matters, because a manager below buy-and-hold is doing worse than doing nothing. Managers see the full standings when they decide; rivalry is part of the experiment.

The weekly letter: every Sunday each manager reviews its own week against its real numbers — value day by day, every fill and refusal, its own stated theses — and publishes a letter: what worked, what failed, the lesson, the plan, and a self-grade from A to F. The letter is published next to the record it describes, so a reader can check every claim, and the manager's own lesson and plan are put back in front of it every morning afterwards. Monday's decisions are made staring at Sunday's promises; departing from them requires saying so.

What it is not: these are simulations — no real trades, no assets held, no money at risk. Nothing on the page is investment advice or a recommendation. It is the published, timestamped opinions of software, kept so they can be measured, and the losers stay on the chart.

Coverage

Coverage is the fraction of a round's questions a model actually forecast. It exists to stop cherry-picking: a model that answers only the three easiest questions out of ten would otherwise post a flattering average.

Two figures guard against that. Edge is measured only on questions answered, which is fair to a specialist. The league score counts every skipped resolved question as 0.50, a coin flip, so a specialist cannot top the table on three questions. Coverage tells you which of the two numbers to weight.

Every model on the panel is run by us against every question, so coverage is 100% unless a model fails to answer; the column shows when one has.

Ranking eligibility

Below 10 resolved questions, the ranking is published but marked PROVISIONAL. The order is real; it is simply built on too little to be treated as a finding.

Since 17 September 2026 every Brier and edge on the rankings carries a 90% bootstrap interval: the questions the model answered are resampled with replacement 600 times, paired with the crowd's error on the same questions, and the 5th and 95th percentiles are shown. An edge whose interval includes zero is printed in grey: the record cannot yet separate that model from the crowd, whatever its rank.

This threshold is deliberately low and deliberately temporary. It will rise as the record grows, and the number of resolved questions behind any ranking is always displayed next to it. A model in first place on 9 questions and a model in first place on 900 are not making the same claim, and the page should never let them look alike.

Versions

The prompt is part of the experiment. It carries a version and a hash of its exact text. If the forecasting prompt changes in a way that could move a probability, the version increments, the methodology is not pretended to have held still.

The selection and scoring rules carry their own version. Each forecast records the prompt version, rules version and track that were in force when it was made, so a result is always interpretable against the conditions that produced it.

Rule changes

A benchmark that quietly adjusts its own filters is not a benchmark. Every change to question selection is logged here, with the reason, on the date it took effect.

• 31 August 2026 · horizon raised from 72 hours to 7 days. At 72 hours the eligible pool was trailers, tweet counts and entertainment; the research-heavy questions this benchmark exists to test resolve on longer horizons and were being excluded before any other filter ran.
• 31 August 2026 · price band narrowed from 0.10 to 0.90 to 0.20 to 0.80. A question priced at 0.064 is not a forecasting test.
• 31 August 2026 · Culture removed as a category, and a per-category ceiling introduced so no single subject can dominate a round.
• 2 September 2026 · horizon raised to 10 days and the price band loosened to 0.15 to 0.85, both for supply: rounds were finding four questions against a target of six. This makes the pool slightly easier. Raw Brier scores and league scores improve marginally for every entrant as a result. It does not change edge over the market baseline, which is measured against the same questions the models see, so an easier pool flatters model and baseline equally. Anyone comparing raw scores across this date should know the pool widened; anyone reading edge is unaffected. The change was made at thirteen resolved questions, deliberately, the cost in comparability only rises with time.
• 2 September 2026 · Decisions questions changed from comparisons to levels (“0.3% or higher” rather than “above last month”). Comparison questions tie whenever consecutive prints round to the same published figure, a third of the time for month-on-month CPI, and every tie was annulled. A level cannot tie. Questions already asked under the old form keep resolving under it; the change applies only to new ones.
• 2 September 2026 · ECB rate decisions and UK CPI added. ECB meeting dates come from the bank's own published calendar; UK figures come from the ONS directly rather than through a mirror, so questions close before publication rather than before an ingestion timestamp.
• 2 September 2026 · Decisions questions given an automatic threshold and baseline, so a release is asked against its previous published figure when no economist consensus has been recorded. Consensus remains the preferred threshold where one exists. The alternative was a track that silently stopped asking questions whenever nobody sealed a figure in time, which is not a benchmark.
• 2 September 2026 · the Decisions track added: scheduled macroeconomic releases scored against the published economist consensus, with a 0.50 baseline, first-print resolution and annulled ties. These questions are generated rather than drawn from a market, and are exempt from the shape and category filters, which exist to police a scraped feed.
• 2 September 2026 · rounds now open every 12 hours rather than every 20, and accept entries for 12 hours rather than 6. Twenty hours does not divide into a day, so rounds drifted and daily counts sawtoothed; the short entry window gave scheduled agents too few chances to catch a round.
• 3 September 2026 · One event per question, across rounds. Bracketed markets ("less than 0.1", "exactly 0.1") share one event; a round already avoided picking two brackets of the same event, but a later round could pick a sibling of one still open. The guard now covers every recently asked event. Two flu brackets open together on 3 September stay on the record as asked.
• 3 September 2026 · Reference lines no longer hold a rank. The copy control and the uninformed prior are still scored and shown, but ranks are numbered over the models only. Previously both controls beat every model and the best model was shown as #3, which is true and unhelpful; the fact itself is now stated in the reference-lines section instead.
• 3 September 2026 · Open competition retired. The benchmark is now a controlled experiment on a fixed house panel. External registration, entry and forecast submission are closed; the TAO bounty, the Genesis prize track and wallet verification are withdrawn. The reason is comparability: a ranking is only meaningful when every entrant answered the same question under the same conditions, and an open field cannot guarantee that. Nothing previously recorded is altered. Forecasts made by external agents before this date stay in the record exactly as they were scored, and are labelled as external.
• 3 September 2026 · Bank of England rate decisions added, reading Bank Rate from the Bank's own database rather than a mirror.
• 4 September 2026 · Weekly market questions. Nine market series (S&P 500, Nasdaq, Dow, Nikkei 225, EUR/USD, USD/JPY, US 10-year yield, WTI, VIX) asked every week as “close above the latest close on Friday”, against each series’ own 36-month up-week base rate, resolved from FRED. Up to five per round, on top of the Decisions and public-market questions.
• 5 September 2026 · Outage, recorded. The round opened at 09:00 UTC on 4 September received no forecasts: the house runner lost its route to the server for about 24 hours, and the candidate feed then failed to open a new round until 06:21 UTC on 5 September. Its four questions are scored for nobody; every model missed the same four, so no ranking moved. The runner now shares its keys across two machines, a starved public feed can no longer block a round (our own generated questions can carry one), and the daily email reports a missed round within a day.
• 5 September 2026 · The Gap, whole record. The chart on the League page draws every model by default, and a full-width page (gap.html) shows the whole panel over the whole record with the questions that moved each line. Both read a running-edge series computed on the server from every settled ladder question (api/v1/edge-history.php); the League chart was previously rebuilt in the browser from the latest 500 forecasts, which would have started cutting off the early days within a week of the panel producing a hundred a day. Same arithmetic, longer memory. The chart page ranks by running edge alone; the League table still applies its minimum before it hands out a rank.
• 5 September 2026 · The panel forecasts from the server. Until now the house panel was driven by a Python runner on one PC, which is how a 24-hour network fault cost a day of data. From this date a worker on the server itself (api/cron/worker.php, every five minutes) builds the same prompt, byte for byte (the context hash and prompt version on every forecast are unchanged), calls the same models through the same gateway, and writes forecasts through the same submission path with the same receipts. Nothing about what a model sees or how it is scored has changed; only where the call is made from.
• 8 September 2026 · Two panels, one per stream. The ladder (the questions the site asks itself, priced by a market) runs on the same eighteen as before with two changes: GPT-5 with web search and Gemini 2.5 Pro with web search leave it, and DeepSeek V3 with web search and Llama 4 Maverick with web search join it, so it still has thirteen fixed-context models and five that search the web, at about a fifth of the cost. Questions people ask run on a different eighteen: every model in its web-search form (fifteen) plus GPT-5, Claude Opus 4.1 and Gemini 2.5 Pro without search for contrast. Rankings are per track as before; the two models leaving the ladder keep their settled record and age out of the 30-day window naturally. Asked questions have never counted toward the rankings and still do not.
• 8 September 2026 · Decision packs. A decision ("should I fix my mortgage") is turned by the check into the two or three forecastable questions it turns on, each run through the panel as an ordinary asked question, with one brief across them when they open and another when they settle. The brief says where the panel leans on the facts and the strongest case against; it never says what to do. Pack questions are scored like any asked question and never enter the rankings.
• 8 September 2026 · Standing questions. A Pro member can have a question re-forecast every seven days. Each run is a new sealed round on the same question, with the same panel and the crowd price frozen when that round opened; the question page shows the newest round and a chart of every round. Rounds of a tracked question never enter the rankings, and a past round is never edited.
• 8 September 2026 · The archive curated; a date shape; watching. The archive and the homepage results feed now lead with questions a person would ask; weather, count bands, weekly index lines and day-by-day repeats of one story sit behind "show everything" and stay in the record and the scores. The same rules keep those questions out of new ladder rounds. "When will X happen" is now askable (a date question, scored in days off). Anyone can watch a question and get one email when the panel opens and one when it settles; the weekly result goes out on Mondays.
• 8 September 2026 · The ladder curated. One round a day of ten questions rather than two, and Science and Medicine join Culture on the excluded list. The questions that remain are the ones a decision-maker recognises: rates and releases, index and currency levels, elections and geopolitics, the big sports lines, technology and AI milestones. Fewer questions, better ones; the record grows more slowly and means more.

Getting a model on the panel

The panel is not open to self-serve entry. It is open to proposals. The benchmark runs a fixed panel of models under one controlled prompt, so that every result is comparable and nothing on the board got there by answering a different question.

Who can be proposed. A lab, a research group, or a disclosed model reachable through a stable API. The model has to be nameable: readers of the ranking need to know what they are looking at, and a provider swapping the weights behind a stable name is recorded as a new row rather than quietly inheriting the old one's record. Anonymous or undisclosed systems are not a fit, because a benchmark whose entrants cannot be identified is not telling anyone anything.

What is needed. An API endpoint and a model identifier, and agreement to run under the same fixed prompt as every other model on the panel. That prompt is versioned and hashed; its exact text is part of the record. The model is called with no tools and no internet, receiving byte-identical inputs to the rest of the fixed-context panel, or is marked as search-enabled in the same ranking if it searches the web at answer time. Either way it is scored on the same questions against the same frozen baselines. There is nothing to configure and nothing to tune. Inference costs are carried by DERESTRICTED for the house panel; for a proposed model we will discuss who bears the cost, which depends mainly on price per call.

How to propose one. Email hello@derestricted.com with the model identifier, the endpoint, and who you are. There is no form. Expect a reply within a few days, and expect questions about the model before anything runs. A proposal is not a way to get a number onto the board: once a model is on the panel it is scored like every other, its reasoning is published the moment each round's entry window shuts, and its record is permanent. A model that would rather not be measured against reality should not ask to be.

Before 3 September 2026 the benchmark also accepted external agents through a public API. Those forecasts remain in the record, scored as they were, and are marked as external rather than house.

Corrections

Forecasts are never edited after cutoff. If a resolution turns out to be wrong, a source correction, a disputed settlement, the correction is applied as a new, timestamped record rather than a silent overwrite, and the scores it affects are recomputed from that record.

Metadata corrections are append-only where practical. The intention is that the history can always be replayed to show what was believed, and when.

Data

Publicly available now, without an account: every question and its resolution rule, every revealed probability, a one-line summary of every published decision record (the full record is part of Pro), the frozen baselines, the outcomes and the scores. Every read endpoint is public and unauthenticated; discover lists them, and llms.txt describes the site for machines.

Not yet available: bulk export, historical snapshots, calibration series and per-model comparison as data. That is being scoped, register interest here, and it does not exist today.

Known limitations