I tried to beat a bookmaker with AI

Before the grand final of The International 2026, the biggest annual Dota 2 tournament, I collected 48 forecasts for the series from twelve LLMs in four configurations. The models studied Dota statistics, head-to-head results and hero picks. In the fixed-context configurations, they received prepared data; as agents, each searched the web on its own. All 48 forecasts favored Team Vision, which was also the bookmaker’s favorite. It ended up losing to Team Spirit.

hy3, for example, gave Team Vision a 78% chance. The explanation sounded quite convincing: the team had beaten Team Spirit before, had a higher Elo rating and was looking stronger at the tournament. But at the end, the model still left room for this:

“Spirit’s best chance is a 3-2 upset.”

Spirit won 3–2.

So the outcome that actually happened was right there in the answer. Except the model had left Spirit with 22%, confidently favoring VISION. It’s a fairly familiar feeling when working with LLMs: you read the explanation, it all seems reasonable, the risks are accounted for, even the unexpected outcome gets a mention. But how accurately do LLMs estimate probabilities overall?

One final can’t tell us that. Events with a 22% chance should happen sometimes, or there’s something wrong with the probabilities. So I looked at the whole tournament, examined the bets separately, and checked what changed when I gave the models tools or showed them the bookmaker’s view.

How I set it up

I named the project Midas, after the Dota item that turns a creep into extra gold. A creep is an in-game unit that gives your hero gold and experience for leveling up when you kill it. The idea was simple: give several models data on the teams and see whether they could find an edge over the bookmaker’s odds. All bets were virtual.

I used the 14 series of the TI 2026 main event, played August 20–23. If you don’t follow Dota, a series consists of individual games, also called maps. Most are first to two wins; the grand final is first to three. Besides betting on the series winner, you can bet on a map handicap. For example, backing a team at −1.5 in a first-to-two series only pays out if it wins 2–0. A 2–1 victory means the bet loses.

Each of the twelve models answered in four configurations:

1. Fixed context without odds. The model receives prepared data and estimates probabilities.
2. Fixed context with odds. It receives the same data plus bookmaker odds, estimates probabilities and selects bets.
3. Agent without odds. It gathers data using tools and estimates probabilities. Searching for odds is forbidden.
4. Agent with odds. It gathers data, checks bookmaker odds, estimates probabilities and selects bets.

The prepared context included Elo, recent results, head-to-head history, drafts, team workload and the bracket. Agents could use OpenDota, Liquipedia, search and page-reading tools. The configurations without odds were there for a fairly obvious reason: if you show a model the bookmaker’s favorite up front, it’s hard to tell where its own estimate came from.

I asked for a probability distribution over exact scores: for example, 2–0, 2–1, 1–2 and 0–2. From that, you can derive a team’s win probability and evaluate handicaps. In the configurations with odds, the model also selected bets and stake sizes.

666

scored forecasts

14 series · 4 configurations

12

models

each in four configurations

8,012

tool calls

across 466 agent trajectories

$25.51

cost in usage logs

excludes separate search costs

The 666 are the selected forecasts made before matches. Tool calls and the $25.51 cover the whole saved experiment, including retries and errors. The cost is the sum of recorded usage, excluding separate search charges and the later Astra run.

And here’s a limitation that will become very noticeable: 666 forecasts still cover just 14 series outcomes. Twelve models betting on the same match several times doesn’t create any extra matches.

Models, run selection and comparison limits

The models were deepseek-v4-flash, deepseek-v4-pro, gemini-3.7-flash, glm-5.3, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra, grok-4.6, hy3, kimi-k3, minimax-3 and qwen3.8-max. Temperature was 1.0; reasoning used the highest setting available through each integration.

Of 1,206 runs, 883 were forecasts made before a series began. Selecting successful canonical runs left 670: the first successful run for each task counted unless it had been explicitly invalidated. The latest audit decisions excluded two more contaminated agent runs, and the timestamp check removed two late answers. That left 666, submitted a median 4.8 hours before the recorded start.

The two late answers surfaced while I was preparing this article: the post_start flag hadn’t fired when the live tracker lacked a start time. The script now compares the answer timestamp directly against the start, falling back to the schedule if it’s missing. Runs 907 and 910 are excluded.

Who estimated probabilities best?

Let me start with this: picking the winner isn’t enough to make a good bet. At decimal odds of 2.0, you need to win more than half the time to turn a profit. At 1.5, more than two-thirds. So “this team is more likely to win” doesn’t get you very far unless you know how much more likely.

minimax-3 is a good example. Across all four configurations, on the shared set of 52 tasks available for every model, it picked the winner correctly in 35 answers, more often than any other model. Yet it ranked ninth for probability quality.

I measured that quality using the Brier score: take the predicted win probability, subtract the actual outcome (1 for a win, 0 for a loss), square the difference, then average. Lower is better. If a team loses, a 70% forecast gets an error of 0.49, while a 51% forecast gets about 0.26. Both picked the wrong winner, but the first overestimated its chances by more.

Always saying “50–50” gives a Brier score of 0.25. The early saved bookmaker line scored 0.1964.

Top 12 model–configuration combinations out of 48, plus two bookmaker lines. Lower Brier is better. Δ vs market = model Brier − early-line Brier on the same series. For example, +0.0018 means the model’s Brier is higher by 0.0018. The closing line is available for 13 of 14 series.

#	Model and configuration	Brier ↓	Δ vs market	Series
—	Early lineFirst saved snapshot	0.1964	—	14
1	deepseek-v4-flashAgent · with odds	0.1983	+0.0018	14
2	deepseek-v4-proFixed context · with odds	0.1994	+0.0030	14
3	grok-4.6Agent · with odds	0.2031	+0.0067	14
4	hy3Fixed context · with odds	0.2050	+0.0086	14
5	gpt-5.6-lunaFixed context · with odds	0.2066	+0.0102	14
6	qwen3.8-maxFixed context · with odds	0.2083	+0.0119	14
7	deepseek-v4-proAgent · with odds	0.2100	+0.0082	12
8	glm-5.3Fixed context · no odds	0.2104	+0.0140	14
9	deepseek-v4-proFixed context · no odds	0.2121	+0.0157	14
10	glm-5.3Agent · with odds	0.2127	+0.0139	13
11	qwen3.8-maxFixed context · no odds	0.2133	+0.0168	14
12	gpt-5.6-solFixed context · with odds	0.2134	+0.0170	14
—	Closing lineLast saved snapshot	0.2155	—	13

All 48 model and configuration results

Each configuration uses all of its eligible forecasts. Its market comparison uses the same series.

#	Model and configuration	Brier ↓	Δ vs market	Series
—	Early lineFirst saved snapshot	0.1964	—	14
1	deepseek-v4-flashAgent · with odds	0.1983	+0.0018	14
2	deepseek-v4-proFixed context · with odds	0.1994	+0.0030	14
3	grok-4.6Agent · with odds	0.2031	+0.0067	14
4	hy3Fixed context · with odds	0.2050	+0.0086	14
5	gpt-5.6-lunaFixed context · with odds	0.2066	+0.0102	14
6	qwen3.8-maxFixed context · with odds	0.2083	+0.0119	14
7	deepseek-v4-proAgent · with odds	0.2100	+0.0082	12
8	glm-5.3Fixed context · no odds	0.2104	+0.0140	14
9	deepseek-v4-proFixed context · no odds	0.2121	+0.0157	14
10	glm-5.3Agent · with odds	0.2127	+0.0139	13
11	qwen3.8-maxFixed context · no odds	0.2133	+0.0168	14
12	gpt-5.6-solFixed context · with odds	0.2134	+0.0170	14
13	gemini-3.7-flashFixed context · no odds	0.2143	+0.0179	14
14	minimax-3Fixed context · no odds	0.2145	+0.0181	14
15	gemini-3.7-flashFixed context · with odds	0.2149	+0.0185	14
16	gpt-5.6-terraFixed context · with odds	0.2155	+0.0191	14
—	Closing lineLast saved snapshot	0.2155	—	13
17	qwen3.8-maxAgent · with odds	0.2156	+0.0192	14
18	kimi-k3Fixed context · no odds	0.2157	+0.0193	14
19	gpt-5.6-solAgent · with odds	0.2164	+0.0200	14
20	glm-5.3Fixed context · with odds	0.2166	+0.0202	14
21	gpt-5.6-lunaAgent · with odds	0.2172	+0.0207	14
22	hy3Fixed context · no odds	0.2175	+0.0211	14
23	kimi-k3Fixed context · with odds	0.2175	+0.0211	14
24	gemini-3.7-flashAgent · with odds	0.2185	+0.0221	14
25	gpt-5.6-terraFixed context · no odds	0.2187	+0.0222	14
26	gpt-5.6-terraAgent · with odds	0.2193	+0.0229	14
27	minimax-3Fixed context · with odds	0.2204	+0.0240	14
28	hy3Agent · with odds	0.2204	+0.0240	14
29	gpt-5.6-lunaFixed context · no odds	0.2212	+0.0248	14
30	grok-4.6Fixed context · with odds	0.2227	+0.0263	14
31	deepseek-v4-flashFixed context · no odds	0.2244	+0.0280	14
32	grok-4.6Fixed context · no odds	0.2281	+0.0317	14
33	gpt-5.6-solFixed context · no odds	0.2288	+0.0324	14
34	kimi-k3Agent · with odds	0.2289	+0.0299	13
35	deepseek-v4-flashFixed context · with odds	0.2301	+0.0337	14
36	minimax-3Agent · with odds	0.2359	+0.0394	14
37	gemini-3.7-flashAgent · no odds	0.2435	+0.0470	14
38	deepseek-v4-proAgent · no odds	0.2462	+0.0498	14
39	qwen3.8-maxAgent · no odds	0.2496	+0.0532	14
40	grok-4.6Agent · no odds	0.2502	+0.0538	14
41	minimax-3Agent · no odds	0.2515	+0.0551	14
42	glm-5.3Agent · no odds	0.2551	+0.0587	14
43	deepseek-v4-flashAgent · no odds	0.2618	+0.0654	14
44	gpt-5.6-terraAgent · no odds	0.2622	+0.0634	13
45	gpt-5.6-solAgent · no odds	0.2623	+0.0659	14
46	hy3Agent · no odds	0.2631	+0.0667	14
47	kimi-k3Agent · no odds	0.2682	+0.0717	14
48	gpt-5.6-lunaAgent · no odds	0.2723	+0.0653	13

Each row is one model–configuration combination. Its Brier score uses all eligible forecasts, with equal weight for each answer. Series counts vary because of missing and excluded forecasts, so the table shows how many went into each score. Market comparisons use the same series as the corresponding combination.

Brier score across 14 series. Each configuration averages individual model forecasts; the number on the right is the forecast count. The dashed line marks a constant 50% prediction.

None of the 48 model–configuration combinations beat the early line. Even the best scored 0.1983 against the market’s 0.1964 on the same series. And that’s with us picking the best result after the tournament, already knowing which configuration did well.

But the agent without odds interested me most. Its 0.2570 is worse than constant 50–50. So the model went looking for extra information, read things, spent tokens, and on average gave worse answers. What on earth did it find?

What the agents did on the web

Without odds, gathering information independently made all twelve models’ forecasts worse by log loss. That’s another probability metric, one that penalizes confident mistakes more heavily. I kept the same series in each paired comparison.

Log loss: lower is better. Both configurations are blind to odds; each model is compared on the same series. The number on the right is the increase in error.

View numbers

Independent research made all 12 models less accurate

Model	Fixed context	Agent
grok-4.6	0.646	0.694
gemini-3.7-flash	0.617	0.679
gpt-5.6-sol	0.645	0.718
deepseek-v4-pro	0.611	0.684
minimax-3	0.618	0.695
qwen3.8-max	0.614	0.694
deepseek-v4-flash	0.634	0.718
gpt-5.6-luna	0.648	0.740
hy3	0.627	0.725
glm-5.3	0.608	0.706
gpt-5.6-terra	0.613	0.718
kimi-k3	0.621	0.734

Tools did noticeably change the answers: the average probability shift between fixed context and the agent was about 8.8 percentage points. The median agent run took just over two minutes; the longest, almost fourteen.

Search used Firecrawl. Its results contained 7,716 links, of which 2,243 pointed to YouTube. Almost 29%. These were links in search results; they don’t establish that agents watched the videos. In 153 of 1,184 calls, Liquipedia returned a redirect such as #REDIRECT [[1w Team]]. The tool call succeeded, but instead of team data, it delivered instructions to go to another page.

One Iron Wing press release was read thirteen times across different runs. There were 220 page-reading requests for 126 distinct pages. That rather calls for a shared cache. Endless loops were rare, though: exact repeats within a single run accounted for just 39 of all 8,012 calls.

It also turned out that forbidding searches for odds doesn’t guarantee they won’t end up in the context. kimi-k3, for example, received a Kalshi price right in the search preview of an ordinary article: the favorite was trading at 52 cents. To check for contamination in agent runs without odds, deepseek-v4-flash acted as an LLM judge, inspecting trajectories for odds that had entered the context. Contaminated runs were excluded according to the latest audit decision. The audit could be wrong too: I reversed two of its decisions after checking them manually.

These findings don’t explain the whole decline on their own. The fixed context had already been assembled for the task, while agents had to find useful information among whatever the tools returned. The prompts differed too. But after this, I’m much less inclined to treat the number of tool calls as a measure of research depth.

As soon as I showed the agents odds, all twelve models improved. That made me curious about what was actually improving: their understanding of Dota, or their agreement with the bookmaker?

Across 162 pairs of answers from the same model on the same series, the average distance from the early line fell from 10.8 to 6.3 percentage points. In 116 of the 162 pairs, the answer with odds was closer to the market. That doesn’t prove copying: the prompts contained prices available at run time, while I’m comparing with the early ones. But the observed effect is quite specific: once given odds, models disagreed with the line less and made smaller errors.

I separately pulled out answers where a model without odds favored the opposite team to the market:

For the agent, 11 of those 49 answers were correct. On those same answers, the model’s Brier score was 0.3175 versus the market’s 0.2311. So this isn’t just about crossing the arbitrary 50% threshold: the probabilities themselves were worse too. But these are 49 answers across just eight series, not 49 independent attempts to catch the bookmaker getting it wrong.

They did sometimes pick an underdog that won. I didn’t see a systematic edge in those disagreements here.

How much the forecasts cost

The overall $25.51 doesn’t tell us much about the price of a good forecast: it includes retries, errors and answers between maps. So I calculated the cost of eligible answers separately for the top three combinations on the leaderboard. Each covered fourteen series.

Model and configuration	Brier ↓	For 14 answers
deepseek-v4-flash Agent · with odds	0.1983	$0.088
deepseek-v4-pro Fixed context · with odds	0.1994	$0.174
grok-4.6 Agent · with odds	0.2031	$0.703

Costs for every model in every configuration

Recorded cost in USD for eligible forecasts. Excludes retries, failures and separately billed search. Counts vary due to missing and excluded forecasts.

Model and configuration	Total	Per forecast
deepseek-v4-flashFixed context · no oddsEligible forecasts: 14	$0.033	$0.0023
deepseek-v4-flashFixed context · with oddsEligible forecasts: 14	$0.057	$0.0041
deepseek-v4-flashAgent · no oddsEligible forecasts: 14	$0.073	$0.0052
deepseek-v4-flashAgent · with oddsEligible forecasts: 14	$0.088	$0.0063
deepseek-v4-proFixed context · no oddsEligible forecasts: 14	$0.151	$0.0108
deepseek-v4-proFixed context · with oddsEligible forecasts: 14	$0.174	$0.0124
deepseek-v4-proAgent · no oddsEligible forecasts: 14	$0.294	$0.0210
deepseek-v4-proAgent · with oddsEligible forecasts: 12	$0.323	$0.0269
gemini-3.7-flashFixed context · no oddsEligible forecasts: 14	$0.074	$0.0053
gemini-3.7-flashFixed context · with oddsEligible forecasts: 14	$0.090	$0.0064
gemini-3.7-flashAgent · no oddsEligible forecasts: 14	$0.409	$0.0292
gemini-3.7-flashAgent · with oddsEligible forecasts: 14	$0.517	$0.0369
glm-5.3Fixed context · no oddsEligible forecasts: 14	$0.377	$0.0269
glm-5.3Fixed context · with oddsEligible forecasts: 14	$0.452	$0.0323
glm-5.3Agent · no oddsEligible forecasts: 14	$0.920	$0.0657
glm-5.3Agent · with oddsEligible forecasts: 13	$1.279	$0.0984
gpt-5.6-lunaFixed context · no oddsEligible forecasts: 14	$0.072	$0.0052
gpt-5.6-lunaFixed context · with oddsEligible forecasts: 14	$0.078	$0.0056
gpt-5.6-lunaAgent · no oddsEligible forecasts: 13	$0.152	$0.0117
gpt-5.6-lunaAgent · with oddsEligible forecasts: 14	$0.188	$0.0134
gpt-5.6-solFixed context · no oddsEligible forecasts: 14	$0.045	$0.0032
gpt-5.6-solFixed context · with oddsEligible forecasts: 14	$0.044	$0.0032
gpt-5.6-solAgent · no oddsEligible forecasts: 14	$0.263	$0.0188
gpt-5.6-solAgent · with oddsEligible forecasts: 14	$0.266	$0.0190
gpt-5.6-terraFixed context · no oddsEligible forecasts: 14	$0.068	$0.0048
gpt-5.6-terraFixed context · with oddsEligible forecasts: 14	$0.082	$0.0058
gpt-5.6-terraAgent · no oddsEligible forecasts: 13	$0.186	$0.0143
gpt-5.6-terraAgent · with oddsEligible forecasts: 14	$0.228	$0.0163
grok-4.6Fixed context · no oddsEligible forecasts: 14	$0.418	$0.0298
grok-4.6Fixed context · with oddsEligible forecasts: 14	$0.570	$0.0407
grok-4.6Agent · no oddsEligible forecasts: 14	$0.598	$0.0427
grok-4.6Agent · with oddsEligible forecasts: 14	$0.703	$0.0502
hy3Fixed context · no oddsEligible forecasts: 14	$0.042	$0.0030
hy3Fixed context · with oddsEligible forecasts: 14	$0.044	$0.0031
hy3Agent · no oddsEligible forecasts: 14	$0.065	$0.0046
hy3Agent · with oddsEligible forecasts: 14	$0.084	$0.0060
kimi-k3Fixed context · no oddsEligible forecasts: 14	$0.650	$0.0465
kimi-k3Fixed context · with oddsEligible forecasts: 14	$0.754	$0.0539
kimi-k3Agent · no oddsEligible forecasts: 14	$2.227	$0.1591
kimi-k3Agent · with oddsEligible forecasts: 13	$1.968	$0.1514
minimax-3Fixed context · no oddsEligible forecasts: 14	$0.065	$0.0046
minimax-3Fixed context · with oddsEligible forecasts: 14	$0.092	$0.0066
minimax-3Agent · no oddsEligible forecasts: 14	$0.153	$0.0109
minimax-3Agent · with oddsEligible forecasts: 14	$0.247	$0.0176
qwen3.8-maxFixed context · no oddsEligible forecasts: 14	$0.408	$0.0292
qwen3.8-maxFixed context · with oddsEligible forecasts: 14	$0.493	$0.0352
qwen3.8-maxAgent · no oddsEligible forecasts: 14	$1.057	$0.0755
qwen3.8-maxAgent · with oddsEligible forecasts: 14	$1.314	$0.0939

All fourteen answers from Flash as an agent with odds cost less than nine cents. Grok in the same configuration cost almost eight times as much and ranked slightly lower. The quality gap is small, and I’ve already picked the best results after the tournament, so I wouldn’t turn this into a general ranking of value for money. But in this experiment, first place among LLMs didn’t cost much.

The difference between configurations is noticeable too. Across 166 pairs with the same model and series, an agent without odds cost an average of 2.67 times as much as fixed context. Meanwhile, independent research made all twelve models’ forecasts worse by log loss, as the chart above shows. The extra calls and tokens didn’t pay off here.

These costs come from saved model-call logs. They exclude separate search charges, as well as the retries and failed attempts included in the overall $25.51.

Did twelve opinions help at all?

Back to Spirit. On the way to the title, it won five of its six series. Before the lower-bracket final against Yandex, the market gave it about 69%, while models on fixed context without odds averaged 47%. Only one of the twelve favored Spirit. Before the grand final, the market gave it 43%, the models 31%, and nobody was on Spirit’s side anymore.

Team Spirit's win probability before each series. Models are the mean of 12 fixed-context forecasts without odds. Spirit won five of these six series; its only loss was the first VISION encounter shown here.

View numbers

How they rated the eventual champion's chances

Opponent	Models	Market
Iron Wing	62.0%	49.3%
VISION	27.6%	35.3%
Liquid	48.9%	59.3%
BoomBoys	60.4%	64.7%
Yandex	46.6%	68.5%
VISION · final	31.0%	42.9%

In the grand final, all twelve models in all four configurations favored VISION: those are the 48 forecasts from the start of the article. But Spirit won. In Liquid versus Yandex, all 48 forecasts also favored the team that eventually lost. The bookmaker favored the same team as the models in both matches. The models, though, gave that team an even higher win probability on average.

The explanations kept coming back to Elo, head-to-head results, form and fatigue from the lower-bracket run. When one model writes that, you take it as one assessment. When twelve different models say roughly the same thing, each in its own words, it starts to feel like the hypothesis has been fairly well checked. Yet they share the underlying data, and all those explanations may rest on the same mistaken assumption.

With fixed context and no odds, the mean pairwise correlation between probabilities was 0.943. A high degree of similarity is expected in itself: they’re looking at the same matches. The more useful check is what averaging buys you. The average forecast of all twelve models had a Brier score of 0.2169, compared with 0.2183 for the average individual model. An improvement of only about 0.6%.

I also tried blending that average forecast with the market’s, 50/50. The result was 0.2038, versus 0.1964 for the line alone. I also tried model weights of 25% and 75%; both blends were worse than the market. This is a check on fourteen series that have already been played, but even this simple approach didn’t reveal anything useful to add to the price.

If one model overestimates a team’s chances and another underestimates them, averaging can bring the forecast closer to reality. But if they all overrate the same team, the average keeps that error. On this tournament, averaging twelve answers brought only a small improvement, so I wouldn’t treat their agreement as a reason to trust the forecast more.

Where the virtual money went

In the configurations with odds, models selected 510 settled bets on 45 distinct market selections. They staked a total of 508 units and lost 143. Return on total stake was −28.2%. With an equal amount on every bet, it would have been −36.3%. No model finished in profit.

Profit / total recorded stake, combining both configurations with odds. The number beside ROI is the bet count. Different models frequently bet on the same outcome.

Looking at that chart, it’s easy to assume the models bet about equally badly throughout the tournament. So I broke the profit down by series, and found something much more interesting:

The five series Spirit won accounted for −140.9 units out of the overall −143.0. The other nine combined came to just −2.1. Those totals already include winning bets, so it would be wrong to call the first row a “share of all losses.”

What happened before the last two matches is particularly telling. All 39 model bets on Yandex versus Spirit lost, costing 38.05 units. In the grand final, they staked another 66.2 and lost 64.4. Yet Spirit had been the bookmaker’s favorite against Yandex. It won, as the market expected, but the models had chosen bets that lost with that result.

We can’t put it all down to the last match either: remove the grand final and the return is still -17.8%. Remove the unanimous miss on Liquid versus Yandex as well, and it’s -14.7%.

Of course, I singled Spirit out after the tournament, already knowing the champion. That doesn’t give us a rule for the next TI. But it does make clear why the bet count is so misleading: a few shared errors were repeated over and over by different models in different configurations. 510 rows in a database look more substantial than the fourteen matches underneath them.

There was a more basic problem too. Models chose the −1.5 handicap 112 times, and every one of those bets lost. Remove the duplicates and there are just ten distinct outcomes across ten series. In four, the chosen team actually won, 2–1. Picking the winner correctly turned out to be perfectly compatible with losing a bet on it. That handicap cost 86.3 units.

Splitting bets at decimal odds of 2.0 is revealing too:

About half the total stake went into each group. The first earned 7.5 units; the second lost 150.5. The higher-odds selections include handicaps on a favorite winning decisively, so this isn’t just about betting on underdogs. And a “bet only below 2.0” rule would, for now, amount to picking a convenient cutoff after seeing the result.

By the models’ own estimates, the mean expected return on their chosen bets was about +16.3%. In other words, they thought they’d found fairly generous prices. It’s just that calculating expected return requires knowing the probability of the event, and that’s exactly where the trouble was.

What happened to bets in the configurations without odds

There, code selected bets after the model answered: it took the model’s probability and the historical price, calculated p × odds − 1, and placed an equal stake whenever the result exceeded 0.06. That produced 226 bets with an ROI of −30.8%.

This check still depends on the model’s probabilities. Also, the two portfolios differ in selections and stake sizes, so the difference in ROI doesn’t isolate the value of using an LLM to choose bets.

Did choosing stake sizes help?

Models chose stake sizes only in the two configurations with odds: fixed context and independent research. They could stake between 0.5 and 2 units. So here I’m comparing only bets from those two configurations: I kept the same selections at the same odds and recalculated the result with an equal stake on each.

Equal stakes produced a return of -36.3%, compared with -28.2% for the model-chosen amounts. All 12 models improved when compared against their own set of bets. Below are the results for every model, combining both configurations with odds, just as in the returns chart:

Model	Equal stakes	Model-chosen stakes
deepseek-v4-flash	-37.5%	-32.5%
deepseek-v4-pro	-30.2%	-19.6%
gemini-3.7-flash	-39.6%	-33.2%
glm-5.3	-38.9%	-34.5%
gpt-5.6-luna	-35.1%	-27.3%
gpt-5.6-sol	-41.7%	-33.7%
gpt-5.6-terra	-44.6%	-32.2%
grok-4.6	-36.9%	-31.0%
hy3	-36.2%	-25.2%
kimi-k3	-42.2%	-31.4%
minimax-3	-25.2%	-21.3%
qwen3.8-max	-26.0%	-18.4%

So allocating money between the chosen bets reduced the losses. It wasn’t enough to turn a profit, but putting an equal amount on each outcome would have made for an even sadder result.

I wouldn’t call this proven risk-management ability yet. Models put less money on outcomes with odds of 2.0 or higher on average, and that’s where most of the losses were. They also often chose the same outcomes, so they could all have benefited from reducing stakes on the same bad bets. I’d be interested in repeating this comparison at the next tournament.

The explanations looked better

After the final, I searched the other answers for upset. It appeared in 71 explanations. In 63 of them, the model still favored the market’s pick. So models were quite happy to allow for an unexpected win in the text, without changing their main assessment.

Momentum was more amusing. It appeared in 84 explanations; the winner was picked correctly in 52.4% of those, versus 64.8% without it. I could have banned models from writing about momentum and congratulated myself on the improved prompt, but first I checked which matches they were describing that way.

Imagine one model uses momentum often and makes worse forecasts overall, while another rarely uses it and gets things right more often. Mix their answers together, and momentum will seem to be associated with errors. Though we may simply have picked up a difference between the two models. The same thing can happen if a word appears more often in forecasts for difficult matches, or in a configuration that performs worse overall.

So I checked whether the association between a word and the error remained after adjusting for which model answered, which match it predicted and which configuration it used. For momentum, the correlation fell from +0.103 to +0.031, close to zero. For experience, the sign flipped from +0.084 to −0.056, while the association between mentioning Elo and making smaller errors almost disappeared. The original numbers largely reflected differences between models, matches and configurations. The presence of a word turned out to be a much less useful clue to forecast quality.

So I didn’t end up with a dictionary of bad forecasts. But I do think it’s worth reconsidering the habit of trusting percentages because the right analytical words surround them. Mentioning the risk of an upset is fairly easy; getting its probability right is harder.

Words, correlations and how they were calculated

Association between a word and Brier error, measured as a correlation from −1 to +1. Positive means the word tends to accompany larger errors; negative means smaller errors. Near zero means a weak association.

Word	Answers	Before adjustment	After adjustment
Elo	312	−0.054	−0.005
draft	205	+0.026	+0.088
momentum	84	+0.103	+0.031
experience	42	+0.084	−0.056
upset	71	−0.058	+0.089

“After adjustment” is the association that remains after removing average differences between matches, models and configurations. For example, it accounts for a model both using a word more often and making larger errors overall. I examined 19 words and word groups; these five examples each appear in at least 40 answers.

I used the visible reasoning field from 666 eligible answers, excluding hidden reasoning tokens and search trajectories. I checked 19 words or word groups; for example, draft includes drafts and drafting. The search ignores case and respects word boundaries: form inside performance doesn’t count.

For the calculation, each answer gets two numbers: 1 if the word appears and 0 if it doesn’t; the second number is the Brier error. The “Before adjustment” column shows the association between them. For “After adjustment,” the script first removes average differences related to the match, model and configuration from both measures, then calculates the association between the remaining deviations. This checks whether the word tells us anything about the error beyond what those three factors already explain. The public-data script independently verifies the calculation using a different method.

What about Astra?

After the tournament, I ran gpt-6-astra on the same fourteen series: fixed context, with and without odds, reasoning=max, temperature 1.0. 28 answers in total.

I took the saved contexts from the original gpt-5.6-sol runs and reconstructed the odds for the same cutoffs using the original price-selection rule. The script checks hashes and snapshots. That makes Sol a useful direct comparison here:

Without odds, Astra narrowly beat Sol; with odds, it did worse. The market was more accurate than either configuration. In the grand final, Astra gave VISION 69% both times, leaving Spirit 31%. The new model repeated the same pattern of underestimating the eventual champion.

Astra’s return on virtual bets was -34.1% without odds in the prompt and -33.5% with odds. Code selected the bets in the first case; the model selected them in the second.

But Astra answered after the tournament, and the results could have been in its training data. So even with the same saved context, there’s no guarantee it was estimating chances without knowing the match outcomes. Comparing it with models that answered before the matches may not be entirely fair. I’m keeping this run as a separate retrospective check and leaving it out of the main leaderboard.

How stable is first place?

The main leaderboard was won by deepseek-v4-flash as an agent with odds: Brier 0.1983. Next came deepseek-v4-pro on fixed context with odds: 0.1994. The gap is just 0.0011. I wanted to understand how much that order depended on the particular matches.

To do that, I removed one series at a time and rebuilt the ranking of all 48 model–configuration combinations. First I calculated it without the first match, then restored the first and removed the second, and so on through all fourteen. The saved forecasts and results stayed unchanged. Each time, I excluded every answer about one match and calculated Brier on each combination’s remaining eligible answers, just as in the table above.

Flash as an agent with odds kept first place in only 4 of 14 checks. In the others, Pro on fixed context or Grok as an agent took the lead, both with odds. Removing a single match was enough to change the winner. The combinations still have slightly different sets of answers: this check doesn’t restore missing forecasts.

These numbers show how much the ranking depends on the set of matches already played. Pro finishing first seven times doesn’t mean it has a 50% chance of being the best at the next tournament. To test that, we need new matches and forecasts made before they start.

In the next experiment, I’d use plenty of models again and test each in every configuration. Models keep improving, and I’d implement the configurations differently too, given what I’ve learned here. So I see the current ranking as the result of a particular experiment, rather than a shortlist for the next one. Flash was the best model here, and the early bookmaker line was more accurate than Flash. What happens with new models and reworked configurations remains to be tested.

I think I got the name Midas right. The mythical Midas nearly starved because even his food turned to gold. If I’d tried to make a living off LLM forecasts, I wouldn’t have needed a golden touch to end up in the same place.