MC PvP Bench
I make LLMs fight each other in Minecraft and calculate their Elo.
The game runs 20× slower, but models respond in real time, so speed matters.
Elo vs. cost
Enable JavaScript for the chart, or use the leaderboard below.
Hover for details. Click or tap to pin.
Cost is per active real-time minute. The frontier uses Elo estimates, not confidence intervals.
More columns →
| # | 95% interval | $ / min | median | W / L / D | ||
|---|---|---|---|---|---|---|
| 1 | OpenAI Frontier | 1,5531,391–1,737 | $0.163 | 6.44s | 73 | 66 / 7 / 0 |
| 2 | Anthropic Frontier | 1,4021,260–1,556 | $0.155 | 6.66s | 62 | 44 / 16 / 2 |
| 3 | Google Frontier | 1,3101,146–1,482 | $0.072 | 7.71s | 29 | 12 / 11 / 6 |
| 4 | OpenAI Frontier | 1,2281,113–1,352 | $0.017 | 5.71s | 89 | 52 / 30 / 7 |
| 5 | 1,2041,091–1,323 | $0.070 | 5.50s | 91 | 65 / 20 / 6 | |
| 6 | Anthropic | 1,1761,030–1,329 | $0.207 | 7.39s | 50 | 28 / 21 / 1 |
| 7 | DeepSeek Frontier | 1,044969–1,120 | $0.0040 | 3.80s | 140 | 81 / 38 / 21 |
| 8 | DeepSeek | 1,000Fixed anchor | $0.0042 | 3.66s | 184 | 101 / 66 / 17 |
| 9 | Anthropic | 981834–1,128 | $0.154 | 4.76s | 31 | 14 / 17 / 0 |
| 10 | xAI | 966822–1,111 | $0.136 | 5.30s | 40 | 18 / 20 / 2 |
| 11 | Anthropic | 931788–1,072 | $0.550 | 6.31s | 43 | 19 / 22 / 2 |
| 12 | Meta | 868720–1,013 | $0.048 | 9.36s | 36 | 10 / 19 / 7 |
| 13 | Meta Frontier | 858774–941 | $0.0013 | 8.24s | 150 | 57 / 60 / 33 |
| 14 | Alibaba | 813704–919 | $0.0048 | 7.29s | 70 | 17 / 18 / 35 |
| 15 | OpenAI | 811712–908 | $0.017 | 5.69s | 91 | 25 / 46 / 20 |
| 16 | Moonshot AI | 788631–938 | $0.094 | 7.16s | 36 | 10 / 22 / 4 |
| 17 | Mistral | 785631–932 | $0.057 | 13.71s | 30 | 3 / 10 / 17 |
| 18 | Xiaomi | 783628–931 | $0.014 | 6.47s | 30 | 6 / 13 / 11 |
| 19 | OpenAI | 777685–866 | $0.0026 | 5.66s | 113 | 25 / 44 / 44 |
| 20 | Z.ai | 764646–879 | $0.0085 | 4.78s | 62 | 12 / 30 / 20 |
| 21 | Anthropic | 752627–873 | $0.0064 | 8.16s | 52 | 6 / 22 / 24 |
| 22 | OpenAI | 745642–845 | $0.029 | 5.88s | 92 | 20 / 49 / 23 |
| 23 | Alibaba | 702556–838 | $0.038 | 10.21s | 41 | 3 / 22 / 16 |
| 24 | OpenAI Frontier | 689578–795 | $0.0009 | 5.72s | 76 | 1 / 34 / 41 |
| 25 | Xiaomi | 676511–826 | $0.016 | 8.36s | 28 | 2 / 14 / 12 |
| 26 | xAI | 517257–718 | $0.219 | 3.24s | 31 | 1 / 27 / 3 |
No matching models. Try another search or reset the filters.
How it works
Duels
Models play from screenshots using mouse and keyboard, through the Pi harness. Each player gets iron armor, a sword, and a shield.
A kill wins. At timeout, more HP wins; equal HP is a draw.
Elo
Ratings are fitted jointly from match records using Bradley–Terry with a Jeffreys penalty. Draws count as half a win. DeepSeek-V4.1-Flash is fixed at 1,000.
The approximate 95% intervals show uncertainty in each rating, not its rank. The fixed anchor has no estimated interval. Opponents and match counts vary between models.
Cost and speed
Cost is recorded API charges divided by active real-time minutes.
Action time is the median from request dispatch to a successful tool call. It includes provider latency and generation, but not the mouse/keyboard action itself.
Pareto frontier
A model is on the frontier if no other model is at least as good for less money, or better for the same money. This uses rounded Elo, without accounting for uncertainty. Filtering doesn’t change the frontier.