LLMs vs a System One Model: A Chess Marathon Benchmark

I built an automatic chess arena: pick two models, press Play, and the backend runs the whole game — move by move, with cost, latency and tokens tracked for every single move.

The players included Claude Opus 5.5, OpenAI models, Gemini, DeepSeek — and Jev, a "System One" model from TypeSafe AI. Every model sees exactly the same thing each move: the position, every piece with its type and square, the material score, and a numbered list of all legal moves.

Here's the thing: the most useful finding wasn't who won. It was how many "losses" were really harness bugs — and what it takes to make a benchmark honest.

Why chess, and why this matchup?

Chess is a clean test. Every move is checkable, games have a clear outcome, and cost compounds over dozens of moves. That makes it a good way to compare two very different kinds of model:

• General LLMs (Opus 5.5, GPT, Gemini, DeepSeek) reason in text, are flexible, and are billed for input and output tokens — including hidden thinking.
• System One models like Jev are built for fast, calibrated judgements. You send a state and ask questions (a choice, a score, a yes/no) and get back probabilities and a confidence. Jev is billed on input tokens only ($0.042 per million); output is free.

The question I wanted to answer: can a small, fast, cheap judgement model hold its own against frontier LLMs on a task that rewards good decisions at every step?

What I built

The stack:

• Frontend: Next.js with react-chessboard — a minimal white UI with a green-and-cream board.
• Backend: FastAPI, with LangGraph orchestrating the game loop and LangChain wrapping the LLMs.
• Tracing: Langfuse — one span per model call, one session per game.
• Tests: DeepEval with custom metrics (legal move, prompt-token budget, finds mate-in-one) plus pytest.
• Rules engine: python-chess, so no model can make an illegal move or fake the board.

The game loop is a small state machine in LangGraph:

prepare -> ask -> validate -> apply -> prepare ...
                     |-- invalid, retries left -> ask
                     '-- invalid, none left    -> fallback -> apply

Each move: build the state, ask the model, validate the reply, retry once if it's invalid, and if it's still invalid, fall back to a simple greedy move and flag it in the UI. Every attempt's cost is counted.

The UI shows the board, a panel per player (model dropdown, a Move / Cost / Latency table, running totals), a pulsing "playing" indicator for whoever is thinking, red-king highlighting for check and checkmate, and a Call Inspector that shows the exact data sent, the prompts, and the reply and token usage for any move.

The state, choices and score design

The biggest design decision was what to send each move. I wanted full information with minimal tokens. Each move, every model gets:

• State: every piece with its type and square (for example, knight on h3), whose turn it is, whether the king is in check, and the last six moves.
• Score: material totals for both sides, and the advantage for the side to move.
• Choices: every legal move as a numbered option, with cheap tactical flags — captures, check, checkmate, promotion, castling, and "destination square is attacked".
• The question: which move is best?

LLMs get a compact text version and reply with only an integer id. Jev gets the real System One structure: the state, a choice question (one option per legal move, keyed by UCI string with a plain-English description), a score question rating the position, and a noul question — "is my king in danger of being checkmated soon?" The score and noul answers are informational; the choice answer drives the move.

Keeping cost and tokens under control

Over a long game you make dozens of calls per model, so small savings add up:

• Compact notation — piece letter plus square, not prose.
• IDs, not move text — models answer with a number.
• Short history — only the last six plies; the state already contains the position.
• A static, cacheable system prompt — the same bytes every call, marked for prompt caching. (Providers only cache above a minimum prompt length, so a short prompt may show little caching.)
• Capped output — the answer is one number.
• Per-move cost accounting — fresh input + cached reads + cache writes + output, per model, per move, per game.
• A prompt-budget test in DeepEval, so prompts can't quietly bloat.

What broke — and why it matters for benchmarks

This is the part I think is most valuable.

1. Opus 5.5 kept "falling back." Mid-game, Opus 5.5's moves started showing a fallback tag and costs spiked to about $0.008 per move at ~7 seconds each. The raw replies gave it away: empty responses with exactly 64 output tokens. Opus 5.5 uses adaptive thinking that can't be switched off — and its hidden thinking consumed the entire 64-token cap, leaving no room for the visible answer. Easy openings needed little thinking, so they worked; complex middlegames didn't. The fix: set effort to low and raise the cap to 1,024 tokens. After that, a 12-move Opus game had 0 fallbacks and 0 invalid replies.

2. A parser that read the wrong number. In one position Opus wrote a short explanation ending in "1". My parser grabbed the first number in the text ("5 points of material") and picked the wrong move. I changed it to use the last valid number and added a test.

3. The same trap with OpenAI reasoning models. gpt-5-mini burned its whole budget on reasoning and returned empty text twice, so the harness fell back. I now set reasoning effort per model family and step it down automatically if a model rejects the setting.

The takeaway: if your harness is leaky, your benchmark measures the harness. Before the Opus fix, games where Jev "beat" Opus were partly Opus being forced into fallback moves. So I track fallbacks and invalid_replies for every player, and only trust results from runs where they're at or near zero.

Jev's side: speed and cost

Integration is one POST to the TypeSafe API with a bearer key — no chat prompt engineering. You describe the state and the options, and you get back a probability for every legal move plus a confidence value. In the opening, Jev's first-move choice was e4 with probability 0.74 and confidence 0.71. It's calibrated, not just a guess.

The efficiency gap (measured here, small samples — treat as indicative, not definitive):

• Jev — ~350 ms per move · ~$0.00006 per move
• Opus 5.5 (after the fix, 12 moves) — ~5.2 s per move · ~$0.006 per move

That's roughly 15× faster and 100× cheaper per move.

The results

The efficiency story above is the part I can stand behind right now — it's measured. The full win / draw / loss marathon (multiple games per matchup, alternating colors, with fallbacks and invalid replies driven to zero) is still being finalized, and I'll update this section with the complete tables once those runs are clean. I'd rather ship an honest methodology than a table of numbers I can't fully vouch for yet.

When I report it, I'll include: the exact matchups and model versions, games per matchup, how draws-by-move-limit are scored, and a per-player efficiency breakdown (total cost, cost/move, latency, tokens/move, fallback rate) — plus a couple of annotated games showing the decisive position and the exact Call Inspector view of what the model was shown.

Caveats (these are what make the benchmark credible)

• The harness favours choose-from-a-list. Models never produce a move from scratch, so this tests selection skill, not full chess ability.
• Tactical flags help everyone (captures, checks, attacked squares) — but they may help some models more than others.
• Small samples. Chess results are noisy; the number of games matters.
• Different billing models. Jev is input-only; LLM costs include output and thinking tokens. Some prices in my config are estimated — verify against current rate cards.
• Reasoning effort was set to low for the Claude 5.x models to fit the token limit. Higher effort could change both results and cost.
• Exclude early games — some used a simulated Jev stand-in before the real API was connected, and some had Opus falling back because of the bug.

How to reproduce

# backend
cd backend && python -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
# add keys to backend/.env: ANTHROPIC_API_KEY, OPENAI_API_KEY,
# TYPESAFEAI_API_KEY, optional LANGFUSE_*
.venv/bin/python -m uvicorn app.main:app --port 8000

# frontend (new terminal)
cd frontend && npm install && npm run dev

Pick two players, press Play, and watch the Call Inspector. Tests run with python -m pytest -q. Jev's docs are at docs.typesafe.ai.

Closing

Chess is a small window into a bigger question: when should you reach for a big reasoning model, and when is a fast, cheap, calibrated judgement model the better tool? On raw efficiency, the gap here wasn't close. But the lesson I keep coming back to is about measurement: log the fallbacks, count the invalid replies, and make sure you're benchmarking the models — not your own plumbing.

I've been digging into Jev and System One models more broadly — you can follow those deep dives and builds on the channel.