Five Qwen3.8-27B Agents Beat Fable 5 on LiveCodeBench Hard

What the five-agent Qwen3.8-27B workflow actually did

The setup is a Memdoor workflow with five roles—planner, two ideators, solver, and manager—plus a repair worker. Each agent starts from a fresh context and communicates only through files on disk. A shell script, not a model, decides success.

The workflow was rebuilt from the method described in a paper by Lee (2026), which claimed five Qwen3.8-27B models match Fable 5 on LiveCodeBench Hard. The Memdoor version used the authors' exact prompts, a 128K reply cap, and the same loop, then ran the first ten problems of the LiveCodeBench Hard list five times each.

The result: a mean of 9.8 out of 10, against 9.6 for the authors' own setup and 8.0 for Fable 5 alone. All 50 runs are published, including every request, reply, program, grader output, and cost. 49 of the 50 problems were solved.

The total bill was $35.79 for 15.3 million output tokens, or $38.77 including 16 attempts that were cut by a clock, stopped for a fix, or paused. A median run took 40 minutes of model time and cost $0.39.

Qwen3.8-27B five-agent workflow: benchmark and cost

The caveats: ten problems, possible contamination, and a different host

The test is small. Ten problems, five passes each, not the authors' 100. The contests date from March 29 to April 6, 2025, and the model may have seen them in training, so contamination cannot be ruled out.

There is also a hardware mismatch. The authors ran their own vLLM in FP8; the Memdoor runs went through OpenRouter, cycling through Modal, CoreWeave, Parasail, and Wafer until Modal slowed to 14 tokens per second, then sticking to FP8 hosts. Modal served 203 calls, CoreWeave 27, and Wafer 115.

Fable 5's column is a single call per problem from the authors' files, not a run in this setup. The loop sees only the public samples; the score is on hidden tests. Two failures in the Memdoor runs came from the same gaps in the authors' loop: a manager inventing a counterexample with a wrong expected answer, and a program passing every hidden test being overwritten because the manager did not say done.

Why this matters for open-weight coding models

If the result holds at larger scale, it undercuts the assumption that only frontier closed models can handle hard competitive programming. The gain comes from the setup, not the model alone.

The paper credits the ledger discipline—every agent prompt says 'fresh context, read the files, do not rely on chat'—for the improvement. That is a design choice any team can copy. The Memdoor workflow is public, and the steps are plain YAML files.

The cost is another signal. At roughly $0.39 per problem, a five-agent Qwen3.8-27B setup is cheap enough to run repeatedly. Fable 5, by contrast, is a closed model with no published per-problem price in this comparison.

The two failure modes are fixable without changing the method: check a manager's counterexample against a brute force on small inputs, and keep the last program that passed the samples. Neither fix is in these numbers.

What it means for you

The workflow is open source and runs on any model you pin, including local ones via Ollama or llama-server. For US, UK, and EU developers, that means you can test the five-agent setup on your own hardware or through OpenRouter without waiting for a vendor.

The published runs used OpenRouter, which bills in US dollars. There is no separate regional pricing for the workflow itself; you pay your model provider. The $35.79 figure is a reference, not a subscription.

The result is a useful data point for anyone building coding agents: the model matters, but the loop around it matters too. Whether five Qwen3.8-27B agents hold up on a full 100-problem set is the test that has not been run yet.