cow-backtester: Backtest a CoW Protocol Solver Against Real Auctions

Every team building a CoW Protocol solver wants one question answered before it commits anything:

Against the last thousand real auctions, would my solver have beaten the ones that won? And on which pairs?

Until recently there was no honest way to answer it without going live. cow-backtester answers it offline, in an afternoon, with no keys, no onboarding and nothing touching production.

pip install cow-backtester

What is cow-backtester?

cow-backtester is an offline backtester, A/B harness and counterfactual scorecard for CoW Protocol solvers. You point it at your solver's /solve endpoint. It replays recent, real CoW auctions against it, then scores each of your solutions against the solutions that actually won on chain.

Point it at two endpoints, and it runs them head to head on identical auctions, so a routing or configuration change can be judged before it ships.

It is free, MIT licensed, published on PyPI, and built from public data only. The code is at github.com/KaiserSolver/cow-backtester.

Why it exists

A solver can produce valid solutions all day and still lose every auction it enters. Your test suite will not tell you that, because your test suite does not contain a field of competitors.

The two tools that already exist are both useful, and neither answers the question.

Shadow mode runs your solver against the live auction stream. The flow is real, but it needs onboarding, it only shows auctions from the moment you connect, and it tells you nothing about last month.

The Playground runs the protocol locally against a fork. It is the right place to check that your solver speaks the protocol correctly. But the orders are not real orders, and the competition is not there.

What was missing is the counterfactual: real order flow, with the real outcome already known, scored against what you would have done. That is what a backtest is, and it is how you would test a trading strategy. It is how you can now test a solver.

How it works

Four steps, all from public data.

1. Pull the archived auctions. The exact /solve requests CoW sent to solvers are kept in a public archive. Each archived file is a valid request body.
2. Send each one to your endpoint, exactly as the driver would, with the same deadline. Your solver does not know it is being replayed.
3. Rebuild what actually won from on-chain settlements, grouped per auction the way combinatorial auctions (CIP-67) define it, and cross-checked against the protocol's competition API where it is available.
4. Score both sides the same way. Your solution and the real winner are scored with the same function, the same limits and the same prices.

That last step is the whole game. It is easy to build something that makes your solver look good, and very easy to do it by accident. So cow-backtester validates every response before scoring it. A solution with an unknown order, a duplicated fill, an over-fill, or a fee padded to manufacture surplus is marked invalid with a reason, never quietly counted. Prices that look too good to be true are flagged.

Quick start

Python 3.10 or later. The only dependency is eth_abi.

pip install cow-backtester

# 1. Baseline: what the field actually captured, no solver needed
cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc>

# 2. Counterfactual: replay through your solver
cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc> \
    --solver-url http://localhost:8080 --json-out results.jsonl

# 3. A/B: two solvers, same auctions, head to head
cow-backtester --chain base --blocks 5000 --rpc-url <your-rpc> \
    --solver-url http://localhost:8080 --solver-name baseline \
    --solver-url http://localhost:8081 --solver-name candidate \
    --html-out ab.html

Your endpoint only needs the standard CoW solver API, POST /solve. If you want to see the output before wiring up an engine, the source checkout includes a mock solver that runs in about a minute.

What you get back

A scorecard on screen, and optionally one JSON row per auction and a single self-contained HTML report you can attach to a pull request.

• Coverage: settlements found, auctions scored, and every auction skipped, with the reason.
• Field surplus: what the winners captured, by order size.
• Winning submitters: who is actually winning.
• Counterfactual: how many auctions your solver answered, how many answers were valid, how many beat the winning set, your capture ratio, and your latency.
• Head to head: with two solvers, side by side on the same auctions.
• Top pairs: where the winners' surplus was, with yours beside it. Not just how much you lose, but where.

Every report prints the exact command that produced it, so anyone who receives one can regenerate it instead of taking your word for it.

Readiness: is your solver ready for production?

--readiness turns a run into a one-screen verdict on whether your endpoint is ready to face real auctions.

cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc> \
    --solver-url http://localhost:8080 --solver-name mine --readiness

It reports six things: answer rate, transport errors, deadline misses, latency against the real budget, solution validity, and how much of the winners' surplus you would have captured. Each is a pass, a warning or a fail. The verdict is READY when everything passes over at least 500 auctions, NOT READY when anything fails, and REVIEW in between.

Add --fail-on not-ready and it becomes a check in your CI pipeline.

The definitions and thresholds are the Solver Readiness Standard, published on the CoW forum as Measuring solver readiness before production. Our own solver's results are at kaisersolver-readiness.

Rank against the field, and estimate rewards

--compete shows where you would have ranked. For every auction, it pulls the historical competition record and inserts your result among the real bids: how often you would have come first, how often top three, and which solvers beat you, how often and by how much. --archive-dir keeps every record it fetches, so the dataset you build stays yours after the protocol's own history rolls off.

--reward-ev estimates consistency rewards under CIP-85, which on most CoW chains is where the money is rather than in winning. It also produces a consistency leaderboard for the whole field, which needs no solver at all.

Which chains it supports

Ten: Ethereum mainnet, Arbitrum, Base, Gnosis, Polygon, BNB Chain, Avalanche, Linea, Ink and Plasma.

It works on chains you are not live on yet, which turns expanding to a new chain from a commitment into a measurement. For readiness verdicts, the driver's time budget ships for Arbitrum, Base and BNB Chain. On the others, you pass the budget yourself, because the tool refuses to grade you against a budget it has guessed.

What a good score does not mean

It replays against today's liquidity. Your solver quotes against current chain state, not the historical block, so results are a strong signal rather than an exact replay. Recent, short windows are best.

It does not include private liquidity. The solvers at the top of the field trade from inventory you cannot reproduce offline.

The archive is recent. The public auction archive keeps about a month. --archive-bodies saves what you use, so your runs can be repeated later.

Beating the winning set is necessary, not sufficient. Real winner selection also applies fairness filters and nets out gas.

All of this is written into the README rather than buried, and any stretch of blocks the tool could not scan is reported, never dropped.

Who builds it

cow-backtester is built by William, who goes by kaisersnipes and builds Kaisersolver, the CoW Protocol solver Fiftysix operates. It started as internal tooling, and it was introduced on the CoW forum in August 2026 so other teams could use it.

Because it is built by a solver in the competition, every ambiguity resolves against us: a result only counts when public data proves it.

Other teams already use it. Another solver team ran it on Plasma, a chain we are not on, found a case it could not see, and contributed the fix. A grant application to take it to version 1.0, with a replay mode for engines that bid one order at a time, was submitted to CoW DAO in October.

Its companion, cow-certify, works on the other side: where cow-backtester asks what your solver would have done, cow-certify verifies what a settlement actually did, from public data. You can try it in your browser at kaisersolver.com.

Frequently asked questions

Is cow-backtester free? Yes. It is open source under the MIT licence.

Does it need my keys or touch production? No. It sends archived auctions to an endpoint you choose and reads public chain data. Nothing is signed or submitted.

Do I need to be onboarded to CoW Protocol? No. You can run it before onboarding, bonding or funding anything, which is the point.

Is a READY verdict a guarantee? No. It is a strong signal. Pair it with a shadow run before going to production.

How do I report a bug or a wrong score? Open an issue on GitHub. Corrections to the scoring are especially welcome. We would rather be told we are wrong than be quietly wrong.

The honest outcome

Most solvers are not profitable, and the top of the field is a long way up. The most useful thing cow-backtester can do for you may be to tell you early, cheaply and offline that the answer is no.

If the answer is yes, and what you are missing is the company side, that is what Fiftysix carries. Get in touch.