What a full audit looks like

What we asked: if we run your code again, on a different computer, do we get the same number?

We ran it 20 times on two different machines, and again with a hidden setting changed that often reshuffles how programs order their work. The yearly return never moved by more than 0.31 points, and the hit rate was identical every time.

What we asked: was the list of markets built as it looked at the time, or from the ones still around today?

Small, and we say so. This check often produces the biggest finding. It doesn't here, because these are 12 large, busy markets rather than hundreds of small ones. We report the size either way: “we checked, and it was minor” is a result too, and it shows the large findings elsewhere were measured rather than guessed. (We have made this mistake ourselves: finding 10 on our front page.)

What we asked: where did the trading-fee figure come from, and is it charged correctly?

Exchanges charge less, and sometimes even pay a small rebate (the maker rebate), when your order waits in the order book to be filled. They charge more when you trade immediately at the current price.

The finding that surprised the client most.

What we asked: was every piece of data checked, and could one source quietly spoil the result?

Nothing was wrong with either of the two data sources. The problem is that there were two, and they weren't the same market.

Re-marking every call against the real traded price, at the right time: 7.1% of “right” calls became wrong and 1.4% of “wrong” calls became right. Net, the hit rate fell by 5.7 points.

The finding that explains the loss the client already knew about.

What we asked: how old is the market picture at the moment the signal makes its call?

The delay is worst exactly when the market moves fastest. On the moments that carried a third of the profit, the live system was typically looking at a picture 4.4 seconds old. In a fast market, the buyers the signal “sees” have often already left. It reads a wall that is no longer there as strength.

This is what happened in the client's known loss. We didn't go looking for it; it came out of the measurements. A buy call into a move that had already happened, because the picture behind it was seconds old.

One event or a pattern? We found every moment in the three years that looked like it: a very fast move, just after one side of the order book thinned out sharply.

A pattern, not an accident. The known loss is these 14 moments behaving normally, and it will happen again on the next one.

−16.1 points on its own (see why the order matters), and the biggest effect on the hit rate.

What we asked: how many versions were tried, and was this one chosen before or after seeing how it did?

This is not an accusation. Nobody did anything dishonest. It's the most common thing we find, and we find it in our own work (finding 7 on our front page). “Try the variations, keep the one that works” is good instinct everywhere except here. The problem is arithmetic: with 41 tries on the same data, the best one is mostly the luckiest one, and luck doesn't repeat.

How we measured it. The client's own history shows no settings were changed after 14 June 2025. So the last six months of 2025 are data no version was ever chosen on. We tested the shipped version, unchanged, on those months:

That last row is the finding. On data it wasn't chosen on, the shipped version is ordinary: 17th of 41, in the middle of its own family.

The biggest correction in this report.

What we asked: could every trade really have happened, at the price used, at that size?

First, where the profit came from. We sorted every 15-minute window by how wild the price moves were:

A third of the profit came from less than 1% of the time. Whatever the test assumed about those moments is very nearly the whole result.

We re-ran the test with one change: when the price had already moved away, the trade happens immediately at the price that actually existed. The client's own exchange data has both. Everything else was left alone. +26.1% became +8.2%.

Why this is a mistake, not a matter of opinion. Whether a waiting order gets filled depends on what happens after the call. The test answered that question with information it couldn't have had yet, and picked the favourable answer every time.

What we asked: is the headline one exact number, or a range? And is the range wide enough to include zero?

So we asked what the corrected number can support. We re-sampled the corrected results 10,000 times, a week at a time, to see how far the figure could move:

These three rows don't say the same thing:

• The return range includes zero. On this evidence, the strategy can't be told apart from one that makes nothing.
• The risk-adjusted range includes zero too. Same message.
• The hit rate range is nowhere near guessing (33.3%). The signal's knowledge is proven. Its profit is not.

That gap is the real state of this strategy, and a common one: a signal that knows something, but whose edge is smaller than the cost of acting on it in real trading.