Know the number is real before the money goes in.

A data migration quietly froze 65 files — they simply stopped updating. Because the filter counted rows instead of reading dates, those frozen assets stayed eligible for about 26 days. The strategy's holdings changed completely from one week to the next, which read like the strategy responding to the market. It was the data breaking underneath it.

Then a second freeze was found inside the restarted clock: three more assets stopped updating on the same day, and one was bought into a portfolio a full week after it had already gone dead.

Underneath that sat the worst of it. One position the strategy held had not traded at all — zero out of 1,441 minutes with a single trade in them. A full-size position, booked at a profit, describing a trade nobody on earth could have made.

The price on your screen is the price for a tiny trade. Put a real order through and you eat into worse and worse prices as you go. On this basket the screen price suggested a cost of 0.004%–0.033%, while getting in and back out actually cost 0.28%–0.78%. A measurement taken at the screen price cannot see that, at all.

So we went back and audited every cost assumption in the research. It found this:

A standing study had assumed the worst possible round trip on that exchange was 0.31%. Filling real orders, 40 of 122 assets cost more than that at $10,000 apiece — the worst of them 0.92%. And those were the easiest-to-trade assets available.

That study's own follow-up note had already tried to fix it with a textbook formula — and overshot by about 1.8× the other way, exactly as the note itself suspected it might. Measured properly, the truth sat between the two. The original conclusion survived at the size it was written for, and failed at ten times that size.

The strategy trades when certain market events happen, and can hold only so many positions at once. On the worst days far more signals arrive at once than there are slots — the busiest single hour had 194. Something has to choose which ones get in, and that choice was being made by the arbitrary order the sorting step happened to leave the data in. 72% of all events shared their hour with at least one other.

Two of our own programs, same data, same settings, produced annual returns 1.4 percentage points apart. Both were running correctly. So we re-ran it 300 times, shuffling only within each tied hour: the answer typically moved by 0.76 points, and the best ordering beat the worst by 4.2.

Worse than the number itself: a cross-check had already been run and had passed. One study cited the other's agreement as proof its simulator reproduced the original. The two figures were 1.4 points apart, and the gap was hidden inside a rounding in the summary. A verification step read as independent confirmation when it was two programs sharing one defect.

The usual way to measure how sure you can be is to shuffle the daily results thousands of times and see how widely the outcomes spread. That assumes each day is independent of the last. This rule held the same position for days at a time, so consecutive days moved together — and treating them as independent counts the same evidence over and over, making the range far too narrow.

Reshuffling in blocks of ten days instead, which keeps that clumping intact, every one of the nine versions had a range that included zero. Splitting the history in half pointed the same way: performance fell in every version, one from a risk-adjusted score of 0.88 to 0.09.

It is worth saying what the problem was not. Trading costs barely mattered: the score went 0.98 before fees, 0.95 at normal fees, 0.85 at four times normal fees. This failed on uncertainty, not on costs — which is exactly the wrong place to have been looking.

The strategy only fires during rare market-wide episodes, so its trades arrive in a small number of clusters — and every trade inside a cluster is the same event happening to different assets. Counting distinct weeks instead of trades, the strongest result in the study rests on six weeks in six and a half years. About one independent observation a year.

The trap that sets is not theoretical. The obvious improvement — sit out the small isolated events, save room for the broad ones — tested better on data held back for the purpose (0.88 → 1.46) and worse on the data it was built with (1.16 → 0.70). A rule that improves on unseen data is the opposite of the usual warning sign and reads like a discovery. Its entire advantage rested on two crash weeks. We did not adopt it.

The strategy places a ladder of buy orders below the price and sells above. Each completed round trip books a small gain — but when the price falls and keeps falling, the buys fill and the matching sells never do, so the position sits there losing. At the end of a test period, what do you do with the purchases still open? Value them at the current price, or leave them out because they never closed?

We ran all eight versions both ways across 20,032 test windows. The accounting choice flips the sign in every configuration. The gap tracks exactly what you would expect: the versions holding the most unsold purchases show the widest gap, one going from +2.4% counting only closed trades to −5.1% counting everything.

One note on how we treated our own claim. The original study made this point in a single line and stated the other figure without ever calculating it — the code tracked both figures and saved only their sum. We measured it before publishing this, and the assertion turned out to be conservative: it said “most versions”, the answer is all of them. Being right by luck is not the same as having checked.

Our own price archive had a known gap: it only held coins that were still listed on the day we downloaded it, so every coin that had already closed was missing. Those are usually the losers. We built a repair, 63 coins came back, and from then on the archive went by a name that said the problem was solved.

It wasn't. The repair had asked the exchange for coins that were about to close, and a coin that has fully closed is no longer on that list at all. So the repair could never have found a single one. It looked complete because of its name, not because anyone had checked it. The exchange's own public archive had every closed coin all along, one download away.

Then we measured whether it changed the answer, and it didn't. Closed coins turned out not to be short-lived oddities: they lived 22 months on average, against 21 for coins still trading, and closed at all points between 2020 and 2026. On the result in question, the missing coins accounted for about 1% of the effect, a rounding error. We had written down what we expected to find before measuring it, and that guess turned out to be wrong.