What we need from you, and the eight checks we run

Think of a recipe. You bake a cake and it comes out perfect. You give the recipe to a friend, they follow it exactly, and something different comes out. Then the recipe was never the whole story — something in your kitchen was doing work nobody wrote down.

The strategy is the recipe. The number is the cake. So we take the recipe to a different kitchen and bake it. More often than anyone expects, a different cake comes out — and then the number was never really about the strategy. It was about that one computer, on that one day.

And if nobody else can get your number, every change you make afterwards — bigger size, new settings — is being judged against something that wobbles.

You go fishing with a net. The holes are big, so the little fish swim straight through. You pull it up, look in, and every fish is a big one. You could easily decide that all fish are big. The net told you that, and the net was wrong.

A plan needs a list of things it was allowed to buy, and nearly everyone builds that list the same way: ask the computer, today, what is available — then run the test back across the last five years. That list is a net with big holes. Anything that died along the way slipped through and is not in it, so the test only ever buys things it already knows survived. Nothing warns you. An absence does not announce itself.

The ones that slip through the net are the ones that failed. So the one thing their test can never show you is what failure costs.

You take money out of the ATM and it charges you $2. Once? You would shrug. But this plan goes to the ATM five times a day, every day, for six years. Now the $2 is not a detail on the receipt. It is the single biggest thing in the plan, and it can turn a winner into a loser on its own.

So how does anyone know it is $2? Somebody said so. They looked at a different machine, a while ago, and wrote it down, and it has been copied from document to document ever since. Nobody has looked at a receipt. And it was never one number anyway — taking out $20 and taking out $2,000 are not charged the same way, and the figure people quote is nearly always the $20 one.

In our own work we found one counted about twenty times too big. Three strategies had already been switched off because of it.

A cost that was copied instead of measured goes wrong in both directions. It flatters plans that do not work, and it quietly kills plans that do — and the ones it kills do not complain. They just go quiet.

A full load of whites, and one red sock goes in with them. Everything comes out pink. And here is the real problem: the damage is not in the sock any more. It is spread thinly across everything, and there is nothing left to pull out and throw away. Every item is a bit pink, and a bit pink is easy to live with.

Market data arrives from outside, and now and then a row is broken. Not slightly. Ridiculously. One day’s figure came in thousands of times bigger than that market had ever moved in a day, went into the wash with thirty months of ordinary days, and came out as a result of nearly 200% a year.

Every safety check passed, because they all checked what came out of the machine rather than what went into it. By then there is no bright red sock, just a faintly rosy month — and a rosy month is not suspicious, it is what everybody was hoping for. Worse, checking two rows and checking every row look identical when you read the code. The only difference is how many it ran on.

Anything checked by sampling is unchecked. The row that breaks everything is the strange one, and the strange row is exactly what a sample is least likely to hold.

A strategy decides from a picture of the market — who wants to buy, who wants to sell. That picture is always slightly old. A second or two.

Most days that changes nothing. On the day everybody runs for the door, it is the whole story. The strategy sees a large crowd of buyers, which looks like safety — but that crowd left a few seconds ago. The signal was not broken. It was reading a photo of a room that was already empty. And that is not bad luck: it is what the strategy does every time the market moves faster than its picture updates.

Which means it will happen again — and your own backtest cannot tell you how often, because it is the thing that recorded those days as wins.

I hide a candy in one hand and you guess. Wrong. Guess again. Wrong. Keep going. On the ninth try you get it, and I write down: guessed right. I do not write down the eight misses. Read my notes and you are a mind reader.

Someone builds the plan, then tries it forty different ways and writes up the one that came out best. The other thirty-nine leave no trace. So the number you are shown is doing two jobs at once: it is the score, and it is also the reason this version is the one in front of you. Pick the highest of forty and of course it is high.

Trade count hides the same trick. Two thousand trades that all landed in the same six panic weeks are six answers, said loudly. So we set days aside before the searching starts and let the plan try them — once. Peek, adjust, peek again, and those days are part of the search too.

Trying lots of things is not cheating — everybody does it. That is exactly why the only real test is data set aside before the searching started, and why you only get one shot at it.

Imagine telling a playground: “I will swap my sandwich for anyone’s cookie.” Who says yes? The ones with the bad cookies. Whoever has the good cookie keeps it.

A strategy that waits for a good price has the same problem: the moments it gets that price are often the moments everyone else is leaving — which is why the price came to it. Add two more: the shelf said $10, but there was one left and you wanted fifty; and by the time you got to the register it was gone. A backtest usually writes down the shelf price, because on paper nothing runs out and nobody else is in the store.

And the days that made most of the money are usually the fastest, wildest ones — which are exactly the days when the price on paper is least likely to have been real.

Imagine someone takes 41 photos of themselves and shows you the best one. The photo is real. Nobody faked it. But it is not how they normally look — it is their luckiest try out of 41.

And a player who scores 8 shots out of 10 is not an 80% shooter. That was one afternoon. So when a result says 61.4%, the question is simply: if you ran it all again, what would you get? If the honest answer is anywhere between 45 and 75, that is not a number. It is a direction — which is still worth something, but it is a different thing to tell an investor.

A number with no wiggle room written next to it is a promise nobody has tested.