So you can tell before you email us whether there is a job here, what it would involve, and whether we are the right people for it. If we aren't, we would rather you knew now.
They come from mistakes we have actually hit in our own research — not from a textbook. All eight are published on our front page as case studies against ourselves, with the dated internal document each one came from.
We number them FC-1 to FC-8 so that findings in your report point at something specific. A finding reads FC-3, not "we had concerns about costs".
We run your code twice, on two different machines, and see whether the same number comes back. Nothing else means anything until this does.
What going wrong looks likeThe author cannot get their own headline number out again — far more common than anyone expects, and usually something unfixed in how the program orders or seeds its work.Was the list of assets built as it looked back then, or from the ones still trading today? And if something was already patched to fix this, does the patch actually test the thing it claims to?
What going wrong looks likeThe names that disappeared are absent from the test — and the ones that disappear are usually the losers, so the record flatters itself.Where did the fee figure come from? Is it charged the right way round — per book, or per position? Was it measured at the order sizes you would actually trade, or read off a screen?
What going wrong looks likeOne cost assumption appears in three documents with three different numbers, and nobody has measured which is right.Was every row checked, or a sample? We run the checks that already exist across the whole set rather than the slice someone looked at.
What going wrong looks likeA single bad print carries thirty months of apparent return, and removing it removes the strategy.Do the filters meant to exclude stale data actually look at a date? We read what the code does, not what the variable is called.
What going wrong looks likeA filter named for recency that never reads a timestamp — it counts rows and wears the name.How many versions were tried before this one, and was this one chosen before or after seeing how it performed? Is there any data it has genuinely never been run on?
What going wrong looks likeHundreds of variants tested, the best one reported, and nothing held back to check it against. The result is the search.Could every position in the result actually have been bought and sold, at the price used, in the size assumed — and using only what was knowable at that moment?
What going wrong looks likePositions opened and closed at prices that existed, in sizes that did not. Or open positions quietly dropped at the end instead of valued, which turns a loss into a profit.Is the headline a single number or a range? We put an error bar around it and ask whether that range is wide enough to include zero.
What going wrong looks likeA figure quoted to two decimal places that is really a distribution — and a perfectly reasonable alternative way of running the same code lands somewhere else entirely.Three things. If you don't have all three, say so on the call — it usually changes what is worth doing, and occasionally it means there is no job, which we will tell you.
None of this is a rule. It is what makes an audit quick and cheap rather than slow and expensive, and anything in the middle column simply means we say so in the quote.
| Ideal | Workable | Ask us first | |
|---|---|---|---|
| Format | Parquet | CSV, a database dump | a live API we must poll; a platform you can't export from |
| Size | Up to ~2 GB | 2–20 GB, adds a day or two | larger than ~20 GB |
| Frequency | Daily or hourly bars | minute bars | every trade, or full order book |
| Code | Python, in version control | R, MATLAB, C++ | a sealed binary; a signal you can't show us |
| Run time | Under 2 hours a run | 2–6 hours, adds days | longer than that |
Run time matters more than people expect. FC-1 alone means running your backtest at least twice, and every variant we test in FC-6 is another run. A six-hour backtest is not a six-hour job.
On confidentiality: tell us if any part of the strategy can't be shown to us — we can usually work around a sealed component, and it is far better known before we start than halfway through. Your code and data are deleted when we deliver, unless you ask us to keep them.
The eight checks are not specific to any market — survivorship, look-ahead, cost realism and selection are older and better documented in equities than anywhere. But knowing a market's mechanics is a different thing from knowing method, and we would rather be straight about where ours stop.
We don't claim to cover every market. Outside crypto we hold no market data of our own, so you would either send us yours or we buy it in — and we tell you which, and what it costs, before any work starts.
Start to finish, for a full audit. A Triage is steps 1 to 3 and stops there.
The fee is for the work, not for finding something. An audit that finds nothing wrong is a real and useful outcome — it is what you are buying the right to conclude. We would rather agree that now than argue about it later.
If a Triage uncovers something bigger, we stop and re-quote. We don't silently absorb it, and we don't silently bill for it.
That is what the first call is for, and it is free. Tell us what the strategy is and what you have. If there isn't enough there to reproduce anything, we'll tell you on that call and turn the work down — before any money changes hands.
Prices, and the five findings we published against our own research, are on the front page.
Research due-diligence and code-audit services only. This is not investment advice, not a recommendation to buy or sell anything, and not an offer to manage money. Our findings say whether a stated result can be reproduced from the material supplied — they are not an opinion on how any strategy will perform in future. The case studies referred to describe defects found in our own internal research; no client work is described.