Do AI Trading Actually Work?

The strongest evidence for AI in trading comes from institutional infrastructure rather than from consumer products. Machine learning is genuinely embedded in order routing, execution timing, fraud surveillance, and risk controls across regulated markets, and supervisors acknowledge these operational uses as established practice [2]. That is a materially different claim from "an AI predicts the market and makes you money," and conflating the two is the single most common source of confusion in this debate.

Where the record is weakest is precisely where retail marketing is loudest: discretionary return generation. Regulatory reviews of how firms describe AI capabilities have found communications that overstate what the technology does, omit material limitations, or imply performance guarantees that cannot be substantiated [1]. FINRA's guidance to member firms makes the same point from the supervisory side, stressing that firms must be able to evidence the models they deploy and the claims they make about them [2].

It is also worth separating three questions that get bundled together. Does the model detect patterns? Usually yes. Do those patterns persist out of sample? Sometimes. Does the persistence survive transaction costs, slippage, and crowding? Rarely, and almost never at the scale implied by retail advertising [3]. A system can be technically sophisticated and still be economically worthless once costs are applied.

Finally, the academic literature is not uniformly negative. There is credible published work showing that machine learning methods can improve forecasting in specific, well-specified settings with careful feature engineering and strict out-of-sample testing [3]. The distinguishing feature of that work is not the algorithm — it is the discipline of the evaluation. Systems that work are the ones whose authors assumed their own results were probably wrong.

Backtest overfitting is the central failure mode, and it is structural rather than accidental. When a researcher tests thousands of model variants against the same historical dataset and keeps the best performer, the winner is selected partly for genuine signal and partly for noise that happened to fit that particular history [3]. The reported Sharpe ratio is then an artifact of the search process, not a property of the market. This problem intensifies with AI because model capacity makes it cheap to search vastly more configurations than a human analyst ever could.

Regulators have flagged a second failure mode: claims that cannot be traced to a verifiable methodology. When firms cannot show the data, assumptions, and validation behind an AI-driven strategy, investors have no way to distinguish a tested system from a plausible narrative [1]. FINRA's framing is that model risk must be governed like any other material risk — documented, monitored, and subject to challenge — which is exactly the discipline most retail-facing products omit [2].

Market impact and regime change complete the picture. A strategy calibrated on a low-volatility period can invert sharply when volatility regimes shift, and strategies that work at small size often degrade as capital scales and the edge gets arbitraged away [3]. Live markets also introduce latency, liquidity gaps, and counterparty behaviour that no historical dataset fully represents.

There is a behavioural layer too. Investors presented with confident AI framing tend to underweight base rates and overweight narrative coherence, which makes them receptive to exactly the claims supervisors have warned about [1]. The failure is not only in the model; it is in the evaluation environment around it.

Start with falsifiability. A credible system states its universe, its holding period, its cost assumptions, and its out-of-sample period before showing you a return chart — and it shows the losses. A marketing claim shows an equity curve with no methodology and no drawdown detail [1]. If you cannot reconstruct how a number was produced, the number is not evidence.

Second, look for independent verification rather than self-reported performance. Third-party audited track records, regulatory registration where the activity requires it, and disclosure of who is accountable for the strategy are all baseline signals [2]. Consumer-protection frameworks exist precisely because unverified performance claims are easy to make and expensive to disprove; firms operating within them accept scrutiny that anonymous signal sellers avoid [4].

Third, apply cost and capacity realism. Ask what happens to the strategy after spreads, commissions, financing, and slippage, and at what capital level the edge disappears [3]. A strategy that only works below a certain size is not scalable, and a strategy that only works in a backtest is not a strategy.

Fourth, treat guaranteed returns as a disqualifier rather than a feature. Any framing that promises a fixed percentage outcome from AI trading contradicts the basic risk structure of markets, and supervisory bodies treat such framing as a red flag for investor harm [4]. The presence of a refund policy, a testimonial wall, or an urgency countdown tells you about the sales funnel, not the model.

The most useful test is one you can run yourself: before trusting any performance claim, ask what evidence would prove it false — and check whether the seller has ever published that evidence. If the answer is nothing, you have learned more from the silence than from the chart.