Skip to main content

Probability and statistics · 12 min read

Hypothesis tests and false discoveries

Why testing many strategies guarantees some that look brilliant by luck, how good luck looks, and how to raise the bar.

Before you start

  • How sure is an average? (this track)
  • Bayesian updating (this track)

By the end you'll be able to

  • Test whether a strategy's returns beat zero using its Sharpe ratio
  • Explain why many tests guarantee false positives, and quantify how good luck looks
  • Apply the Bonferroni correction and name its refinements
  • Run a research process that keeps count of what was tried

Test 200 trading ideas that have no edge at all, and about ten of them will pass a standard significance test. Each one will come with a respectable backtest and a p-value under 5%. This is the single most common way quant research goes wrong, and it doesn't need anyone to make a mistake in their code. This lesson shows how big the effect is and what to do about it.

SymbolMeaning
SRAnnualised Sharpe ratio: mean return over volatility, per year
TTLength of the test, in years
ttThe t-statistic for "the true mean return is zero"
α\alphaThe significance level, the false positive rate you accept (5%)
mmThe number of strategies (or variations) tested
Φ−1\Phi^{-1}The inverse of the standard normal cumulative distribution function

The rest of this lesson needs lifetime access

It will be part of lifetime access, which is coming soon: every lesson in all four tracks, including lessons added later, for one payment.

See pricing

Already have access? Sign in