Skip to main content

Data and machine learning · 12 min read

Overfitting

Why training error always falls and test error doesn't: bias, variance, the noise floor, and how to keep financial models honest.

Before you start

  • Conditional expectation
  • Thinking in arrays with NumPy

By the end you'll be able to

  • Distinguish training error from test error
  • Decompose test error into bias, variance and noise
  • Explain why financial data is especially prone to overfitting
  • Use holdout data, simplicity and regularisation to control it

Give a model enough freedom and it will fit any history perfectly, including all the noise in it. The fit then tells you nothing about the future. In finance this is the default outcome rather than an occasional risk, because the signal is weak and the noise is large. This lesson makes overfitting visible, measures it, and sets out the habits that keep it under control.

TermMeaning
training errorHow badly the model fits the data it was fitted on
test errorHow badly it predicts new data from the same source
biasError from a model too simple to capture the real pattern
varianceError from a model so flexible its fit changes with every sample of noise
noise floorThe error no model can beat: the variance of the unpredictable part
regularisationPenalising complexity so the fit prefers simple explanations

The rest of this lesson needs lifetime access

It will be part of lifetime access, which is coming soon: every lesson in all four tracks, including lessons added later, for one payment.

See pricing

Already have access? Sign in