Skip to main content

Programming for quant · 13 min read

Floating point and testing numerical code

How computers store numbers, a textbook formula that fails on real prices, and how to test numerical code so both bugs and bad maths get caught.

Before you start

  • Thinking in arrays with NumPy (this track)
  • Variance and standard deviation

By the end you'll be able to

  • Explain how doubles work and why 0.1 + 0.2 is not 0.3
  • Recognise catastrophic cancellation and use stable variance algorithms
  • Sum long lists of numbers accurately
  • Test numerical code with tolerances, reference values and properties

Numerical code can be wrong in two ways. It can have a bug, or it can be a correct formula that the computer can't evaluate accurately. The second kind is harder to spot, because the code looks right and usually gives sensible answers. This lesson explains how computers store numbers, shows a textbook formula failing on realistic data, and sets out how to test numerical code so that both kinds of error get caught.

TermMeaning
doubleA 64-bit floating-point number, Python's float and NumPy's float64
ulp"Unit in the last place": the gap between a double and the next one
ε\varepsilonMachine epsilon, 2−52≈2.2×10−162^{-52} \approx 2.2 \times 10^{-16}: the ulp of 1
cancellationSubtracting two nearly equal numbers, which wipes out their shared digits
relative error∣computed−true∣/∣true∣\lvert \text{computed} - \text{true} \rvert / \lvert \text{true} \rvert

The rest of this lesson needs lifetime access

It will be part of lifetime access, which is coming soon: every lesson in all four tracks, including lessons added later, for one payment.

See pricing

Already have access? Sign in