Numerical code can be wrong in two ways. It can have a bug, or it can be a correct formula that the computer can't evaluate accurately. The second kind is harder to spot, because the code looks right and usually gives sensible answers. This lesson explains how computers store numbers, shows a textbook formula failing on realistic data, and sets out how to test numerical code so that both kinds of error get caught.
| Term | Meaning |
|---|---|
| double | A 64-bit floating-point number, Python's float and NumPy's float64 |
| ulp | "Unit in the last place": the gap between a double and the next one |
| Machine epsilon, : the ulp of 1 | |
| cancellation | Subtracting two nearly equal numbers, which wipes out their shared digits |
| relative error |
The rest of this lesson needs lifetime access
It will be part of lifetime access, which is coming soon: every lesson in all four tracks, including lessons added later, for one payment.
Already have access? Sign in

