Relative error and correct digits¶
Level: 201 · for anyone who has ever said "correct to three digits"
One line: "Correct to p significant digits" has only p + 1 possible values and no definition that behaves, so the number to report is the relative error |x − x̂| / |x|: continuous, the same in any unit and any base, hidden by a norm for the small components of a vector, and the thing a float promises to keep below 2⁻⁵³ on every operation.
Two measures, and why the second one¶
Let x be the true value and x̂ (read "x hat") an approximation of it. There are two obvious ways to say how far apart they are:
- the absolute error |x − x̂|, in whatever units x has;
- the relative error |x − x̂| / |x|, a pure number, undefined when x = 0.
A second way to write the relative error is the one numerical analysis actually uses: put x̂ = x(1 + ρ), and the relative error is |ρ|. Section 1 checks, with exact fractions, that the two forms agree on every row, and shows the ZeroDivisionError at x = 0. Some authors drop the absolute values; when the sign matters, the convention is to talk about "the error x − x̂".
Section 2 is the reason the relative error is the one that gets reported. Scale x and x̂ by the same α — change metres to kilometres, dollars to cents — and the absolute error scales by α while the relative error does not move. An absolute error of 5 is enormous for a length of 10 and negligible for the distance to the Moon; a relative error of 5 × 10⁻³ means the same thing for both. This is the measure the significant figures rule for × and ÷ was compressing, and uncertainty propagation is what happens to it in arithmetic.
What the program prints¶
Verified output of relative_error.py — regenerated by tools/run_examples.py, never hand-typed.
1. TWO MEASURES OF ERROR
x is the truth, xhat the approximation.
x xhat |x - xhat| |x - xhat| / |x| rho, where xhat = x(1 + rho)
1.00000 1.00499 0.00499 4.99e-03 +4.99e-03
9.00000 8.99899 0.00101 1.12e-04 -1.12e-04
100 101 1 1.00e-02 +1.00e-02
0.001 0.002 0.001 1.00e+00 +1.00e+00
-2 -1.9 0.1 5.00e-02 -5.00e-02
|rho| equals E_rel in every row (checked with exact fractions): True
At x = 0 the relative error is undefined: rel_error(0, 0.001) -> ZeroDivisionError
Higham's own convention for a signed quantity is 'the error x - xhat', which is -rho times x.
2. RELATIVE ERROR DOES NOT CARE ABOUT THE UNITS
x = 1.00000 and xhat = 1.00499, both multiplied by the same alpha:
alpha |x - xhat| E_rel
1 4.990e-03 4.99e-03
1000 4.990e+00 4.99e-03
1/7 7.129e-04 4.99e-03
1e-9 4.990e-12 4.99e-03
E_rel unchanged by every alpha: True absolute error scaled by exactly alpha: True
Metres or kilometres, dollars or cents: the relative error is the same number.
3. COUNTING CORRECT DIGITS GIVES ONLY p + 1 POSSIBLE ANSWERS
Significant digits: the first nonzero digit and everything after it.
1.7320 5 significant digits
0.0491 3 significant digits
An approximation written with 5 significant digits can have 0, 1, 2, 3, 4 or 5
of them correct: six possible answers. Relative error is a continuous scale.
Two approximations, each correct to 3 digits but not 4, by either definition:
x xhat E_rel A: p=3 p=4 B: p=3 p=4
1.00000 1.00499 4.99e-03 yes no yes no
9.00000 8.99899 1.12e-04 yes no yes no
Same digit count; the relative errors differ by a factor of 44.5.
4. DEFINITION A: p CORRECT DIGITS IF x AND xhat ROUND TO THE SAME p-DIGIT NUMBER
x = 0.9949, xhat = 0.9951
p x rounds to xhat rounds to agree?
1 1 1 yes
2 0.99 1.0 no
3 0.995 0.995 yes
4 0.9949 0.9951 no
One and three correct digits, but not two. A count that is not monotone is not a count.
5. DEFINITION B: |x - xhat| IS LESS THAN HALF A UNIT IN THE p-TH SIGNIFICANT DIGIT OF x
x = 0.123, xhat = 0.127, |x - xhat| = 0.004
p unit in digit p half agree?
1 0.10 0.05 yes
2 0.010 0.005 yes
3 0.0010 0.0005 no
'Agree to two significant digits', although they round to 0.12 and 0.13.
Definition B is monotone (p digits implies p - 1 digits), on 294 pairs: True
...so it has no anomaly: 0.9949 vs 0.9951: p=1 yes, p=2 yes, p=3 yes, p=4 no, p=5 no -> 3 correct digits
Neither definition implies the other:
0.123 vs 0.127 B: 2 digits; A at p=2: no
0.9946 vs 0.9954 A at p=3: yes (both round to 0.995); B at p=3: no (|x - xhat| = 0.0008, half a unit = 0.0005)
6. DEFINITION B IS A BAND OF RELATIVE ERROR, ONE DECADE WIDE
On the same 294 pairs xhat = x(1 + rho), for p = 1..8:
E_rel <= 0.5 x 10^-p implies B holds at p failures: 0
B holds at p implies E_rel < 5 x 10^-p failures: 0
The same absolute error under two leading digits:
x = 1.000, |x - xhat| = 0.004: B gives 3 digits, E_rel = 4.0e-03
x = 9.999, |x - xhat| = 0.004: B gives 3 digits, E_rel = 4.0e-04
'3 correct digits' spans a factor of 10 in relative error. The relative error itself does not.
7. VECTORS: A NORM HIDES THE SMALL COMPONENTS
x = (1, 0.001) xhat = (1.0001, 0.0011)
norm ||x - xhat|| / ||x||
inf 1.00e-04
1 2.00e-04
2 1.41e-04
About four correct digits, says every norm. Component by component:
i x_i xhat_i |x_i - xhat_i| |x_i - xhat_i| / |x_i|
1 1.0 1.0001 0.0001 1.0e-04
2 0.001 0.0011 0.0001 1.0e-01 <- 10% wrong, and invisible above
componentwise relative error = max_i |x_i - xhat_i| / |x_i| = 1.0e-01
A normwise bound of 1e-4 promises the small component only an ABSOLUTE error
of about 1e-4 x ||x||, which is 10% of that component.
8. PYTHON'S math.isclose IS A RELATIVE-ERROR TEST
isclose(a, b, rel_tol=r) is |a - b| <= r x max(|a|, |b|) (abs_tol defaults to 0)
isclose(1.00000, 1.00499, rel_tol=5e-03) True |a - b| / max = 4.97e-03
isclose(1.00000, 1.00499, rel_tol=4e-03) False |a - b| / max = 4.97e-03
isclose(9.00000, 8.99899, rel_tol=4e-03) True |a - b| / max = 1.12e-04
It divides by the larger of the two, so it cannot tell which one is the truth.
E_rel divides by the truth, and the two can differ (section 11, kata 1).
9. ROUNDING INTO A FLOAT IS A RELATIVE ERROR, AND A BOUNDED ONE
binary64 keeps 53 significant bits. Rounding a real x to the nearest float gives
fl(x) = x(1 + rho) with |rho| <= u = 2^-53 = 1.11e-16, the unit roundoff.
Checked exactly on 2,000 values k/7 and k/1000:
largest |rho| seen 9.80e-17 <= 2^-53: True
And every single operation: fl(a op b) = (a op b)(1 + delta), |delta| <= u.
checked +, -, x, / on 61^2 pairs of floats (14,815 results; exact zeros skipped):
largest |delta| seen 1.11e-16 <= 2^-53: True
Correct decimal digits of fl(x), by definition B, over the 1,850 inexact values:
fewest 15, most 18. Section 6 promised at least 15, since u <= 0.5 x 10^-15.
10. THE TABLEMAKER'S DILEMMA
Tabulate y = exp(pi x sqrt(163)) to t significant digits
(computed with five guard digits, then rounded to nearest):
t y
10 262537412600000000
15 262537412640769000
20 262537412640768744.00
25 262537412640768744.0000000
30 262537412640768743.999999999999
35 262537412640768743.99999999999925007
40 262537412640768743.9999999999992500725972
Is the last digit before the point a 4 or a 3? Every table up to t = 25 says 4.
It is 3, followed by twelve 9s. Rounded to the nearest integer the entry is ...744,
but a table that prints the integer part cannot settle it with fewer than 30 digits,
and nothing about the number said in advance how many digits that would take.
An exact tie needs an algebraic value:
sqrt(0.01550025) = 0.1245 exactly, so to 3 digits it is 0.124 (ties to even) or 0.125 (ties away):
more digits will never settle it; only the tie rule does.
A transcendental value is never exactly a tie, but it can sit as close as it likes.
Tabulating sin(k/1000) for k = 1..1000 to 4 significant digits, the digits past the
fourth were read until the rounding was decided:
digits read entries
1 800
2 185
3 13
4 2
Hardest entry: sin(0.368) = 0.3597|500552... needed 4 digits past the cut.
One entry in five needs a second digit, and each digit after that is needed about
one time in ten. No entry comes with a bound in advance.
11. PRACTICE, SOLVED
Kata 1 (Higham, problem 1.1). Measuring against xhat instead of x:
E = |x - xhat| / |x|, E' = |x - xhat| / |xhat|. With xhat = x(1 + rho): E' = E / |1 + rho|,
so E / (1 + E) <= E' <= E / (1 - E) whenever E < 1.
checked for rho = -0.99 .. 0.99 in steps of 0.01: True
rho = +0.5: E = 0.500 E' = 0.333
rho = -0.5: E = 0.500 E' = 1.000
Small errors: the two agree to first order. Large ones: it matters what you divide by.
Kata 2. Truth x = 3,213,468 (a count), estimate xhat = 3,200,000:
E_abs = 13,468 E_rel = 4.19e-03
A: 2 correct digits (3.2E+6 vs 3.2E+6; at p=3, 3.21E+6 vs 3.20E+6)
B: 2 correct digits (half a unit in the 2nd digit is 50,000; in the 3rd, 5,000)
Kata 3. x = (1000, 1, 0.001), xhat = (1000.1, 1.001, 0.0011):
normwise (inf-norm) 1.0e-04
per component 1.0e-04, 1.0e-03, 1.0e-01
componentwise 1.0e-01
The normwise 1e-4 certifies four digits of the first component only; the third is 10% off.
Counting correct digits is a crude instrument¶
The significant digits of a number are its first nonzero digit and everything after it: 1.7320 has five, 0.0491 has three. That is the counting rule from two lessons ago, now applied to a computed answer rather than a measured one.
Suppose x̂ is written with p significant digits. Then the question "how many of them are correct?" has at most p + 1 answers — none, one, …, all p — so it is a coarse scale by construction. Section 3 shows how coarse. The two approximations
x = 1.00000 x̂ = 1.00499 relative error 4.99 × 10⁻³
x = 9.00000 x̂ = 8.99899 relative error 1.12 × 10⁻⁴
are each correct to three significant digits but not four, under either definition below, yet their relative errors differ by a factor of about 44. The count sees two identical cases. The relative error sees a factor of 44, and it is the same leading-digit effect the uncertainty propagation page opens with: 1.0 and 9.9 are both two significant figures and one is known ten times better than the other.
Two definitions of "correct", and both misbehave¶
The intuition that x̂ "has three correct digits" feels clear, and pinning it down turns out to be genuinely awkward. Higham gives two candidate definitions, and the program runs both.
Definition A. x̂ has p correct significant digits if x and x̂ round to the same number to p significant digits. It is mathematically clean and usually matches intuition. Section 4 is the counterexample: x = 0.9949 and x̂ = 0.9951 round to the same number at one digit (1) and at three (0.995), but not at two (0.99 against 1.0). One and three correct digits, but not two — a count that is not monotone in p is not a count.
Definition B. x̂ agrees with x to p significant digits if |x − x̂| is less than half a unit in the p-th significant digit of x. This one is monotone — section 5 checks it on 294 pairs — and it has no anomaly: 0.9949 against 0.9951 comes out as three correct digits. Its cost is generosity. By definition B, 0.123 and 0.127 agree to two significant digits, because they differ by 0.004 and half a unit in the second digit is 0.005 — although they round to 0.12 and 0.13, and most people would say they agree to less than two.
The two definitions are not even nested. 0.123 / 0.127 passes B at two digits and fails A; 0.9946 / 0.9954 passes A at three digits (both round to 0.995) and fails B (they differ by 0.0008, and half a unit is 0.0005).
Section 6 says what definition B really is: a band of relative error one decade wide. Half a unit in the p-th digit of x is 5 × 10^(e−p), where 10^e is the leading power of ten of x, so
and where in that band a given x̂ falls depends on the leading digit of x. The program shows the same absolute error, 0.004, giving "three correct digits" at x = 1.000 and at x = 9.999, with relative errors 4 × 10⁻³ and 4 × 10⁻⁴.
Higham's conclusion is the one to keep. A count of correct digits is a useful way to think about accuracy. The relative error is the precise measure, and it is also independent of the base — which matters once the machine works in base 2 and you read in base 10. Whenever you give an approximate answer, give an estimate or a bound for its relative error with it.
Vectors: a norm hides the small components¶
When x and x̂ are vectors, the relative error is usually taken with a norm, ‖x − x̂‖ / ‖x‖, and the three norms in common use are the largest component (∞), the sum of the components (1) and the Euclidean length (2). A bound of ½ × 10⁻ᵖ on that quantity says less than it seems to: components about as large as ‖x‖ have about p correct digits, but for the smaller components it only bounds the absolute error, which relative to them may be no promise at all.
Section 7 makes it concrete. For x = (1, 0.001) and x̂ = (1.0001, 0.0011), every norm reports a relative error of about 10⁻⁴ — four correct digits — while the second component is 10% wrong. The measure that puts every component on an equal footing is the componentwise relative error
which here is 0.1, and which is the standard tool of error analysis and perturbation analysis for exactly this reason.
Where the definition shows up in code¶
Python's math.isclose(a, b, rel_tol=r) is a relative-error test: it asks whether |a − b| ≤ r × max(|a|, |b|). Section 8 runs it on the two approximations from section 3.
math.isclose(1.00000, 1.00499, rel_tol=5e-3) # True |a - b| / max(|a|, |b|) = 4.97e-3
math.isclose(1.00000, 1.00499, rel_tol=4e-3) # False
Notice the 4.97 × 10⁻³ against the 4.99 × 10⁻³ above. isclose divides by the larger of the two values, because a test between two computed numbers has no truth to divide by, and that choice makes the test symmetric. Higham's relative error divides by the truth. Kata 1 below is his own exercise on how much the choice of denominator can matter. The sibling Python library's float equality and NaN ↗ covers the rest of isclose: why an absolute epsilon that works near 0.3 fails near 2000, and the abs_tol=0.0 trap near zero. The Rust library runs a textbook's EPSILON tolerance that never actually gets tested in Rust in Action §2.3.3, run ↗.
The bridge to the machine¶
Section 9 is where this lesson hands over to machine numbers. Rounding a real number x into binary64 — the nearest of a finite set — gives
and u is called the unit roundoff. The program checks it exactly on 2,000 fractions and finds the largest |ρ| at 9.8 × 10⁻¹⁷. More than that: every single operation on floats satisfies fl(a op b) = (a op b)(1 + δ) with |δ| ≤ u, and the program checks that too, on 14,815 additions, subtractions, products and quotients, where the largest |δ| seen is u itself to three digits. This is the standard model of floating-point arithmetic, equation (1.1) in Higham's book and the assumption every error bound in it rests on. The whole of rounding error analysis is the study of what happens to a relative error of 10⁻¹⁶ per operation over many operations.
In definition B's terms, the 1,850 inexact values in the sample came out with between 15 and 18 correct decimal digits: never fewer than the 15 that section 6's sufficient condition promises for u ≤ ½ × 10⁻¹⁵, and more when the leading digit and the position between two floats happen to be kind. Catastrophic cancellation is what happens to that 10⁻¹⁶ when it is divided by a difference of 10⁻¹⁷.
The tablemaker's dilemma¶
Higham closes the section with an aside. You are tabulating a transcendental function, and one entry comes out as 0.124|5000000, correct to a few digits in the last place shown, with the bar after the last digit you intend to print. Is that digit a 4 or a 5? It depends on whether a nonzero digit is hiding further out, and there is no simple bound on how many digits you must compute to find out.
Section 10 runs the book's own exercise on it (problem 1.2). The number e^(π√163), a famous near-integer, tabulated to t significant digits:
t = 10 262537412600000000
t = 20 262537412640768744.00
t = 25 262537412640768744.0000000
t = 30 262537412640768743.999999999999
Every table up to 25 digits says the integer part ends in 4. It ends in 3, followed by twelve 9s, and a table that prints the integer part cannot settle it with fewer than 30 digits. Nothing about the number announced that in advance.
Two more things the program shows. An exact tie needs an algebraic value — √0.01550025 is exactly 0.1245, and to three digits it is 0.124 under ties-to-even and 0.125 under ties-away, so no number of extra digits settles it; only the tie rule does. A transcendental value is never exactly a tie, but it can sit as close to one as it likes: tabulating sin(k/1000) for k = 1 … 1000 to four digits, 800 entries were decided by one digit past the cut, 185 needed two, 13 needed three and two needed four, the worst being sin(0.368) = 0.3597|5005…. Each further digit is needed about one time in ten, and no entry comes with a bound in advance. That is why correctly rounded elementary functions were a research problem for decades, and why the Handbook of Floating-Point Arithmetic gives the dilemma a section of its own.
Practice¶
Kata 1 — Higham's problem 1.1. In error analysis it is sometimes convenient to divide by |x̂| instead of |x|. With E = |x − x̂| / |x| and E′ = |x − x̂| / |x̂|, find inequalities relating E′ to E.
Solution
Write x̂ = x(1 + ρ). Then E = |ρ| and E′ = |ρ| / |1 + ρ| = E / |1 + ρ|. For E < 1 the denominator lies between 1 − E and 1 + E, so
and symmetrically E′ / (1 + E′) ≤ E ≤ E′ / (1 − E′). Section 11 checks the first pair on ρ from −0.99 to 0.99 and shows both ends being reached: at ρ = +0.5, E = 0.5 and E′ = 0.333; at ρ = −0.5, E = 0.5 and E′ = 1.0. For small errors the two agree to first order, which is why isclose can afford to divide by whichever value is larger.
Kata 2 — the blackboard, revisited. The significant figures page reported a population as 3,200,000. Suppose the census count is x = 3,213,468 and the blackboard's figure is the approximation x̂. Give the absolute error, the relative error, and the number of correct significant digits under definitions A and B.
Solution
Section 11: the absolute error is 13,468 and the relative error 4.19 × 10⁻³. Under A, x and x̂ both round to 3.2 × 10⁶ at two digits and to 3.21 × 10⁶ against 3.20 × 10⁶ at three, so two correct digits. Under B, half a unit in the second digit of x is 50,000, which exceeds 13,468, and half a unit in the third is 5,000, which does not: two correct digits. Both definitions agree here, and the relative error 4.19 × 10⁻³ sits inside definition B's band for p = 2, between 0.5 × 10⁻² and 5 × 10⁻².
Kata 3 — three components. For x = (1000, 1, 0.001) and x̂ = (1000.1, 1.001, 0.0011), compute the normwise relative error in the ∞-norm and the componentwise relative error. Which components does the normwise number certify to four digits?
Solution
Section 11: the normwise relative error is 0.1 / 1000 = 10⁻⁴. The individual relative errors are 10⁻⁴, 10⁻³ and 10⁻¹, so the componentwise relative error is 0.1. The normwise 10⁻⁴ certifies four digits of the first component only; the second has three and the third is 10% off, and the norm cannot see either.
Run it yourself¶
From the root of your clone of this repository:
Further reading¶
- Nicholas J. Higham, Accuracy and Stability of Numerical Algorithms, 2nd edition, SIAM 2002 (doi:10.1137/1.9780898718027 ↗) — §1.2 "Relative Error and Significant Digits" is this page; problem 1.1 is kata 1 and problem 1.2 is the tablemaker's table; §1.5 "Backward and Forward Errors" and §1.6 "Conditioning" are where the relative error goes next, and §2.2 "Model of Arithmetic" is the standard model of section 9 in full.
- Jean-Michel Muller et al., Handbook of Floating-Point Arithmetic, 2nd edition, Springer 2018 (doi:10.1007/978-3-319-76526-6 ↗) — chapter 10 "Evaluating Floating-Point Elementary Functions", §10.5 "The Table Maker's Dilemma".
- PEP 485 — A function for testing approximate equality ↗ — the design of
math.isclose, including why it divides by the larger value and whyabs_toldefaults to zero.
Auf Deutsch: Stichwörter¶
Statt „auf p Stellen richtig“ meldet man den relativen Fehler, die eine Zahl, die unabhängig von der Größenordnung ist.
Stichwörter: relativer Fehler, absoluter Fehler, gültige Stellen (correct digits), Größenordnung, Fehlerschranke.
See also¶
- Significant figures — the previous lesson: the counting rule, and the two arithmetic rules that absolute and relative error explain
- Uncertainty propagation — the next lesson: what × and ÷ do to a relative error and + and − to an absolute one
- Machine numbers — the finite set behind section 9, and the five rounding functions
- Catastrophic cancellation — a relative error amplified 17,458 times, and conditioning against stability
- Float equality and NaN ↗ —
math.isclosein full, in the sibling Python library - Rust in Action §2.3.3, run ↗ — an
EPSILONtolerance checked against a compiler, in the sibling Rust library - Did the rounding decide it? ↗ — instead of a bound on the error, an interval that is guaranteed to contain the answer