The tension between two errors
A derivative is defined by a limit as a step tends to zero, but a computer uses a finite step and approximate function values. Making the step smaller improves the analytic approximation while magnifying errors in the difference. We will distinguish a bound on function-evaluation errors from a detailed floating-point model: the former can be proved without pretending that all evaluations have the same relative error.
Definition 1.3.1
(Forward and centered differences)
Let . The forward and centered differences of at are
Definition 1.3.2
(Absolute evaluation-error bound)
The available values satisfy at each point actually used. This is an absolute error bound. It can include measurement and evaluation errors, but arithmetic used in subtracting and dividing introduces additional rounding unless the model explicitly includes it.
Truncation and noise bounds
Theorem 1.3.3
(Forward-difference error bound)
If on , then
Proof
(Taylor remainder plus two value errors)
Taylor's theorem gives for some . Dividing by bounds the truncation error by . The difference of the two value errors has magnitude at most ; dividing by and using the triangle inequality gives the claim.
Theorem 1.3.4
(Centered-difference error bound)
If on , then
Proof
(Cancellation of the even terms)
Expand and to second order with third-derivative remainders. The constant and quadratic terms cancel, leaving and a remainder of magnitude at most . Divide by . Two evaluation errors each bounded by contribute at most .
Proof X-ray
Why symmetry improves the analytic order
The forward formula retains a second-derivative term proportional to . In the centered formula the even Taylor terms cancel between opposite points, so the first uncontrolled term is proportional to . Both formulas still divide the difference of imperfect values by , so neither eliminates noise amplification.
Choosing a step from the bound
Proposition 1.3.5
(Steps that minimize the error bounds)
Assume and that the respective derivative bounds hold throughout the candidate intervals. The forward upper bound is minimized at
and the centered upper bound at
Proof
(Setting the derivative to zero)
The function on has , which is negative below and positive above it, so is the minimizer. There and , so the minimum is . For , vanishes where , and it changes sign from negative to positive there.
Common mistake
(Treating the bound-optimal step as the best step)
These steps optimize worst-case upper bounds, not necessarily the actual error: the two signed errors may cancel, and the derivative bounds may be pessimistic. Also, if the available domain or valid derivative bound restricts , the candidate must be checked against that restriction. The case does not produce a positive minimizing step in these bounds; decreasing then reduces the displayed truncation term, subject to the assumptions.
Worked example 1.3.6
(A centered difference of the exponential)
Estimate for with a centered difference and value errors bounded by . On , take . The candidate lies below one. The truncation term is at most about and the value-error term about , for a total bound near . This bound is not a prediction that every computation has that error.
Worked example 1.3.7
(A forward difference of the exponential)
Estimate for with the forward difference, again with . On take . Then , and the minimized bound is . At the optimum the truncation term equals the noise term . The centered bound above is roughly twenty times smaller, which is the practical gain from the higher analytic order. Both bounds use for the derivative bound over or , although near the true derivatives are close to ; the bounds are pessimistic.
Worked example 1.3.8
(Exact values and a quadratic function)
Take at with exact values, . The forward difference is , so its error is exactly and decreases to zero with : there is no positive minimizing step. The centered difference is for every , an exact result. This is consistent with the centered bound, since gives , and the formula for does not apply.
Counterexample mode
Smaller h can make the available information worse
Let and suppose the two values in a forward difference have errors at and at . Then the computed difference is . Its error grows as , even though the exact forward difference has zero truncation error. A step smaller than the scale of uncertainty may amplify noise rather than reveal a better derivative.
Common mistake
(Ignoring relative errors and the rounding of sample points)
Relative error needs separate care near zeros. An assumption gives an absolute bound depending on the sampled values, and it may fail to model cancellation or underflow in the computation of . Moreover, if rounds to in floating point, the intended distinct sample points may become identical. A complete implementation must account for these effects and the rounding in the final arithmetic.
Practice and solutions
Quick check
(Derive the exact forward-difference truncation error for at .)
Expand .
Solution · Answer
, so the signed error is .
Quick check
(If and , find the step minimizing the stated forward bound.)
Use .
Solution · Answer
The candidate is , provided the derivative bound is valid on the required interval.
Quick check
(Why does the centered estimate require a third-derivative bound in this argument?)
Which Taylor terms cancel?
Solution · Answer
The constant and quadratic Taylor terms cancel, and the two third-order remainders give the stated bound. Other regularity assumptions can yield other bounds.
Quick check
(Suppose and the exact function values are used. What are both difference formulas for every ?)
Substitute .
Solution · Answer
Both equal one exactly; the instability in the counterexample comes from errors in the values, not from truncation for this linear function.