Motivation
A formula becomes a computing method only after someone specifies where its inputs come from, which records are accepted, what state survives between records, and how the final number is checked. These choices are mathematical. If a retry duplicates a record, the program solves a different least-squares problem. If an invalid number enters an accumulator, later algebra cannot repair it. If a data file is loaded into a list before a streaming loop starts, the loop alone does not establish a small memory requirement.
We will construct a complete small pipeline: generate a synthetic CSV file, read it in bounded batches, validate its schema and record identifiers, update a summary, fit an affine response, and verify the answer against an exact design calculation. The example uses only Python's standard library and ordinary files in a temporary directory.
The statistical interpretation is deliberately limited. We first solve the descriptive problem posed by these particular records. Their construction is deterministic, so the residual sum of squares is not automatically an estimate with a confidence level. A later analysis could add a sampling model, but that would be an additional argument. Keeping this boundary visible makes a runnable demonstration more useful than a reassuring output message.
Definitions
The mathematics starts only after we say exactly which records count as data.
Definition 4.1.1
(Record and ordered input contract)
A record has three fields: an integer identifier, a finite real predictor , and a finite real response . The input contract requires the identifiers in that order, each exactly once. A record is accepted when it has the right number of fields, parses to finite values and carries the next expected identifier.
This is stronger than uniqueness and is convenient for a sequential synthetic file. An unordered source would need a different duplicate-detection scheme; a single expected-next counter cannot certify uniqueness of arbitrary identifiers. The contract also has a completeness boundary: the sequential check verifies that the rows read form the prefix , but it cannot detect a missing trailing suffix without knowing the expected total . To verify a complete fixed snapshot, compare the final count with a trusted expected . The demonstration separately checks its expected count 110.
Definition 4.1.2
(Centered streaming state)
For the first accepted pairs, define means and centered sums
The state consists of the count , the two means and the three centered sums.
A batch of at most parsed rows is an input buffer, rather than an additional mathematical sample. Changing should not change which records are accepted or the real-arithmetic target. It may change an implementation's arithmetic order if it changes the reduction strategy.
The fitted model is . Its descriptive loss is the sum of squared residuals over all accepted records. When , every accepted predictor is equal, so the intercept and slope cannot be separately identified. A valid program must report that degeneracy instead of dividing by zero. A positive but tiny also warns that the slope may be sensitive to perturbations.
Theorem/proposition
Proposition 4.1.3
(A one-record update for paired centered sums)
Suppose a nonempty state summarizes accepted pairs. For a new pair, let , , and . Set
Then the exact new sums are , , and . The first pair initializes its two means and three zero sums.
Theorem 4.1.4
(The fitted line and its residual certificate)
For and , the unique minimizer of is
These are exact identities for the accepted finite data. They do not by themselves supply an uncertainty interval or a causal interpretation.
Proof sketch or proof idea
Proof
(Proof of the one-record update)
Each old deviation shifts by the same amount: and . Expanding over the old records,
because deviations from the mean of exactly these records sum to zero. The new record contributes . Adding, the increment is . Since , this equals , which is the stated cross update. The square updates follow by setting the two coordinates equal. For the first pair, its own coordinates are the means and every centered sum is zero. By induction, after every accepted record the state equals the definitions applied to exactly the accepted prefix.
Proof
(Proof of the fitted-line theorem)
First hold fixed. With , the loss is minimized exactly at the mean of the , by completing the square in . Substituting it leaves , which expands to
Positive gives a unique slope and hence a unique intercept. The final term is a sum of squared residuals at that slope, proving nonnegativity. Equivalently, Cauchy–Schwarz applied to the two centered data vectors gives .
Concept lensGeometric
The residual as the uncorrelated part of the response
When , write , the cosine of the angle between the centered predictor and response vectors. The certificate becomes : the fitted line removes the share of the centered response energy, the squared length of its projection onto the centered predictor. The bound is the Cauchy–Schwarz step of the proof, and zero residual means exactly . In the synthetic example below, .
The algebra in the first proof is short; what deserves attention is the condition under which it may be applied.
Proof X-ray
Three invariants with different jobs
The key move in the update proof is that the old deviations sum to zero. That is true only because the stored means are the means of exactly the records already accepted, so the proof is an induction whose hypothesis is the algebraic invariant: the summary describes the accepted prefix. Two further invariants are needed before this algebra means anything. The schema invariant says every accepted item is a valid finite pair; a NaN would make every later sum NaN, and no identity could repair it. The identifier invariant says the accepted sequence is the intended prefix, without omissions or duplicates. The recurrence establishes the algebraic claim conditional on the other two; it does not validate a file format or detect a duplicated input on its own. In the second proof, positivity of the predictor spread is what allows the division and makes the square in the slope strictly determine its minimizer.
Worked examples
Worked example 4.1.5
(An exact synthetic target before running any code)
Generate ten blocks, each with predictors . Within block , set , where for even and for odd . Each block's predictor sum is zero, and there are five blocks of each sign. Thus , , , and . The cross term involving vanishes, so . Also . The fitted line is exactly , and its residual sum is . This analytic answer lets us check the implementation against an independent calculation.
The following complete program creates and removes its own temporary file. The file is written a record at a time; neither generation nor analysis loads all records into memory. Copy the block into a Python file and run it as one program.
import csv
import math
import tempfile
from itertools import islice
from pathlib import Path
def fit_file(path, batch_size=8):
if batch_size < 1:
raise ValueError("batch_size must be positive")
n = 0
mx = my = sxx = sxy = syy = 0.0
with open(path, newline="", encoding="utf-8") as handle:
rows = csv.reader(handle)
if next(rows, None) != ["row_id", "x", "y"]:
raise ValueError("unexpected schema")
while True:
batch = list(islice(rows, batch_size))
if not batch:
break
for row in batch:
if len(row) != 3:
raise ValueError("wrong field count")
rid, xs, ys = row
if int(rid) != n:
raise ValueError("missing, repeated or reordered id")
x, y = float(xs), float(ys)
if not (math.isfinite(x) and math.isfinite(y)):
raise ValueError("nonfinite measurement")
dx, dy = x - mx, y - my
n += 1
mx += dx / n
my += dy / n
sxx += dx * (x - mx)
sxy += dx * (y - my)
syy += dy * (y - my)
if n < 2 or sxx <= 0:
raise ValueError("line is not identified")
slope = sxy / sxx
return n, my - slope * mx, slope, syy - sxy * slope
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / "measurements.csv"
with open(path, "w", newline="", encoding="utf-8") as handle:
writer = csv.writer(handle)
writer.writerow(["row_id", "x", "y"])
for i in range(110):
x = i % 11 - 5
noise = -1 if (i // 11) % 2 == 0 else 1
writer.writerow([i, x, 2 + 3 * x + noise])
answer = fit_file(path)
expected = (110, 2.0, 3.0, 110.0)
assert answer[0] == expected[0]
assert all(math.isclose(a, b, rel_tol=1e-11, abs_tol=1e-11)
for a, b in zip(answer[1:], expected[1:]))
print(answer[0], *(round(value, 8) for value in answer[1:]))
It prints 110 2.0 3.0 110.0. The tolerance acknowledges floating-point evaluation; the exact design proof supplies the reference. The numeric parsers permit forms such as surrounding whitespace and digit underscores. This example validates converted values and field positions rather than a canonical textual spelling of numbers. Parse failures deliberately stop the calculation. Silently skipping bad rows would define another dataset, so that policy would need its own accepted/rejected counts and target interpretation.
What a second pass can certify
There are two useful ways to compute a residual after fitting. The sufficient-statistic identity subtracts explained centered energy from total centered energy. A second pass instead evaluates each residual of the computed coefficients and sums its square directly. The latter avoids subtracting two potentially large energy terms when their difference is small. It measures the loss of the coefficients actually returned, which need not equal the exact minimum after rounding. A careful summation of those nonnegative terms can therefore be an independent numerical diagnostic, even though both calculations have the same real-arithmetic target when the coefficients are exact.
The second pass must revisit the same accepted data. A file replaced between passes, a different missing-value rule, or a changing input stream invalidates the comparison. For a continuing stream, retain an input snapshot or record a finite prefix with a stable version identifier before making a two-pass claim. Extra input work is the price of this diagnostic; constant summary memory does not mean every useful verification is free.
Slope sensitivity can also be quantified directly. Suppose the computed centered sums differ from their exact values by in and in , with . Subtracting the two ratios gives
The small denominator explains why a near-constant predictor is troublesome even when the accumulators have small absolute errors. Recentering the predictor changes its coordinate origin and can improve intermediate arithmetic; it does not create variation that the data lack. Rescaling predictor units changes the numerical slope and its units as well, so a reported coefficient should always accompany the transformation used to define its input. An intercept can become large after an input shift while the fitted predictions remain the same.
Worked example 4.1.6
(Memory and precision are separate budgets)
With a batch bound , the analysis holds at most CSV rows plus a fixed-sized numerical state. Its working memory is records and its arithmetic work is . The temporary input file occupies disk space. Actual bytes depend on field widths and language object overhead, so six accumulators do not imply a six-number total memory footprint.
Adding to every predictor changes the fitted intercept but leaves the exact centered sums and slope unchanged. The recurrence avoids subtracting two raw second moments of order . It does not prove perfect accuracy: input rounding, overflow, ill-conditioned predictor variation and accumulated updates still matter. More precision or a second pass may be justified when a small residual is obtained by subtracting two large fitted-energy terms.
Worked example 4.1.7
(A file can be well formed and mathematically uninformative)
Take accepted pairs . The file can satisfy the schema and identifier rules, yet . Every line with makes the same fitted predictions. The algorithm correctly refuses to return an identified slope. Replacing the zero denominator by a tiny constant would invent a parameter choice and disguise a missing-information problem as a numerical calculation.
Common mistakes
Batching changes how records are buffered, not which problem is solved. The commonest error is to combine batch answers instead of batch statistics.
Counterexample mode
Averaging batch slopes
Claim: the slope of the pooled data is the average of the slopes fitted to each batch. Take batch one and batch two . Each batch has slope , so the average is . The pooled data have , , centered predictors and centered responses , so , and the pooled slope is . Each slope solves a different fitted problem, and some batches may even have constant predictors. The repair is to combine raw or centered sufficient statistics, never fitted coefficients; the next unit develops that reduction.
Two further errors concern what a check actually certifies.
Common mistake
(Checking only the number of accepted rows)
A final count of does not detect a duplicate paired with an omission: identifiers also give four rows. Identity information supplies a distinct contract, which the ordered identifier check enforces record by record.
Common mistake
(Clamping a negative residual to zero)
The computed expression for the residual can become slightly negative through rounding, although its exact value is nonnegative. Do not erase a substantial negative result with an unconditional maximum against zero. Investigate the scale and numerical method, and recompute residuals in a second pass when needed.
Input validation also does not make the chosen linear model adequate: a systematic curved residual pattern can remain after every code assertion passes.
Summary
A useful mathematical pipeline specifies records, acceptance rules, state invariants, resource costs and an independent target calculation. Centered paired sums lead to a one-pass fitted line when predictor variation is positive. The program demonstrates that entire path on synthetic data while leaving uncertainty, model adequacy and severe numerical conditioning as separate questions.
Exercises
Quick check
(For pairs , compute all five mean/centered statistics, the fitted line and the residual sum. Why is the slope identified?)
Center the data first.
Solution · Answer
The means are , . Centered predictor values are and centered responses are , giving , , . Hence , and the minimum residual is . Positive makes the two parameters unique; zero residual alone would not establish this if every predictor were equal.
Quick check
(The expected identifiers are , but a file contains . Explain what the program does, and why merely checking a final count of four would be insufficient.)
Follow the expected-identifier counter record by record.
Solution · Answer
After two records the expected identifier is two, so the second occurrence of one raises an error before being accumulated. A final count alone can match even though one intended record is missing and another appears twice. This program uses ordered identifiers; its guarantee should not be advertised for an unordered input without changing the checking method.
Quick check
(Replace every response by . Prove how the means, centered sums, fitted intercept, slope and minimum residual change.)
Track what happens to each centered response.
Solution · Answer
The new response mean is , while predictor mean is unchanged. Every centered response is unchanged, so all three centered sums are unchanged. Thus the slope stays , the intercept becomes , and every residual remains the original residual. The minimum residual sum is therefore unchanged. This is an exact translation-equivariance statement, not a promise of bitwise equality after floating-point input conversion.
Quick check
(For a billion fixed-width records, compare and in asymptotic arithmetic, buffer memory and statistical target. Does either batch size prove that a nearly constant predictor yields an accurate slope?)
Separate arithmetic work, buffer memory and the target problem.
Solution · Answer
Both perform arithmetic and read all records. Their buffers hold at most 100 and 10,000 records, respectively, a factor of one hundred apart; numerical summary size stays constant. They solve the same finite-data problem under the same acceptance and update order. Predictor near-degeneracy is a conditioning issue, so neither buffer choice supplies a slope-accuracy theorem. A larger buffer may reduce input overhead, but that is a performance question requiring measurement.