This is original self-study exposition, not an official examination, marking scheme, or weighting forecast. Follow each calculation on paper before using code to check it.
1. Identify the inputs, unknowns, and outputs
An observation is one recorded case. Let n be the number of observations and d the number of original features. The jth feature of observation i is xᵢⱼ; its observed response is yᵢ. Subscripts are labels, not multiplication. Our final task is to return a prediction ŷ for a new input, not merely an error score.
An ordinary affine model is ŷᵢ=a+Σⱼbⱼxᵢⱼ. The intercept a is the prediction when all features are zero. Holding the other features fixed, increasing feature j by one unit changes the prediction by bⱼ. If several inputs change together, one coefficient alone does not describe the total change.
The design matrix M has one observation per row and one model term per column. Its p columns match the order of the p coefficients in c. For a+bx, a row is [1,x] and c=[a,b]ᵀ. For a+bx+qx², a row is [1,x,x²]. Thus M is n×p, c is p×1, and Mc is n×1. Only an ordinary affine model with an intercept and all original features has p=d+1. A quadratic in one original feature already has three design columns.
The residual is r=y−Mc: observation minus prediction. RSS=Σᵢrᵢ²=‖y−Mc‖₂². Squaring prevents positive and negative residuals from cancelling and penalizes large residuals more heavily. An argmin returns minimizing parameters; it does not return the minimum loss value.
2. Fit once, without hiding the elimination
Use (0,1),(1,2),(2,4). For ŷ=a+bx, M has rows [1,0],[1,1],[1,2] and y=[1,2,4]ᵀ.
RSS=(1−a)²+(2−a−b)²+(4−a−2b)².
Differentiating with respect to a and setting the result to zero gives −2[(1−a)+(2−a−b)+(4−a−2b)]=0, or 3a+3b=7. Differentiating with respect to b gives −2[(2−a−b)+2(4−a−2b)]=0, or 3a+5b=10.
Subtract the first equation from the second: 2b=3, so b=3/2. Substitution gives 3a=7−9/2=5/2, hence a=5/6. The fitted model is ŷ=5/6+(3/2)x.
Its training predictions are 5/6,7/3,23/6. Residuals are 1/6,−1/3,1/6, so RSS=1/36+1/9+1/36=1/6. Check their sum: 1/6−1/3+1/6=0. Check their inner product with x: 0(1/6)+1(−1/3)+2(1/6)=0. These are residual orthogonality to the intercept column and feature column, respectively.
MᵀMc=Mᵀy is just a compact form of the same equations. Write c=(MᵀM)⁻¹Mᵀy only when M has full column rank. For a counterexample, two identical columns allow different coefficient pairs with the same sum, hence the same prediction. The parameters need not be unique. QR or SVD can solve least squares without explicitly forming an inverse.
3. LOOCV: remove the observation before fitting again
Compare a constant model ŷ=a with an affine model ŷ=a+bx. Fitting on all observations and then scoring each one in turn computes training error, not leave-one-out cross-validation.
LOOCV takes development data, candidate specifications, and a scoring loss. Here each fold holds out one observation and fits on the other two. Average the three held-out squared errors; smaller is better. A held-out response is available for scoring, not for fitting that fold.
Why is the constant prediction the training mean? With m training responses, minimize their squared deviations from one real constant a. Differentiate the sum, set the derivative to zero, and divide by the positive count m. The second derivative is positive, so this is the unique minimum, not merely a stationary point. The mean is computed anew from each fold’s training responses, never from its held-out response.
Fold 1: hold out (0,1)
The training points are (1,2),(2,4). The constant least-squares prediction is their response mean, (2+4)/2=3. At x=0 its squared error is (1−3)²=4.
The line has slope (4−2)/(2−1)=2. From 2=a+2×1, the intercept is zero. The fold's line is ŷ=2x; it predicts 0 at x=0, giving squared error 1.
Fold 2: hold out (1,2)
The training points are (0,1),(2,4). The constant prediction is (1+4)/2=5/2, giving (2−5/2)²=1/4. The line has slope (4−1)/(2−0)=3/2 and intercept 1. At x=1 it also predicts 5/2, with squared error 1/4.
Fold 3: hold out (2,4)
The training points are (0,1),(1,2). Their constant prediction is 3/2, with squared error (4−3/2)²=25/4. Their line is ŷ=1+x. It predicts 3 at x=2, with squared error 1.
| Held-out x | Constant prediction | Constant squared error | Line prediction | Line squared error |
|---|---|---|---|---|
| 0 | 3 | 4 | 0 | 1 |
| 1 | 5/2 | 1/4 | 5/2 | 1/4 |
| 2 | 3/2 | 25/4 | 3 | 1 |
Constant LOOCV=(4+1/4+25/4)/3=7/2=3.5. Line LOOCV=(1+1/4+1)/3=3/4=0.75. Select the affine model specification, not a winning fold's coefficients.
Now refit on all three development observations, obtaining ŷ=5/6+(3/2)x as above. Do not arbitrarily deploy 2x,1+1.5x, or1+x. For new input x=3, the refitted prediction is 5/6+9/2=16/3. This is not a known true response. Without its observed y, no prediction error can be computed at that point.
If a final test set exists, keep it outside model selection, feature selection, and λ tuning. A test score repeatedly consulted to revise a model is not an independent final evaluation. In this three-point example there is no separate test set; LOOCV must not be relabelled as an independent final-test result.
4. Scores, fold sizes, and leakage
The training response mean is 7/3. TSS=(1−7/3)²+(2−7/3)²+(4−7/3)²=(16+1+25)/9=14/3. The refitted training R² is 1−(1/6)/(14/3)=27/28. This high training score is not the same quantity as LOOCV0.75: one is a dimensionless comparison to the training-mean baseline, the other an average held-out squared prediction error.
For m evaluated observations, MSE=RSS/m, RMSE=√MSE, and MAE=Σ|rᵢ|/m. RMSE has the units of y. Here training MSE=1/18 and MAE=2/9. A single held-out observation has TSS=0, so ordinary R² is undefined for that fold; do not average such foldwise R² values. Test R² may be negative: true values[2,4] with predictions [6,6] give RSS20,TSS2,R²−9.
For unequal fold sizes, a per-observation MSE pools squared errors before dividing by the total number of evaluated observations. A one-point fold with MSE9 and a three-point fold with MSE1 give (1×9+3×1)/4=3, not (9+1)/2=5. Equal-fold and equal-observation weighting answer different questions.
Every learned preprocessing step belongs inside training. In each fold, estimate means, standard deviations, missing-value rules, and feature selection from the training partition alone, then apply those fixed rules to validation. Only the final refit re-estimates them from all development data. Computing a mean on the full dataset before splitting lets held-out observations influence the fitted coordinate system: leakage can occur without directly using their responses.
Why a k-fold training partition has n(k−1)/k observations
Here n counts development observations and k counts folds, not features or parameters. Divide the observations into k disjoint, approximately equal validation groups. In one round exactly one group is held out; its complement is the training partition. If n is divisible by k, the held-out group has n/k observations, so subtract this count from n. For n=100 and k=5, hold out 20 and train on 80 in each round; run all five rounds so each observation is validated once. For n=103 and k=5, a balanced partition has group sizes 21,21,21,20,20, hence the training sizes are 82,82,82,83,83. The expression n(k−1)/k=82.4 is then an average or approximation, not a possible fractional sample count. LOOCV is the special case k=n and trains on n−1 observations per round.
Monte Carlo validation: repeated random splits, not input sensitivity
Choose the candidate models or λ values, a loss, a training/validation ratio, and a number of repetitions before examining outcomes. For each repetition draw a fresh random split of the development observations, for example 80 training and 20 validation observations from n=100. Different repetitions may reuse observations, so these are not disjoint k-fold groups. Use the same split for all candidates in a repetition to make their comparison fair. This simple random-split procedure assumes exchangeable observations; time-ordered or grouped observations require an appropriate time/group-respecting split.
Within each repetition: (1) estimate means, scales and any other learned preprocessing only on its 80 training observations; (2) freeze and apply those rules to both partitions; (3) fit each candidate using those 80 training observations; (4) predict the 20 validation responses and calculate the stated validation loss. Store a loss for every candidate on every split. Average each candidate’s validation losses over repetitions, then select the candidate with the smaller average. Do not select the single luckiest split or take its fitted coefficients as the final model.
For a small bookkeeping example, suppose the validation MSEs for λ=0 are 4,9,5 over three repetitions and those for λ=2 are 6,5,4. The averages are 6 and 5, so select λ=2, even though λ=0 achieved 4 on the first split. These are illustrative recorded losses, not newly claimed empirical results. After selection, re-estimate preprocessing and refit the λ=2 model on all 100 development observations. Evaluate a separate final test set only after the choices are fixed; its observations and responses must not participate in split selection, λ choice or preprocessing estimates.
Monte Carlo validation resamples training/validation partitions and studies out-of-sample loss for model selection. Monte Carlo input sensitivity instead perturbs an input or uncertain quantity and propagates that perturbation through a model. Section 8 illustrates input sensitivity with fixed perturbations. They ask different questions. A lower LOOCV or repeated-split score means better performance in this specified comparison on these observed data; it does not guarantee the genuinely best population model, the best untested candidate, or the smallest future error. Reusing the same validation evidence to make many choices can itself overfit that evidence.
5. Undo standardization: restore the response first
Let zⱼ=(xⱼ−μⱼ)/sⱼ, where μⱼ and nonzero sⱼ are training-feature statistics. If t=(y−μᵧ)/sᵧ and the fitted model is t̂=β₀+Σβⱼzⱼ, first restore the response: ŷ=μᵧ+sᵧt̂.
Substitution gives ŷ=μᵧ+sᵧβ₀+Σ(sᵧβⱼ/sⱼ)(xⱼ−μⱼ). Separate variable terms from constants: bⱼ=sᵧβⱼ/sⱼ and a=μᵧ+sᵧβ₀−Σbⱼμⱼ. If y was not transformed, use μᵧ=0,sᵧ=1; do not introduce a response standard deviation that was never used.
Example: μₓ=4,sₓ=2,μᵧ=10,sᵧ=5,β₀=0,β₁=2. Then b=5×2/2=5 and a=10−5×4=−10. The restored model is ŷ=−10+5x. At x=6 it gives 20. Alternatively z=(6−4)/2=1,t̂=2, then ŷ=10+5×2=20. Agreement checks the entire restoration.
The incorrect intercept 10 would predict40 at x=6: it fails to subtract the 20 contributed by input centering. A zero-variance feature has sⱼ=0 and cannot be divided by its standard deviation; remove the redundant constant feature or specify a constant-column rule. Standardization is not unit-norm scaling: under the population-variance convention, n standardized values have squared sum n, not 1.
6. Ridge: write the full objective before differentiating
Specify J(c)=‖y−Mc‖²+λcᵀDc, λ≥0, with diagonal penalty matrix D. Penalizing every coefficient means D=I. Leaving the first coefficient, an intercept, unpenalized means D=diag(0,1,…,1). λ is not implicitly1, and missing original matrices do not justify invented numerical answers.
The gradient is 2Mᵀ(Mc−y)+2λDc. Setting it to zero gives (MᵀM+λD)c=Mᵀy. An inverse formula requires invertibility. For λ>0 and positive-semidefinite D, the quadratic form is ‖Mv‖²+λvᵀDv. Uniqueness holds exactly when Null(M) and Null(D) intersect only at zero. Full penalization ensures this; unpenalized directions must still be identified by the data.
A fully specified fresh example uses x=[−1,0,1],y=[0,1,2], model a+bx, and objective RSS+2b², with no intercept penalty. The design columns are orthogonal: MᵀM=diag(3,2),Mᵀy=[3,2]ᵀ. Thus 3a=3 and (2+2)b=2, yielding a=1,b=1/2. Predictions are [1/2,1,3/2],RSS=1/2,penalty=2(1/2)²=1/2,total objective 1. OLS gives a=1,b=1 and RSS 0, but its value under the same ridge objective is 2. Ridge sacrifices fit to reduce its penalty; this is not a calculation error.
If the intercept is also penalized, the first equation becomes 5a=3 and a=3/5: a different problem. For RSS/n+α‖c‖², multiply the whole objective by n to get λ=nα. Matching parameter names or numerical values does not establish matching penalty strength. With orthogonal columns of squared norms dⱼ, the coordinate shrinkage factor is dⱼ/(dⱼ+λ). It is a common factor only when those norms agree; correlated columns cannot use the formula coordinate by coordinate.
7. Lasso: do not omit zero
Consider F(c)=(c−3)²+λ|c|,λ≥0. For c>0, the derivative is 2(c−3)+λ and its candidate is 3−λ/2, positive only for λ<6. For c<0, the derivative is 2(c−3)−λ, giving3+λ/2, which contradicts c<0. Thus no negative interior candidate is valid.
For λ>0, the ordinary derivative at zero does not exist. When λ=0 the penalty vanishes and the minimizer is c=3. The left derivative is −6−λ≤0; the right derivative is −6+λ≥0 exactly when λ≥6. The complete solution is c*=max(3−λ/2,0). Values λ=0,2,6,8 give 3,2,0,0; include the boundary 6 in the zero branch. Convexity makes these conditions globally sufficient.
By contrast, ridge (c−3)²+λc² has solution 3/(1+λ), nonzero for every finite λ in this example. Lasso reaches exact zero. This does not mean every ridge coefficient in every dataset is necessarily nonzero.
If the objective becomes (1/2)(c−3)²+λ|c|, the positive candidate is 3−λ and the zero threshold is 3. Only the squared-loss factor changed, not the entire objective, so the minimizer changes. More generally, for a nonzero column m and target residual r, minimizing ‖r−mc‖²+λ|c| gives S(z,λ/2)/d, where d=mᵀm>0,z=mᵀr and S(z,t)=sign(z)max(|z|−t,0). With nonorthogonal design columns, this is a coordinate subproblem, not permission to independently threshold all original correlations and call the result a global solution.
8. Sensitivity: state what changes and what stays fixed
Fix ŷ=5/6+1.5x and change only a new input. The exact change is Δŷ=1.5Δx. Moving x from 3 to 3.1 changes the prediction from 16/3 to 329/60, an increase 0.15. No refitting occurs.
Now instead change the third training response from 4 to 4+ε and refit with the same design. The input mean remains 1. The slope numerator increases byε while its denominator is 2, giving b=3/2+ε/2. The response mean increases byε/3; thereforea=5/6−ε/6. The new prediction at x=3 is 16/3+4ε/3. A training-response changeε can produce a larger prediction change, especially in extrapolation. The old slope 1.5 does not describe this experiment.
Near-collinear features can make coefficients sensitive to small perturbations even when training predictions are relatively stable. Ridge often stabilizes penalized directions but does not guarantee lower test error. A sensitivity report must specify the perturbed quantity, its magnitude, whether fitting is repeated, and whether coefficients or predictions are measured.
9. Check your understanding
Checkpoint
Predict at x=8 using both the standardized and restored models in Section 5.
Solution · Solution 1
At x=8 in Section 5, z=2,t̂=4,ŷ=30. The restored formula also gives −10+5×8=30.
Checkpoint
Minimize (c−3)²+4|c| and compare its value with the zero candidate.
Solution · Solution 2
For (c−3)²+4|c|, the threshold is 2 and c*=1. Its objective is (1−3)²+4=8, below the zero candidate's9.
Checkpoint
What happens to the Section 6 ridge fit when λ=0? What does this establish about future prediction?
Solution · Solution 3
Setting λ=0 in Section 6 gives a=1,b=1,predictions [0,1,2],RSS 0. Perfect training fit alone does not establish error-free prediction on unknown data.
Checkpoint
Compare changing a new input from 3 to 2.8 with lowering the third training response by 0.2 and refitting.
Solution · Solution 4
With the final fixed model, changing a new input from 3 to 2.8 gives Δŷ=1.5(−0.2)=−0.3. Changing the third training response by −0.2 and refitting instead changes the prediction at x=3 by (4/3)(−0.2)=−4/15. These are different questions.
fx-50FH II: check arithmetic without skipping the mathematics
Use ordinary scalar arithmetic on the fx-50FH II here: write the full expression before executing it. For constant LOOCV, enter (4+1/4+25/4)/3 and expect 3.5. For line LOOCV, enter (1+1/4+1)/3 and expect 0.75. For the final prediction, enter 5/6+(3/2)*3 and expect 16/3 (approximately 5.333333333). For training R², enter 1-(1/6)/(14/3) and expect 27/28 (approximately 0.9642857143). Here * means multiplication; keep complete denominator brackets.
Obtaining 0.75 does not establish that each fold used only its two training observations. A calculator cannot detect leakage. Derive the coefficients and residuals on paper, then use scalar arithmetic for checking. Ordinary single-feature REG output is not automatically a ridge, lasso, or multivariable-regression solution. These checks require neither CAS nor full matrix functionality.