A quadratic term keeps a variable near an observation, while an absolute-value penalty can set it exactly to zero. Their balance leads to soft thresholding. We derive the rule at and away from the nondifferentiable point, then prove that the map does not amplify perturbations.
The scalar problem
Definition 9.1.1
(The scalar soft-thresholding problem)
For and , consider
Write for the objective. The number is the observation and is the threshold.
The objective is continuous and tends to as , so a minimizer exists. The quadratic is strictly convex and is convex, hence their sum is strictly convex and the minimizer is unique. Thus is a well-defined number, and the task is to compute it.
Theorem 9.1.2
(Soft-thresholding formula)
For every and every ,
where .
In particular . A large observation is moved toward zero by ; an observation within the threshold, including either boundary , is mapped to zero.
Why the zero branch is an interval
The proof uses the convex subdifferential of and the fact that, for a convex function, belongs to its subdifferential exactly at a minimizer.
Proof
(Three cases of the optimality condition)
The convex subdifferential of is for , for , and at . The last claim follows directly from for all : this holds exactly when .
The convex optimality condition is
If , the equation gives , consistent exactly when . If , it gives , consistent exactly when . At , the condition is , or . These three cases cover every and include ; uniqueness makes the piecewise rule the minimizer.
Proof X-ray
The kink supplies a range of slopes
Away from zero the objective is differentiable, and the equation produces one candidate. That candidate must still be checked against the sign that was assumed, and this sign check is what produces the strict conditions and . The key move happens at : the subdifferential is the whole interval , so the equation becomes the inclusion , and an entire interval of observations shares the minimizer zero. The three conditions , and partition the real line, so each observation has exactly one consistent case. The argument closes because the optimality condition is sufficient as well as necessary, and that sufficiency is where is used: it makes the objective convex.
The sign condition on the threshold cannot be dropped, even though the objective still has a minimizer when it fails.
Counterexample mode
A negative threshold breaks the formula
One might try the same formula with . At the objective is , and the formula predicts . But while , so zero is not a minimizer; the two minimizers also show that uniqueness fails. Here is not convex, so the inclusion no longer certifies a minimum. The formula is restored only under the stated hypothesis .
A perturbation cannot grow
Theorem 9.1.3
(Soft thresholding is nonexpansive)
For every and all ,
Proof
(Monotone subgradients and one subtraction)
Let and . The optimality conditions give and for some , . The subgradient inequalities and , added together, imply
Subtracting the optimality equations and using gives
If , the desired inequality is immediate. Otherwise take absolute values and divide by to obtain
This is a stability statement for the map, not a claim that every iterative optimization algorithm using it converges. It also includes without division by .
A vector calculation
For , the objective is a sum of independent scalar objectives. Its unique minimizer is obtained by applying to each coordinate. Coordinatewise nonexpansiveness gives the vector estimate after squaring and summing. No claim about a general coupled quadratic is implied by this separable calculation.
Worked example 9.1.4
(Three coordinates, one of them inside the threshold)
For and , the first two coordinates exceed the threshold in absolute value and the third does not. Hence
Its objective value is : the quadratic part is and the penalty is , and .
Worked example 9.1.5
(An observation exactly at the threshold)
Take and . The formula gives . Directly, , and for
while for both terms of are larger than at zero. So is the unique minimizer. The positive-branch expression also equals here, but the case is not consistent at equality; the boundary point belongs to the zero branch.
Worked example 9.1.6
(When the nonexpansive bound is an equality and when it is strict)
Let . For and , both lie on the positive branch, so : the bound is an equality. For and , the outputs are and , so
In general, if and , then : crossing the zero branch removes exactly its width .
Checks and exercises
Common mistake
(Shrinking by the threshold without the maximum)
Writing for every is wrong inside the threshold. With and it gives , which even changes sign; yet while . The maximum with is exactly the zero branch .
Common mistake
(Confusing soft thresholding with hard thresholding)
The rule “keep if , otherwise set it to ” is hard thresholding, and it does not minimize this objective. With and it returns , but while . The absolute-value penalty shrinks surviving values by as well as removing small ones.
Quick check
(What happens at ?)
Compare the zero branch with the positive branch at equality.
Solution · Answer
The output is . The positive-branch expression equals there, but is not valid at equality, so the zero branch applies.
Quick check
(For , what is the map?)
Set the penalty to zero in the objective.
Solution · Answer
It is the identity, , and the nonexpansive inequality is an equality.
Quick check
(Why is applying the scalar rule separately invalid for an arbitrary quadratic with cross terms?)
Ask whether the objective is still a sum of one-variable functions.
Solution · Answer
Changing one coordinate then changes terms involving other coordinates, so the objective need not separate into independent scalar problems.
Quick check
(Does the threshold rule mean a small nonzero output is impossible?)
Try an observation slightly above the threshold.
Solution · Answer
No. If with , the output is .
The derivation assumes the stated Euclidean quadratic and nonnegative penalty.