Binary Logistic Regression: Likelihood, Convexity and Finite-Data Existence
How the logistic model turns affine scores into model probabilities, why its negative log-likelihood is a smooth convex objective, how to evaluate it stably, and sufficient conditions and examples for existence or nonexistence of a finite minimizer.
§5.2Estimated reading time: 30 min
From scores to a probability model
An affine classifier computes a score z=aTθ and predicts by its sign. The perceptron uses only that sign. Logistic regression goes further. It treats the score as the input to a probability model, and then it chooses θ by maximizing the likelihood of the observed labels. This note derives the resulting objective, computes its derivatives, and examines a question that is easy to overlook: whether a finite best parameter exists at all.
Setting. Fix integers n,d≥1, feature vectors a1,…,an∈Rd, and observed labels y1,…,yn∈{0,1}. Let A∈Rn×d be the matrix whose i-th row is aiT. If an intercept is wanted, one coordinate of every ai is set to the constant 1, so the intercept is simply one coordinate of θ. Write zi=aiTθ. Throughout, log is the natural logarithm.
Definition 5.2.1
(Sigmoid and softplus)
The sigmoid and softplus functions R→R are
σ(z)=1+e−z1,softplus(z)=log(1+ez).
The sigmoid maps every score into the open interval (0,1), which is what allows it to be read as a probability.
Definition 5.2.2
(Logistic regression model)
For a parameter θ∈Rd, the logistic regression model postulates
P(Yi=1∣ai)=σ(aiTθ).
This statement is an assumption, and it should be read with care. The number σ(zi) is the probability that the model with parameter θ assigns to the label 1. Nothing in the data proves that the real data-generating process has this form. A value of σ(zi) close to 1 expresses strong confidence within the model. It is not certainty about the world.
The likelihood and the loss
To write the likelihood as a product, we assume that Y1,…,Yn are conditionally independent given the fixed design, under the model. Because each yi is 0 or 1, the probability of the observed label can be written as σ(zi)yi(1−σ(zi))1−yi. One of the two factors equals 1, and the other is the correct probability. The likelihood is therefore
L(θ)=i=1∏nσ(zi)yi(1−σ(zi))1−yi.
Two elementary identities simplify the logarithm. The first is 1−σ(z)=1+e−ze−z=1+ez1=σ(−z). The second is logσ(z)=−log(1+e−z)=−softplus(−z). Together they give log(1−σ(z))=−softplus(z). The negative log-probability of one label, as a function of its score, is therefore
ℓ(z,1)=softplus(−z),ℓ(z,0)=softplus(z).
Next, softplus(−z)=logezez+1=softplus(z)−z. Both cases therefore merge into a single formula:
ℓ(z,y)=softplus(z)−yz.
Averaging the per-example losses gives the negative average log-likelihood
L(θ)=−n1logL(θ)=n1i=1∑n[softplus(zi)−yizi].
Once L has been written down, it is a deterministic function of θ for the given list of features and labels. Everything below analyzes that function. None of it uses independence, identically distributed sampling, or any claim that the model is correctly specified. The conditional likelihood interpretation uses conditional independence and the stated model; it does not require identical response probabilities across different features.
The coefficients are fitted to the training data only. Minimizing L guarantees nothing about calibration on new cases or about out-of-sample accuracy. The loss is also unbounded: ℓ(z,1)=softplus(−z)≥−z, so a confidently wrong score can incur an arbitrarily large loss on a new example.
Stable evaluation. Computing log(1+ez) literally overflows in floating point when z is large and positive. Split by sign. If z≥0, then log(1+ez)=z+log(1+e−z). If z<0, the expression is already safe. Both cases are covered by
softplus(z)=max(z,0)+log(1+e−∣z∣),
and hence by
ℓ(z,y)=max(z,0)−yz+log1p(exp(−∣z∣)).
Here the exponential never receives a positive argument.
Common mistake
(Evaluating softplus literally)
Coding log(1+exp(z)) as written fails at both ends. For large positive z the exponential overflows. For large negative z the sum 1+ez rounds to exactly 1, the logarithm returns 0, and the small positive loss is lost. The split form above fixes the first problem, and log1p, which evaluates log(1+u) accurately for tiny u, fixes the second.
Worked example 5.2.3
(The loss at extreme scores in double precision)
Double precision overflows above about 1.8×10308, that is, for ez with z above about 709.8, and its relative spacing near 1 is about 2.2×10−16.
Take z=800 and y=0. The literal formula log(1+e800) overflows, since e800≈10347. The stable formula gives max(800,0)−0+log1p(e−800)=800, because e−800 underflows to 0, which is harmless here. The exact loss exceeds 800 by less than 10−347.
Take z=40 and y=1. The exact loss is softplus(−40)=log(1+e−40) with e−40≈4.25×10−18. Evaluated literally, 1+4.25×10−18 rounds to 1 and the result is 0. The stable formula gives 40−40+log1p(e−40)≈4.25×10−18, correct to full relative precision because log(1+u)=u−u2/2+⋯.
Every loss term is strictly positive, because each one equals softplus of some number. This gives L(θ)>0 for every θ. That fact is used repeatedly below.
Derivatives
Lemma 5.2.4
(Sigmoid derivative)
For every real z, σ′(z)=σ(z)(1−σ(z)), and 0<σ(z)(1−σ(z))≤41, with equality exactly at z=0. Moreover softplus′(z)=σ(z).
Proof
(Differentiating the sigmoid)
Differentiate σ(z)=(1+e−z)−1:
σ′(z)=(1+e−z)2e−z=σ(z)⋅1+e−ze−z=σ(z)(1−σ(z)).
Put p=σ(z)∈(0,1). Then p(1−p)>0, and p(1−p)=41−(p−21)2≤41, with equality iff p=21, that is, iff z=0. Finally, dzdlog(1+ez)=1+ezez=σ(z).
It follows that ∂zℓ(z,y)=σ(z)−y and ∂z2ℓ(z,y)=σ(z)(1−σ(z)). The first derivative is the model probability minus the observed label: a residual.
The Hessian satisfies 0⪯∇2L(θ)⪯4n∥A∥22I. Consequently L is convex.
Proof
(Chain rule and a weighted quadratic form)
The score is linear in θ, so ∇θzi=ai. The chain rule applied to ℓ(zi,yi) gives the gradient term (σ(zi)−yi)ai. Differentiating that term once more gives σ′(zi)aiaiT.
Write wi=σ(zi)(1−σ(zi)). For any v∈Rd,
vT∇2L(θ)v=n1i=1∑nwi(aiTv)2.
Every wi is positive, so the sum is at least 0. Every wi is also at most 41, so the sum is at most 4n1i∑(aiTv)2=4n∥Av∥2≤4n∥A∥22∥v∥2.
A function with positive semidefinite Hessian everywhere is convex.
Convexity is not the same as strict convexity. Suppose Av=0 for some v=0, which happens whenever A has rank below d. Then every score is unchanged along the direction v, so L(θ+tv)=L(θ) for every real t. Any minimizer, if one exists, is then part of a whole line of minimizers. Convexity also does not imply that a minimizer exists. The examples below show both failures.
Concept lensAlgebraic
Stationarity as moment matching
Setting the gradient to zero says AT(σ(Aθ)−y)=0: the vector of fitted probabilities minus labels is orthogonal to every column of A. Equivalently, i∑σ(zi)ai=i∑yiai, so the fitted total of each feature matches its observed total. With an intercept column of ones, the average fitted probability equals the training proportion of ones. When the distinct feature vectors are linearly independent, as in the worked examples below, matching these totals forces each group's fitted probability to equal its observed proportion.
The penalized objective
For λ>0, define
Lλ(θ)=L(θ)+2λ∥θ∥2,
where the penalty covers every coordinate of θ, including the intercept coordinate if there is one. Its Hessian is ∇2L(θ)+λI, so
λI⪯∇2Lλ(θ)⪯(λ+4n∥A∥22)I.
Theorem 5.2.6
(Unique finite penalized minimizer)
Let λ>0. Then Lλ has exactly one global minimizer θ⋆. It satisfies ∥θ⋆∥≤2log2/λ, and it is characterized by the stationarity condition ∇Lλ(θ⋆)=0, that is,
λθ⋆=−n1AT(σ(Aθ⋆)−y),
where σ acts on vectors entrywise.
Proof
(Existence by coercivity, uniqueness by curvature)
Existence. Since L≥0, we have Lλ(θ)≥2λ∥θ∥2. At θ=0 every score is 0 and softplus(0)=log2, so Lλ(0)=log2.
Consider the sublevel set S={θ:Lλ(θ)≤log2}. It contains 0, so it is nonempty. It is closed because Lλ is continuous. It is contained in the ball of radius 2log2/λ, so it is bounded. Hence S is compact, and the continuous function Lλ attains a minimum on S at some point θ⋆. Outside S the values exceed log2≥Lλ(θ⋆), so θ⋆ is a global minimizer.
Uniqueness. Take θ=θ′ and set g(t)=Lλ(θ+t(θ′−θ)). Then g′′(t)≥λ∥θ′−θ∥2>0, so g is strictly convex. If θ and θ′ were both minimizers, then g(21)<21(g(0)+g(1)), which is the minimum value. That is a contradiction.
Characterization. At any minimizer of a differentiable function the gradient vanishes. Conversely, if ∇Lλ(θ)=0, then each restriction g through θ is convex with g′(0)=0, so g(1)≥g(0), and θ is a global minimizer.
Proof X-ray
Where the penalty does the work
The data enter the existence argument only through two facts: L≥0 and L is continuous. Neither bounds a sublevel set. For separable data the set {L≤log2} contains a whole ray, as the section on separable data shows. The single inequality Lλ(θ)≥2λ∥θ∥2 is what traps every candidate in a ball; compactness of that ball then supplies the minimizer.
Uniqueness uses the penalty a second time, through the curvature floor ∇2Lλ⪰λI. Without it, a null direction v of A leaves L constant along θ+tv, and a minimizer comes with a whole line of others; the exercises exhibit such a line.
The characterization step uses neither use of λ: for any convex differentiable function a stationary point is a global minimizer. So for λ=0 it remains true that a zero of ∇L is a maximum-likelihood estimate; what may fail is that such a zero exists or is unique.
The theorem does not cover the common variant that leaves the intercept unpenalized.
Common mistake
(Quoting the theorem with an unpenalized intercept)
When the intercept is left out of the penalty, the lower bound 2λ∥θ∥2 fails along the intercept direction, and existence can fail. For instance, suppose all labels equal 1. Set every slope coordinate to 0 and let the intercept b tend to ∞. The objective equals softplus(−b), which is positive and tends to 0, so its infimum is never attained.
A gradient descent guarantee
Theorem 5.2.7
(Gradient descent with a fixed step)
Let λ>0, M=λ+∥A∥22/(4n) and η=1/M, and iterate θk+1=θk−η∇Lλ(θk). Write Lλ⋆=Lλ(θ⋆). Then
Take h=−g/M with g=∇f(θ). This yields f(θ+h)≤f(θ)−∥g∥2/(2M).
For f=Lλ, the Hessian bound ∇2f⪰λI gives f(θ⋆)≥f(θ)+gTu+2λ∥u∥2 with u=θ⋆−θ. Over all u, the right side is smallest when u=−g/λ, which gives f⋆≥f(θ)−∥g∥2/(2λ). Hence ∥g∥2≥2λ(f(θ)−f⋆).
Combining the two inequalities gives f(θk+1)−f⋆≤(1−λ/M)(f(θk)−f⋆). Iterating this proves the first bound. The same curvature bound applied at θ⋆, where the gradient is zero, gives f(θ)−f⋆≥2λ∥θ−θ⋆∥2.
For λ=0, sum the descent inequality over k=0,…,K. Since L>0, the sum of the decreases is at most L(θ0), so k=0∑K∥∇L(θk)∥2≤2M0L(θ0). The smallest of the K+1 terms is at most their average.
Worked example: an intercept-only model
Worked example 5.2.8
(Four labels, one constant feature)
Let n=4, d=1, ai=1 for every i, labels (0,1,1,1), and λ=0. Every score equals the scalar parameter z. Because i∑yi=3,
L is strictly convex, so it has at most one stationary point, and any stationary point is the unique global minimizer. Solve σ(z)=43:
1+e−z=34⟹e−z=31⟹z⋆=log3.
The fitted probability is σ(log3)=43, which is exactly the training proportion of ones. Since softplus(log3)=log4, the minimum value is
L(log3)=log4−43log3≈0.5623,
which is below L(0)=log2≈0.6931.
One gradient step. Here A is a column of four ones, so ∥A∥22=4 and M0=4/16=41. This matches the direct bound L′′≤41. Start at θ0=0 with step η=1, which is well below 1/M0=4:
L′(0)=21−43=−41,θ1=0−1⋅(−41)=41.
All four examples now receive the model probability σ(41)=1/(1+e−1/4)≈0.5622, up from 0.5.
Direct evaluation gives L(41)=softplus(41)−163≈0.82594−0.1875=0.63844. This is consistent with the bound.
The conflicting labels are what keep this minimizer finite. All four rows have the same feature vector, so no parameter can give them different scores. The best the model can do is one shared probability. These label ties are a property of the data. They should not be confused with a score tie. The decision rule "predict 1 iff σ(z)≥21", equivalently "iff z≥0", assigns a score of exactly 0 to the positive class by convention. That is a deterministic feature of the rule and says nothing about the labels. At the fitted θ⋆=log3, every row is predicted positive and the row labeled 0 is misclassified. For any θ, at least one row is misclassified because conflicting labels share the same score.
Counterexample mode
Convex, even strictly convex, yet no minimizer
Change the labels to (1,1,1,1) and keep ai=1 and λ=0. Then L(z)=softplus(−z). For every finite z this is positive, and L′(z)=σ(z)−1<0. As z→∞, L(z)→0. The infimum 0 is approached but never attained, so no maximum-likelihood estimate exists, even though L′′>0 everywhere.
Now add a penalty with λ>0:
Lλ(z)=softplus(−z)+2λz2,Lλ′(z)=λz−σ(−z).
The derivative Lλ′ is strictly increasing because Lλ′′=λ+σ(z)(1−σ(z))>0. At the left end, Lλ′(0)=−21. At the right end, Lλ′(2λ1)=21−σ(−2λ1)>0. So there is exactly one root zλ⋆∈(0,2λ1).
The root is defined implicitly by λz=1/(1+ez). It has no elementary closed form and is computed numerically. For λ=21 it is approximately 0.675, which gives a fitted probability of about 0.663.
Separable data in general
Definition 5.2.9
(Strict separability)
The training data are strictly separable if some θˉ∈Rd satisfies aiTθˉ>0 whenever yi=1 and aiTθˉ<0 whenever yi=0.
Proposition 5.2.10
(Strict separation rules out a finite minimizer)
If the training data are strictly separable, then θinfL(θ)=0, the infimum is not attained, and ∇L(θ)=0 for every θ.
Proof
(Scaling a separating direction)
Along the ray tθˉ, each label-1 term is softplus(−taiTθˉ), which tends to 0 as t→∞. Each label-0 term is softplus(taiTθˉ) with a negative score, which also tends to 0. So L(tθˉ)→0. Since L>0 everywhere, the infimum 0 is not attained, and the unpenalized problem has no finite minimizer.
Because L is convex and differentiable, a point where the gradient vanished would be a minimizer. So under strict separation ∇L never vanishes.
The converse is not claimed here. Non-existence of a minimizer does not by itself certify strict separability, and weaker degenerate configurations are not analyzed in this note.
Worked example 5.2.11
(One conflicting label per group restores a finite fit)
Use the intercept-and-slope features a=(1,−1)T and a=(1,1)T, three rows each. The first group has labels (0,0,1) and the second (0,1,1), so n=6. Neither group is pure, so no θˉ separates the data strictly.
Write θ=(b,c)T; the two scores are b−c and b+c. Each group contributes 3σ(z)−nones times its feature vector to n∇L, so the gradient vanishes when σ(b−c)=31 and σ(b+c)=32. Since σ(log2)=1/(1+21)=32 and σ(−log2)=31, this gives b−c=−log2 and b+c=log2, so θ⋆=(0,log2)T.
The two feature vectors are linearly independent and every weight σ(zi)(1−σ(zi)) is positive, so the Hessian is positive definite and θ⋆ is the unique minimizer. The fitted probabilities reproduce the group proportions, and the minimum value is −32log32−31log31=log3−32log2≈0.6365. Removing the minority label from both groups makes the data separable, and the minimizer disappears.
Fitted probabilities are not population probabilities
Counterexample mode
A confident fit to a fair coin
Suppose the population label really is a fair coin, independent of the single constant feature, so the population probability of 1 is 21. Three draws all come out 1, which happens with probability 81. Fit the penalized intercept model with λ=0.01.
The stationarity condition λz=σ(−z) has root z⋆≈3.36. Because σ(z⋆)=1−σ(−z⋆)=1−λz⋆, the fitted probability is about 1−0.0336≈0.966.
The model reports roughly 97% for an event whose population frequency is 50%. The optimization is correct; the training sample is simply unrepresentative.
As λ→0 the fitted probability tends to 1. Each root is positive. If some sequence λj↓0 had roots bounded by M, then λjzλj⋆≤Mλj→0, while σ(−zλj⋆)≥σ(−M)>0, contradicting the stationarity equation. Hence zλ⋆→∞. A high fitted probability is a property of the fit, not a calibrated statement about future cases.
Exercises
Exercise 5.2.12
(Let n=2, a1=(1,1)T with y1=1, and a2=(1,−1)T with y2=0. The first coordinate is the intercept. Compute ∇L(0) and ∇2L(0), and compare the Hessian with the bound ∥A∥22/(4n).)
Because: Both scores are 0, so σ=21 and each weight is wi=41.
2
Evaluate the Hessian
∇2L(0)=21⋅41[(1111)+(1−1−11)]=41I.
3
Compare with the bound
ATA=2I, so ∥A∥22=2 and ∥A∥22/(4n)=41. The bound is attained at θ=0, where every weight takes its maximum value 41.
Exercise 5.2.13
(Show that the data of the previous exercise are strictly separable, find L along a separating ray, and conclude that no maximum-likelihood estimate exists.)
Foundation
Solution(3)
1
Exhibit a separating direction
Take θˉ=(0,1)T. The scores are a1Tθˉ=1>0 for the label-1 example and a2Tθˉ=−1<0 for the label-0 example.
2
Evaluate L along the ray θ=(0,t)T
L=21[softplus(−t)+softplus(−t)]=softplus(−t), which tends to 0 as t→∞.
3
Conclude
The infimum 0 is not attained, and no maximum-likelihood estimate exists. Along the ray the gradient is (0,−σ(−t))T, which is never zero.
Because: L>0 everywhere.
Exercise 5.2.14
(Let n=4, let ai=(1,1)T for every i (a duplicated constant feature), and let the labels be (0,1,1,1). Describe the minimizers for λ=0. Then show that for every λ>0 there is a unique minimizer, and locate it.)
Core
Solution(4)
1
Reduce to one score
Every score equals s=θ1+θ2, so L=softplus(s)−43s. By the worked example, this is minimized exactly when s=log3.
2
Describe the unpenalized minimizers
For λ=0 the minimizers form the entire line θ1+θ2=log3. The rank of A is 1, and the null direction is (1,−1)T.
3
Solve the penalized stationarity condition
Key step
Stationarity reads (σ(s)−43)(1,1)T+λθ=0. This forces θ1=θ2=t with h(t)=λt+σ(2t)−43=0. The function h is strictly increasing, h(0)=−41 and h(4λ1)=σ(2λ1)−21>0, so there is a unique root t⋆∈(0,4λ1).
Because: The penalty covers both coordinates, so the theorem guarantees a unique minimizer.
4
Locate the minimizer
The minimizer is (t⋆,t⋆)T. Since σ(2t⋆)=43−λt⋆<43, we get 2t⋆<log3: the penalty shrinks the fitted score. The minimizer lies in the row space of A, as the stationarity equation λθ⋆=−ATr/n requires.
Exercise 5.2.15
(Models P and Q give four cases the scores (−1,0,1,2) and (−2,0,2,4) respectively. Compare their rankings and their decisions at probability thresholds 21 and 54. Then explain where the threshold 54 comes from when a false positive costs 4 and a false negative costs 1.)
Core
Solution(4)
1
Compare the rankings
Q's scores are twice P's, so both models rank the four cases identically.
2
Apply the threshold 21
The rule "predict positive iff σ(z)≥21" is equivalent to z≥0. Both models predict positive for cases 2, 3 and 4. Case 2 is predicted positive only because of the ≥ convention applied to a score of exactly 0.
3
Apply the threshold 54
σ(z)≥54 holds iff e−z≤41, that is, iff z≥log4≈1.386. P flags only case 4. Q flags cases 3 and 4. The rankings agree, but the decisions differ.
4
Derive the threshold from the costs
Predicting positive has expected cost 4(1−p), and predicting negative has expected cost p. Positive is the better choice iff 4(1−p)≤p, that is, iff p≥54. A ranking alone cannot choose between P and Q, and neither fit is known to be calibrated on new data.
Because: The argument treats p as a calibrated probability.