Motivation
A collection of measurements can have several centers even when no class labels were recorded. One Gaussian must explain all observations with a single mean and covariance. A mixture instead proposes an unobserved component label for each observation. The difficulty is that the likelihood contains a sum over labels inside a logarithm, so the simple calculations available for a labeled Gaussian sample no longer apply directly.
Expectation–maximization, abbreviated EM, addresses this difficulty by alternating between two coherent questions. Given the current parameters, how much posterior weight should each observation assign to each label? Given those weights, how should the parameters change? Soft assignments are consequences of the probability model, not arbitrary confidence scores. A proof of likelihood monotonicity explains why this iteration is sensible, but does not promise a global optimum or even a finite maximum.
Latent variables can also describe continuous hidden factors instead of discrete groups. Factor analysis attributes correlations among measured variables to shared latent coordinates, while leaving separate measurement noise. Comparing the two models helps prevent a common mistake: treating every latent representation as uniquely recoverable simply because the observable distribution is well defined.
Definitions
Definition 2.1.1
(A finite Gaussian mixture)
For components in , let , , , and be symmetric positive definite. A latent label has , and has Gaussian density . The marginal density is
For independent observations the log likelihood is . The responsibility is the normalized weight .
Responsibilities for one observation sum to one. They depend on mixture weights, covariances and the point's position. The component with the nearest mean need not have the largest responsibility when covariances or weights differ. Numerically, log densities and a log-sum-exp normalization avoid losing tiny positive contributions through underflow; this is a numerical implementation issue separate from the probabilistic definition.
For distributions on the labels, introduce a lower bound
using by continuity. In the strictly positive mixture model all joint label weights are positive, so this expression and the posterior are well behaved for any label distribution. The term is entropy. It is part of the bound, not an optional regularizer added to the observed likelihood.
Definition 2.1.2
(A Gaussian factor model)
Let and be independent, where is positive diagonal. For a loading matrix , set . Then . The shared factors create off-diagonal covariance; the diagonal noise describes variation specific to each measured coordinate.
Unlike a mixture label, a factor is continuous. Unlike a general principal-component reconstruction, this model explicitly separates latent variation from measurement noise. Principal component analysis is a geometric approximation criterion; factor analysis is a distributional covariance model. Their relationship requires specifying the noise model rather than simply giving their latent coordinates the same name. A mixture says that each observation comes from one of several populations. A factor model says that several hidden continuous influences contribute to each observation. Both use latent variables, but they answer different questions. A mixture can produce multiple modes; a single Gaussian factor model remains Gaussian after integrating out its factors.
For fixed factor parameters, hidden coordinates can still be inferred even though their orientation is a convention. Multiplying the standard normal prior density of the factor by the conditional observation density gives a quadratic expression in the factor. Let
Completing the square shows that the conditional distribution of the factor given an observation is Gaussian with mean and covariance . The matrix is positive definite because its quadratic form is the squared norm of a vector plus a nonnegative term. Conditional uncertainty is therefore retained, rather than replacing the hidden coordinate by a supposedly known value.
This calculation also explains what an EM treatment of factor analysis would need. A posterior mean supplies a first moment, but quadratic likelihood terms require a second moment as well. That moment is , not merely the outer product of the posterior mean. Omitting the covariance treats an uncertain factor as observed and changes the update equations. We have derived the conditional inference step here; a complete constrained maximum likelihood algorithm for the loadings and diagonal noise would additionally require its own M-step and admissibility analysis.
Theorem/Proposition
Theorem 2.1.3
(The EM identity and monotonicity)
For the positive finite mixture above,
Thus , with equality exactly when each equals the current posterior. If an E-step uses those posteriors at and a subsequent step finds admissible with , then .
Theorem 2.1.4
(Gaussian mixture M-step under nonsingular weighted scatter)
Fix responsibilities, put , and suppose each weighted scatter matrix below is positive definite. Maximizing over unconstrained positive Gaussian mixture parameters gives
These are maximum likelihood updates; the covariance divisor is , not a correction intended to make a sample covariance unbiased. If weighted scatter is singular, these formulas do not provide an admissible positive definite covariance, and the stated unconstrained maximization claim no longer applies.
Theorem 2.1.5
(Orthogonal ambiguity of Gaussian factors)
For the factor model, any orthogonal matrix gives the same observable distribution after replacing by . Consequently, the distribution of alone cannot identify the orientation or signs of the latent factors without additional restrictions.
Proof sketch or proof idea
Proof
(The lower bound touches the likelihood at the E-step)
Write and . The posterior is . Expanding the divergence gives
Summing proves the identity. Nonnegativity of divergence follows, for example, from : on the positive support of , with the posterior, , and equality requires everywhere. At the E-step the divergence vanishes. Therefore
No claim about global optimality entered this chain.
Proof X-ray
A moving lower bound, not a fixed surrogate
The bound is made tight at the old parameter by recomputing responsibilities. Improving an arbitrary weighted Gaussian objective would not establish the final equality in the chain. An approximate E-step needs a separate argument. A generalized M-step may merely increase the correct bound, but it still must keep all parameters admissible.
Proof
(Deriving the updates and the factor ambiguity)
Ignoring entropy, which is fixed during the M-step, the weight terms are . A Lagrange multiplier for their sum gives ; strict concavity gives the maximum. For a fixed positive definite covariance, completing the weighted quadratic in makes its minimizer the weighted mean.
Let be the weighted covariance divided by . The remaining covariance objective is to minimize . If is positive definite, set . Apart from a constant this is , minimized when every eigenvalue . Hence .
For factors, and independence gives . Replacing by leaves unchanged. Gaussian distributions are determined by mean and covariance, proving observational equivalence.
Worked examples
Worked example 2.1.6
(Responsibilities and a logistic ratio)
Consider two equally weighted one-dimensional Gaussian components with means and , both of variance one. At their densities are equal, so the responsibilities are . At a general , the density ratio of the second to the first is . Thus the second responsibility is . At it is approximately , although the observation lies exactly at that component's mean. Finite overlap leaves positive probability for both labels.
Worked example 2.1.7
(One complete weighted M-step)
Suppose observations are and , and a valid E-step has produced responsibilities for the first component and for the second. These responsibilities occur, for example, with equal weights, means zero and two, and common variance . Both effective counts equal one. The new weights remain , while the new means are and . For the first covariance, ; symmetry gives the same result for the second. These are soft weighted fits. Assigning each point completely to its most likely component would instead give zero variances here and leave the positive variance model.
Worked example 2.1.8
(A factor explains correlation without fixing its sign)
Take one standard normal factor, loading , and independent noise with . The observable covariance is
The correlation is . Replacing the factor by its negative and the loading by produces exactly the same measurements in distribution. The covariance eigenvalues are seven and three, with leading direction . This direction differs from the loading , because the noise variances are unequal. Recovering loadings by simply taking a leading covariance eigenvector is not a general factor-analysis method.
Concept lensAlgebraic
The loadings through row inner products
Write for the th row of . Then the covariance entries are for and . The data therefore constrain only the inner products among the rows of , and a rotation preserves exactly these: . In the example, , , so and .
Common mistakes
There is a precise bridge from soft Gaussian assignments to nearest-center geometry. If all components have the same isotropic covariance and equal weights, their responsibilities are proportional to . As the variance decreases toward zero with centers fixed, a uniquely nearest component receives responsibility approaching one. At a tie the limiting weights are shared among the tied nearest components. Thus hard assignment is a particular limiting behavior, not the definition of EM. Unequal weights or covariance shapes change the finite-variance assignments, and a covariance-collapse singularity is not a principled way to select this limit.
Finally, the word “expectation” in the E-step refers to averaging a complete-data log likelihood under the current posterior. It does not mean drawing new observations, replacing missing measurements by their marginal means, or taking the expectation under the unknown true parameter. The distribution used in that expectation is available from the current fitted model. Keeping this conditioning explicit is the simplest way to reconstruct the algorithm when the latent variable is more complicated than a single mixture label.
Counterexample mode
A Gaussian mixture likelihood can be unbounded
It is tempting to assume that maximizing the mixture likelihood always has a finite answer for EM to approach. It does not. With at least two components and at least one observation , keep one component as a fixed positive density at every sample point and give another a fixed weight , mean and covariance . Its density at is proportional to , so that observation's log density tends to infinity as . At every other observation the fixed component supplies a positive lower bound. Hence the total likelihood is unbounded above. Monotone improvement does not solve the absence of a finite unconstrained maximum. Covariance constraints or explicit regularization change the optimization problem and must be stated.
Common mistake
(The nearest mean need not carry the largest responsibility)
Take two unit-variance components with means and and weights and . At the first mean is nearer, yet the ratio of the second density term to the first is . The second component therefore has responsibility about . Responsibilities weigh the mixing proportions and covariances as well as distance, so they coincide with nearest-center assignment only in the equal-weight, equal-isotropic-covariance case described above.
Common mistake
(Convergence and interpretation need separate evidence)
A nondecreasing sequence of likelihoods need not yield convergent parameters, especially when the likelihood is unbounded. Even a stationary parameter can be a poor local solution. Permuting mixture labels leaves the density unchanged, and rotating factors leaves their covariance unchanged. A component name or factor direction therefore acquires meaning from an identification convention or external interpretation, not from the likelihood alone.
Summary
EM alternates posterior inference and improvement of a tight lower bound. Weighted means and covariances follow from that bound, with admissibility conditions that matter for small samples. Factor analysis explains covariance through shared coordinates and diagonal noise, while its rotations illustrate nonidentifiability. Good likelihood values alone do not establish unique interpretation, global optimality, or numerical stability.
Exercises
Quick check
(In the equal-weight, unit-variance mixture with means minus one and one, calculate the posterior responsibility of the second component at x equal to minus one. Give an exact expression and check that responsibilities sum to one.)
Use the likelihood ratio rather than evaluating two normalizing constants.
Solution · Answer
The second responsibility is . The first is . They are nonnegative and sum to one. Symmetry exchanges the responsibilities at observations and .
Quick check
(For observations zero and three, fixed responsibilities for component one are 0.75 and 0.25. Compute its effective count, updated mean and updated variance.)
The variance is centered at the updated mean.
Solution · Answer
The effective count is one, the mean is , and the variance is . Dividing by an effective count minus one would be undefined and would not be the likelihood update.
Quick check
(Explain why one component collapsing onto an observation does not force the other observations' log densities to minus infinity in the counterexample.)
Retain the positive background component and its fixed weight.
Solution · Answer
If the background density is and its weight is , each sample density is at least , independently of . Its logarithm therefore has a finite lower bound. At the collapsing component makes the log density diverge upward, so there is no opposing downward divergence in the finite sum.
Quick check
(Let L be the two by two identity and noise covariance be the identity. Give two distinct loading matrices with the same observable covariance, and explain why EM monotonicity cannot distinguish them.)
Use a nontrivial orthogonal rotation.
Solution · Answer
Take and . Both satisfy , so both observable covariances are with the specified noise. They induce identical likelihoods for every dataset. A theorem about increasing that likelihood cannot identify a difference the observable model does not contain.