Motivation
A factor model is useful only if we can estimate it and describe what its factors mean. Writing a covariance as a low rank contribution plus diagonal noise explains a pattern of correlations, but does not yet tell us how to fit the loadings. The hidden coordinates are not observations. Regressing measurements on guessed factor scores and ignoring their uncertainty solves a different problem.
The earlier Gaussian-model unit derived the conditional distribution of a factor. Here we carry that calculation through a complete expectation–maximization iteration. The essential bridge is a second moment: the outer product of a posterior mean must be supplemented by posterior covariance. Once those moments have been retained, the loading update is a matrix least-squares calculation and the noise update is an expected residual variance. This makes the algorithm reconstructible rather than a list of formulas to memorize.
We also separate ordinary factor analysis from probabilistic principal component analysis, or PPCA. The former allows different noise variances in different coordinates; the latter assumes the same noise variance everywhere. That restriction is exactly what makes principal covariance directions appear in a likelihood solution. Neither model identifies an intrinsic orientation of the hidden axes.
Definitions
Definition 4.1.1
(Centered Gaussian factor analysis)
For observations , fix and write . For , let
independently across observations, with factors independent of noise. Here and has strictly positive entries. Define . The marginal covariance is .
Fixing the sample mean is justified by the marginal Gaussian likelihood: for every positive definite covariance its maximizing mean is the sample mean. We therefore optimize the covariance after centering once. A different EM scheme could update the mean jointly using augmented sufficient statistics, but mixing that scheme's mean update with the centered formulas below would be an error. The divisor in is because we are maximizing likelihood, not constructing an unbiased covariance estimator.
At an old admissible parameter , put
The E-step statistics, held fixed throughout the next M-step, are
Their dimensions are and . The presence of makes positive definite even if the posterior means span only a proper subspace. Computationally, linear systems are preferable to explicitly forming inverses; the inverse notation states the mathematics, not a requirement for numerical implementation.
Definition 4.1.2
(Probabilistic PCA and a noise floor)
PPCA is the factor model with , where . A declared noise floor replaces positivity by or for a fixed . This is a constrained likelihood model. It is different from computing an unconstrained update and silently adding an arbitrary constant afterward.
Theorem/Proposition
Theorem 4.1.3
(Posterior moments and the complete centered M-step)
For the model above, . Hence . Let
If every , the factor-analysis M-step is . With a noise floor it is instead . For PPCA, the corresponding update is , provided this is positive, or under the floor.
The same loading update applies to both noise structures. In each row, a positive noise weight multiplies the whole expected squared regression residual and therefore does not change its minimizer. This fact would fail for arbitrary coupled constraints on , or if one used a non-diagonal structured noise family without rederiving the optimization.
Theorem 4.1.4
(Likelihood improvement and what it does not identify)
An exact E-step followed by the admissible M-step above cannot decrease the observed Gaussian likelihood, within the stated centered and possibly noise-constrained model. For every orthogonal , and give the same marginal distribution. Monotonicity therefore neither selects a unique loading orientation nor proves convergence to a global likelihood maximum.
Theorem 4.1.5
(The principal PPCA fit when the sample covariance is positive definite)
Suppose has eigenvalues and . Let contain leading orthonormal eigenvectors and set . If , an unconstrained maximum-likelihood PPCA covariance is obtained from
for any orthogonal . Repeated sample eigenvalues permit further equivalent choices of eigenvectors. The positive-definiteness and strict-gap assumptions exclude boundary cases from this statement.
Proof sketch or proof idea
Proof
(Complete the square, then retain both moments)
Conditional on , terms depending on in minus twice the log joint density are
Completing the square gives plus a constant. The precision is positive definite because for . Gaussian normalization proves the posterior formula. Expanding yields the second moment ; the two cross terms vanish because the centered conditional variable has mean zero.
Ignoring constants and multiplying by a positive factor, the M-step minimizes
Complete the matrix square inside brackets:
The last contribution has nonnegative trace after weighting by , and vanishes exactly at . Thus the loading is a global minimizer of this M-step, not just a stationary solution. At that loading the remaining objective is . Differentiating each summand shows decrease up to and increase afterward when . Under a floor, clipping at the floor gives the exact constrained minimizer. Isotropic noise reduces the expression to , with the stated minimizer.
The matrix is positive semidefinite because it is an average conditional expected residual outer product at . A zero diagonal entry needs attention: without a floor, tends to minus infinity as decreases to zero, so no positive minimizing variance exists for that M-step. Reporting zero as an admissible Gaussian noise variance would conceal a boundary failure.
Posterior inference also has a useful shrinkage interpretation. Whiten the measured coordinates by their noise standard deviations and consider the singular directions of the whitened loading. In a latent singular direction with singular value , posterior variance is rather than the prior variance one. A direction strongly represented relative to measurement noise is learned more precisely. A zero singular direction receives no information: its posterior mean remains zero and its posterior variance remains one. These are statements about a fitted probability model, so they do not prove that the model's noise assumptions match an actual experiment.
For example, suppose a second independent measurement will be made on the same underlying factor, with fresh noise from the fitted model. Conditional on the first measurement, its predictive mean is the loading times the factor posterior mean, plus the fitted observation mean. Its predictive covariance is the loading times the posterior covariance times the transposed loading, plus the fresh noise covariance. Both terms matter. Substituting a single factor score and reporting only fresh noise would understate uncertainty. This interpretation explains why preserving posterior covariance benefits prediction as well as the EM derivation.
Proof
(One EM iteration cannot lower the likelihood)
Let be the observed log-likelihood of the centered data, and let be the old admissible parameter. For every and every , Bayes' rule gives . Take the expectation over and sum over :
where is the expected complete-data log-likelihood and . Up to terms free of , is minus times the M-step objective above, so the admissible M-step output maximizes over a feasible set that contains . Hence . Gibbs' inequality (non-negativity of the Kullback–Leibler divergence between the posteriors at and ) gives for every . Subtracting the two expressions of ,
The second statement of the theorem is immediate: for orthogonal , so , and hence the marginal Gaussian, is unchanged.
Proof X-ray
Two inequalities and one fixed posterior
The argument has one fixed object and two inequalities. The fixed object is the posterior at : it defines both and , which is why and are computed once and held through the whole M-step. The first inequality comes from the M-step and compares only with , so must belong to the feasible set; with a noise floor, an old parameter below the floor would not be covered. The second inequality is Gibbs' inequality and needs no optimization at all. Neither inequality compares with a global maximizer, so the conclusion is that does not decrease, not that it converges to its maximum. If the stored moments were recomputed at a partly updated parameter, would no longer be an expectation under one fixed posterior, and the decomposition would not describe the objective being minimized.
Proof
(Why equal noise produces principal directions)
Apart from constants the negative marginal log likelihood is . A PPCA covariance has at most eigenvalues larger than , with all remaining eigenvalues equal to . For fixed covariance eigenvalues , the trace is minimized by pairing the largest with the largest . For and , the ordered pairing minus the exchanged pairing is . Thus exchanging an ordered pair cannot decrease the trace. More generally, squared overlaps of two orthonormal eigenbases form a doubly stochastic matrix, and a linear function of those overlaps is minimized at an ordered permutation; the elementary exchange gives the ordering.
With that alignment, minimize . Each free is minimized at , and differentiation in gives the trailing average. The gap hypothesis ensures all proposed loading variances are positive and the order constraints hold. Allowing fewer than elevated eigenvalues cannot improve this optimum: such covariance choices belong to the same rank-at-most- feasible family already minimized. This proves the stated covariance formula. Orthogonal loading ambiguity follows directly from .
Proof X-ray
Align first, then optimize one variable at a time
The real idea is a separation. First fix the covariance eigenvalues and show that the trace term is smallest when the eigenvectors of are those of in the same order; this is the exchange inequality. After alignment the objective splits into one-variable problems, which produce and the trailing average. The equal-noise structure enters only as the requirement that eigenvalues of coincide, and this is what pools the trailing eigenvalues. The gap hypothesis is used last, to make every positive so that the loadings exist. Positive definiteness of keeps finite. The loadings are determined only up to , since only enters.
Worked examples
Worked example 4.1.6
(A complete scalar EM iteration)
Use centered observations , one factor, old loading and noise variance one. Then , , , , and . Therefore and . The fitted marginal variance changes from two to , toward the sample second moment four. If the posterior covariance had been omitted, the loading would instead be two and the noise update zero. That false calculation treats hidden factors as known exactly. In one observed dimension loading variance and noise variance are not separately identifiable; the example demonstrates an iteration, not unique recovery.
Worked example 4.1.7
(Uncertainty survives two measurements)
Take , , and . Posterior precision is , so . The weighted score is , hence . Its second moment is . The noisier second coordinate receives smaller weight. Using its raw magnitude without its variance would misstate the evidence, while using alone would omit of posterior uncertainty.
Worked example 4.1.8
(One PPCA direction and two residual variances)
If and , the noise estimate is . A loading is , giving fitted covariance . PPCA preserves the leading empirical variance but pools the two trailing ones. It does not reproduce exactly, and setting noise equal to the smallest eigenvalue one would solve a different problem. A diagonal-noise factor model has more flexibility but also greater identification difficulties; adding parameters is not automatically better with a small sample.
Common mistakes
Counterexample mode
Identical covariance, different explanations
A tempting claim is that the observed covariance determines the loading and the noise variance. It does not. In dimension one, marginal variance four is produced by loading and noise one, or loading one and noise three. Both factors have standard normal priors and induce exactly the same observed distribution. This ambiguity is more than a sign change. In higher dimensions, diagonal-noise factor analysis also requires appropriate dimension and structural restrictions before any uniqueness claim can be made. A lower triangular loading convention can fix some rotations, but it does not by itself prove that every covariance has a unique factor/noise decomposition.
Common mistake
(A small residual is not a model-selection theorem)
Increasing expands the covariance family, so training likelihood can improve simply because the model is more flexible. Choosing requires a declared validation or model-comparison procedure. Neither an EM stopping tolerance nor a visually impressive factor plot estimates prediction uncertainty. With repeated eigenvalues, individual principal axes can change while their shared subspace remains the same; comparing axes without allowing rotations can report a difference that the model does not identify.
History of an idea
The probabilistic bridge to PCA
Tipping and Bishop's 1999 PPCA paper explicitly develops an isotropic latent Gaussian model and an EM method for principal-subspace estimation. The connection concerns that probability model; it does not turn every diagonal-noise factor model into ordinary PCA.
Summary
Posterior means and posterior second moments play different roles. Keeping both gives a positive definite latent scatter, a loading regression and an expected residual noise update. A declared noise floor has an exact constrained M-step. Equal noise links the maximum-likelihood covariance to principal eigenspaces, while unequal diagonal noise and latent rotations require more cautious interpretation.
Exercises
Quick check
(In the scalar EM example, compute the expected squared residual at the new loading separately for each observation. Explain why it equals the updated noise variance.)
Use the squared posterior-mean residual plus the loading squared times posterior variance.
Solution · Answer
For , , and . The expected residual square is . For , symmetry gives the same answer. Averaging yields , exactly the M-step variance. Using only the first term would underestimate it.
Quick check
(Suppose is the two by two identity, has rows and , and is diagonal with entries and . Find the loading and diagonal-noise M-step; then impose a noise floor of .)
The floor changes the noise update, not the loading minimizer.
Solution · Answer
The loading is , and . Without a floor the noise is . With floor it becomes . For every positive diagonal noise matrix the same loading minimizes the completed square, so the floor does not require another loading solution in this unconstrained-loading model.
Quick check
(For sample covariance diag(10,6,2,2), fit PPCA with two factors. State the marginal eigenvalues and explain the rotational freedom.)
Average the two trailing empirical eigenvalues.
Solution · Answer
Noise is two. One loading has columns and , giving loading variances eight and four. The marginal covariance has eigenvalues . Right multiplication by any orthogonal two by two matrix preserves , and hence the likelihood. It may mix the two factor columns even though the leading observed eigenvalues differ; the factors themselves retain an isotropic prior.
Quick check
(Derive the posterior mean when L is zero. Show what the loading M-step does, and explain why this initialization can be undesirable.)
A latent factor independent of the observation has its original prior as posterior.
Solution · Answer
If , then and every . Thus , and . The next noise estimate absorbs the observed coordinate variances, or their average under PPCA. Subsequent iterations keep zero loading because factors remain independent of measurements. This fixed initialization can miss an informative nonzero-loading solution; likelihood monotonicity does not force an algorithm to escape a stationary configuration.