CSCI3320
Fundamentals of Machine Learning
Statistical decisions, linear discriminants, density estimation, latent and factor models, clustering, regularization, model selection and evaluation, decision trees and multilayer networks.
Your progress
0/ 28 sections marked complete
Next section: Decisions, estimation and linear representations
Completion marks are saved in this browser.
This course in the web
In the CSCI3320 notes
- Theorem cards
- 70
- Definitions
- 53
- Counterexamples
- 30
- Concept lenses
- 21
- Histories of ideas
- 6
- Proof X-rays
- 33
Sections with a history of an idea
- Estimating factor models and probabilistic PCA4.1
- Normalized cuts and multiway spectral clustering5.1
- NMF solvers: multiplicative descent and nonnegative least squares6.1
- RPCL and model selection during learning7.1
- Optimism, effective dimension and information criteria26.1
- Probability calibration: Brier decomposition and decision costs28.1
None of these is on the timeline yet.
No concept lens from this course is recorded in the web yet.
Counts cover the 28 public sections of this course and what is written in them, not recorded connections. 0 of these sections are indexed into the web in detail so far.
In the webTheorem cards 70 · Definitions 53 · Counterexamples 30 · Concept lenses 21 · Histories of ideas 6 · Proof X-rays 33
None of these is on the timeline yet.
No concept lens from this course is recorded in the web yet.
Counts cover the 28 public sections of this course and what is written in them, not recorded connections. 0 of these sections are indexed into the web in detail so far.
Chapter 1 · 1 section
Statistical foundations and linear geometry
Separate prediction loss from likelihood, derive least squares, and compare the geometric purposes of SVD and Fisher discrimination.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 1.1
Decisions, estimation and linear representations
Separate prediction loss from likelihood, derive least squares, and compare the geometric purposes of SVD and Fisher discrimination.
Chapter 2 · 1 section
Latent probability models
Derive the EM lower bound, compute Gaussian mixture updates, and examine covariance collapse and latent-factor ambiguity.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 2.1
Gaussian mixtures, EM and factor models
Derive the EM lower bound, compute Gaussian mixture updates, and examine covariance collapse and latent-factor ambiguity.
Chapter 3 · 1 section
Clustering, regularization and model selection
Prove the k-means descent and graph-Laplacian identities, then derive ridge and sparse shrinkage while keeping model selection honest.
Theorems 3 · Definitions 4Counterexamples 2Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 3.1
Clustering geometry, graph structure and small sample control
Prove the k-means descent and graph-Laplacian identities, then derive ridge and sparse shrinkage while keeping model selection honest.
Chapter 4 · 1 section
Estimating factor models and probabilistic PCA
Derive Gaussian posterior sufficient statistics, complete factor-analysis EM updates, and explain the isotropic-noise link to principal components.
Theorems 3 · Definitions 2Counterexamples 1History 1Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 4.1
Estimating factor models and probabilistic PCA
Derive Gaussian posterior sufficient statistics, complete factor-analysis EM updates, and explain the isotropic-noise link to principal components.
Chapter 5 · 1 section
Normalized cuts and multiway spectral clustering
Build an affinity graph, derive the normalized-cut trace relaxation, and follow a complete spectral embedding and rounding algorithm.
Theorems 2 · Definitions 2Counterexamples 1History 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 5.1
Normalized cuts and multiway spectral clustering
Build an affinity graph, derive the normalized-cut trace relaxation, and follow a complete spectral embedding and rounding algorithm.
Chapter 6 · 1 section
NMF solvers: multiplicative descent and nonnegative least squares
Prove a valid auxiliary-function descent result, diagnose zero locking, and derive an alternating nonnegative least-squares alternative.
Theorems 2 · Definitions 2Counterexamples 1Lenses 1History 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 6.1
NMF solvers: multiplicative descent and nonnegative least squares
Prove a valid auxiliary-function descent result, diagnose zero locking, and derive an alternating nonnegative least-squares alternative.
Chapter 7 · 1 section
RPCL and model selection during learning
Specify frequency-sensitive rival penalization, analyze its local effects and limitations, and contrast it with an explicit penalized birth–death procedure.
Theorems 2 · Definitions 2Counterexamples 1History 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 7.1
RPCL and model selection during learning
Specify frequency-sensitive rival penalization, analyze its local effects and limitations, and contrast it with an explicit penalized birth–death procedure.
Chapter 8 · 1 section
Sparse learning with correlated predictors and honest validation
Derive lasso coordinate and proximal updates for general designs, prove descent and fitted-value uniqueness, and separate tuning from evaluation.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 8.1
Sparse learning with correlated predictors and honest validation
Derive lasso coordinate and proximal updates for general designs, prove descent and fitted-value uniqueness, and separate tuning from evaluation.
Chapter 9 · 2 sections
Information measures
Finite cross-entropy, relative entropy and log loss.
Theorems 6 · Definitions 5Counterexamples 2Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 9.1
Cross-entropy and relative entropy
Prove nonnegativity for finite distributions and connect log loss to empirical frequencies.
- 9.2
Mutual information and data processing
Derive finite mutual information, its chain rule and a data-processing inequality.
Chapter 10 · 2 sections
Risk and regularization
Expected squared loss under repeated training and shrinkage.
Theorems 3 · Definitions 2Counterexamples 2Lenses 2Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 10.1
Bias, variance and shrinkage under squared loss
Derive pointwise prediction risk and quantify the bias–variance tradeoff in a shrinkage example.
- 10.2
Ridge regression as spectral shrinkage
Derive ridge coefficients and fitted values using the singular-value decomposition, including rank-deficient designs.
Chapter 11 · 1 section
Ranking and evaluation
ROC curves, score ties and ranking probabilities.
Theorems 1 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 11.1
ROC curves, tied scores and ranking probability
Construct an empirical ROC curve and prove the tie-adjusted pairwise interpretation of trapezoidal AUC.
Chapter 12 · 1 section
Stochastic optimization
Unbiased stochastic gradients and expected descent.
Theorems 3 · Definitions 2Counterexamples 1Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 12.1
Mini-batch stochastic gradients and expected descent
Derive an unbiased mini-batch gradient and a smooth finite-sum descent bound with explicit sampling assumptions.
Chapter 18 · 1 section
Uncertainty in density estimation
Separate finite-sample sampling error from smoothing bias on a declared grid.
Theorems 2 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 18.1
Density uncertainty on a finite grid
Derive a simultaneous finite-grid confidence band for box-kernel estimates, separate sampling error from smoothing bias, and handle a finite adaptive bandwidth choice.
Chapter 19 · 1 section
Linear discriminants
Fisher direction, classification and singular scatter.
Theorems 3 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 19.1
Fisher discriminants: scatter, direction and classification
Derive Fisher's direction and distinguish it from a classifier threshold.
Chapter 20 · 1 section
Multiclass decisions and the accuracy of kernel estimation
Minimize a full conditional cost matrix, compare plug-in estimators, and prove one-dimensional kernel bias, variance and integrated-error rates.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 20.1
Multiclass decisions and the accuracy of kernel estimation
Minimize a full conditional cost matrix, compare plug-in estimators, and prove one-dimensional kernel bias, variance and integrated-error rates.
Chapter 21 · 1 section
Computing and checking low-rank representations
Derive both finite approximation bounds, control loading perturbations with a gap, and execute a centered PCA fit and power iteration.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 2
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 21.1
Computing PCA: singular values, spectral gaps and reliable reconstruction
Fit a centered low-rank representation, prove spectral approximation and a finite perturbation bound, and diagnose the assumptions behind power iteration.
Chapter 22 · 1 section
Hierarchical clustering
Interpret agglomerative trees through linkage and threshold graphs.
Theorems 2 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 22.1
Hierarchical clustering: linkage and dendrograms
Build and interpret agglomerative trees, prove the single-link graph connection, and compare linkage rules.
Chapter 23 · 1 section
Ward clustering
Derive variance-based merge costs and construct a hierarchy.
Theorems 2 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 23.1
Ward clustering: variance and merge cost
Derive the increase in within-cluster squared error when two groups merge and use it to construct an agglomerative hierarchy.
Chapter 24 · 1 section
Local density construction and multivariate smoothing
Construct neighbor and reflected kernel estimates, prove pointwise consistency, and compute a complete matrix-bandwidth comparison.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 24.1
Constructing local density estimates: neighbors, boundaries and bandwidths
Compute nearest-neighbor density heights, prove pointwise consistency, correct a support boundary, and select a multivariate Gaussian bandwidth without self-counting.
Chapter 25 · 1 section
Nearest-neighbor classification
Connect local class votes to Bayes decisions and pointwise consistency.
Theorems 1 · Definitions 1Counterexamples 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 25.1
Nearest-neighbor classification: local votes and consistency
Construct local class votes, relate them to Bayes decisions, and prove a pointwise consistency statement under explicit assumptions.
Chapter 26 · 1 section
Optimism, effective dimension and information criteria
Prove finite Gaussian prediction corrections, derive the linear-model AICc adjustment, and delimit the likelihood and Laplace arguments behind AIC and BIC.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1History 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 26.1
Optimism, effective dimension and information criteria
Prove finite Gaussian prediction corrections, derive the linear-model AICc adjustment, and delimit the likelihood and Laplace arguments behind AIC and BIC.
Chapter 27 · 1 section
Nested selection and procedure assessment
Execute inner selection and outer refitting, compute exact finite examples, and distinguish smaller-sample procedure risk from final-predictor risk.
Theorems 3 · Definitions 2Counterexamples 2Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 27.1
Nested cross-validation: inner selection and outer evaluation
Execute a finite nested selection algorithm, derive its precise outer-risk target, reproduce exact six-point calculations, and diagnose pooled leakage and dependent fold errors.
Chapter 28 · 1 section
Probability calibration and decision costs
Separate probability reliability, retained information and decision risk through exact finite Brier identities.
Theorems 2 · Definitions 3Counterexamples 1Lenses 1History 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 28.1
Probability calibration: Brier decomposition and decision costs
Prove the finite binary Brier decomposition, compute an exact synthetic calibration table, and distinguish score reliability, retained information and cost-sensitive decisions.
Chapter 29 · 1 section
Linear classifiers
Perceptron, logistic regression and maximum-margin classification through the margin.
Theorems 4 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 29.1
Linear classifiers: perceptron, logistic regression and the maximum margin
Define margins of a linear classifier, prove the perceptron mistake bound, derive the logistic-loss gradient and its convexity, and compute a hard-margin separating hyperplane exactly.
Chapter 30 · 1 section
Multilayer networks
Feed-forward networks and the exact backpropagation computation of gradients.
Theorems 2 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 30.1
Multilayer perceptrons and backpropagation
Define a feed-forward network, prove why nonlinear activations are needed, derive the backpropagation recursion with proof, and verify an exact gradient and an XOR network by hand.
Chapter 31 · 1 section
Decision trees
Greedy classification and regression trees with cost-complexity pruning.
Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1
Chapter outline →See this chapter on the web →Not yet indexed in the web
- 31.1
Decision trees: Gini impurity, regression trees and cost-complexity pruning
Grow classification and regression trees by greedy axis-aligned splits, prove the concavity and variance-reduction identities behind the split rules, and compute cost-complexity pruning exactly.