Skip to content

CSCI3320

Fundamentals of Machine Learning

Statistical decisions, linear discriminants, density estimation, latent and factor models, clustering, regularization, model selection and evaluation, decision trees and multilayer networks.

26 chapters · 28 sectionsEN · 繁 · 简
My course notes

Your progress

0/ 28 sections marked complete

Next section: Decisions, estimation and linear representations

Completion marks are saved in this browser.

This course in the web

In the CSCI3320 notes

Theorem cards
70
Definitions
53
Counterexamples
30
Concept lenses
21
Histories of ideas
6
Proof X-rays
33

Sections with a history of an idea

None of these is on the timeline yet.

No concept lens from this course is recorded in the web yet.

Open this course in the knowledge map

Counts cover the 28 public sections of this course and what is written in them, not recorded connections. 0 of these sections are indexed into the web in detail so far.

In the webTheorem cards 70 · Definitions 53 · Counterexamples 30 · Concept lenses 21 · Histories of ideas 6 · Proof X-rays 33

None of these is on the timeline yet.

No concept lens from this course is recorded in the web yet.

Open this course in the knowledge map

Counts cover the 28 public sections of this course and what is written in them, not recorded connections. 0 of these sections are indexed into the web in detail so far.

Chapter 1 · 1 section

Statistical foundations and linear geometry

Separate prediction loss from likelihood, derive least squares, and compare the geometric purposes of SVD and Fisher discrimination.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 1.1

    Decisions, estimation and linear representations

    Separate prediction loss from likelihood, derive least squares, and compare the geometric purposes of SVD and Fisher discrimination.

Chapter 2 · 1 section

Latent probability models

Derive the EM lower bound, compute Gaussian mixture updates, and examine covariance collapse and latent-factor ambiguity.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 2.1

    Gaussian mixtures, EM and factor models

    Derive the EM lower bound, compute Gaussian mixture updates, and examine covariance collapse and latent-factor ambiguity.

Chapter 3 · 1 section

Clustering, regularization and model selection

Prove the k-means descent and graph-Laplacian identities, then derive ridge and sparse shrinkage while keeping model selection honest.

Theorems 3 · Definitions 4Counterexamples 2Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 3.1

    Clustering geometry, graph structure and small sample control

    Prove the k-means descent and graph-Laplacian identities, then derive ridge and sparse shrinkage while keeping model selection honest.

Chapter 4 · 1 section

Estimating factor models and probabilistic PCA

Derive Gaussian posterior sufficient statistics, complete factor-analysis EM updates, and explain the isotropic-noise link to principal components.

Theorems 3 · Definitions 2Counterexamples 1History 1Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 4.1

    Estimating factor models and probabilistic PCA

    Derive Gaussian posterior sufficient statistics, complete factor-analysis EM updates, and explain the isotropic-noise link to principal components.

Chapter 5 · 1 section

Normalized cuts and multiway spectral clustering

Build an affinity graph, derive the normalized-cut trace relaxation, and follow a complete spectral embedding and rounding algorithm.

Theorems 2 · Definitions 2Counterexamples 1History 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 5.1

    Normalized cuts and multiway spectral clustering

    Build an affinity graph, derive the normalized-cut trace relaxation, and follow a complete spectral embedding and rounding algorithm.

Chapter 6 · 1 section

NMF solvers: multiplicative descent and nonnegative least squares

Prove a valid auxiliary-function descent result, diagnose zero locking, and derive an alternating nonnegative least-squares alternative.

Theorems 2 · Definitions 2Counterexamples 1Lenses 1History 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 6.1

    NMF solvers: multiplicative descent and nonnegative least squares

    Prove a valid auxiliary-function descent result, diagnose zero locking, and derive an alternating nonnegative least-squares alternative.

Chapter 7 · 1 section

RPCL and model selection during learning

Specify frequency-sensitive rival penalization, analyze its local effects and limitations, and contrast it with an explicit penalized birth–death procedure.

Theorems 2 · Definitions 2Counterexamples 1History 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 7.1

    RPCL and model selection during learning

    Specify frequency-sensitive rival penalization, analyze its local effects and limitations, and contrast it with an explicit penalized birth–death procedure.

Chapter 8 · 1 section

Sparse learning with correlated predictors and honest validation

Derive lasso coordinate and proximal updates for general designs, prove descent and fitted-value uniqueness, and separate tuning from evaluation.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 8.1

    Sparse learning with correlated predictors and honest validation

    Derive lasso coordinate and proximal updates for general designs, prove descent and fitted-value uniqueness, and separate tuning from evaluation.

Chapter 9 · 2 sections

Information measures

Finite cross-entropy, relative entropy and log loss.

Theorems 6 · Definitions 5Counterexamples 2Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 9.1

    Cross-entropy and relative entropy

    Prove nonnegativity for finite distributions and connect log loss to empirical frequencies.

  2. 9.2

    Mutual information and data processing

    Derive finite mutual information, its chain rule and a data-processing inequality.

Chapter 10 · 2 sections

Risk and regularization

Expected squared loss under repeated training and shrinkage.

Theorems 3 · Definitions 2Counterexamples 2Lenses 2Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 10.1

    Bias, variance and shrinkage under squared loss

    Derive pointwise prediction risk and quantify the bias–variance tradeoff in a shrinkage example.

  2. 10.2

    Ridge regression as spectral shrinkage

    Derive ridge coefficients and fitted values using the singular-value decomposition, including rank-deficient designs.

Chapter 11 · 1 section

Ranking and evaluation

ROC curves, score ties and ranking probabilities.

Theorems 1 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 11.1

    ROC curves, tied scores and ranking probability

    Construct an empirical ROC curve and prove the tie-adjusted pairwise interpretation of trapezoidal AUC.

Chapter 12 · 1 section

Stochastic optimization

Unbiased stochastic gradients and expected descent.

Theorems 3 · Definitions 2Counterexamples 1Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 12.1

    Mini-batch stochastic gradients and expected descent

    Derive an unbiased mini-batch gradient and a smooth finite-sum descent bound with explicit sampling assumptions.

Chapter 18 · 1 section

Uncertainty in density estimation

Separate finite-sample sampling error from smoothing bias on a declared grid.

Theorems 2 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 18.1

    Density uncertainty on a finite grid

    Derive a simultaneous finite-grid confidence band for box-kernel estimates, separate sampling error from smoothing bias, and handle a finite adaptive bandwidth choice.

Chapter 19 · 1 section

Linear discriminants

Fisher direction, classification and singular scatter.

Theorems 3 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 19.1

    Fisher discriminants: scatter, direction and classification

    Derive Fisher's direction and distinguish it from a classifier threshold.

Chapter 20 · 1 section

Multiclass decisions and the accuracy of kernel estimation

Minimize a full conditional cost matrix, compare plug-in estimators, and prove one-dimensional kernel bias, variance and integrated-error rates.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 20.1

    Multiclass decisions and the accuracy of kernel estimation

    Minimize a full conditional cost matrix, compare plug-in estimators, and prove one-dimensional kernel bias, variance and integrated-error rates.

Chapter 21 · 1 section

Computing and checking low-rank representations

Derive both finite approximation bounds, control loading perturbations with a gap, and execute a centered PCA fit and power iteration.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 2

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 21.1

    Computing PCA: singular values, spectral gaps and reliable reconstruction

    Fit a centered low-rank representation, prove spectral approximation and a finite perturbation bound, and diagnose the assumptions behind power iteration.

Chapter 22 · 1 section

Hierarchical clustering

Interpret agglomerative trees through linkage and threshold graphs.

Theorems 2 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 22.1

    Hierarchical clustering: linkage and dendrograms

    Build and interpret agglomerative trees, prove the single-link graph connection, and compare linkage rules.

Chapter 23 · 1 section

Ward clustering

Derive variance-based merge costs and construct a hierarchy.

Theorems 2 · Definitions 1Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 23.1

    Ward clustering: variance and merge cost

    Derive the increase in within-cluster squared error when two groups merge and use it to construct an agglomerative hierarchy.

Chapter 24 · 1 section

Local density construction and multivariate smoothing

Construct neighbor and reflected kernel estimates, prove pointwise consistency, and compute a complete matrix-bandwidth comparison.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 24.1

    Constructing local density estimates: neighbors, boundaries and bandwidths

    Compute nearest-neighbor density heights, prove pointwise consistency, correct a support boundary, and select a multivariate Gaussian bandwidth without self-counting.

Chapter 25 · 1 section

Nearest-neighbor classification

Connect local class votes to Bayes decisions and pointwise consistency.

Theorems 1 · Definitions 1Counterexamples 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 25.1

    Nearest-neighbor classification: local votes and consistency

    Construct local class votes, relate them to Bayes decisions, and prove a pointwise consistency statement under explicit assumptions.

Chapter 26 · 1 section

Optimism, effective dimension and information criteria

Prove finite Gaussian prediction corrections, derive the linear-model AICc adjustment, and delimit the likelihood and Laplace arguments behind AIC and BIC.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1History 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 26.1

    Optimism, effective dimension and information criteria

    Prove finite Gaussian prediction corrections, derive the linear-model AICc adjustment, and delimit the likelihood and Laplace arguments behind AIC and BIC.

Chapter 27 · 1 section

Nested selection and procedure assessment

Execute inner selection and outer refitting, compute exact finite examples, and distinguish smaller-sample procedure risk from final-predictor risk.

Theorems 3 · Definitions 2Counterexamples 2Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 27.1

    Nested cross-validation: inner selection and outer evaluation

    Execute a finite nested selection algorithm, derive its precise outer-risk target, reproduce exact six-point calculations, and diagnose pooled leakage and dependent fold errors.

Chapter 28 · 1 section

Probability calibration and decision costs

Separate probability reliability, retained information and decision risk through exact finite Brier identities.

Theorems 2 · Definitions 3Counterexamples 1Lenses 1History 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 28.1

    Probability calibration: Brier decomposition and decision costs

    Prove the finite binary Brier decomposition, compute an exact synthetic calibration table, and distinguish score reliability, retained information and cost-sensitive decisions.

Chapter 29 · 1 section

Linear classifiers

Perceptron, logistic regression and maximum-margin classification through the margin.

Theorems 4 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 29.1

    Linear classifiers: perceptron, logistic regression and the maximum margin

    Define margins of a linear classifier, prove the perceptron mistake bound, derive the logistic-loss gradient and its convexity, and compute a hard-margin separating hyperplane exactly.

Chapter 30 · 1 section

Multilayer networks

Feed-forward networks and the exact backpropagation computation of gradients.

Theorems 2 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 30.1

    Multilayer perceptrons and backpropagation

    Define a feed-forward network, prove why nonlinear activations are needed, derive the backpropagation recursion with proof, and verify an exact gradient and an XOR network by hand.

Chapter 31 · 1 section

Decision trees

Greedy classification and regression trees with cost-complexity pruning.

Theorems 3 · Definitions 2Counterexamples 1Lenses 1Proof X-rays 1

Chapter outline →See this chapter on the web →Not yet indexed in the web

  1. 31.1

    Decision trees: Gini impurity, regression trees and cost-complexity pruning

    Grow classification and regression trees by greedy axis-aligned splits, prove the concavity and variance-reduction identities behind the split rules, and compute cost-complexity pruning exactly.

Vercel visit statistics and public-page performance measurements each require separate consent. Neither is needed to use the website.

Website function storage (used when needed)

Security, sign-in and your chosen preferences use storage when needed. This is separate from optional statistics.

Withdrawing stops future statistics. The page reloads if analytics code needs unloading. It does not delete data already sent.

Read Cookie and storage details