Supervised Machine Learning Engineering

Welcome to the definitive guide to Supervised Machine Learning Engineering. Rather than treating machine learning frameworks as opaque black boxes, this curriculum derives every core formula from mathematical first principles and implements every algorithm from scratch in both pure Python and pure Rust without third-party dependencies.

Explore curriculum lessons from the sidebar to inspect step-by-step mathematical proofs, zero-dependency implementations, interactive zoomable diagrams, and knowledge verification checkpoints.

Curriculum Architecture at a Glance#

This course provides comprehensive mathematical derivations and production-grade implementations across the supervised learning lifecycle:

1. Linear Classification & Logistic Regression: Convexity proofs of the binary cross-entropy loss, log-loss Hessians, multi-class Softmax cross-entropy gradient derivations, and Newton-Raphson / L-BFGS second-order optimization updates.
2. Instance-Based Learning & Metric Spaces ($k$-NN): Minkowski distance metrics, metric space topologies, curse of dimensionality geometry, and spatial KD-Tree construction with hyper-rectangle pruning bounds.
3. Probabilistic Graphical Models (Naive Bayes): Conditional independence assumptions, maximum likelihood estimation for Gaussian, Multinomial, and Bernoulli distributions, Laplace smoothing ($\alpha$), and log-sum-exp numerical underflow prevention.
4. Support Vector Machines & Convex Duality (SVM): Maximum margin hyperplanes, primal quadratic programming formulations, Karush-Kuhn-Tucker (KKT) stationarity conditions, Wolfe dual representation, and the Mercer condition for reproducing kernel Hilbert spaces (RBF).
5. Decision Trees & Partition Theory (CART): Impurity metrics (Shannon Entropy vs. Gini Impurity), split criteria optimization, sub-tree cost-complexity pruning ($\alpha$), and binary feature partitioning algorithms.
6. Ensemble Mechanics (Bagging, Random Forests & Gradient Boosting): Bias-variance decomposition, variance reduction via bootstrap aggregation, AdaBoost exponential loss gradient descent, and second-order Taylor expansion loss approximations for XGBoost.
7. Neural Networks & Matrix Backpropagation: Forward propagation matrix formulations, activation function derivatives, full multivariable chain rule proofs, and batch gradient updates with momentum and Adam dynamics.
8. Model Evaluation, Calibration & Generalization: ROC curves, Precision-Recall AUC dynamics, expected calibration error (ECE), Platt scaling sigmoid calibration, and temperature scaling proofs.

Core Pedagogical Standard#

• Step-by-Step Algebraic Derivations: Every loss function, Hessian matrix, and backpropagation chain is derived algebraically step by step.
• Zero-Dependency Implementations: Every algorithm includes complete, runnable implementations in pure Python (standard library only) and pure Rust (std only).
• Interactive Visual Architecture: Explore interactive flowcharts, mathematical formulas rendered with KaTeX, expandable diagrams, and knowledge verification checkpoints.

The complete course source code and implementations are open on the Source Repository.