Machine Learning Foundations: Inductive Learning, Loss Optimization & Core Paradigms

Tom Mitchell's formal definition states: "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."

1. The Four Primary Machine Learning Paradigms

Supervised Learning

Inductive Mapping with Ground Truth

Given a training dataset D = {(x_1, y_1), ..., (x_n, y_n)} sampled i.i.d. from an unknown joint distribution P(X, Y), the objective is to learn a hypothesis function h_θ(x)that minimizes Empirical Risk:

R_emp(θ) = (1/n) ∑ L( h_θ(x_i), y_i ) + λ Ω(θ)

Regression (Continuous Y): Linear Regression, Ridge, Lasso, Support Vector Regression (SVR).
Classification (Discrete Y): Logistic Regression, Decision Trees, Random Forests, Gradient Boosted Trees (XGBoost).

Unsupervised Learning

Latent Structure & Density Estimation

Given unlabeled data D = {x_1, ..., x_n} sampled from P(X), the objective is to uncover the underlying data manifold, discover cluster memberships, or estimate the generative probability density:

min_C ∑ ∑ || x_i - μ_k ||^2 | max E[ log P(X; θ) ]

Centroid & Density Clustering: K-Means, DBSCAN, Hierarchical Agglomerative Clustering.
Dimensionality Reduction: Principal Component Analysis (PCA), t-SNE, UMAP.
Generative Density: Gaussian Mixture Models (GMMs via Expectation-Maximization), Variational Autoencoders (VAEs).

Self-Supervised Learning (SSL)

Pretext Task Foundation Pretraining

The cornerstone of modern Foundation Models. The supervisory training signal is derived autonomously from the unannotated data itself by masking or corrupting parts of the input and training the network to predict the missing segments:

L_SSL = - ∑ log P( x_masked | x_unmasked; θ )

Masked Language Modeling (MLM): BERT predicting masked tokens in sentences.
Causal Autoregressive Prediction: GPT predicting token t+1 given tokens 1...t.
Contrastive Representation (SimCLR/CLIP): Pulling positive image-text pairs together and repelling negative distractors.

Reinforcement Learning (RL)

Markov Decision Processes & Value Iteration

An agent learns an optimal policy π*(a|s) mapping environmental states to actions to maximize the expected discounted cumulative reward:

Q*(s, a) = R(s, a) + γ ∑ P(s'|s, a) max_a' Q*(s', a')

Value-Based: Q-Learning, Deep Q-Networks (DQN with experience replay).
Policy-Based: REINFORCE, Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC).
Human Alignment: RLHF guiding LLMs to conform to human preference distributions.

2. The Bias-Variance Decomposition & Model Generalization

The Expected Generalization Error of any machine learning model on unseen test data can be decomposed mathematically into three orthogonal components:

E[ (y - h_θ(x))^2 ] = [ Bias(h_θ(x)) ]^2 + Var(h_θ(x)) + σ^2_irreducible

Bias Squared (Underfitting)

The difference between the expected prediction of our model and the true underlying mathematical function. High bias indicates the model is too restrictive (e.g. fitting a straight line to sinusoidal data).

Variance (Overfitting)

The variability of a model prediction for a given data point across different training splits. High variance indicates the model memorizes random noise in the training set rather than the signal.

Irreducible Error

The intrinsic stochastic noise in the data generating process σ^2. Cannot be reduced by any algorithmic design regardless of parameter scale.

3. Rigorous Evaluation Metrics: Beyond Raw Accuracy

In imbalanced real-world datasets (e.g. 99% non-fraud, 1% fraud), simple Accuracy is dangerously misleading. Engineers evaluate systems using rigorous statistical metrics:

Metric	Mathematical Formulation	Optimal Use Case
Precision	TP / (TP + FP)	When False Positives are catastrophic (e.g. Spam filters, autonomous emergency braking).
Recall (Sensitivity)	TP / (TP + FN)	When False Negatives are fatal (e.g. Cancer detection, financial fraud detection).
F1-Score	2 · (Precision · Recall) / (Precision + Recall)	Harmonic mean balancing precision and recall on severely skewed class distributions.
ROC-AUC	∫ TPR(FPR) d(FPR)	Threshold-independent measure of class separation across all operating points.
Log-Loss (Cross-Entropy)	- (1/n) ∑ [ y_i log(p_i) + (1-y_i) log(1-p_i) ]	Quantifies probabilistic calibration; heavily penalizes confident incorrect predictions.