Bias-Variance Tradeoff

The bias-variance tradeoff is arguably the most important concept in machine learning. Prediction errors come from two sources: bias (the model is too simple to capture the true pattern, leading to systematic underprediction or overprediction) and variance (the model is too sensitive to the specific training data, performing well on training examples but poorly on new data).

High-bias models underfit -- they have high error on both training and test data because they cannot represent the complexity of the underlying relationship. High-variance models overfit -- they have low training error but high test error because they memorize noise in the training data rather than learning the true signal. The classic illustration is fitting polynomials: a degree-1 polynomial underfits a sine wave (high bias), while a degree-15 polynomial overfits to noise (high variance).

The goal is to find the sweet spot that balances both. Techniques like regularization, cross-validation, and early stopping help manage this tradeoff. In practice, you start simple (high bias, low variance), evaluate whether the model underfits or overfits, and adjust complexity accordingly. Understanding this tradeoff is essential for choosing model complexity, tuning hyperparameters, and diagnosing why a model fails in production.