Chapter 26 — Underfitting and Overfitting
Model complexity • Bias–variance trade-off • Learning curves • Validation curves
Diagnosing whether a model is too simple, too complex, or limited by data
| BRIDGE FROM CHAPTER 25 Residual analysis showed where a trained model makes systematic errors. This chapter asks a complementary question: is the model fundamentally too simple, excessively flexible, or constrained by the amount and quality of training data? |
Chapter map
| Section | Core question | Primary diagnostic |
|---|---|---|
| 26.1 Underfitting | Is the model too simple to learn the pattern? | Training and validation performance are both weak. |
| 26.2 Overfitting | Is the model fitting training-specific noise? | Training is excellent but validation is much worse. |
| 26.3 Bias–variance trade-off | How much complexity is appropriate? | Generalization gap and validation performance. |
| 26.4 Learning curves | Would more data help? | Scores versus training-set size. |
| 26.5 Validation curves | Which complexity range is useful? | Scores versus a hyperparameter. |
Chapter overview
A model can fail because it has not learned enough structure, because it has learned too much training-specific detail, or because the available data do not support the desired level of complexity. Underfitting and overfitting therefore cannot be diagnosed from training performance alone. The key is to compare performance across data that were used for fitting and data that were kept separate for validation.
This chapter develops a practical diagnostic toolkit. Students will read training and validation scores together, reason about bias and variance, use learning curves to distinguish data shortages from model limitations, and use validation curves to identify useful hyperparameter ranges.
Learning objectives
- Recognize the typical training-versus-validation pattern of underfitting.
- Recognize the generalization gap associated with overfitting.
- Explain high bias, high variance, model complexity, and generalization in practical terms.
- Interpret learning curves and decide whether additional training data are likely to help.
- Interpret validation curves and identify a useful hyperparameter region.
- Distinguish true model overfitting from validation problems caused by data leakage.
- Diagnose underfitting and overfitting in a reproducible scikit-learn workflow.
Table 26.1. Quick diagnostic patterns
| Training performance | Validation performance | Likely diagnosis | Typical response | |
|---|---|---|---|---|
| Weak | Weak and similar | Underfitting / high bias | Increase useful complexity; improve features; reduce excessive regularization. | |
| Very strong | Clearly weaker | Overfitting / high variance | Reduce complexity; regularize; get more data; improve validation discipline. | |
| Strong | Strong and close | Good generalization | Keep the model region; verify on a protected test set. | |
| Strong | Unstable across folds | Variance / data sensitivity | Use cross-validation; inspect sample size, groups, drift, and leakage. | |
| CORE IDEA A low training error is not the final objective. Supervised learning aims for low error on future, unseen observations drawn from the intended deployment population. | ||||
26.1 Underfitting
Underfitting occurs when the model class, feature representation, or training configuration is too limited to capture important structure in the data. The result is poor performance even on the observations that the model was allowed to learn from.
Table 26.2. Common signs and causes of underfitting
| Signal or cause | What it means | Example |
|---|---|---|
| High training error | The model cannot fit the training relationships adequately. | A depth-1 tree is asked to represent a strongly curved boundary. |
| High validation error | Unseen-data performance is also weak. | Validation accuracy remains close to training accuracy, but both are low. |
| Insufficient features | Important predictive information is absent from X. | Demand is modeled without season, price, or promotion variables. |
| Excessive regularization | The model is constrained too strongly. | A very large Ridge penalty shrinks useful effects too aggressively. |
| Excessively simple model | The hypothesis class lacks flexibility. | A straight line is used for a strongly nonlinear relationship. |
Training and validation errors move together
A defining feature of underfitting is that training and validation performance are both unsatisfactory and often fairly close. Because the model does not fit the training data well, there is little reason to expect it to perform much better on validation data.
High training error + High validation error → likely underfitting The word “high” is relative to an appropriate baseline and the task’s operational requirements. |
Typical remedies
- Increase model flexibility only when the validation design is trustworthy.
- Engineer features that expose relevant nonlinearities, interactions, or domain information.
- Reduce excessive regularization or overly restrictive hyperparameter settings.
- Check whether preprocessing has destroyed useful signal.
- Verify that the target is predictable from the available inputs before increasing complexity.
| CAUTION Poor validation performance does not automatically mean underfitting. If training performance is strong while validation is weak, the problem is more consistent with overfitting, leakage, distribution shift, or an unsuitable validation split. | |
PYTHON • A deliberately underfit decision tree from sklearn.tree import DecisionTreeClassifier |
26.2 Overfitting
Overfitting occurs when a model learns patterns that are specific to the training sample rather than patterns that generalize reliably. An overfit model can reproduce the training observations extremely well while making noticeably worse predictions on unseen data.
Table 26.3. Main overfitting mechanisms
| Mechanism | Why it increases risk | Illustration |
|---|---|---|
| Excessive model complexity | The model has enough flexibility to reproduce small sample-specific details. | An unrestricted tree creates very small terminal leaves. |
| Small training dataset | There is not enough evidence to distinguish stable signal from chance patterns. | A high-dimensional model is trained on only a few dozen rows. |
| Model memorization | Rules become specific to individual observations instead of reusable structure. | Training accuracy approaches 100% while validation stalls. |
| Data leakage | Validation information reaches training or feature construction. | Scaling or feature selection is fitted before the split, or future information is included. |
| Noisy or irrelevant features | Flexible models can exploit accidental correlations. | Random identifiers appear predictive in one sample. |
The generalization gap
The difference between training performance and validation performance is often called the generalization gap. A large gap is a warning that the fitted model behaves much better on familiar observations than on unseen ones.
Generalization gap = Training score − Validation score For error metrics, compare the corresponding training error and validation error instead of subtracting scores blindly. |
Data leakage can mimic excellent modeling
Leakage deserves special attention because it can create a misleading version of overfitting: both validation and test-like metrics may look unrealistically strong when the evaluation pipeline accidentally includes information that would not be available at prediction time. Leakage is therefore not solved simply by reducing model complexity; the data pipeline and split logic must be repaired.
| LEAKAGE CHECK Before interpreting a suspiciously high score, verify that preprocessing, imputation, feature selection, target encoding, temporal aggregation, and hyperparameter tuning were all confined to appropriate training folds. | |
PYTHON • An intentionally high-variance tree from sklearn.tree import DecisionTreeClassifier |
26.3 Bias–variance trade-off
Bias and variance provide a useful conceptual language for reasoning about model complexity. Bias describes error caused by restrictive assumptions that prevent the model from representing the underlying relationship. Variance describes sensitivity to the particular training sample: if small changes in the data cause large changes in the fitted model, variance is high.
Table 26.4. Bias, variance, and complexity
| Model region | Bias | Variance | Training fit | Generalization risk |
|---|---|---|---|---|
| Too simple | High | Low | Weak | Underfitting |
| Useful complexity | Moderate / controlled | Moderate / controlled | Good | Best validation region |
| Too complex | Low on training data | High | Extremely strong | Overfitting |
Model complexity is not one universal number
Complexity depends on the model family. For a decision tree, depth and minimum leaf size are important. For k-nearest neighbors, a very small k can be highly flexible. For polynomial regression, degree controls flexibility. For regularized linear models, smaller regularization penalties generally allow more flexible coefficients. The direction of a hyperparameter must therefore be interpreted in context.
Table 26.5. Examples of complexity-controlling hyperparameters
| Model | Lower-complexity direction | Higher-complexity direction | |
|---|---|---|---|
| Decision tree | Smaller max_depth; larger min_samples_leaf | Larger / unlimited max_depth; very small leaves | |
| Random forest tree components | Shallower trees; larger leaves | Deeper trees; smaller leaves | |
| Polynomial regression | Lower polynomial degree | Higher polynomial degree | |
| k-nearest neighbors | Larger k | Smaller k | |
| Ridge / Lasso | Larger regularization strength | Smaller regularization strength | |
| IMPORTANT The best model is not the model with the lowest training error. It is the model configuration that achieves the most reliable validation performance under a validation design that matches deployment. | |||
26.4 Learning curves
A learning curve evaluates model performance as the amount of training data increases. It usually displays a training score and a cross-validated validation score for several training-set sizes. The shape of the two curves helps distinguish a data shortage from a model-capacity limitation.
Table 26.6. Reading learning-curve patterns
| Observed pattern | Likely interpretation | Would much more similar data help? |
|---|---|---|
| Training and validation both plateau at a weak level | High bias / model limitation | Usually not much unless the model or features change. |
| Training strong; validation weaker; gap narrows as data grows | High variance / data shortage | Often yes. |
| Both curves strong and close | Good generalization | Additional data may still help modestly. |
| Validation curve unstable | Small sample, heterogeneous groups, drift, or noisy evaluation | Potentially, but first inspect the split and data structure. |
Training score versus dataset size
With very small training subsets, a flexible model can often fit the available observations extremely well. As more observations are added, the training task becomes harder and the training score may decline toward a stable level. This decline is not automatically bad; it can indicate that the score is becoming more realistic.
Validation score versus dataset size
The validation score often improves as the model receives more training examples. If it continues to rise while the gap to the training curve shrinks, collecting more similar data may be valuable. If both curves have already converged at a weak score, the model family or feature representation is likely the more important bottleneck.
PYTHON • Compute a learning curve with cross-validation import numpy as np |
Diagnosing data shortage
A data-shortage diagnosis is strongest when the training score is substantially better than validation performance and the validation curve is still improving as more data are added. The gap suggests variance, while the upward validation trend suggests that additional representative examples may reduce it.
Diagnosing model limitations
If training and validation curves converge early at a weak level, more examples alone are unlikely to solve the problem. The model may be too constrained, the features may omit relevant information, or the target may contain substantial irreducible uncertainty.
| VALIDATION DISCIPLINE Learning curves should be computed inside cross-validation or another appropriate resampling scheme. Repeatedly looking at the final test set while increasing training size turns the test set into a tuning instrument. |
26.5 Validation curves
A validation curve holds the dataset and evaluation procedure fixed while varying one hyperparameter. It shows how training and validation performance change as model complexity is adjusted. This is especially useful for identifying parameter regions that are clearly too simple or too flexible before a more focused hyperparameter search.
Table 26.7. Validation-curve interpretation
| Region | Training score | Validation score | Interpretation | |
|---|---|---|---|---|
| Low complexity | Low / moderate | Low / moderate | Underfitting likely. | |
| Intermediate complexity | High | Highest or near-highest | Useful generalization region. | |
| Excessive complexity | Very high | Declining or unstable | Overfitting risk. | |
PYTHON • Validation curve for decision-tree depth import numpy as np | ||||
Selecting useful parameter ranges
A validation curve should usually be treated as a diagnostic and search-range tool, not as permission to choose a parameter from a single noisy maximum. If several nearby values perform similarly, prefer a stable region and confirm the choice with cross-validation or a systematic search procedure.
One hyperparameter at a time
A validation curve changes one parameter while the others remain fixed. This makes interpretation easier but does not reveal all interactions among hyperparameters. After identifying plausible ranges, GridSearchCV, RandomizedSearchCV, or another search strategy can evaluate combinations more systematically.
| PRACTICAL RULE Use learning curves to ask “Would more data help?” and validation curves to ask “Is this complexity range useful?” They answer related but different diagnostic questions. |
Practical lab — Diagnose underfitting and overfitting
In this lab, students work with the same nonlinear binary-classification dataset throughout. They first compare three decision-tree complexities, then use a learning curve and a validation curve to justify the diagnosis. The objective is not merely to obtain the highest score; it is to explain why each model behaves the way it does.
Lab objectives
- Create a nonlinear dataset with a protected test split.
- Train deliberately underfit, intermediate, and overfit decision trees.
- Compare training accuracy, validation accuracy, and the generalization gap.
- Build a learning curve and interpret whether more data are likely to help.
- Build a validation curve for max_depth and identify a useful complexity range.
- Evaluate the selected model once on the protected test set.
- Write a short diagnosis supported by quantitative and graphical evidence.
Step 1 — Create the dataset and protected split
PYTHON • Generate a nonlinear classification problem from sklearn.datasets import make_moons | |
| WHY THREE SPLITS? The validation set supports diagnosis and model selection. The test set stays untouched until the final model configuration has been chosen. |
Step 2 — Train models with three complexity levels
PYTHON • Underfit, intermediate, and high-variance trees from sklearn.tree import DecisionTreeClassifier |
Step 3 — Compare training and validation performance
PYTHON • Inspect the generalization gap import pandas as pd |
Interpretation framework
| Observation | Diagnosis |
|---|---|
| Low training and low validation accuracy, small gap | Underfitting / high bias. |
| Very high training accuracy, lower validation accuracy, large gap | Overfitting / high variance. |
| Strong validation accuracy with a controlled gap | Better generalization region. |
Step 4 — Plot a learning curve for the unrestricted tree
PYTHON • Does additional data reduce the variance problem? import numpy as np |
Students should describe both the absolute validation level and the gap between curves. If the validation curve improves as the training size grows while the gap narrows, more representative data may help reduce variance. If both curves were to converge at a weak level, the diagnosis would shift toward model or feature limitations.
Step 5 — Build a validation curve for max_depth
PYTHON • Find a useful depth region from sklearn.model_selection import validation_curve |
PYTHON • Plot the validation curve plt.plot(param_range, train_mean, marker="o", label="Training") |
Step 6 — Retrain the selected depth and evaluate once on test data
PYTHON • Final protected test evaluation final_model = DecisionTreeClassifier( | |
| FINAL-EVALUATION RULE Do not return to the test set after seeing its score and then choose a different depth. If you do, the test set has become part of the tuning process. |
Step 7 — Optional extension: demonstrate excessive regularization
Underfitting is not limited to shallow trees. Students can repeat the same diagnostic logic with a regularized linear model and observe how excessively strong regularization can weaken both training and validation performance.
PYTHON • Optional Ridge-style classification extension from sklearn.pipeline import make_pipeline |
Student error-analysis report
Students should submit a short diagnosis rather than only screenshots or raw metric values. A strong report explains the evidence, identifies the likely failure mode, and recommends the next experiment.
Table 26.8. Suggested report structure
| Report section | Required content |
|---|---|
| 1. Model comparison | Training accuracy, validation accuracy, and generalization gap for the three trees. |
| 2. Underfitting diagnosis | Evidence showing why the shallow model is too simple. |
| 3. Overfitting diagnosis | Evidence showing why the unrestricted tree has excessive variance. |
| 4. Learning-curve interpretation | State whether more similar training data appear likely to help and why. |
| 5. Validation-curve interpretation | Identify the useful depth region and where excessive complexity begins. |
| 6. Final evaluation | Report the protected test score for the selected model. |
| 7. Recommendation | Propose one next step: more data, feature improvement, regularization, pruning, or a different model family. |
Discussion questions
1. Why can an unrestricted decision tree have nearly perfect training accuracy and still be a poor final model?
2. If both training and validation accuracy are low, why is collecting more data not always the first remedy?
3. What learning-curve pattern would make you more confident that additional data could improve generalization?
4. Why should a validation curve be interpreted as a region rather than blindly choosing one noisy maximum?
5. How can data leakage produce misleadingly optimistic evidence and invalidate an underfitting/overfitting diagnosis?
6. Which evidence in this lab supports the final choice of max_depth?
Chapter summary
- Underfitting is associated with a model that is too simple or too constrained: training and validation performance are both weak.
- Overfitting is associated with excessive sensitivity to the training sample: training performance is very strong but validation performance is materially worse.
- High bias is linked to insufficient flexibility; high variance is linked to excessive sensitivity to the particular training data.
- The goal is not minimum training error but reliable generalization to unseen observations.
- Learning curves vary training-set size and help diagnose whether more representative data are likely to help.
- Validation curves vary a hyperparameter and help identify underfit, useful, and overfit complexity regions.
- Data leakage must be ruled out before trusting unusually strong validation performance.
- A protected test set should be used only after model selection decisions are complete.
Knowledge check
1. What training-versus-validation pattern is most typical of underfitting?
2. What is the generalization gap, and why can a large gap be concerning?
3. How do high bias and high variance relate to model complexity?
4. How can a learning curve distinguish a data shortage from a model-capacity limitation?
5. What does a validation curve show that a learning curve does not?
6. Why can data leakage invalidate conclusions about generalization?
7. Why should the final test set remain untouched during model selection?
| NEXT STEP Next: cross-validation, hyperparameter tuning, and robust model selection. |