Chapter 27 — Cross-Validation
K-fold • Stratification • Groups • Time series • Stable model comparison
Estimating generalization performance more reliably than with a single split
| BRIDGE FROM CHAPTER 26 Underfitting and overfitting were diagnosed by comparing training and validation behavior. Cross-validation makes that validation evidence more reliable by repeating evaluation across several carefully constructed data partitions. |
Chapter map
Table 27.1. From one split to structured resampling
Section | Core question | Primary tool |
|---|---|---|
| 27.1 Single split | How much can one random validation set influence the result? | Repeated train/validation comparisons |
| 27.2 K-fold | How can every observation contribute to validation? | KFold / cross_val_score |
| 27.3 Stratified K-fold | How do we preserve class proportions? | StratifiedKFold |
| 27.4 Group CV | How do we keep related entities together? | GroupKFold |
| 27.5 Time-series CV | How do we respect temporal order? | TimeSeriesSplit |
| 27.6 Reporting | How stable is performance across folds? | Mean, SD, min, max |
Chapter overview
A validation score is an estimate, not a permanent property of a model. If the estimate depends strongly on which observations happened to enter one validation set, model selection can become unstable. Cross-validation reduces this dependence by evaluating the learning procedure repeatedly on different training and validation partitions.
The important idea is not simply to “use more folds.” The partitioning strategy must reflect the data-generating process. Ordinary K-fold is suitable for many independent observations, stratification is often preferable for classification, group-aware splitting is essential when observations share an entity, and time-series validation must preserve chronological order.
Learning objectives
- Explain why a single train-validation split can produce an unstable estimate of model performance.
- Describe the full K-fold cross-validation procedure and the role of k.
- Use stratified folds to preserve class proportions in classification tasks.
- Use group-aware folds to prevent the same entity from appearing in both training and validation data.
- Use forward, time-ordered validation without allowing future observations to influence the past.
- Report cross-validation results with mean, standard deviation, minimum, and maximum scores.
- Compare several models fairly under exactly the same cross-validation strategy.
- Keep preprocessing inside pipelines so it is refitted separately inside every fold.
27.1 Limitations of a single train-validation split
A single split is easy to understand and fast to compute, but its result can depend substantially on the random sample assigned to validation. This sensitivity is especially important when the dataset is small, the target is imbalanced, rare subgroups exist, or several models have similar true performance.
Table 27.2. Why a single split can be misleading
Problem | What happens | Consequence |
|---|---|---|
| Sensitivity to random sampling | A different random seed moves different observations into validation. | The estimated score can change noticeably. |
| Unstable evaluation | One unusually easy or difficult validation subset dominates the conclusion. | Model ranking may reverse after another split. |
| Small datasets | The validation set contains relatively few observations. | The estimate has high uncertainty. |
| Rare classes or subgroups | Some important cases may be underrepresented or absent. | The score hides performance weaknesses. |
| Hyperparameter search | Many choices are judged on the same small validation sample. | The workflow can overfit the validation set. |
| CORE IDEA Cross-validation does not create new data. It uses the available data more efficiently to obtain a more stable estimate of how the training procedure behaves across plausible validation samples. |
Example 27.1. Demonstrating split-to-split variability
| from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score X, y = load_breast_cancer(return_X_y=True) for seed in [2, 7, 18, 31, 44]: X_train, X_valid, y_train, y_valid = train_test_split( X, y, test_size=0.25, stratify=y, random_state=seed ) model = DecisionTreeClassifier(max_depth=4, random_state=0) model.fit(X_train, y_train) score = accuracy_score(y_valid, model.predict(X_valid)) print(f"seed={seed:>2} validation accuracy={score:.3f}") |
If the scores differ, that variability is useful information: the performance estimate itself is uncertain. Cross-validation summarizes this variability instead of hiding it behind one random seed.
27.2 K-fold cross-validation
K-fold cross-validation divides the dataset into k non-overlapping subsets called folds. Each fold is used once as validation data while the remaining k − 1 folds are used for training. Every observation therefore participates in validation exactly once during one complete K-fold run.
- Divide the data into k folds.
- Train on k − 1 folds.
- Validate on the remaining fold.
- Repeat until every fold has served as validation.
- Aggregate the fold scores.
Figure 27.1. Five-fold cross-validation rotation
Round | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5 |
|---|---|---|---|---|---|
1 | Validate | Train | Train | Train | Train |
2 | Train | Validate | Train | Train | Train |
3 | Train | Train | Validate | Train | Train |
4 | Train | Train | Train | Validate | Train |
5 | Train | Train | Train | Train | Validate |
Mean CV score = (s₁ + s₂ + ⋯ + sₖ) / k The fold scores s₁, …, sₖ are performance estimates from different validation subsets. |
Example 27.2. Basic 5-fold cross-validation
| from sklearn.model_selection import KFold, cross_val_score from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler cv = KFold(n_splits=5, shuffle=True, random_state=42) model = make_pipeline( StandardScaler(), LogisticRegression(max_iter=2000) ) scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy") print(scores) print("Mean:", scores.mean()) print("Standard deviation:", scores.std()) |
Common choices are 5-fold and 10-fold cross-validation. Increasing k gives each model more training data per round, but it also requires more fits. The best choice depends on dataset size, computational budget, and the stability required for the decision.
Table 27.3. Effect of the number of folds
Choice | Training fraction per round | Typical trade-off |
|---|---|---|
| k = 5 | 80% | Good default balance between stability and computation. |
| k = 10 | 90% | More computation; often useful for smaller datasets. |
| Large k | Close to 100% | Expensive; fold scores can be strongly correlated. |
| PREPROCESSING RULE When scaling, encoding, imputation, feature selection, or other learned preprocessing is required, place it inside a Pipeline. Each fold must learn preprocessing from its training portion only. |
27.3 Stratified K-fold
In classification, ordinary K-fold does not explicitly guarantee that every fold has a similar class distribution. Stratified K-fold constructs folds so that the proportion of each class is approximately preserved. This is particularly valuable when one class is much less frequent than another.
Table 27.4. K-fold versus stratified K-fold
Property | KFold | StratifiedKFold |
|---|---|---|
| Preserves class proportions | Not guaranteed | Yes, approximately |
| Typical use | General independent observations | Classification |
| Imbalanced classification | Can create uneven folds | Usually preferable |
| Works with shuffle | Yes | Yes |
| Prevents group leakage | No | No — use a group-aware splitter |
Example 27.3. Preserving class proportions across folds
| from sklearn.model_selection import StratifiedKFold, cross_val_score from sklearn.metrics import make_scorer, f1_score cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) model = make_pipeline( StandardScaler(), LogisticRegression(max_iter=2000) ) f1_scores = cross_val_score(model, X, y, cv=cv, scoring="f1") print("Fold F1 scores:", f1_scores) print(f"Mean F1: {f1_scores.mean():.3f}") |
| IMPORTANT Stratification balances class proportions, not every possible subgroup. If observations belong to repeated patients, customers, devices, schools, or other entities, group-aware validation may be more important than ordinary stratification. |
27.4 Group cross-validation
Many datasets contain multiple observations from the same real-world entity. For example, a patient may have several visits, a customer may have many transactions, a machine may generate repeated sensor windows, and a student may appear in multiple assessments. Randomly separating those rows can leak entity-specific information from training into validation.
Group cross-validation assigns all observations from the same group to the same side of a fold. The model is therefore validated on groups that were not seen during that fold’s training stage.
Table 27.5. Examples of groups that should stay together
Dataset | Observation unit | Possible group |
|---|---|---|
| Medical | Visit or image | Patient |
| Retail | Transaction | Customer |
| Industrial IoT | Sensor window | Machine / device |
| Education | Assessment record | Student |
| Marketing | Ad impression | Campaign or account |
Example 27.4. Keeping entities in separate folds
| import numpy as np from sklearn.model_selection import GroupKFold, cross_val_score from sklearn.ensemble import RandomForestClassifier # Example: every three consecutive rows belong to one entity. groups = np.arange(len(y)) // 3 cv = GroupKFold(n_splits=5) model = RandomForestClassifier( n_estimators=300, random_state=42, n_jobs=-1 ) scores = cross_val_score( model, X, y, cv=cv, groups=groups, scoring="accuracy" ) print("Group-CV accuracy:", scores) |
| LEAKAGE CHECK Ask: “Could two rows contain information about the same real-world entity?” If yes, a random row-level split may be optimistic even when no target column was explicitly leaked. |
27.5 Time-series cross-validation
Time-ordered data require a different logic. A model intended to predict the future should never be evaluated by training on future observations and validating on earlier observations. Time-series cross-validation therefore uses forward validation: training occurs on earlier periods and validation occurs on later periods.
Table 27.6. Expanding-window forward validation
Round | Training period | Validation period |
|---|---|---|
| 1 | Periods 1–4 | Period 5 |
| 2 | Periods 1–5 | Period 6 |
| 3 | Periods 1–6 | Period 7 |
| 4 | Periods 1–7 | Period 8 |
| NO FUTURE-TO-PAST LEAKAGE Do not randomly shuffle a forecasting dataset before cross-validation. Feature engineering must also respect time: rolling statistics, normalizers, and target-derived aggregates must be computed without using future observations. |
Example 27.5. Forward validation with TimeSeriesSplit
| import numpy as np from sklearn.model_selection import TimeSeriesSplit from sklearn.linear_model import Ridge from sklearn.metrics import mean_absolute_error X_time = np.arange(120).reshape(-1, 1) y_time = 0.4 * X_time.ravel() + np.sin(X_time.ravel() / 6) tscv = TimeSeriesSplit(n_splits=5) mae_scores = [] for train_idx, valid_idx in tscv.split(X_time): model = Ridge(alpha=1.0) model.fit(X_time[train_idx], y_time[train_idx]) pred = model.predict(X_time[valid_idx]) mae_scores.append(mean_absolute_error(y_time[valid_idx], pred)) print("Fold MAE values:", mae_scores) |
27.6 Reporting cross-validation results
A cross-validation result should communicate both central performance and variability. Reporting only the mean can hide an unstable model that performs well on some folds and poorly on others.
Mean = (1/k) Σ sᵢ | SD = √[(1/k) Σ(sᵢ − mean)²] For descriptive reporting, the fold-score standard deviation summarizes variation across folds. |
Table 27.7. Recommended cross-validation summary
Statistic | Interpretation | Why report it? |
|---|---|---|
| Mean | Average performance across folds | Primary estimate of typical performance |
| Standard deviation | Spread of fold scores | Indicates stability across validation subsets |
| Minimum | Worst observed fold | Highlights a potential weak case |
| Maximum | Best observed fold | Shows optimistic end of observed range |
| Individual fold scores | Full pattern | Helps detect one unusually difficult fold |
Example 27.6. Reporting mean, variability, and range
| import numpy as np from sklearn.model_selection import cross_val_score scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy") print(f"Mean: {scores.mean():.3f}") print(f"SD: {scores.std():.3f}") print(f"Min: {scores.min():.3f}") print(f"Max: {scores.max():.3f}") print("Fold scores:", np.round(scores, 3)) |
A small standard deviation is encouraging only when the cross-validation strategy itself is appropriate. Stable scores from a leaky split are still misleading. Validation design comes before summary statistics.
Table 27.8. Reading mean and variability together
Pattern | Possible interpretation | Next action |
|---|---|---|
| High mean, low SD | Strong and stable under these folds | Confirm on a protected final test set |
| High mean, high SD | Good average but subgroup/sampling sensitivity | Inspect difficult folds and data segments |
| Low mean, low SD | Consistently weak model | Improve features/model; diagnose underfitting |
| Low mean, high SD | Weak and unstable | Check data size, leakage, distribution shifts, and model fit |
Practical lab — Compare several models using the same cross-validation strategy
Goal: compare multiple classifiers fairly by holding the cross-validation design constant. We will use the Breast Cancer Wisconsin dataset, a 5-fold StratifiedKFold splitter, and several metrics. Logistic regression is placed in a scaling pipeline so that preprocessing is learned independently inside each fold.
Lab step 1 — Load the data and define one shared splitter
| from sklearn.datasets import load_breast_cancer from sklearn.model_selection import StratifiedKFold, cross_validate X, y = load_breast_cancer(return_X_y=True) cv = StratifiedKFold( n_splits=5, shuffle=True, random_state=42 ) |
| FAIR-COMPARISON RULE Reuse the exact same splitter for every model. Otherwise differences in fold difficulty can be mistaken for differences in model quality. |
Lab step 2 — Define the models
| from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier models = { "Logistic regression": make_pipeline( StandardScaler(), LogisticRegression(max_iter=2000) ), "Decision tree": DecisionTreeClassifier( max_depth=4, random_state=42 ), "Random forest": RandomForestClassifier( n_estimators=300, random_state=42, n_jobs=-1 ), } |
Table 27.9. Why these models are compared
Model | Role in the comparison |
|---|---|
| Logistic regression | Linear baseline; scaling is learned inside its Pipeline. |
| Decision tree | Nonlinear single-tree model with low preprocessing requirements. |
| Random forest | Ensemble model that reduces the variance of individual trees. |
Lab step 3 — Evaluate several metrics on the same folds
Step 3A. Define the metrics and result containers
| import pandas as pd scoring = { "accuracy": "accuracy", "precision": "precision", "recall": "recall", "f1": "f1", "roc_auc": "roc_auc", } rows = [] fold_details = {} |
Step 3B. Run cross-validation and summarize each model
| for name, model in models.items(): result = cross_validate( model, X, y, cv=cv, scoring=scoring, n_jobs=-1 ) fold_details[name] = result rows.append({ "Model": name, "Accuracy mean": result["test_accuracy"].mean(), "F1 mean": result["test_f1"].mean(), "F1 SD": result["test_f1"].std(), "F1 min": result["test_f1"].min(), "F1 max": result["test_f1"].max(), "ROC AUC mean": result["test_roc_auc"].mean(), }) summary = pd.DataFrame(rows).sort_values("F1 mean", ascending=False) print(summary.round(3)) |
Lab step 4 — Inspect the fold-level stability
| import numpy as np for name, result in fold_details.items(): f1 = result["test_f1"] print(f"\n{name}") print("F1 by fold:", np.round(f1, 3)) print(f"mean={f1.mean():.3f} sd={f1.std():.3f} " f"min={f1.min():.3f} max={f1.max():.3f}") |
Students should not choose a winner from the mean alone. Compare the mean, spread, worst fold, operational metric priorities, computational cost, and interpretability requirements.
Lab step 5 — Write the comparison report
Table 27.10. Required student report
Question | Evidence to include |
|---|---|
| Which model has the strongest average performance? | Mean F1 and ROC AUC |
| Which model is most stable? | F1 standard deviation and fold range |
| Does one model have a weak fold? | Minimum score and fold-level values |
| Is the ranking consistent across metrics? | Accuracy, precision, recall, F1, ROC AUC |
| Why is the comparison fair? | Same folds, same data, pipeline-contained preprocessing |
| What would you do before deployment? | Tune on training/CV data, then evaluate once on a protected test set |
Discussion questions
1. Why might a model with the highest mean score still be a poor operational choice?
2. What does a large fold-to-fold standard deviation suggest about the dataset or model?
3. Why is StratifiedKFold preferable to ordinary KFold for this classification lab?
4. Why must StandardScaler stay inside the logistic-regression pipeline?
5. When would GroupKFold be more appropriate than StratifiedKFold?
6. Why must time-series validation preserve chronological order?
Common mistakes to avoid
- Comparing models with different random folds and attributing split differences to the models.
- Scaling or selecting features on the full dataset before cross-validation.
- Using ordinary random K-fold when the same entity appears in several rows.
- Shuffling time-series data and allowing future observations to enter training folds.
- Reporting only the mean score and ignoring instability across folds.
- Using cross-validation as a substitute for a final untouched test set when an unbiased final estimate is required.
- Tuning repeatedly on the same cross-validation results without recognizing that model selection itself can overfit the resampling process.
Chapter summary
Table 27.11. Cross-validation checklist
Concept | Key message |
|---|---|
| Single split | Fast but can be sensitive to one random sample. |
| K-fold | Rotates the validation fold so every observation is evaluated once. |
| Stratified K-fold | Preserves class proportions and is a strong default for classification. |
| Group CV | Keeps related observations together and prevents entity leakage. |
| Time-series CV | Uses forward validation and prevents future-to-past leakage. |
| Reporting | Use mean, SD, min, max, and fold-level scores to describe stability. |
| Fair comparison | Use the same folds and leakage-safe pipelines for all models. |
Knowledge check
1. What limitation of a single validation split does cross-validation primarily address?
2. Describe the five main steps of K-fold cross-validation.
3. Why is stratification useful in classification?
4. Give two examples where GroupKFold is necessary.
5. Why is random shuffling inappropriate for forecasting validation?
6. What do a cross-validation mean and standard deviation communicate?
7. Why should preprocessing be placed inside a pipeline during cross-validation?
8. Why should several models be compared with exactly the same cross-validation strategy?
| NEXT STEP Cross-validation provides a reliable framework for model comparison and hyperparameter evaluation. The next stage is to use that framework deliberately during model selection and tuning without leaking information from the final test set. |