Chapter overview

A model that performs perfectly on the observations it has already seen may still fail on new cases. Dataset splitting creates a controlled simulation of deployment: the model learns from one portion of the data, development decisions are made on another, and the final result is measured on observations that were protected from the entire modeling process.

This chapter develops a practical decision framework for four common splitting strategies: random, stratified, group-based, and time-based splitting. It also treats leakage as a design problem rather than a last-minute debugging problem. Every example uses Python and scikit-learn, and the practical lab asks students to justify the split that matches the way predictions will be used in reality.

Learning objectives

• Explain the difference between training performance and generalization performance.
• Assign clear roles to training, validation, and test sets.
• Use train_test_split reproducibly and interpret its main parameters.
• Preserve class proportions with stratification when appropriate.
• Keep related observations together with group-aware splitting.
• Respect chronology and prediction timing with time-based splitting.
• Recognize preprocessing, feature, target, temporal, group, and duplicate leakage.
• Audit a split with quantitative checks before training a model.

Chapter map

Table 9.1. Questions answered in this chapter

Prerequisites

Students should be comfortable with pandas DataFrames, selecting columns and rows, and the ideas of features X and target y. Familiarity with a simple classifier is helpful but not required; the chapter emphasizes evaluation design rather than algorithm complexity.

9.1 Why data splitting is necessary

Data splitting separates learning from evaluation. Without that separation, a model can appear successful simply because it is being asked to reproduce patterns, noise, or individual cases that influenced its training.

Training performance versus generalization

Training performance answers a limited question: how well does the fitted model reproduce the examples used to estimate its parameters? Generalization performance asks the question that matters after deployment: how well will the model perform on relevant observations that were not used during learning?

The difference between the two is the generalization gap. A small gap is not automatically good: both scores could be poor because the model underfits. A large gap is a warning that the model may have memorized idiosyncrasies of the training set, exploited leakage, or become too complex for the available data.

Table 9.2. Reading training and unseen-data performance together

Unseen data as a simulation of deployment

A holdout set is not valuable merely because its rows have a different index. It is valuable when it represents the type of novelty expected in production. For a one-row-per-customer churn dataset, random unseen customers may be appropriate. For repeated hospital visits, the meaningful novelty may be an entirely unseen patient. For forecasting, it is future time. For image recognition, it may be a new person, device, site, or acquisition session.

Honest model evaluation

An evaluation is honest when the test observations do not influence model design, preprocessing choices, feature selection, threshold selection, or hyperparameter tuning. The test set is therefore not a convenient dataset for repeated experimentation. It is a final examination whose questions remain sealed until the development process is complete.

• Use training data to fit model parameters and preprocessing statistics.
• Use validation data or cross-validation to compare models and tune decisions.
• Use the test set once, after the entire modeling recipe has been fixed.
• Report the test result even when it is less favorable than the validation result.

Why evaluating on training data is optimistic

The learning algorithm explicitly searches for a model that reduces error on the training observations. Reusing those observations for evaluation rewards both useful structure and accidental fit. The resulting score is an in-sample score, not an estimate of performance on new cases.

The exact scores depend on the data and software version, but the unrestricted tree commonly obtains a much higher training score than test score. The purpose of the example is not to condemn decision trees; it is to show why an unseen set is necessary to reveal the gap.

Checkpoint

A model obtains 99% accuracy on its training data and 73% on a correctly protected test set. Which number estimates deployment performance more directly? What additional evidence would you request before trusting it? Record a two-sentence answer.

9.2 Basic dataset split

A three-way split assigns a distinct job to each subset. The separation prevents development decisions from quietly adapting to the same data used for the final claim.

Figure 9.1. The three-way holdout design

The training set

The training set is the only subset used to estimate model parameters. The model learns coefficients, tree rules, neighbor structures, or other internal representations from it. Data-dependent preprocessing steps must also be fitted on the training set: imputation values, scaling statistics, encoding categories, selected features, and resampling decisions.

The validation set

The validation set supports development choices. It is used to compare candidate algorithms, tune hyperparameters, select features, choose probability thresholds, and diagnose underfitting or overfitting. When cross-validation is used, several temporary validation folds are created within the training portion, and a fixed standalone validation set may be unnecessary.

The test set

The test set estimates final performance after the full modeling procedure is frozen. It should not influence model selection, preprocessing decisions, or metric choice. If the test result triggers another round of tuning, the test set has become validation data; a new independent test set is then needed for an unbiased final estimate.

Table 9.3. The role of each subset

Typical proportions

There is no universally correct percentage. The relevant quantities are the absolute number of training observations, the number of independent evaluation units, the rarity of important outcomes, and the precision required from the final metric.

Table 9.4. Starting points for split proportions

The effect of dataset size

A test set of 20% means 200 observations in a dataset of 1,000 but 200,000 observations in a dataset of one million. The second test set may be unnecessarily large for many metrics, while the first may contain too few cases from a rare class. Always inspect counts after splitting.

For classification, count each class in every subset. For grouped data, count unique groups as well as rows. For time-based data, report the date range and duration. For regression, compare the distribution and range of the target, especially the number of extreme or operationally important values.

9.3 Using train_test_split

The scikit-learn train_test_split function is a convenient way to create a random holdout. It accepts aligned arrays or DataFrames and returns training and test subsets in matching order.

A basic split

Main parameters

Table 9.5. Important train_test_split parameters

Creating training, validation, and test sets

train_test_split creates two subsets per call. A three-way split is commonly produced in two stages. First, protect the test set. Second, divide the remaining development data into training and validation sets. The second proportion must be calculated relative to the remainder.

Random state and reproducibility

Passing the same integer random_state to the same splitting code and same ordered input data makes the random partition repeatable across calls. This is essential for fair experiments: two candidate models should not appear different merely because they were evaluated on different random rows.

A seed does not make the whole project reproducible by itself. The dataset version, row order, preprocessing code, software versions, model randomness, and evaluation settings must also be recorded. The seed is part of an experiment specification, not a guarantee that all environments will behave identically forever.

Shuffling

Shuffling is useful when the row order is arbitrary or organized by class, source file, or collection batch. It helps prevent one contiguous region from becoming the entire test set. Shuffling is inappropriate when order carries information that deployment must respect, particularly forecasting and other future-prediction problems.

Audit the output, not only the code

9.4 Stratified splitting

A stratified split preserves the target-class proportions approximately across subsets. It is often the safest basic strategy for classification when observations are independent and no group or time boundary is required.

Why class proportions matter

Suppose 5% of observations are fraudulent. A purely random split can, by chance, place a noticeably different percentage in training and testing, especially when the dataset or minority class is small. This affects both learning and evaluation. Stratification reduces that accidental imbalance by allocating samples according to their class labels.

Stratification is approximate

Integer sample counts constrain the result. If a class contains seven observations, it is impossible to reproduce its overall proportion exactly in every subset. Scikit-learn aims to preserve relative frequencies as closely as the requested split and class counts allow.

Limitations for very rare classes

• A class represented by one observation cannot appear in both training and testing.
• A tiny test count produces an unstable recall estimate: one additional error can change the metric dramatically.
• Rare-class examples may be correlated by source, patient, device, or event, so stratifying rows can still leak groups.
• If the minority class appears only in recent time, class balance and chronological realism may conflict.
• Collecting more labeled cases or redesigning the evaluation may be more responsible than forcing a split.

When stratification is not enough

Standard stratification protects a single class-label distribution; it does not automatically protect group membership, time order, geographic independence, or multiple labels attached to each observation. If several constraints matter, prioritize the deployment boundary. StratifiedGroupKFold can help during cross-validation when both class balance and group separation are needed, but perfect balance may be impossible when groups have different sizes or class mixtures.

A class-count audit

Decision rule

Use a stratified row split when the target is categorical, each row is an approximately independent unit, chronology is not part of the prediction problem, and each class contains enough examples. If any of these conditions fails, move to a more appropriate splitting design.

9.5 Group-based splitting

Repeated observations from the same entity are usually correlated. A random row split may therefore place near-duplicates or entity-specific signatures on both sides of the evaluation boundary.

What is a group?

A group identifier marks observations that must remain together. Common examples include patient_id, customer_id, person_id, machine_id, household_id, school_id, store_id, acquisition_session, or source_document. The identifier is used by the splitter even when it should not be used as a predictive feature.

Table 9.6. Examples of dependent observations

Why entity overlap is leakage

Imagine a model that predicts whether a machine will fail. Each machine contributes hundreds of sensor windows. If windows from the same machine appear in training and testing, the model may recognize that machine's vibration profile. The test score then estimates performance on more windows from familiar machines, not on a new machine. Whether this is leakage depends on the intended deployment, but it must be an explicit choice.

Using GroupShuffleSplit

Group-aware validation options

Table 9.7. Common group-aware strategies

Audit group separation

Questions to settle with domain experts

• Will deployment predict new observations for known entities, entirely new entities, or both?
• Which identifier captures the strongest dependency: person, household, site, device, or session?
• Are some groups so large that they dominate the row count or class distribution?
• Should sites, geographic regions, or acquisition batches be held out to test broader transfer?
• Can a group appear under multiple identifiers because of data-integration errors?

9.6 Time-based splitting

When predictions are made about the future, evaluation must preserve chronology. Training on later observations and testing on earlier ones gives the model access to information that would not exist at the historical prediction time.

Figure 9.2. A chronological holdout with an optional gap

Training on the past and testing on the future

A chronological holdout first sorts observations by the relevant event or prediction time, selects a cutoff, trains on observations before the cutoff, and evaluates on observations after it. The test window should represent a realistic deployment horizon such as the next week, quarter, season, or maintenance cycle.

Why random splitting is dangerous for forecasting-like problems

A random split mixes old and new rows. The training data can then include later market conditions, newer equipment behavior, updated policies, or post-event measurements that help predict earlier test cases. Because adjacent observations are often similar, the model may also benefit from autocorrelation: a training row from tomorrow can closely resemble a test row from today.

Temporal leakage

• Computing a customer's total purchases using transactions that occur after the prediction date.
• Using a final hospital discharge code to predict an outcome at admission.
• Filling missing historical values with an average calculated from future observations.
• Randomly splitting overlapping sensor windows so neighboring windows occur in both subsets.
• Using a data extract or revised label that was not available at the intended prediction time.

Concept drift

Concept drift occurs when the relationship between inputs and target changes over time. Customer behavior, fraud tactics, equipment wear, clinical practice, economic conditions, and data-collection systems can evolve. A future holdout naturally tests whether the model survives some of this change; a random split can conceal it by averaging different periods together.

Table 9.8. Common temporal evaluation designs

Using TimeSeriesSplit

TimeSeriesSplit creates successive training sets that contain earlier observations and validation sets that follow them. When equal-duration metrics are required, observations should be equally spaced or first aggregated to a consistent interval. The gap parameter excludes a number of observations between train and validation boundaries.

9.7 Data leakage

Data leakage occurs when information unavailable to the real prediction process influences training, model selection, or evaluation. Leakage makes validation results look better without improving genuine future performance.

A practical definition

Information has leaked when a model receives a shortcut that crosses the intended prediction boundary. The shortcut may be a feature that directly reveals the answer, a preprocessing statistic computed from test rows, a future observation, a repeated entity, or a duplicate. The essential question is not whether a column is technically present; it is whether that information would be legitimately available when the prediction is made.

Table 9.9. Common leakage mechanisms and controls

Preprocessing leakage

Preprocessing can learn from data even when it does not use the target. For example, the mean used for imputation and the mean and standard deviation used for scaling are statistics estimated from the dataset. If they are computed before splitting, the test distribution affects the representation of the training data.

Pipelines protect the boundary

A scikit-learn Pipeline bundles preprocessing and modeling. During fitting, each transformation learns only from the training subset supplied to the pipeline. During prediction, the already-fitted transformations are applied to validation or test data. Pipelines reduce accidental leakage and make the modeling recipe reproducible.

Leakage-safe pipeline example

Feature and target leakage

Feature leakage arises when a feature is unavailable, incomplete, or not yet known at prediction time. Target leakage is a particularly severe form in which a feature directly or indirectly encodes the outcome. A churn prediction dataset might include account_closed_date; a credit-risk dataset might include final_collection_status; a medical diagnosis model might include a treatment code assigned after diagnosis.

Leakage through duplicates

Exact duplicates are easy to detect, but near-duplicates can be more dangerous. Two crops of the same image, overlapping windows from the same signal, repeated exports of the same transaction, or lightly edited text can cross the split. Deduplication should be performed with a rule defined before model evaluation, and duplicate clusters should be assigned as groups when all copies cannot simply be removed.

Duplicate audit in Python

A leakage audit before training

• Prediction time: write the exact moment at which the prediction is made.
• Availability: confirm that every feature exists at that moment.
• Transformation fit: identify which steps learn statistics and fit them only on training data.
• Entity overlap: test whether relevant groups appear in more than one subset.
• Time order: confirm that no training timestamp occurs after a protected future test period.
• Duplicates: search for exact and domain-specific near-duplicates across subsets.
• Target construction: verify that labels and label-derived fields do not enter X.
• Human process: ensure the test set was not used to choose features, thresholds, or metrics.

Practical lab — Compare four splitting strategies

In this lab, students work with a synthetic customer-event dataset containing repeated customers, a chronological variable, and an imbalanced binary target. They create random, stratified, grouped, and time-based splits; audit each split; then explain which strategy is valid under different deployment assumptions.

Lab objectives

• Implement the four splitting strategies with reproducible code.
• Compare class distributions and sample counts.
• Measure customer overlap between training and testing.
• Check whether chronology is respected.
• Connect each technical split to a concrete deployment scenario.

Scenario

A subscription service records several monthly observations per customer. The target, churn_next_month, indicates whether the customer leaves during the following month. The company is considering two uses: scoring future months for existing customers and scoring customers from a newly acquired portfolio. These uses imply different evaluation boundaries.

Part A — Create the dataset

Part B — Define reusable audits

Part C — Random and stratified splits

Part D — Group-based split

Part E — Time-based split

Part F — Interpret the comparison

Complete the interpretation table after running the code. Do not select a winner based only on similar class proportions. Each column reveals a different property of the evaluation design.

Table 9.10. Expected qualitative differences among the four splits

Student tasks

1. Run all four splitting strategies and save the comparison table.
2. Explain why random and stratified splits can have many overlapping customers.
3. Explain why the grouped split can have a different row percentage from 20%.
4. Measure the number of positive test cases for every strategy.
5. For the time split, report the training and test date ranges.
6. Choose the split for predicting future months for existing customers.
7. Choose the split for predicting a newly acquired portfolio of unseen customers.
8. Propose a design when both new customers and future months must be tested.

Part G — Optional model comparison

The following extension trains the same pipeline under each split. The purpose is to demonstrate that a higher score is not necessarily more trustworthy. A random row split may score well because repeated customers occur on both sides; a future split may be harder because the target process changes over time.

Expected findings

• Stratification should preserve the overall positive rate more closely than an unconstrained random split.
• Random and stratified row splits should contain many of the same customer IDs in training and testing.
• The grouped split should have zero customer overlap but may have less stable class balance.
• Only the time-based split should guarantee that the latest training time precedes the earliest test time.
• The most optimistic model score may come from the least realistic split.

Challenge extension

Design a combined future-and-new-customer evaluation. One defensible approach is to choose a future cutoff, identify customers first appearing in the future test window, and evaluate only those customers. Another is to create outer time blocks and keep entities separated within each block. State precisely which population the resulting metric represents and how much data remains.

Lab deliverables

Table 9.11. Practical-lab submission checklist

Split-selection decision guide

Use the following sequence before writing splitting code. The questions are ordered by constraints that usually dominate simple class balancing.

1. Is the prediction about a future period? If yes, begin with a chronological design.
2. Do several rows belong to the same entity, site, device, session, or source? If yes, protect the relevant group boundary.
3. Is the target categorical and imbalanced? If yes, preserve or at least audit class proportions within the valid boundary.
4. Are observations approximately independent and identically distributed? If yes, a shuffled random or stratified split may be appropriate.
5. Is the dataset small? Protect a final test set and use cross-validation within training rather than repeatedly consuming the test set.

Table 9.12. Strategy selection at a glance

Minimum split documentation

Every project report should state the following information so another analyst can reconstruct and challenge the evaluation design.

• Dataset version and number of observations before and after exclusions.
• Unit of observation and target definition.
• Training, validation, and test sizes in both rows and percentages.
• Class counts or target-distribution summary for every subset.
• Random seed and splitter parameters, when randomness is used.
• Group identifier and unique-group counts, when groups are used.
• Date ranges, cutoff, gap, and horizon, when time is used.
• Duplicate handling and leakage checks.
• Explicit statement that the test set was not used during development.

Common mistakes and corrections

Table 9.13. Frequent splitting errors

Knowledge check

Choose one answer for each question. Complete the questions before consulting the answer key.

1. What is the main purpose of a test set?

• A. Fit model parameters
• B. Select features repeatedly
• C. Estimate final performance on protected unseen data
• D. Increase the training score

2. Which parameter makes a shuffled train_test_split repeatable?

• A. test_size
• B. random_state
• C. stratify
• D. train_size

3. In classification, stratify=y is mainly used to:

• A. Scale features
• B. Preserve class proportions approximately
• C. Remove duplicates
• D. Sort observations by time

4. Several visits belong to each patient. Which starting strategy best tests new-patient generalization?

• A. Random row split
• B. Stratified row split only
• C. Group-based split by patient_id
• D. Sort by target value

5. Why can random splitting be invalid for forecasting?

• A. It always creates small sets
• B. It can train on future observations
• C. It cannot handle numeric targets
• D. It removes rare classes

6. Which operation is preprocessing leakage?

• A. Fitting a scaler on training data
• B. Transforming test data with the fitted scaler
• C. Fitting a scaler on all data before splitting
• D. Recording the random seed

7. GroupShuffleSplit test_size primarily refers to:

• A. Features
• B. Target classes
• C. Groups
• D. Calendar days

8. A model is tuned many times after inspecting test results. What has happened?

• A. The test set effectively became validation data
• B. The model became stratified
• C. The target was standardized
• D. Group leakage was removed

9. What should be checked for a time-based holdout?

• A. Training times follow test times
• B. Training and test times are identical
• C. The latest training time precedes the earliest test time
• D. Rows are shuffled after sorting

10. Which statement is most accurate?

• A. The highest score always identifies the best split
• B. Every classification dataset should use only stratification
• C. The split should reproduce the deployment boundary
• D. A fixed random seed prevents all leakage

Answer key and explanations

Table 9.14. Knowledge-check solutions

Score interpretation

• 9–10 correct: ready to justify and audit split strategies.
• 7–8 correct: solid understanding; revisit the boundaries that caused errors.
• 5–6 correct: review group, time, and preprocessing leakage before proceeding.
• 0–4 correct: repeat the chapter examples and practical lab with guided support.

Chapter summary

• Training performance measures fit to familiar data; generalization requires unseen data.
• Training, validation, and test sets have different roles and must not be treated interchangeably.
• train_test_split is appropriate for simple random or stratified row-level holdouts.
• Stratification protects class proportions but does not protect groups or chronology.
• Group-based splitting keeps all observations from the same entity on one side of the boundary.
• Time-based splitting trains on the past and evaluates on the future, revealing drift and preventing future-to-past leakage.
• Leakage can enter through preprocessing, features, targets, time, groups, and duplicates.
• A split must be audited with counts, overlaps, date ranges, and feature-availability checks.
• The most realistic evaluation can produce a lower score, but that score is more useful for decision-making.

Key vocabulary

Table 9.15. Essential terminology

Further reading

• scikit-learn: train_test_split
• scikit-learn: GroupShuffleSplit
• scikit-learn: TimeSeriesSplit
• scikit-learn user guide: cross-validation and splitters

What’s next?

The next stage is to fit preprocessing transformations without crossing the boundaries established here. Numerical scaling, imputation, and categorical encoding should be learned from training data and applied consistently to validation, test, and future observations. A reusable scikit-learn pipeline will make that discipline much easier to maintain.