Chapter 33 — Final Test Evaluation
Freeze the model • Refit once • Evaluate once • Report honestly
Turning model development into a defensible final evaluation
| BRIDGE FROM CHAPTER 32 After a fair comparison has produced one final candidate, the protected test set becomes relevant for the first time. Chapter 33 formalizes the last evaluation step: freeze every modeling choice, refit the final pipeline on the available development data, evaluate once on the untouched test set, and report both strengths and limitations. |
Chapter map
Table 33.1. Chapter roadmap
Section | Main question | Core discipline |
|---|---|---|
| 33.1 Role of the test set | What is the test set for? | A final unbiased estimate after all model and threshold decisions are frozen. |
| 33.2 Retraining the final model | What should be refit? | Use finalized preprocessing and hyperparameters; combine train + validation when appropriate. |
| 33.3 Test report | What evidence belongs in the final report? | Use task-appropriate metrics, residual/confusion analysis, threshold disclosure, and subgroup checks. |
| 33.4 Honest reporting | How should results be communicated? | Show favorable and unfavorable findings, uncertainty, limits, and intended use conditions. |
| Practical activity | What should students deliver? | A formal final-test evaluation section with reproducible code and a decision-ready interpretation. |
Chapter overview
The final test evaluation is not another opportunity to improve the model. It is a measurement step. The test set should represent data that the model-development process has not used for fitting, feature selection, hyperparameter tuning, threshold experimentation, or model choice. Once the test results are observed, changing the model in response to those results turns the test set into a new validation set.
A defensible final report therefore begins with process discipline. The model architecture, preprocessing, hyperparameters, feature set, decision threshold, and primary metrics should already be fixed. The final pipeline can then be retrained on the complete development data and evaluated once under the conditions that best approximate intended deployment.
Learning objectives
- Explain why a protected test set is used only after model selection is complete.
- Distinguish validation-driven model development from final test measurement.
- Retrain a finalized pipeline using training and validation data without changing its specification.
- Build a formal classification or regression test report using task-appropriate metrics and diagnostic analysis.
- Report thresholds, subgroup behavior, residuals, error ranges, and uncertainty rather than only one headline score.
- Recognize selective reporting, test-set overuse, and post-test tuning as threats to honest evaluation.
- Write a concise, decision-ready final-test evaluation section.
33.1 Role of the test set
The test set is a final measurement instrument. It estimates how the selected model is likely to perform on new observations drawn under conditions similar to those represented by the test data. Its value comes from independence: the test labels must not influence model-selection decisions.
Table 33.2. Development data versus test data
Activity | Training / validation data | Protected test data |
|---|---|---|
| Fit model parameters | Yes | No |
| Choose preprocessing | Yes | No |
| Engineer/select features | Yes | No |
| Tune hyperparameters | Yes | No |
| Choose algorithm family | Yes | No |
| Tune decision threshold | Validation only | No, unless threshold is fixed elsewhere |
| Estimate final generalization | Not the final estimate | Yes |
| GOLDEN RULE If a test result changes what model you deploy, the test set has participated in model selection. A new untouched evaluation set would then be needed for a genuinely final estimate. |
Why repeated test evaluation is dangerous
- Trying many models and reporting only the one with the best test score creates implicit test-set tuning.
- Changing a threshold because test recall looks disappointing leaks test information into the operational decision.
- Adding or removing features after viewing test errors uses the test set as a feature-selection signal.
- Repeatedly checking the same test set can lead to an optimistic estimate even when each individual change seems small.
A simple data-flow discipline
Train → Validation / Cross-validation → Freeze choices → Refit → Test once The test set is measured only after the development loop is complete. |
PYTHON • Create a protected train / validation / test split from sklearn.datasets import load_breast_cancer |
33.2 Retraining the final model
After model selection, the development phase should produce a complete frozen specification: preprocessing steps, feature set, model family, hyperparameters, random seeds where relevant, and any operational threshold. The final refit does not reopen these choices; it simply uses more labeled development data to estimate the already-chosen model parameters.
Table 33.3. What must be frozen before the final refit
Component | Example frozen choice | Can test results change it? |
|---|---|---|
| Preprocessing | StandardScaler inside a pipeline | No |
| Feature set | All selected features from development process | No |
| Algorithm | Logistic Regression | No |
| Hyperparameters | C = 1.0, L2 penalty | No |
| Threshold | 0.37 selected on validation data | No |
| Primary metrics | Recall, F1, ROC AUC | No |
Combining training and validation data
Once the model specification is fixed, it is common to combine the original training and validation partitions and refit the final pipeline. This gives the final model access to more labeled development observations while preserving the untouched test set for evaluation.
| IMPORTANT Do not combine the test set into the final fit before reporting test performance. That would measure the model on observations it has already seen during training. |
PYTHON • Choose a threshold on validation data before the final refit import numpy as np |
PYTHON • Refit the finalized pipeline on all development data import pandas as pd |
33.3 Test report
A final test report should answer more than “What is the score?” It should document the test population, model version, threshold or prediction rule, primary metrics, important secondary metrics, error patterns, subgroup behavior, and any limitations that affect interpretation.
Classification test report
Table 33.4. Recommended classification evidence
Item | What to report | Why it matters |
|---|---|---|
| Confusion matrix | TP, TN, FP, FN | Shows the actual error types at the deployed threshold. |
| Precision | TP / (TP + FP) | Quantifies how often positive predictions are correct. |
| Recall | TP / (TP + FN) | Quantifies how many positive cases are detected. |
| F1-score | Harmonic mean of precision and recall | Summarizes the precision-recall trade-off. |
| ROC AUC / PR AUC | Threshold-independent ranking quality | Provides probability/score discrimination evidence. |
| Threshold | Exact fixed operating threshold | Makes the classification rule reproducible. |
| Subgroup analysis | Metrics by relevant groups | Checks whether aggregate performance hides weak segments. |
PYTHON • Evaluate the final classifier once on the test set from sklearn.metrics import ( |
Classification subgroup analysis
Subgroup analysis should be defined before inspecting test outcomes whenever possible. Groups may reflect operational segments, data sources, acquisition channels, geography, device type, target prevalence, or demographic attributes when their use is legally and ethically appropriate.
PYTHON • Compare classification performance across a pre-defined feature band import pandas as pd |
| INTERPRETATION A subgroup result based on very few observations should be treated as uncertain evidence, not as a definitive ranking. Always report subgroup sample sizes together with performance. |
Regression test report
Table 33.5. Recommended regression evidence
Item | What to report | Why it matters |
|---|---|---|
| MAE | Mean absolute error | Average absolute error in target units. |
| RMSE | Root mean squared error | Emphasizes larger errors and stays in target units. |
| R² | Coefficient of determination | Compares squared error with a mean-prediction baseline. |
| Residual analysis | Residual distribution and plots | Reveals bias, heteroscedasticity, outliers, and nonlinearity. |
| Error by target range | MAE / RMSE across target bands | Shows whether performance changes with target magnitude. |
| Subgroup analysis | Metrics by operational groups | Checks whether aggregate accuracy hides weak segments. |
PYTHON • Regression: train a finalized pipeline and evaluate the protected test set from sklearn.datasets import load_diabetes |
PYTHON • Regression: residual and target-range analysis import pandas as pd |
33.4 Honest reporting
A final evaluation has scientific and operational value only when it is reported honestly. Strong reporting describes what the model does well, where it fails, how certain the estimates are, and under what conditions the results are expected to generalize. Hiding unfavorable evidence makes the report less useful for deployment decisions.
Table 33.6. Honest-reporting checklist
Principle | Good reporting practice | Weak reporting practice |
|---|---|---|
| Report positive and negative results | Show both strong metrics and important error modes. | Show only the strongest metric. |
| Avoid metric cherry-picking | Use pre-specified primary/secondary metrics. | Choose whichever metric makes the model look best. |
| State data limits | Describe sample size, coverage, missing populations, and collection conditions. | Present results as universal without discussing coverage. |
| State expected use conditions | Describe intended population, operating range, and threshold. | Leave deployment conditions implicit. |
| State uncertainty | Report variability, confidence intervals, or sample-size caveats when useful. | Treat one point estimate as exact. |
| Document deviations | Explain any protocol changes or post-hoc analyses. | Quietly alter the process after seeing test results. |
Uncertainty around a final score
A test metric is an estimate based on a finite sample. If the test set is small, a few observations can materially change the reported value. Confidence intervals or bootstrap intervals can communicate this uncertainty when the application requires more than a point estimate.
PYTHON • Bootstrap a confidence interval for classification F1 import numpy as np |
| CAUTION A confidence interval quantifies sampling uncertainty under the observed test distribution. It does not automatically capture future distribution shift, data-quality changes, or deployment feedback effects. |
Practical activity — Produce a formal final-test evaluation section
Students now act as the final evaluation team. They receive a model whose architecture, preprocessing, hyperparameters, selected features, and decision threshold have already been finalized. Their task is to evaluate the model once on the protected test set and write a formal report section that could be included in a technical document or project submission.
Activity requirements
- State the model specification and confirm that all development choices were frozen before test evaluation.
- Describe the test set: size, target distribution, important inclusion/exclusion conditions, and whether it reflects the expected use environment.
- Evaluate the final model using the primary and secondary metrics defined before opening the test results.
- Include the confusion matrix for classification or residual/error analysis for regression.
- Report the fixed classification threshold when applicable.
- Perform at least one pre-defined subgroup or target-range analysis and include subgroup sample sizes.
- Identify at least two strengths and two limitations revealed by the final evaluation.
- State uncertainty and any conditions under which the reported results may not generalize.
- Conclude with a deployment recommendation: proceed, proceed with constraints/monitoring, or do not proceed.
Formal classification template
Table 33.7. Suggested final-test report structure
Subsection | What the student should write |
|---|---|
| Final model | Pipeline, features, algorithm, hyperparameters, frozen threshold, model version. |
| Test protocol | How the test set was protected and when it was opened. |
| Test population | Sample size, class balance, relevant operational characteristics. |
| Main results | Confusion matrix, precision, recall, F1, ROC AUC / PR AUC, threshold. |
| Subgroup results | Metrics and sample size for pre-defined groups. |
| Error analysis | Important false positives / false negatives and plausible patterns. |
| Uncertainty & limits | Sampling uncertainty, dataset coverage, likely shift risks. |
| Recommendation | Deployment decision and required monitoring or constraints. |
Formal regression template
Table 33.8. Suggested regression final-test report structure
Subsection | What the student should write |
|---|---|
| Final model | Pipeline, features, algorithm, hyperparameters, model version. |
| Test protocol | Evidence that the test set was not used during selection or tuning. |
| Main results | MAE, RMSE, R² and target-unit interpretation. |
| Residual analysis | Bias, spread, outliers, heteroscedasticity, nonlinearity. |
| Target-range results | Error by low / medium / high target bands or quantiles. |
| Subgroup results | Operational-group error and sample sizes. |
| Uncertainty & limits | Finite-sample uncertainty and dataset coverage. |
| Recommendation | Deployment decision, safeguards, and monitoring requirements. |
| SUBMISSION STANDARD The final-test section should be reproducible, concise, and self-contained. A reader should be able to understand what was tested, how it was tested, the exact operating rule, the observed performance, the main weaknesses, and the deployment recommendation without reading the entire development notebook. |
Discussion questions
1. Why is a high validation score not a substitute for a final test evaluation?
2. What should happen if the final test recall is lower than expected?
3. Why must a tuned decision threshold be frozen before the test set is opened?
4. When is it reasonable to combine training and validation data for the final refit?
5. Why should subgroup sample sizes appear next to subgroup metrics?
6. What is the difference between sampling uncertainty and distribution-shift risk?
7. Which result would make you recommend “proceed with constraints” rather than “proceed”?
Chapter summary
Table 33.9. Final-test discipline at a glance
Stage | Correct action | Key question |
|---|---|---|
| Before test | Freeze model, preprocessing, hyperparameters, features, threshold, and metrics. | Are all decisions complete? |
| Final refit | Train the fixed pipeline on the full development data. | Am I only re-estimating parameters? |
| Test evaluation | Evaluate once under fixed conditions. | Is this a measurement rather than another experiment? |
| Diagnostics | Inspect confusion/residuals, target ranges, and pre-defined subgroups. | Where does the model fail? |
| Reporting | Include positive and negative findings, uncertainty, and dataset limits. | Would a skeptical reader understand the risks? |
| Decision | Recommend proceed / constrained proceed / do not proceed. | Is the evidence sufficient for the intended use? |
| NEXT STEP A final test report closes model development, but not the model lifecycle. Deployment requires versioning, monitoring, drift detection, incident procedures, and a plan for future re-evaluation when data or operating conditions change. |