Chapter 30 — Hyperparameter Tuning
Manual tuning • Grid search • Randomized search • Successive halving • Nested CV
Searching model configurations systematically without turning validation data into another training set
| BRIDGE FROM CHAPTER 29 Feature selection decides which inputs should enter the model. Hyperparameter tuning decides how the learning algorithm should behave. Both are model-selection activities and therefore must be evaluated with the same leakage-safe validation discipline. |
Chapter map
| Section | Main question | Primary concern |
|---|---|---|
| 30.1 Parameters vs. hyperparameters | What is learned and what is configured? | Avoid confusing fitted quantities with search choices. |
| 30.2 Manual tuning | How can a small search be organized? | Controlled experiments and record keeping. |
| 30.3 Grid search | When is exhaustive search practical? | Combinations, CV, and computational cost. |
| 30.4 Randomized search | How can a larger space be explored efficiently? | Distributions and search budget. |
| 30.5 Successive halving | How can weak candidates be discarded early? | Resource allocation across iterations. |
| 30.6 Pipelines | How are preprocessing and model settings tuned together? | Double-underscore names and leakage control. |
| 30.7 Nested CV | How is tuning itself evaluated fairly? | Outer evaluation and inner selection loops. |
Chapter overview
Hyperparameters control how an algorithm learns. Examples include tree depth, regularization strength, the number of neighbors, the number of trees, and preprocessing choices such as polynomial degree. Unlike fitted coefficients or learned split thresholds, hyperparameters are selected by the practitioner or by a search procedure.
A tuning procedure is an experiment. The candidate configurations must be compared on validation information rather than on the final test set. When preprocessing, feature selection, and model fitting are combined in a Pipeline, cross-validated search can reproduce the complete training workflow independently inside each fold.
Learning objectives
- Distinguish learned model parameters from externally configured hyperparameters.
- Plan a small manual tuning experiment and record results systematically.
- Build parameter grids for exhaustive GridSearchCV experiments.
- Use RandomizedSearchCV with lists or probability distributions and a fixed search budget.
- Explain successive halving as progressive resource allocation and candidate elimination.
- Tune preprocessing and estimator options together using Pipeline step names and double underscores.
- Recognize leakage caused by fitting preprocessing outside the cross-validation search.
- Use nested cross-validation when an unbiased estimate of the entire tuning procedure is important.
- Compare candidate algorithms using the same folds and the same target metric.
Table 30.1. Search strategies at a glance
| Strategy | Candidate selection | Main strength | Main limitation | |
|---|---|---|---|---|
| Manual | Human-selected experiments | Easy to reason about | Can miss interactions and is hard to scale. | |
| Grid search | Every combination in a finite grid | Complete within the stated grid | Cost grows multiplicatively. | |
| Randomized search | Random samples from lists/distributions | Efficient coverage of large spaces | Does not guarantee every combination is tested. | |
| Successive halving | Many candidates, then progressive elimination | Concentrates resources on promising candidates | Early scores may be noisy; implementation details matter. | |
| CORE PRINCIPLE Hyperparameter tuning is model selection. If the same validation evidence is repeatedly used to choose configurations, the chosen model can overfit that validation process. Protect a final test set—or use an outer cross-validation loop for evaluation. | ||||
30.1 Parameters versus hyperparameters
Parameters
Parameters are quantities learned from the training observations during fitting. The practitioner does not normally assign their final values directly. They emerge from optimization or from the structure of the fitted estimator.
- Linear and logistic regression coefficients.
- The intercept of a fitted linear model.
- Decision-tree split features and split thresholds.
- Leaf predictions in a regression tree.
- Support vectors and fitted dual coefficients in an SVM.
Hyperparameters
Hyperparameters configure the learning procedure before a particular fit. Their useful values are usually selected from validation evidence, domain constraints, computational budgets, or established defaults.
- Decision-tree max_depth and min_samples_leaf.
- Random-forest n_estimators and max_features.
- K-nearest-neighbors n_neighbors.
- Regularization strength C or alpha.
- RBF-kernel gamma.
- Gradient-boosting learning_rate and number of estimators.
Table 30.2. Parameter or hyperparameter?
| Item | Learned during fit? | Configured before fit? | Type | |
|---|---|---|---|---|
| Linear-regression coefficient beta_1 | Yes | No | Parameter | |
| Tree split threshold | Yes | No | Parameter | |
| Tree max_depth | No | Yes | Hyperparameter | |
| KNN n_neighbors | No | Yes | Hyperparameter | |
| Ridge alpha | No | Yes | Hyperparameter | |
PYTHON • Inspect configurable hyperparameters from sklearn.ensemble import RandomForestClassifier | ||||
| IMPORTANT Do not call fitted attributes such as coef_, feature_importances_, or tree_.threshold hyperparameters. The trailing underscore convention in scikit-learn commonly marks attributes that are created or learned during fit. | ||||
30.2 Manual tuning
Manual tuning is useful when only one or two influential hyperparameters need to be explored or when the purpose is educational. A disciplined manual experiment changes a clearly defined value, keeps the validation protocol fixed, records the score, and avoids reacting to every small fluctuation.
Selecting meaningful parameter values
Candidate values should reflect how a hyperparameter changes model behavior. Tree depths such as 2, 4, 6, 10, and unlimited explore increasingly flexible structures. Regularization strengths are often tested on a logarithmic scale because useful values may differ by orders of magnitude.
Changing one parameter at a time
One-at-a-time experiments are easy to interpret because the performance change can be associated with a single modification. Their limitation is that hyperparameters can interact: the best tree depth may depend on min_samples_leaf, and the best RBF gamma may depend on C.
PYTHON • A controlled manual tuning loop from sklearn.model_selection import cross_val_score, StratifiedKFold |
Table 30.3. Minimum experiment log
| Run | Changed setting | CV strategy | Primary metric | Mean ± SD | Decision | |
|---|---|---|---|---|---|---|
| 1 | Baseline defaults | 5-fold stratified | ROC AUC | record result | Reference | |
| 2 | max_depth = 4 | same folds | ROC AUC | record result | Compare | |
| 3 | max_depth = 8 | same folds | ROC AUC | record result | Compare | |
| PRACTICE Keep the data splits, preprocessing, random seeds, and scoring metric constant while comparing settings. Otherwise, a score difference may come from a different experiment rather than from the hyperparameter change. | ||||||
30.3 Grid search
Grid search evaluates the Cartesian product of all values listed in a parameter grid. It is attractive when the search space is small and discrete because every stated combination is evaluated under cross-validation.
Number of candidates = |A| × |B| × |C| × ... Three parameters with 4, 3, and 5 values create 60 candidate configurations before cross-validation is considered. |
Parameter grids
PYTHON • Define and run GridSearchCV from sklearn.model_selection import GridSearchCV |
Computational cost
If a grid contains 18 candidates and 5-fold cross-validation is used, at least 90 candidate-fold fits are required, plus a final refit when refit=True. This multiplication becomes expensive when models are slow or the grid is large.
Table 30.4. Why grid size grows quickly
| Hyperparameters | Values per parameter | Grid candidates | Fits with 5-fold CV | |
|---|---|---|---|---|
| 2 | 5 × 5 | 25 | 125 + refit | |
| 3 | 5 × 5 × 5 | 125 | 625 + refit | |
| 4 | 5 × 5 × 5 × 5 | 625 | 3,125 + refit | |
| INTERPRETATION Grid search is exhaustive only inside the grid you define. A dense search over an unhelpful range can be more expensive and less useful than a well-designed smaller search. | ||||
30.4 Randomized search
Randomized search samples a fixed number of configurations instead of evaluating every possible combination. This separates the search budget from the size of the complete parameter space and is often useful when several hyperparameters are continuous or when only a few dimensions strongly influence performance.
Lists and probability distributions
Discrete alternatives can be supplied as Python lists. Continuous or integer ranges can be represented by probability distributions that implement an rvs method. A logarithmic distribution is often appropriate for positive scale parameters such as C, alpha, or gamma because it explores orders of magnitude rather than equal arithmetic intervals.
PYTHON • RandomizedSearchCV with a search budget from scipy.stats import loguniform, randint |
Search budget
The n_iter setting is a budget: it controls how many candidate configurations are sampled. More iterations increase the probability of visiting useful regions but also increase computation. Reproducible teaching experiments should set random_state when the sampler is stochastic.
Table 30.5. Grid search versus randomized search
| Question | GridSearchCV | RandomizedSearchCV | |
|---|---|---|---|
| How are candidates chosen? | Every grid combination | Random samples | |
| Best for | Small, discrete spaces | Large or continuous spaces | |
| Budget control | Indirect through grid size | Direct through n_iter | |
| Can use distributions? | No, finite values | Yes | |
| Reproducibility | Deterministic given CV/model | Set random_state for sampling | |
| GOOD SEARCH DESIGN Start broad enough to learn where performance is promising, then narrow the range if a second-stage search is justified. Do not repeatedly refine a search by looking at the final test score. | |||
30.5 Successive halving
Successive halving treats tuning like a tournament. Many candidates begin with a small resource allocation. Weak configurations are eliminated, and only promising candidates receive more resources in later iterations. The resource is commonly the number of training samples, but some searches can use an estimator parameter such as n_estimators when configured appropriately.
1. Start with many candidate configurations.
2. Evaluate each candidate with a limited resource budget.
3. Rank the candidates using the validation score.
4. Remove a fraction of the weaker candidates.
5. Increase resources for the survivors.
6. Repeat until the final candidates receive the largest budget.
PYTHON • Successive halving in scikit-learn from sklearn.experimental import enable_halving_search_cv # noqa: F401 | |
| SCIKIT-LEARN NOTE HalvingGridSearchCV and HalvingRandomSearchCV are still exposed through scikit-learn's experimental enable_halving_search_cv import. Their API may change without the normal deprecation cycle, so version-sensitive production code should check the installed documentation. |
When halving helps
- The original candidate space is too large for exhaustive full-resource evaluation.
- Partial-resource scores are informative enough to eliminate poor candidates.
- Training cost grows strongly with the number of samples or another tunable resource.
- The goal is efficient search rather than evaluating every candidate at full fidelity.
Successive halving can discard a candidate that would have performed better with more resources if its early score is noisy. For this reason, resource choice, minimum resources, factor, and data size should be considered rather than treating halving as an automatic replacement for all other search strategies.
30.6 Tuning within a pipeline
A Pipeline lets preprocessing and model fitting be treated as one estimator. During cross-validated search, the pipeline is fitted independently inside each training fold. This prevents validation-fold information from entering learned preprocessing statistics such as means, scales, imputations, selected features, or encodings.
Step names and double-underscore notation
Pipeline parameters are addressed with the pattern step_name__parameter_name. For a pipeline containing a scaler named scale and a logistic-regression model named model, the model hyperparameter C is written model__C. A tunable preprocessing parameter follows the same rule.
PYTHON • Tune preprocessing and model settings together from sklearn.pipeline import Pipeline |
Table 30.6. Pipeline parameter naming
| Pipeline step | Underlying parameter | Search key |
|---|---|---|
| scale | with_mean | scale__with_mean |
| pca | n_components | pca__n_components |
| model | C | model__C |
| model | class_weight | model__class_weight |
What can be tuned jointly?
A pipeline search can treat preprocessing decisions as part of the model configuration. This is useful when the correct preprocessing choice depends on the estimator or on the strength of regularization.
- Imputation strategy or indicator options.
- Scaling choices and dimensionality-reduction settings.
- Feature-selection method or number of retained features.
- Estimator family, regularization, class weights, and complexity controls.
Leakage-safe preprocessing
PYTHON • Wrong: preprocessing before cross-validation # WRONG: the scaler sees all training/validation folds at once | |
PYTHON • Right: preprocessing inside the Pipeline pipe = Pipeline([ | |
| PIPELINE RULE Any step that learns statistics from the data belongs inside the cross-validation workflow: scaling, imputation, encoding, feature selection, dimensionality reduction, and many target-dependent transformations. |
30.7 Nested cross-validation
Ordinary cross-validation can be used to choose hyperparameters, but the best cross-validation score is not automatically an unbiased estimate of future performance. The search has optimized configurations to those validation folds. Nested cross-validation separates hyperparameter selection from performance estimation by adding an outer evaluation loop.
Inner tuning loop
For each outer training split, an inner search uses only that outer training data to choose hyperparameters. GridSearchCV or RandomizedSearchCV commonly performs this inner selection.
Outer evaluation loop
The tuned search object is then evaluated on the outer validation fold, which played no role in choosing that fold's hyperparameters. Averaging the outer scores estimates the performance of the complete model-selection procedure rather than the optimistic score of one selected configuration.
Outer score = Evaluate( InnerSearch(outer training data), outer held-out data ) Repeat across outer folds; summarize the outer held-out scores. | |
PYTHON • Nested CV — define the inner search from sklearn.model_selection import StratifiedKFold, GridSearchCV, cross_val_score | |
PYTHON • Nested CV — evaluate the tuned procedure outer_scores = cross_val_score( |
Table 30.7. Non-nested versus nested evaluation
| Approach | Uses validation to tune? | Independent evaluation of tuning procedure? | Typical use | |
|---|---|---|---|---|
| Single GridSearchCV | Yes | No | Choose a final configuration with a separate test set available. | |
| Nested CV | Inner loop | Yes, outer loop | Small datasets or high-stakes performance estimation. | |
| WHY NESTED CV If the goal is to report how well a tuned modeling procedure is expected to generalize, the outer loop must remain untouched by the inner hyperparameter search. This reduces selection bias from reporting the best inner-CV score as if it were an independent test result. | ||||
Practical lab — Optimize two candidate algorithms
Objective: compare two classification algorithms under the same cross-validation protocol, tune each algorithm with an appropriate search strategy, evaluate the selected models on a protected test set, and explain the trade-off between predictive quality and search cost.
| LAB DISCIPLINE The final test set is used only after each search has selected its best configuration from the training data. Both candidates use the same primary metric—ROC AUC—and the same 5-fold StratifiedKFold object during tuning. |
Step 1 — Load data and create a protected test set
PYTHON • Breast Cancer Wisconsin dataset from sklearn.datasets import load_breast_cancer |
Lab checkpoint — Fix the experiment before searching
- Keep X_test and y_test untouched until both searches are complete.
- Use the same StratifiedKFold object for both candidate algorithms.
- Use ROC AUC as the primary tuning metric for both searches.
- Record random_state values so stochastic searches and estimators are reproducible.
- Record the candidate count or n_iter so computational budgets can be compared fairly.
| WHY THIS MATTERS If the evaluation protocol changes while the hyperparameters change, the experiment no longer isolates the effect of model configuration. Fair tuning requires a fixed comparison framework. |
Step 2 — Candidate A: tune logistic regression with grid search
PYTHON • Import the Logistic Regression tuning tools from sklearn.pipeline import Pipeline |
PYTHON • Define the Logistic Regression pipeline and grid logit_pipe = Pipeline([("scale", StandardScaler()), |
PYTHON • Run the Logistic Regression grid search logit_search = GridSearchCV( |
Step 3 — Candidate B: tune random forest with randomized search
PYTHON • Define the Random Forest search space from scipy.stats import randint |
PYTHON • Run the Random Forest randomized search rf_search = RandomizedSearchCV( |
Step 4 — Compare the best cross-validation results
PYTHON • Summarize the two searches import pandas as pd |
The best_cv_auc values are model-selection scores, not independent test estimates. They are useful for choosing among configurations, but they should not be reported as the final unbiased performance of the search procedure.
Step 5 — Inspect the search tables, not only best_params_
PYTHON • Inspect top-ranked candidates def top_candidates(search, n=5): | |
| ANALYSIS QUESTION Is the top-ranked configuration clearly better than the next few candidates, or are several settings statistically similar across folds? When scores are nearly tied, prefer a simpler, faster, or more stable configuration when operationally appropriate. |
Step 6 — Evaluate both selected models once on the test set
PYTHON • Protected test-set evaluation from sklearn.metrics import accuracy_score, f1_score, roc_auc_score |
Step 7 — Compare computational cost
PYTHON • Candidate counts and fit times for name, search in { |
Search quality is not only a predictive-performance question. A marginal AUC improvement may not justify a search that is much more expensive, especially when retraining is frequent or compute is constrained.
Step 8 — Optional: nested cross-validation for Candidate A
PYTHON • Estimate the tuned procedure with an outer loop from sklearn.model_selection import cross_val_score |
cross_val_score clones the GridSearchCV object for each outer fold. Each clone performs its own inner search using only the corresponding outer training data, then the tuned estimator is scored on the untouched outer validation fold.
Step 9 — Produce the tuning report
1. State the protected test-set policy and the cross-validation strategy used for tuning.
2. List the hyperparameters and ranges explored for each candidate algorithm.
3. Explain why grid search was reasonable for Candidate A and randomized search for Candidate B.
4. Report the best mean cross-validated ROC AUC and the best parameter configuration for each search.
5. Inspect at least five top-ranked configurations and discuss whether the ranking is stable or nearly tied.
6. Report final test Accuracy, F1, and ROC AUC once for each selected estimator.
7. Compare computational cost using candidate counts and fit-time information.
8. Recommend one candidate algorithm using both predictive and operational evidence.
9. Explain when nested cross-validation would be preferable to a single inner search plus test split.
Extension — Tune preprocessing choices too
A pipeline search can include preprocessing alternatives as hyperparameters. For example, a PCA step can be turned on or bypassed, or the number of selected components can be searched jointly with model regularization. This tests the complete modeling workflow rather than tuning only the last estimator.
PYTHON • Search over a pipeline step from sklearn.decomposition import PCA |
Chapter summary
- Model parameters are learned during fit; hyperparameters configure how the learning procedure behaves.
- Manual tuning is useful for small, interpretable experiments but scales poorly when hyperparameters interact.
- GridSearchCV evaluates every combination in a finite grid, so cost grows multiplicatively with the number of values.
- RandomizedSearchCV samples a fixed number of configurations and can use probability distributions for broad searches.
- Successive halving allocates small resources to many candidates, removes weak candidates, and increases resources for survivors.
- Preprocessing and model hyperparameters should be searched inside a Pipeline using step__parameter names.
- Tuning preprocessing outside cross-validation can leak validation information into the training procedure.
- The best inner cross-validation score is a selection score, not automatically an unbiased estimate of future performance.
- Nested cross-validation uses an inner tuning loop and an outer evaluation loop to estimate the generalization of the complete search procedure.
- Fair algorithm comparison requires the same folds, metric, data policy, and test-set protection for every candidate.
| NEXT STEP After a model has been tuned, the workflow should shift from repeated optimization toward final evaluation, error analysis, reproducibility, and deployment readiness. Continuing to tune against the final test set would destroy its role as independent evidence. |