Chapter 29 — Feature Selection
Filter methods • Wrapper methods • Embedded methods • Leakage control
Keeping the variables that add reliable predictive value while simplifying the modeling system
| BRIDGE FROM CHAPTER 28 Feature engineering can create many potentially useful variables. Feature selection asks the complementary question: which variables should remain in the final modeling system after predictive value, stability, cost, leakage risk, and domain meaning are considered? |
Chapter map
| Section | Main question | Primary concern |
|---|---|---|
| 29.1 Why select? | Why use fewer features? | Generalization, speed, cost, interpretability. |
| 29.2 Filter methods | Can features be screened before model fitting? | Variance, correlation, χ², ANOVA, mutual information. |
| 29.3 Wrapper methods | Which subset works best with this estimator? | RFE, sequential selection, cross-validated selection. |
| 29.4 Embedded methods | Can selection occur during model fitting? | L1 sparsity, tree importance, regularization. |
| 29.5 Pitfalls | How can feature selection mislead evaluation? | Leakage, test-set use, instability, domain blind spots. |
Chapter overview
Feature selection is the process of identifying a smaller subset of input variables that provides enough predictive information for the modeling objective. The goal is not to remove features merely to obtain a shorter table. A good selection strategy should preserve or improve generalization, reduce unnecessary complexity, and remain reproducible under cross-validation.
Feature selection is especially valuable when a dataset contains many weak, redundant, noisy, expensive, or highly correlated variables. It can also simplify deployment when each retained feature requires a sensor, database join, questionnaire item, external API, or manual measurement.
Learning objectives
- Explain why reducing the feature set can improve generalization, speed, cost, and interpretability.
- Distinguish filter, wrapper, and embedded feature-selection strategies.
- Apply variance threshold, correlation screening, chi-square, ANOVA, and mutual-information methods appropriately.
- Use recursive feature elimination and sequential feature selection with cross-validation discipline.
- Use L1 regularization and tree-based importance as embedded selection mechanisms.
- Recognize leakage caused by selecting features before splitting or by consulting the final test set.
- Compare predictive performance before and after selection using the same cross-validation folds.
- Assess selection stability and combine statistical evidence with domain analysis.
Table 29.1. The three families of feature selection
| Family | How it decides | Model dependence | Typical cost | Example | |
|---|---|---|---|---|---|
| Filter | Scores features using data statistics. | Low | Low | ANOVA SelectKBest. | |
| Wrapper | Repeatedly fits a predictive estimator on subsets. | High | High | RFE or sequential selection. | |
| Embedded | Selection happens as the estimator is fitted. | High | Medium | L1 logistic regression. | |
| CORE PRINCIPLE Feature selection is part of model training. Any selector that learns from the data must be fitted only on the training portion of each validation split, usually by placing it inside a scikit-learn Pipeline. | |||||
29.1 Why select features?
More features increase the amount of information available to the model, but they also increase the number of opportunities to fit noise. When the sample size is limited, irrelevant or weak predictors can increase variance and make the fitted relationship less stable across datasets.
Reduce overfitting
A flexible model can exploit accidental patterns in noisy variables. Removing variables that do not contribute reproducible signal can reduce the effective complexity of the learning problem and improve out-of-sample behavior. Selection is not a guaranteed cure for overfitting, but it is one useful control when dimensionality is high relative to the amount of data.
Improve training speed and simplify models
Many algorithms become slower as the number of input columns grows. Fewer variables reduce memory use, matrix operations, tree split searches, and sometimes prediction latency. Simpler models are also easier to debug, document, monitor, and explain to domain experts.
Reduce data collection cost
Some features are expensive. A laboratory measurement may require a test, an industrial feature may require a physical sensor, and a business feature may require a paid data source. A model that loses almost no predictive performance after removing an expensive variable can be operationally superior.
Improve interpretability
A compact feature set focuses analysis on a smaller number of relationships. This can make coefficient tables, partial-dependence analysis, error investigations, and stakeholder discussions more manageable. However, interpretability also depends on feature meaning and model form—not only on feature count.
Table 29.2. What should improve after selection?
| Objective | Desired effect | What to monitor |
|---|---|---|
| Generalization | Same or better validation performance. | Cross-validation mean and variability. |
| Efficiency | Lower training and prediction cost. | Fit time, latency, memory. |
| Collection cost | Fewer expensive measurements. | Acquisition cost per prediction. |
| Interpretability | Smaller, defensible feature set. | Feature meanings and stability. |
29.2 Filter methods
Filter methods rank or remove features using statistical properties that are mostly independent of the final predictive estimator. They are fast and useful for initial screening, especially when the feature count is large. Their main limitation is that a univariate filter can miss variables that are weak alone but useful through interactions.
Variance threshold
A feature with zero variance has the same value for every training observation and therefore cannot help distinguish outcomes. Near-constant features may also be uninformative, although a rare indicator can still be important in some domains. VarianceThreshold should therefore be used with a defensible threshold rather than blindly.
PYTHON • Remove constant or near-constant features from sklearn.feature_selection import VarianceThreshold |
Correlation screening
Correlation is commonly used to detect redundant numerical predictors. If two variables carry nearly the same linear information, one may be removed for simplicity. A high feature-feature correlation does not tell us which variable is better for the target, and low Pearson correlation does not imply that a feature is useless because nonlinear relationships can exist.
|corr(x_j, x_k)| > threshold A correlation threshold is a redundancy rule, not a universal statistical law. | |
PYTHON • Inspect highly correlated numerical features corr = X_train.corr(numeric_only=True).abs() | |
| CAUTION Never delete a feature solely because it correlates with another variable. Prefer the feature with better domain meaning, data quality, availability, stability, or collection cost—or let a regularized model handle the redundancy. |
Chi-square test
The chi-square feature score evaluates dependence between each nonnegative input feature and a classification target. It is often applied to count data such as term frequencies or nonnegative occurrence measurements. Continuous features containing negative values should not be passed directly to scikit-learn chi2 without an appropriate transformation.
χ² = Σ (Observed − Expected)² / Expected Larger deviations from independence produce larger chi-square scores. | |
PYTHON • Select classification features with chi-square from sklearn.feature_selection import SelectKBest, chi2 |
ANOVA F-test
For classification, the ANOVA F-test compares between-class variation with within-class variation for each numerical feature. A high F statistic indicates that class means are separated relative to within-class spread. It is a univariate test, so it does not capture interactions among predictors.
F = between-class variation / within-class variation For regression, scikit-learn provides f_regression rather than f_classif. | |
PYTHON • ANOVA feature selection from sklearn.feature_selection import SelectKBest, f_classif |
Mutual information
Mutual information measures how much knowing a feature reduces uncertainty about the target. Unlike simple correlation and ANOVA mean comparisons, it can detect broader nonlinear dependencies. Estimates can be noisy on small datasets, so rankings should be checked across resamples or folds.
MI(X;Y) = 0 when X and Y are statistically independent Higher mutual information indicates stronger statistical dependence, not causality. | |
PYTHON • Mutual-information ranking from sklearn.feature_selection import mutual_info_classif |
Table 29.3. Choosing a filter method
| Method | Target type | Feature requirement | Strength | Main limitation |
|---|---|---|---|---|
| Variance threshold | Any | Numeric representation | Very fast unsupervised screen. | Does not use target information. |
| Correlation | Any | Usually numeric | Finds linear redundancy. | Misses nonlinear dependence. |
| Chi-square | Classification | Nonnegative values | Natural for counts/frequencies. | Univariate; input constraints. |
| ANOVA F | Classification | Numeric | Fast class-separation score. | Mostly linear/univariate view. |
| Mutual information | Class. or reg. | Numeric/categorical settings | Can detect nonlinear dependence. | Estimates may be variable. |
29.3 Wrapper methods
Wrapper methods evaluate subsets by repeatedly fitting a predictive estimator. Because they directly consider estimator performance, they can produce strong subsets, but their computational cost can be much higher than filter methods. The selected subset may also depend strongly on the estimator and validation strategy.
Recursive feature elimination
Recursive feature elimination starts with a set of features, fits an estimator that exposes coefficients or feature importance, removes the least important feature or group of features, and repeats. RFE requires the desired number of retained features to be specified. RFECV extends this idea by selecting the number of features using cross-validation.
PYTHON • Recursive feature elimination from sklearn.feature_selection import RFE |
PYTHON • Cross-validated RFE from sklearn.feature_selection import RFECV |
Sequential feature selection
Sequential selection builds a subset iteratively. Forward selection starts with no features and adds the candidate that improves cross-validated performance the most. Backward selection starts with all features and removes the least useful candidate. Sequential methods can be easier to interpret than exhaustive search, but they are greedy and can miss a globally optimal subset.
PYTHON • Forward sequential feature selection from sklearn.feature_selection import SequentialFeatureSelector | |
| COMPUTATION Wrapper methods may fit the estimator dozens or hundreds of times. For wide datasets, first remove obviously unusable features with domain rules or inexpensive filters, then apply wrappers to a smaller candidate set. |
29.4 Embedded methods
Embedded methods perform feature selection as part of fitting the model itself. They often offer a useful compromise between computational cost and model awareness. The selected features are still model-dependent, and the regularization or importance threshold should be tuned using validation data rather than the final test set.
Lasso and L1 regularization
L1 regularization adds the sum of absolute coefficient magnitudes to the loss. With sufficient regularization, some fitted coefficients become exactly zero. In regression this is Lasso; in classification, logistic regression with an L1 penalty can provide analogous sparse coefficients.
objective = data loss + λ Σ |β_j| A coefficient equal to zero effectively removes that feature from the fitted linear model. | |
PYTHON • L1-regularized logistic selection from sklearn.feature_selection import SelectFromModel |
Decision-tree and random-forest importance
Tree-based models can expose impurity-based feature importance. SelectFromModel can retain variables above a chosen importance threshold. Tree importance can favor high-cardinality or highly variable predictors, and correlated features can share importance, so permutation importance or repeated validation can provide useful complementary evidence.
PYTHON • Embedded selection with a random forest from sklearn.ensemble import RandomForestClassifier |
Regularized models as selectors
Ridge regression and L2-regularized logistic regression shrink coefficients but generally do not set them exactly to zero, so they are not sparse selectors by themselves. Elastic Net combines L1 and L2 behavior and can be useful when correlated predictors should be stabilized while still allowing sparsity.
Table 29.4. Embedded-method behavior
| Method | Selection mechanism | Correlated features | Main caution |
|---|---|---|---|
| Lasso / L1 | Some coefficients become exactly zero. | May choose one of several correlated variables. | Selection can change with regularization strength. |
| Elastic Net | L1 sparsity + L2 stabilization. | Often more stable for correlated groups. | Two regularization controls require tuning. |
| Decision tree | Split-based importance. | Importance may concentrate on one variable. | Single-tree importance is unstable. |
| Random forest | Average split importance across trees. | Can distribute importance across correlated features. | Impurity importance has known biases. |
29.5 Feature-selection pitfalls
Feature selection can improve a model only when evaluation remains honest. The most serious mistakes occur when the selector sees information that belongs to validation or test data, or when a purely statistical rule removes variables whose operational meaning has not been considered.
Performing selection before splitting
If target-aware selection is fitted on the complete dataset before the train-validation split, validation outcomes influence which variables are retained. The subsequent validation score is then optimistic because the validation set has already participated in model design.
Table 29.5. Leakage-safe versus leaking workflows
| Workflow | Verdict | Reason | |
|---|---|---|---|
| Select on all rows → split → fit model | Leaking | Validation targets influence selected subset. | |
| Split → fit selector on training → transform validation | Correct | Validation remains unseen during selection. | |
| Pipeline(selector, model) inside cross-validation | Preferred | Selection is refitted independently in every training fold. | |
| Use test score to choose k | Leaking | Test set becomes development data. | |
PYTHON • Leakage-safe selector inside a Pipeline from sklearn.feature_selection import SelectKBest, f_classif | |||
Selecting features using the test set
The final test set should estimate performance only after feature selection, hyperparameter tuning, and model choice are complete. If the analyst repeatedly checks test performance and modifies the selected subset, the test set gradually becomes part of training and loses its role as an unbiased final evaluation.
Instability of importance rankings
When features are correlated, sample size is modest, or signal is weak, the top-ranked variables can change substantially across folds. Stability matters because a production system built around a fragile ranking may behave differently after retraining. Record how often each feature is selected and how its rank or coefficient varies.
Removing variables without domain analysis
A statistically weak feature may still be necessary for fairness monitoring, regulatory reporting, safety constraints, or causal adjustment in a separate analysis. Conversely, a highly predictive feature may be ethically inappropriate or unavailable at scoring time. Feature selection is therefore a statistical and domain decision, not only a ranking exercise.
| DOMAIN REVIEW Before removing or retaining a variable, consider measurement quality, acquisition cost, availability at prediction time, ethical and legal constraints, operational meaning, and whether a correlated alternative is more reliable. |
Practical lab — Compare performance before and after feature selection
Goal: compare a full-feature classification model with three selection strategies—filter, wrapper, and embedded—using exactly the same cross-validation folds. Students will report predictive performance, number of retained features, selected feature names, and evidence of selection stability.
| LAB RULE The selector must be inside the Pipeline used by cross-validation. Never fit a target-aware selector once on the full dataset and then cross-validate only the classifier. |
Step 1 — Load and inspect the dataset
PYTHON • Breast Cancer Wisconsin dataset import numpy as np |
The dataset contains 30 continuous predictors. This is large enough to make selection meaningful while remaining small enough for wrapper methods to run in a classroom environment.
Step 2 — Define one common evaluation strategy
PYTHON • Same folds and same metrics for every model from sklearn.model_selection import StratifiedKFold, cross_validate |
Step 3 — Build filter, wrapper, and embedded selectors
PYTHON • Filter selector — ANOVA top 10 from sklearn.feature_selection import SelectKBest, f_classif |
PYTHON • Wrapper selector — RFE top 10 from sklearn.feature_selection import RFE |
PYTHON • Embedded selector — L1 sparsity from sklearn.feature_selection import SelectFromModel |
Step 4 — Compare cross-validation performance
PYTHON • Evaluate all pipelines on identical folds models = { |
Interpretation task: determine whether any selected subset matches or improves the baseline while using fewer variables. A tiny difference in the mean may not be meaningful when fold-to-fold standard deviation is larger than the difference.
Step 5 — Fit on the training data and inspect selected names
After choosing the evaluation design, fit each selector on a training partition—not on the protected final test set—to inspect which features it retains. The code below demonstrates the mechanics using one train split.
PYTHON • Recover selected feature names from sklearn.model_selection import train_test_split |
Step 6 — Measure selection stability across folds
PYTHON • Count how often ANOVA selects each feature selection_count = pd.Series(0, index=X.columns, dtype=int) |
A feature selected in 5 of 5 folds is more stable under this procedure than one selected in only 1 or 2 folds. Stability does not prove the feature is causal or necessary, but it helps identify fragile rankings.
Step 7 — Optional: compare training time
PYTHON • Measure computational cost import time | |
| EXPECTED PATTERN A filter method is usually cheaper than RFE because RFE repeatedly refits an estimator. Feature selection should be evaluated not only for accuracy but also for computational and operational cost. |
Step 8 — Produce the feature-selection report
1. Report baseline AUC, F1, accuracy, and fold variability using all features.
2. Report the same metrics for ANOVA, RFE, and L1 embedded selection.
3. State how many features each method retains and list the selected variables.
4. Identify which selector gives the best trade-off between predictive performance and feature count.
5. Compare the stability of selected features across folds.
6. Discuss whether strongly correlated predictors lead different methods to choose different representatives.
7. Identify any selected feature that is expensive, unavailable at prediction time, or questionable from a domain perspective.
8. Recommend a final feature-selection strategy and justify it with validation evidence rather than test-set tuning.
Extension — Compare different values of k without leaking
The number of retained features is itself a hyperparameter. Instead of trying k = 5, 10, 15, and 20 on the final test set, tune k inside cross-validation. A GridSearchCV over a Pipeline ensures that feature selection and model fitting remain nested inside each training fold.
PYTHON • Tune the number of selected features from sklearn.model_selection import GridSearchCV |
Chapter summary
- Feature selection seeks a smaller subset that preserves useful predictive information while reducing unnecessary complexity.
- Filter methods use statistical properties such as variance, correlation, chi-square, ANOVA, or mutual information.
- Wrapper methods such as RFE and sequential selection repeatedly fit an estimator and can be computationally expensive.
- Embedded methods select features during model fitting, for example through L1 sparsity or tree-based importance.
- Selection must be fitted inside each training fold; performing target-aware selection before splitting causes leakage.
- The final test set must not be used to choose features or the number of retained variables.
- Importance and selection rankings can be unstable, especially when predictors are correlated.
- Statistical selection should be combined with domain knowledge, production availability, cost, ethics, and data quality.
- The best subset is not necessarily the smallest one—it is the subset that provides the best validated operational trade-off.
| NEXT STEP After designing and selecting features, the next stage is usually systematic hyperparameter tuning. The same validation discipline remains essential: tune only with training/validation information and protect the final test set. |