Chapter 28 — Feature Engineering
Domain knowledge • Transformations • Aggregations • Interactions • Leakage control
Designing input variables that expose useful predictive structure without contaminating evaluation
| BRIDGE FROM CHAPTER 27 Cross-validation provides a more reliable way to estimate model performance. Feature engineering now asks a different question: can we represent the same raw information in a form that makes useful patterns easier for the model to learn? |
Chapter map
| Section | Main question | Primary concern |
|---|---|---|
| 28.1 Purpose | Why create new features? | Signal, domain knowledge, noise, interpretability. |
| 28.2 Methods | How can raw variables be transformed? | Ratios, differences, interactions, bins, logs, time/frequency. |
| 28.3 Group features | How can history be summarized? | Counts, averages, maxima, rolling statistics. |
| 28.4 Leakage | When does a feature reveal unavailable information? | Future data, targets, global statistics, prediction-time availability. |
| 28.5 Evaluation | Did the engineered feature truly help? | Ablation, cross-validation, importance, fold stability. |
Chapter overview
Feature engineering is the process of creating, transforming, or selecting input variables so that a learning algorithm can represent the underlying problem more effectively. A strong engineered feature does not merely increase the number of columns. It expresses relevant structure in a form that is available at prediction time, reproducible in production, and useful across validation folds.
Modern tree ensembles and deep models can learn many interactions automatically, but feature engineering remains important for tabular data, smaller datasets, linear models, operational constraints, and domains where meaningful ratios, temporal summaries, or historical aggregates encode knowledge that would otherwise be difficult to learn.
Learning objectives
- Explain why feature engineering can improve predictive performance and interpretability.
- Construct ratios, differences, totals, counts, aggregations, interactions, polynomial terms, bins, log transforms, time-since-event variables, and frequency features.
- Create group-based historical summaries without mixing future information into the past.
- Recognize target leakage, future leakage, global-statistic leakage, and prediction-time availability problems.
- Evaluate engineered features with ablation studies and a consistent cross-validation strategy.
- Inspect feature importance and assess whether usefulness is stable across folds.
- Design and test at least five engineered features in a reproducible scikit-learn workflow.
Table 28.1. Raw variables versus engineered representations
| Raw information | Possible engineered feature | Why it may help | |
|---|---|---|---|
| Spend and transaction count | spend_per_transaction | Separates purchase size from purchase frequency. | |
| Visits and purchases | conversion_rate | Represents efficiency of turning visits into purchases. | |
| Current date and last purchase date | days_since_last_purchase | Makes customer recency explicit. | |
| Income with strong right skew | log_income | Compresses extreme values and can make relationships smoother. | |
| Price and quantity | order_total = price × quantity | Represents the economic quantity that directly matters. | |
| QUALITY RULE Every engineered feature should have a defensible meaning, a clear prediction-time data source, and evidence that it helps on unseen data. More columns are not automatically better. | |||
28.1 Purpose of feature engineering
The raw columns stored in a database are often chosen for operational reasons rather than predictive modeling. Feature engineering reorganizes those columns into variables that better correspond to the mechanisms of the problem.
Making useful patterns easier to learn
Suppose a customer spends 900 currency units over 30 transactions while another spends 900 over 3 transactions. Total spend is identical, but average transaction size is very different. A model can sometimes infer this relationship from both columns, but an explicit ratio makes the pattern available directly—especially to a linear model.
average_purchase_value = total_spend / number_of_transactions Use a safe denominator when zero counts are possible. |
Incorporating domain knowledge
Domain knowledge helps translate business, scientific, or engineering concepts into measurable predictors. Examples include debt-to-income ratio in lending, power-per-unit-load in energy systems, body-mass index in health research, or temperature change over time in industrial monitoring. The feature should reflect a plausible mechanism, not a post-hoc search for an accidental correlation.
Reducing noise and improving interpretability
Transformations can reduce the influence of extreme scales or irrelevant variation. Aggregating noisy repeated measurements can produce a more stable signal. Meaningful features can also simplify model explanations: “purchases per month” may be easier to discuss than separate transaction and tenure coefficients.
Table 28.2. Four reasons to engineer a feature
| Purpose | What changes | Example | Possible benefit |
|---|---|---|---|
| Expose structure | Combine raw columns | revenue / transactions | Easier relationship to learn. |
| Inject domain knowledge | Encode a meaningful mechanism | pressure × flow | Better physical or operational meaning. |
| Reduce noise | Summarize repeated values | 7-day average sensor value | More stable signal. |
| Improve interpretation | Use decision-relevant units | tenure in years | More understandable effects. |
28.2 Common feature engineering methods
Feature engineering methods can be grouped into arithmetic combinations, nonlinear transformations, discretization, temporal representations, and frequency or aggregation summaries. The best method depends on the data-generating process and the model family.
Table 28.3. Common methods and typical use cases
| Method | Generic form | Example | Main use |
|---|---|---|---|
| Ratio | a / b | spend / transactions | Normalize one quantity by another. |
| Difference | a − b | actual − planned | Measure a gap or change. |
| Total | a + b + … | online + store purchases | Combine related components. |
| Count | number of events | support tickets in 30 days | Represent activity or exposure. |
| Aggregation | mean / max / min / std | mean weekly usage | Summarize repeated observations. |
| Interaction | a × b | price × promotion | Represent conditional effects. |
| Polynomial | a², a³, … | temperature² | Represent smooth nonlinearity. |
| Binning | continuous → intervals | age bands | Capture regimes or improve interpretation. |
| Log transform | log(1 + a) | log revenue | Compress skewed positive values. |
| Time since event | now − event time | days since service | Represent recency. |
| Frequency | events / time | orders per month | Normalize activity by exposure time. |
Ratios, differences, totals, and counts
Arithmetic features are often the most interpretable. Ratios can compare scale-adjusted behavior, differences can represent gaps, totals can reconstruct the quantity that drives the target, and counts can summarize exposure or activity. Always define how missing and zero denominators are handled.
PYTHON • Simple arithmetic features import numpy as np |
Interaction and polynomial terms
An interaction term allows the effect of one feature to depend on another. Polynomial terms allow a linear estimator to represent curved relationships. These methods can be powerful, but they increase dimensionality and can amplify multicollinearity or overfitting if used indiscriminately.
PYTHON • Polynomial and interaction terms with scikit-learn from sklearn.preprocessing import PolynomialFeatures | |
| INTERPRETATION With degree 2, PolynomialFeatures can create x₁², x₂², and x₁×x₂ terms. That expands the hypothesis space, so cross-validation becomes essential. |
Binning and log transformations
Binning replaces a continuous variable with intervals. This may be useful when the relationship changes by operational regime or when a simple categorical interpretation is desirable. A log transform is especially common for positive, right-skewed variables such as revenue, counts, or transaction values.
x_log = log(1 + x) log1p is numerically convenient and is defined when x = 0. | |
PYTHON • Binning and log transformation import numpy as np |
Time-since-event and frequency features
Time-based features convert timestamps into durations or rates that match the prediction moment. Recency, frequency, and exposure-normalized rates are widely useful because an event yesterday often has a different meaning from the same event one year ago.
PYTHON • Recency and frequency features prediction_date = pd.Timestamp("2026-09-01") |
28.3 Group-based features
Many machine-learning tables contain one row per prediction entity—such as a customer, machine, account, or patient—while raw history contains many events per entity. Group-based feature engineering summarizes that event history into predictors that can be joined to the modeling table.
Table 28.4. Examples of group-based historical features
| Entity | Historical records | Engineered feature | Interpretation |
|---|---|---|---|
| Customer | Purchases | Average purchase value | Typical monetary size of an order. |
| Customer | Transactions | Number of transactions | Historical activity volume. |
| Machine | Sensor readings | Maximum temperature | Extreme operating condition. |
| Machine | Failure log | Historical failure count | Prior reliability experience. |
| Account | Daily balances | 30-day average balance | Recent level smoothed over time. |
Aggregation with pandas
PYTHON • Create customer-level historical summaries customer_history = ( |
Windowed and historical features
A single lifetime average can hide recent change. Time-window features—such as the number of failures in the previous 30 days or average usage in the previous 7 days—often align better with operational decisions. The critical requirement is that each window ends at or before the prediction timestamp.
historical window = [prediction time − window length, prediction time] Never allow observations after the prediction time to enter the aggregation. | |
| ENTITY SAFETY If multiple rows from the same customer, patient, machine, household, or site appear in the dataset, combine group-aware feature construction with group-aware validation when appropriate. Otherwise the model may benefit from near-duplicate history across folds. |
28.4 Leakage risks
A feature is useful only if it can be computed with information legitimately available when the prediction is made. Leakage occurs when training features include information about the future, the target, or the held-out validation data. Leakage can create spectacular validation scores that collapse in deployment.
Table 28.5. Common leakage patterns
| Leakage type | Problematic example | Why it leaks | Safer alternative |
|---|---|---|---|
| Future aggregation | Next 30 days of purchases used to predict tomorrow | Uses events that have not happened yet. | Aggregate only history before prediction time. |
| Target-derived feature | Mean target by category computed on all rows | Validation targets influence validation features. | Fit target encoding inside each training fold. |
| Global statistic | Standardization mean computed before CV | Validation fold contributes to preprocessing. | Put fitted preprocessing inside a Pipeline. |
| Prediction-time absence | “final claim status” used at application time | Feature is not known when decision is required. | Use only variables available at scoring time. |
Statistics computed from the complete dataset
Some feature transformations must learn parameters: means, medians, category frequencies, target encodings, vocabulary, principal components, and selected features. When such parameters are estimated from the full dataset before cross-validation, information from the validation fold influences the training representation.
PYTHON • Leakage-safe preprocessing with a Pipeline from sklearn.impute import SimpleImputer |
Target-based features
Target encoding can be legitimate, but it must be learned using training targets only. For each validation fold, category-to-target statistics must be fitted on the corresponding training folds and then applied to validation rows. More advanced implementations use smoothing and nested or out-of-fold strategies to reduce noise and leakage.
| PREDICTION-TIME TEST Before keeping a feature, ask: “At the exact moment the model must produce a prediction, can this value be computed without knowing the future or the answer?” If not, remove or redesign it. |
28.5 Evaluating engineered features
Feature engineering should be treated as an experiment. A feature that sounds reasonable may add no predictive value, may duplicate existing information, may help only one fold, or may increase variance. Evaluation should therefore compare a baseline representation with controlled alternatives under the same validation strategy.
Ablation studies
An ablation study removes one feature or one group of features from an otherwise unchanged system. If performance falls consistently when a feature is removed, that feature is contributing useful information. If performance improves, the feature may be noisy, redundant, or harmful.
ablation effect = score(all features) − score(with feature removed) Report the effect across folds rather than from one split whenever possible. |
Cross-validation comparison
Use the same folds, metric, preprocessing, and estimator for baseline and engineered feature sets. Paired fold-by-fold results are especially informative because each version is evaluated on the same validation observations.
Feature importance and stability
Feature importance can indicate which engineered variables a fitted model uses, but importance is not proof of causality. Correlated features can share or exchange importance. Stability across folds is therefore valuable: a feature whose coefficient or importance changes sign dramatically may be unstable even if its average importance appears large.
Table 28.6. Evidence that an engineered feature is useful
| Evidence | Strong sign | Warning sign | |
|---|---|---|---|
| Cross-validation mean | Improves the chosen metric consistently. | Tiny change within normal fold variability. | |
| Ablation | Removing feature degrades performance. | Removing feature improves performance. | |
| Fold stability | Effect direction is reasonably stable. | Large sign/rank changes across folds. | |
| Interpretation | Mechanism is plausible and available at scoring time. | Feature is difficult to justify operationally. | |
| Production feasibility | Can be computed reliably with low latency. | Requires future, delayed, or fragile data. | |
| EXPERIMENTAL DISCIPLINE Do not repeatedly engineer features against the final test set. Use cross-validation or a validation set for development, then evaluate the selected feature recipe once on the protected test set. | |||
Practical lab — Design and test engineered features
Goal: build a baseline classifier, create at least five interpretable features, compare baseline and engineered representations with the same cross-validation folds, perform an ablation study, and inspect whether the engineered effects are stable.
| LAB RULE All feature formulas below use only information available at prediction time. The cross-validation object is created once and reused for every comparison. |
Step 1 — Create a reproducible customer dataset
PYTHON • Generate raw customer features and a binary target import numpy as np |
Step 2 — Define baseline and engineered features
PYTHON • Define the baseline feature set target = "will_buy_next_month" |
PYTHON • Engineer more than five new features engineered["spend_per_transaction"] = ( |
Step 3 — Build one evaluation strategy
PYTHON • Use the same folds for every feature set from sklearn.linear_model import LogisticRegression |
Step 4 — Compare baseline and engineered representations
PYTHON • Cross-validation comparison def evaluate(feature_list): |
Interpretation task: decide whether the engineered representation improves the primary metric by more than normal fold-to-fold variability. Do not rely on the mean alone; inspect the standard deviation as well.
Step 5 — Perform an ablation study
PYTHON • Remove one engineered feature at a time full_auc = evaluate(all_features)["AUC mean"] | |
| READ THE SIGN A positive ablation effect means the full model scored better than the version without that feature. A negative value means performance improved when the feature was removed. |
Step 6 — Inspect coefficient stability across folds
PYTHON • Collect standardized logistic-regression coefficients from sklearn.base import clone |
Step 7 — Produce the feature-engineering report
1. List the engineered features you created and explain the domain meaning of each one.
2. Report baseline and engineered cross-validation AUC, F1, accuracy, and fold variability.
3. Identify which engineered feature has the largest positive ablation effect.
4. Identify any feature whose removal improves performance and explain why it may be redundant or noisy.
5. Inspect coefficient signs and variability across folds. Which engineered effects are stable?
6. For every feature, state whether it is guaranteed to be available at prediction time.
7. Recommend a final feature set and justify the decision with validation evidence rather than intuition alone.
Extension — A leakage audit for group-based features
Imagine that each customer also has a transaction-history table. Before adding average purchase, transaction count, maximum amount, or a rolling 30-day total, write down the prediction timestamp and prove that every contributing event occurred before that timestamp. If customers have multiple prediction rows, decide whether group-aware or time-aware validation is required.
Table 28.7. Suggested student audit
| Feature | Available at prediction time? | Uses future events? | Needs fitting inside CV? | Keep / redesign / remove |
|---|---|---|---|---|
| spend_per_transaction | Yes | No | No | Student decision |
| 30-day purchase count | Depends on window definition | Must be No | No if strictly historical | Student decision |
| category target mean | Potentially | No | Yes | Student decision |
| global category frequency | Potentially | No | Yes if learned from data | Student decision |
Chapter summary
- Feature engineering transforms raw data into representations that can expose useful predictive structure.
- Ratios, differences, totals, counts, interactions, polynomial terms, bins, logs, recency, and frequency features are common tools.
- Group-based features summarize historical events but must respect entity boundaries and prediction time.
- Future information, target-derived statistics, global preprocessing, and unavailable production variables are major leakage risks.
- Ablation studies and cross-validation are stronger evidence than intuition about whether a feature helps.
- Importance should be interpreted together with correlation, stability across folds, operational feasibility, and domain meaning.
- The final feature set should be reproducible, leakage-safe, explainable enough for the application, and validated on unseen data.
| NEXT STEP Once useful features have been designed, the next modeling task is usually to tune model hyperparameters systematically while preserving the validation discipline established in Chapters 26–28. |