Chapter 8 — Feature and Target Preparation
Roles • semantic types • conversions • temporal features • entity-aware splitting
A practical chapter for transforming a cleaned table into a semantically valid, leakage-resistant feature matrix and target vector ready for supervised learning. |
Chapter Overview
Cleaning removes obvious defects, but a clean table is not automatically a valid machine learning dataset. Every column must be assigned a precise role. The target must represent the future outcome that the model is intended to predict. Features must be available at prediction time, carry stable meaning, use appropriate data types, and remain aligned with the target after filtering, sorting, joining, or splitting operations.
This chapter develops the semantic and technical preparation stage that sits between data cleaning and model preprocessing. It explains how to construct the feature matrix X and target vector y, classify feature types, convert raw values safely, engineer time-based variables, and prevent identifiers from causing memorization or entity leakage.
Learning objectives
- Create a feature matrix X and target vector y while preserving row-level alignment.
- Distinguish identifiers, predictors, target variables, metadata, and target-derived leakage fields.
- Classify columns by semantic feature type rather than relying only on their storage dtype.
- Convert numeric, categorical, Boolean, and date values without silently changing meaning.
- Handle invalid conversions explicitly and produce auditable validation reports.
- Extract calendar, duration, recency, seasonal, and cyclical time features using a defensible reference time.
- Explain why identifiers usually harm generalization and detect accidental information encoded in them.
- Use group-aware splitting when several rows belong to the same customer, patient, machine, account, or other entity.
- Build a data dictionary that specifies each variable’s meaning, role, availability, and required preprocessing.
- Deliver a reproducible preparation function that can be applied consistently to future data.
Running case study
The examples use a customer-renewal dataset. Each row represents one contract observed at a defined snapshot time. The modeling objective is to predict whether the contract will renew within the following 30 days. The raw table includes valid predictors, identifiers, dates, free text, and several fields that would reveal the outcome if used incorrectly.
Column | Example | Initial concern |
|---|---|---|
| customer_id | C004812 | Entity identifier; repeated across contracts and events. |
| contract_id | CTR-2025-00981 | Row identifier; should not be a predictor. |
| monthly_spend | "1,249.50" | Numeric value stored as text. |
| plan | Premium | Nominal category. |
| satisfaction_level | high | Ordered category. |
| auto_pay | Yes | Boolean meaning stored as text. |
| signup_date | 2023-04-12 | Date requiring parsing and age calculation. |
| last_activity_at | 2026-07-20 17:42 | Timestamp for recency features. |
| support_notes | Asked about cancellation fee | Free text; may need NLP preprocessing. |
| renewed_30d | 1 | Target variable. |
| cancellation_date | 2026-08-18 | Post-outcome field; direct target leakage. |
LEAKAGE WARNING | The prediction-time test A feature is eligible only if its value would be known, reliable, and legally usable at the exact moment when the prediction is produced. A column can be historically present in the database and still be unavailable at prediction time. |
8.1 Separating Features and Target
Separating X and y is not merely a syntactic operation. It formalizes the learning problem. The feature matrix contains the information supplied to the model, while the target vector contains the outcome that the model must learn to predict. An incorrect role assignment can invalidate every later result even if the code runs without error.
Figure 8.1 — A semantic workflow for constructing X and y.
8.1.1 Creating the feature matrix X
For n observations and p predictors, the feature matrix X has n rows and p columns. Row i must describe the same observation represented by target value yᵢ. Columns may initially contain mixed raw types, but each column should have a stable definition and a known transformation plan.
X ∈ ℝⁿˣᵖ or, before encoding, X = [xᵢⱼ] with mixed semantic feature types |
In pandas, X is usually a DataFrame because its index and column labels help preserve alignment and meaning. Converting too early to a NumPy array removes these labels and makes auditing more difficult.
| PYTHON • Example 8.1 — Create the initial feature matrix |
| import pandas as pd # Each row is one contract at the prediction snapshot. df = pd.read_csv( "customer_renewal_snapshot.csv", parse_dates=["snapshot_at", "signup_date", "last_activity_at"] ) target_col = "renewed_30d" X = df.drop(columns=[target_col]).copy() print("X shape:", X.shape) print("Feature columns:", X.columns.tolist()) |
8.1.2 Creating the target vector y
The target vector y contains one outcome per observation. For binary classification it may contain 0 and 1; for multiclass classification it contains one class label per row; for regression it contains a continuous numeric value. The target definition must include the event, horizon, reference time, and labeling rules.
Element | Renewal example | Why it matters |
|---|---|---|
| Event | Contract renews | Defines the observable outcome. |
| Horizon | Within 30 days | Prevents mixing short- and long-term outcomes. |
| Reference time | Snapshot timestamp | Separates past predictors from future outcome. |
| Positive label | 1 = renewed | Ensures consistent interpretation of metrics. |
| Negative label | 0 = not renewed | Must distinguish mature negatives from unknown outcomes. |
| Exclusions | Contracts without a complete 30-day follow-up | Avoids labeling delayed outcomes as negative. |
| PYTHON • Example 8.2 — Create and validate a binary target |
| target_col = "renewed_30d" y = df[target_col].copy() # Validate that the target is complete and uses the intended labels. assert y.notna().all(), "The supervised target contains missing labels." assertset(y.unique()).issubset({0, 1}), "Unexpected target labels." print(y.value_counts(dropna=False)) print(y.value_counts(normalize=True).rename("proportion")) |
LEAKAGE WARNING | Do not impute supervised labels A missing target usually means the outcome is unknown, delayed, censored, or unavailable. Inventing a target with mean, mode, or model-based imputation changes the learning task and can create circular evidence. |
8.1.3 Removing identifiers
Identifiers distinguish records or entities but normally do not describe a generalizable relationship. Examples include database primary keys, customer numbers, UUIDs, claim IDs, transaction references, filenames, and row counters. Their high cardinality allows flexible models to memorize examples, especially when the same entity appears repeatedly.
| PYTHON • Example 8.3 — Remove identifiers explicitly |
| identifier_cols = [ "customer_id", "contract_id", "source_row_id", ] X = X.drop(columns=identifier_cols, errors="raise") print("Remaining features:", X.columns.tolist()) |
Identifiers should still be retained outside X when they are needed for traceability, group-aware splitting, error analysis, or joining predictions back to operational systems. A useful pattern is to store them in a separate metadata table with the same index.
| PYTHON • Example 8.4 — Preserve identifiers as metadata |
| metadata = df[[ "customer_id", "contract_id", "snapshot_at", ]].copy() X = df.drop(columns=[ "renewed_30d", "customer_id", "contract_id", ]).copy() y = df["renewed_30d"].copy() assert metadata.index.equals(X.index) assert X.index.equals(y.index) |
8.1.4 Removing target-derived variables
A target-derived variable contains information created by, after, or in direct response to the outcome. It can make validation scores appear exceptional while producing a model that is impossible to use prospectively. Leakage can be explicit, such as cancellation_date, or subtle, such as a workflow status that is updated only after the renewal outcome is known.
Candidate variable | Leakage mechanism | Decision |
|---|---|---|
| renewal_processed_at | Created after the renewal event | Remove. |
| cancellation_date | Directly reveals non-renewal | Remove. |
| final_account_status | Updated after the outcome window | Remove or reconstruct as-of snapshot. |
| refund_amount_30d | Uses transactions inside the prediction horizon | Remove. |
| days_since_last_activity | Valid only if calculated from events before snapshot | Keep after temporal validation. |
| support_tickets_prior_90d | Historical window before snapshot | Usually eligible. |
| PYTHON • Example 8.5 — Remove documented leakage fields |
| target_derived_cols = [ "renewal_processed_at", "cancellation_date", "final_account_status", "refund_amount_30d", ] unexpected = sorted(set(target_derived_cols) - set(df.columns)) if unexpected: print("Optional leakage columns not present:", unexpected) X = X.drop(columns=target_derived_cols, errors="ignore") |
LEAKAGE WARNING | Name-based filtering is not enough A leakage column may have an innocent name. Confirm the business timestamp and data-generation process for every candidate feature. “Status”, “score”, “approved”, “closed”, and “resolved” fields deserve particular scrutiny. |
8.1.5 Checking alignment between X and y
Alignment means that row i in X and row i in y refer to the same observation. pandas aligns objects by index, which is valuable when used deliberately but dangerous when indices are duplicated, reset inconsistently, or created by independent filtering. Shape equality alone does not prove alignment.
| PYTHON • Example 8.6 — Validate row alignment |
| defvalidate_xy_alignment(X: pd.DataFrame, y: pd.Series) ->None: if len(X) != len(y): raise ValueError("X and y have different row counts.") if not X.index.equals(y.index): raise ValueError("X and y indices are not identical and ordered.") if X.index.has_duplicates: raise ValueError("The observation index contains duplicates.") if y.isna().any(): raise ValueError("The target contains missing labels.") validate_xy_alignment(X, y) |
Filtering should be performed with a shared Boolean mask or on a combined DataFrame before separation. Filtering X and y independently is a common source of subtle misalignment.
| PYTHON • Example 8.7 — Apply one eligibility mask |
| # Safe: construct one mask and apply it to all aligned objects. eligible = ( df["snapshot_at"].notna() & df["renewed_30d"].notna() & df["followup_complete"].eq(True) ) filtered = df.loc[eligible].copy() y = filtered["renewed_30d"].copy() X = filtered.drop(columns=["renewed_30d"]).copy() metadata = filtered[["customer_id", "contract_id"]].copy() assert X.index.equals(y.index) assert metadata.index.equals(y.index) |
8.1.6 A role registry
A role registry records which columns are targets, predictors, identifiers, metadata, exclusions, or post-outcome fields. It converts informal knowledge into executable controls and makes schema changes visible.
| PYTHON • Example 8.8 — Define column roles as configuration |
| COLUMN_ROLES = { "target": ["renewed_30d"], "identifier": ["customer_id", "contract_id"], "metadata": ["snapshot_at", "data_source"], "leakage": ["cancellation_date", "renewal_processed_at"], "feature": [ "monthly_spend", "plan", "satisfaction_level", "auto_pay", "signup_date", "last_activity_at", "support_notes" ], } all_registered = [ col for cols in COLUMN_ROLES.values() for col in cols ] assertlen(all_registered) == len(set(all_registered)), "A column was assigned to more than one role. |
QUALITY CHECK | Separation checklist Before training, verify the target definition and horizon, remove direct and indirect leakage, retain identifiers only as metadata, confirm one row per observation definition, and prove that X, y, and metadata share the same unique index. |
8.2 Feature Types
The storage dtype reported by pandas does not fully describe a feature. An integer may represent age, a count, a postal code, an ordered category, or a database key. Correct preprocessing depends on semantic meaning, measurement scale, and expected model behavior.
Figure 8.2 — Semantic feature types and typical subtypes.
8.2.1 Continuous numerical variables
A continuous variable can conceptually take any value within an interval, even when recorded to a finite precision. Examples include temperature, income, distance, response time, pressure, and monthly spend. Typical operations include missing-value imputation, scaling, robust transformation, clipping based on validated limits, and nonlinear feature generation.
- Units and measurement precision must be documented.
- Skewed distributions may benefit from logarithmic or power transformations.
- Outliers may be valid, erroneous, or operationally important; treatment should not be automatic.
- A numeric code is not continuous merely because it is stored as a number.
8.2.2 Discrete numerical variables
A discrete numerical variable takes countable values, often non-negative integers. Examples include number of support tickets, previous purchases, defects, visits, or devices. Counts retain numerical order and distance, but their distributions may be highly skewed or zero-inflated.
Question | Continuous example | Discrete example |
|---|---|---|
| What is measured? | Monthly spend | Number of support tickets |
| Possible values | Any non-negative monetary amount | 0, 1, 2, … |
| Meaning of difference | A 10-unit difference is meaningful | One additional event is meaningful |
| Common concerns | Units, skew, outliers | Zeros, overdispersion, rare large counts |
8.2.3 Nominal categorical variables
Nominal categories identify distinct groups without an intrinsic order. Examples include plan type, region, channel, device family, and payment method. Assigning arbitrary integers can create false distance and order. One-hot encoding, frequency-aware grouping, or model-specific categorical handling is usually more appropriate.
DECISION RULE | Preserve the distinction “Basic”, “Plus”, and “Premium” may appear ordered commercially, but the order must represent the modeled concept. If the categories describe product identities rather than increasing levels, treat them as nominal. |
8.2.4 Ordinal categorical variables
Ordinal categories have a defensible order but not necessarily equal spacing. Satisfaction levels such as low, medium, and high are ordered, yet the difference between low and medium is not guaranteed to equal the difference between medium and high. An explicit ordered mapping preserves order while making the assumption visible.
| PYTHON • Example 8.9 — Declare an ordered category |
| satisfaction_order = ["very_low", "low", "medium", "high", "very_high"] df["satisfaction_level"] = pd.Categorical( df["satisfaction_level"], categories=satisfaction_order, ordered=True, ) df["satisfaction_code"] = df["satisfaction_level"].cat.codes # Note: missing or unknown categories receive code -1 and need handling. |
8.2.5 Boolean variables
Boolean variables express two logical states such as yes/no, true/false, active/inactive, or present/absent. Raw data often contains multiple textual representations or nullable states. Unknown is not the same as false, so nullable Boolean types or explicit categories may be needed.
| PYTHON • Example 8.10 — Normalize Boolean values safely |
| boolean_map = { "yes": True, "no": False, "true": True, "false": False, "1": True, "0": False, } normalized = df["auto_pay"].astype("string").str.strip().str.lower() df["auto_pay"] = normalized.map(boolean_map).astype("boolean") unmapped = normalized[df["auto_pay"].isna()].dropna().unique() print("Unmapped Boolean values:", unmapped) |
8.2.6 Date and time variables
Raw timestamps rarely belong directly in a classical tabular model. Their useful information is usually expressed through calendar components, elapsed durations, recency, periodic cycles, and event ordering. Date engineering must respect the prediction snapshot and time zone.
8.2.7 Free-text variables
Free text includes notes, messages, descriptions, titles, and comments. It cannot be treated as an ordinary category because almost every string may be unique. Typical approaches include bag-of-words, TF-IDF, embeddings, keyword indicators, language detection, length features, or specialized language models. Text may also contain personal data, identifiers, or post-outcome statements that require governance and leakage review.
Feature type | Raw example | Typical representation | Main risk |
|---|---|---|---|
| Continuous | 1249.50 | Scaled numeric value | Units and outliers |
| Discrete | 3 tickets | Count or transformed count | Zero inflation |
| Nominal | Premium | One-hot or native categorical | False ordering |
| Ordinal | High | Ordered code | Assumed spacing |
| Boolean | Yes | Boolean / 0–1 | Unknown treated as false |
| Date/time | 2026-07-20 17:42 | Recency and calendar features | Temporal leakage |
| Free text | Asked about fees | TF-IDF or embedding | Privacy and leakage |
8.2.8 Building a semantic type inventory
| PYTHON • Example 8.11 — Register semantic feature groups |
| feature_groups = { "continuous": ["monthly_spend", "account_balance"], "discrete": ["support_tickets_90d", "login_count_30d"], "nominal": ["plan", "region", "payment_method"], "ordinal": ["satisfaction_level"], "boolean": ["auto_pay", "has_mobile_app"], "datetime": ["signup_date", "last_activity_at"], "text": ["support_notes"], } registered = [c for cols in feature_groups.values() for c in cols] assertlen(registered) == len(set(registered)) assertset(registered).issubset(X.columns) |
KEY IDEA | Semantic type beats dtype Use pandas dtypes as evidence, not as the final classification. The data dictionary should answer what the variable means, how it was produced, which values are valid, and how the model should represent it. |
8.3 Data Type Conversion
Conversion transforms raw storage formats into types that preserve the intended meaning. Safe conversion is explicit, validated, and reversible in the sense that unexpected values are reported rather than silently discarded. Conversion code should be fitted into a reproducible preparation pipeline or function, not performed manually in an undocumented notebook cell.
8.3.1 Converting strings to numbers
Numeric values may contain currency symbols, thousands separators, decimal commas, percent signs, whitespace, unit suffixes, or placeholders. A robust process first standardizes the representation, then converts with errors set to coerce, and finally audits the newly created missing values.
| PYTHON • Example 8.12 — Convert currency-like strings |
| raw = df["monthly_spend"].astype("string") cleaned = ( raw .str.strip() .str.replace("$", "", regex=False) .str.replace(",", "", regex=False) ) df["monthly_spend_num"] = pd.to_numeric(cleaned, errors="coerce") conversion_failed = raw.notna() & df["monthly_spend_num"].isna() print(df.loc[conversion_failed, ["monthly_spend"]].drop_duplicates()) |
For decimal-comma data, the order of replacements must match the locale. For example, “1.234,50” should become “1234.50”, whereas the same replacements would misinterpret “1,234.50”. Source-specific parsing rules should be documented rather than guessed row by row.
8.3.2 Converting categorical columns
Converting repeated strings to pandas category dtype reduces memory use and records the allowed levels. It does not by itself encode values for a model. Category declarations can also expose unexpected levels and provide stable schemas across training and prediction data.
| PYTHON • Example 8.13 — Validate and declare nominal categories |
| allowed_plans = ["Basic", "Plus", "Premium"] plan = df["plan"].astype("string").str.strip().str.title() unexpected = sorted(set(plan.dropna()) - set(allowed_plans)) if unexpected: raise ValueError(f"Unexpected plan categories: {unexpected}") df["plan"] = pd.Categorical(plan, categories=allowed_plans) |
8.3.3 Converting date columns
Date parsing must specify the expected format when possible. Automatic parsing may swap month and day, interpret mixed formats inconsistently, or ignore time-zone differences. Invalid values should become explicit missing timestamps and appear in a conversion report.
| PYTHON • Example 8.14 — Parse a known date format |
| raw_date = df["signup_date"].astype("string") df["signup_date_parsed"] = pd.to_datetime( raw_date, format="%Y-%m-%d", errors="coerce", utc=True, ) failed = raw_date.notna() & df["signup_date_parsed"].isna() print("Invalid signup dates:", failed.sum()) print(raw_date[failed].value_counts().head(10)) |
8.3.4 Handling invalid conversions
The errors="coerce" option is useful because it exposes invalid values as missing, but it should never be the final step. Compare missingness before and after conversion, inspect failed raw values, classify the cause, and decide whether to correct, map, reject, or retain them as unknown.
| PYTHON • Example 8.15 — Produce an auditable conversion report |
| defconversion_report(raw: pd.Series, converted: pd.Series) -> pd.DataFrame: failed = raw.notna() & converted.isna() return ( raw.loc[failed] .astype("string") .value_counts(dropna=False) .rename_axis("raw_value") .reset_index(name="failed_rows") ) report = conversion_report(df["monthly_spend"], df["monthly_spend_num"]) print(report.head(20)) |
8.3.5 Preserving category meaning
Conversion should preserve semantics, including leading zeros, ordered levels, explicit unknown states, and distinctions between missing, not applicable, and false. Postal codes, product codes, classroom numbers, and diagnosis codes often look numeric but should remain strings or categories.
Raw column | Tempting conversion | Semantic problem | Safer representation |
|---|---|---|---|
| postal_code = "00120" | Integer 120 | Leading zeros and geographic identity are lost | String or nominal category |
| risk_level = 1,2,3 | Continuous number | Assumes equal spacing | Ordered category with documented order |
| auto_pay = missing | False | Unknown becomes a negative fact | Nullable Boolean or Unknown category |
| product_code = 4021 | Continuous number | Arithmetic distance has no meaning | String/category |
| percentage = "8.5%" | String category | Magnitude is lost | Numeric 0.085 or 8.5 with documented unit |
8.3.6 A reusable conversion function
| PYTHON • Example 8.16 — Centralize conversion logic |
| defconvert_customer_columns(frame: pd.DataFrame) -> pd.DataFrame: out = frame.copy() spend = out["monthly_spend"].astype("string") spend = spend.str.strip().str.replace(",", "", regex=False) out["monthly_spend"] = pd.to_numeric(spend, errors="coerce") out["auto_pay"] = ( out["auto_pay"].astype("string").str.strip().str.lower() .map({"yes": True, "no": False}) .astype("boolean") ) out["signup_date"] = pd.to_datetime( out["signup_date"], format="%Y-%m-%d", errors="coerce", utc=True ) out["plan"] = pd.Categorical( out["plan"].astype("string").str.strip().str.title(), categories=["Basic", "Plus", "Premium"], ) return out |
QUALITY CHECK | Fail loudly on schema drift When a new category, unit, date format, or Boolean representation appears, the safest default is to report it. Silent coercion can turn a data pipeline change into widespread missing values and degraded predictions. |
8.4 Date and Time Feature Extraction
Temporal variables provide information about calendar position, elapsed time, recency, seasonality, and periodic behavior. Every derived value must use only timestamps available at the prediction snapshot. A date feature calculated against the current system clock can change between training and inference and may not reproduce the historical state.
8.4.1 Calendar components: year, month, day, weekday, and hour
Calendar components can represent long-term trends, monthly seasonality, operational schedules, weekday effects, and time-of-day behavior. Their usefulness depends on the domain. The numerical order of month or hour is not fully appropriate for periodic variables because the endpoints are adjacent in reality.
| PYTHON • Example 8.17 — Extract calendar components |
| snapshot = pd.to_datetime(df["snapshot_at"], utc=True) last_activity = pd.to_datetime(df["last_activity_at"], utc=True) X_time = pd.DataFrame(index=df.index) X_time["snapshot_year"] = snapshot.dt.year X_time["snapshot_month"] = snapshot.dt.month X_time["snapshot_day"] = snapshot.dt.day X_time["snapshot_dayofweek"] = snapshot.dt.dayofweek # Monday = 0 X_time["snapshot_hour"] = snapshot.dt.hour X_time["snapshot_is_weekend"] = snapshot.dt.dayofweek.ge(5) |
8.4.2 Duration features
A duration is the difference between two events. Examples include contract age, time since account creation, session length, repair duration, and time between purchases. Duration often generalizes better than absolute dates because it expresses the relationship relevant to the decision.
| PYTHON • Example 8.18 — Calculate duration with temporal validation |
| signup = pd.to_datetime(df["signup_date"], utc=True) snapshot = pd.to_datetime(df["snapshot_at"], utc=True) contract_age = snapshot - signup X_time["contract_age_days"] = contract_age.dt.total_seconds() / 86_400 invalid_order = X_time["contract_age_days"].lt(0) if invalid_order.any(): raise ValueError("Found signup dates after the prediction snapshot.") |
8.4.3 Recency features
Recency measures how long it has been since an event. It is often predictive of churn, demand, maintenance, engagement, and fraud. The reference time should be the row-specific prediction snapshot, not the date when the notebook happens to run.
| PYTHON • Example 8.19 — Engineer a leakage-safe recency feature |
| last_activity = pd.to_datetime(df["last_activity_at"], utc=True) snapshot = pd.to_datetime(df["snapshot_at"], utc=True) recency = snapshot - last_activity X_time["days_since_last_activity"] = recency.dt.total_seconds() / 86_400 future_activity = X_time["days_since_last_activity"].lt(0) if future_activity.any(): raise ValueError("Activity occurred after the prediction snapshot.") |
8.4.4 Seasonal indicators
Seasonal indicators encode known periods such as weekends, quarters, academic terms, holidays, heating seasons, billing cycles, or peak hours. They should be based on domain definitions and prediction geography. Holiday calendars differ by country and sometimes by region.
| PYTHON • Example 8.20 — Create simple seasonal indicators |
| month = snapshot.dt.month X_time["quarter"] = snapshot.dt.quarter X_time["is_year_end"] = month.isin([11, 12]) X_time["is_summer_north"] = month.isin([6, 7, 8]) # Domain-defined billing cycle position. X_time["billing_day"] = snapshot.dt.day X_time["near_month_end"] = snapshot.dt.day.ge(25) |
8.4.5 Cyclical encoding
Month, weekday, and hour are periodic. Encoding hour as the integer 0–23 makes 23 and 0 appear far apart, although they are adjacent. Sine and cosine coordinates map a periodic variable onto a circle and preserve this adjacency.
sin(2πv/P) and cos(2πv/P), where v is the value and P is the period |
Figure 8.3 — Periodic variables can be represented on a circle.
| PYTHON • Example 8.21 — Encode hour and month cyclically |
| import numpy as np hour = snapshot.dt.hour X_time["hour_sin"] = np.sin(2 * np.pi * hour / 24) X_time["hour_cos"] = np.cos(2 * np.pi * hour / 24) month_zero_based = snapshot.dt.month - 1 X_time["month_sin"] = np.sin(2 * np.pi * month_zero_based / 12) X_time["month_cos"] = np.cos(2 * np.pi * month_zero_based / 12) |
8.4.6 Time zones and daylight-saving changes
Store timestamps with a time-zone policy. UTC is useful for storage and ordering, but local calendar features may require conversion to the business or user time zone. Daylight-saving transitions can create repeated or nonexistent local times. The data dictionary should specify the source and intended zone for each timestamp.
| PYTHON • Example 8.22 — Convert timestamps before local calendar extraction |
| # Parse as UTC, then derive local operational time. event_utc = pd.to_datetime(df["last_activity_at"], utc=True) event_local = event_utc.dt.tz_convert("Africa/Casablanca") X_time["local_hour"] = event_local.dt.hour X_time["local_dayofweek"] = event_local.dt.dayofweek |
8.4.7 A temporal feature function
| PYTHON • Example 8.23 — Build temporal features reproducibly |
| defbuild_time_features(frame: pd.DataFrame) -> pd.DataFrame: snapshot = pd.to_datetime(frame["snapshot_at"], utc=True) signup = pd.to_datetime(frame["signup_date"], utc=True) activity = pd.to_datetime(frame["last_activity_at"], utc=True) out = pd.DataFrame(index=frame.index) out["contract_age_days"] = (snapshot - signup).dt.days out["activity_recency_days"] = ( snapshot - activity ).dt.total_seconds() / 86_400 out["snapshot_dayofweek"] = snapshot.dt.dayofweek out["snapshot_is_weekend"] = snapshot.dt.dayofweek.ge(5) hour = snapshot.dt.hour out["snapshot_hour_sin"] = np.sin(2 * np.pi * hour / 24) out["snapshot_hour_cos"] = np.cos(2 * np.pi * hour / 24) if (out[["contract_age_days", "activity_recency_days"]] < 0).any().any(): raise ValueError("A source event occurs after its prediction snapshot.") return out |
KEY IDEA | Reference time is part of the feature definition “Days since last activity” is incomplete documentation. State “days from last eligible activity to the contract snapshot timestamp, excluding events after the snapshot.” This definition can be reproduced for training and live predictions. |
8.5 Identifier Columns
Identifiers are useful for database operations and audit trails but generally unsuitable as model inputs. They often have nearly one unique value per row, carry arbitrary formatting, and change when data is migrated. Their apparent predictive power frequently comes from memorization, repeated entities, source-system artifacts, or time encoded in sequential IDs.
8.5.1 Why IDs usually should not be model features
- The numerical distance between IDs has no domain meaning.
- High-cardinality identifiers allow trees or embeddings to memorize entities.
- New entities at prediction time have unseen IDs.
- Sequential IDs may proxy for time, batches, locations, or data-source changes.
- Repeated IDs can cause the same entity to appear in training and testing data.
- Identifiers may expose personal or operational information that the model does not need.
| PYTHON • Example 8.24 — Flag near-unique columns for review |
| id_audit = pd.DataFrame({ "dtype": df.dtypes.astype(str), "unique": df.nunique(dropna=False), "rows": len(df), }) id_audit["uniqueness_ratio"] = id_audit["unique"] / id_audit["rows"] possible_ids = id_audit.query("uniqueness_ratio > 0.95") print(possible_ids.sort_values("uniqueness_ratio", ascending=False)) |
8.5.2 Cases where an identifier contains accidental information
An identifier can encode a creation year, branch, device family, batch, geography, or source system. Using the entire identifier is rarely defensible. When the embedded component is genuinely available and meaningful, extract it into an explicitly named feature and validate its stability.
Identifier pattern | Accidental information | Recommended action |
|---|---|---|
| CTR-2026-004812 | Contract creation year | Extract creation year only if it is valid and not redundant with dates. |
| MA-07-93821 | Country and branch code | Extract branch as a category after governance review. |
| DEVICE-A3-... | Hardware family | Use a documented device-family feature, not the serial number. |
| 000001, 000002, ... | Record order or ingestion time | Remove; use an actual timestamp if appropriate. |
| Image filename with class name | Target class | Remove or sanitize; direct leakage. |
| PYTHON • Example 8.25 — Extract meaning, not identity |
| # Extract a documented semantic component; discard the full ID. parts = df["contract_id"].astype("string").str.extract( r"^CTR-(?P<creation_year>\d{4})-(?P<sequence>\d+)$" ) df["contract_creation_year"] = pd.to_numeric( parts["creation_year"], errors="coerce" ) # Never use the sequence or the complete identifier as a predictor. df = df.drop(columns=["contract_id"]) |
8.5.3 Entity-aware splitting
When several rows belong to the same entity, a random row split can place one entity’s records in both training and test sets. The model then benefits from entity-specific behavior that will not be available for genuinely new entities. Group-based splitting keeps each entity entirely within one partition.
Figure 8.4 — Group-based splitting prevents entity overlap.
| PYTHON • Example 8.26 — Split by customer |
| from sklearn.model_selection import GroupShuffleSplit groups = df["customer_id"] splitter = GroupShuffleSplit( n_splits=1, test_size=0.20, random_state=42, ) train_idx, test_idx = next(splitter.split(X, y, groups=groups)) X_train, X_test = X.iloc[train_idx], X.iloc[test_idx] y_train, y_test = y.iloc[train_idx], y.iloc[test_idx] train_customers = set(groups.iloc[train_idx]) test_customers = set(groups.iloc[test_idx]) assert train_customers.isdisjoint(test_customers) |
8.5.4 Choosing the correct generalization target
The split must reflect the intended use. If future predictions will concern new rows from known customers, a time-based split within customer may be relevant. If the model must generalize to entirely new customers, use a group split. If it must generalize across hospitals, factories, schools, or devices, group by that higher-level domain.
Deployment question | Appropriate separation | Example |
|---|---|---|
| New events for known entities? | Chronological split | Predict next purchase for existing customers. |
| Completely new entities? | Group split by entity | Predict risk for newly enrolled customers. |
| New sites or institutions? | Group split by site | Validate across hospitals or factories. |
| Future period with recurring entities? | Time split, with leakage controls | Predict next quarter using prior quarters. |
| Multiple nested levels? | Group by the level that must generalize | Patients nested within clinics. |
8.5.5 Keep IDs for traceability, not learning
| PYTHON • Example 8.27 — Reattach identifiers after prediction |
| # Modeling objects X = prepared_features.copy() y = df["renewed_30d"].copy() # Audit object retained separately prediction_keys = df[[ "customer_id", "contract_id", "snapshot_at", ]].copy() # Later, attach predictions without exposing IDs to the estimator. results = prediction_keys.loc[X_test.index].copy() results["predicted_probability"] = model.predict_proba(X_test)[:, 1] |
PRACTICE NOTE | Identity and grouping are separate roles A column can be excluded from X yet remain essential for splitting, auditing, subgroup evaluation, deduplication, and delivering predictions. “Do not model with it” does not mean “delete it from the project.” |
Practical Activity — Build a Feature and Target Data Dictionary
Students receive a deliberately mixed customer-renewal dataset. Their task is to define the observation, select the target, classify each variable by semantic type and role, specify prediction-time availability, and propose required preprocessing. The final output is a data dictionary plus executable preparation checks.
Activity scenario
A subscription company wants to contact customers who are unlikely to renew within the next 30 days. A prediction is generated at midnight on each contract snapshot date. Each row in the supplied table represents one active contract at that snapshot. Some customers own several contracts, and the dataset contains operational fields updated after the renewal process.
Laboratory dataset
Variable | Sample value | Initial description |
|---|---|---|
| customer_id | C01982 | Customer key; may repeat across contracts. |
| contract_id | CTR-2024-00819 | Unique contract key. |
| snapshot_at | 2026-07-01 00:00:00+00:00 | Prediction reference time. |
| age_text | "42 years" | Customer age stored as text. |
| monthly_spend | "1,249.50" | Monthly amount stored as text. |
| support_tickets_90d | 3 | Count before snapshot. |
| plan | premium | Subscription plan. |
| satisfaction_level | high | Ordered survey response. |
| auto_pay | YES | Boolean stored as text. |
| signup_date | 2022-10-14 | Contract start date. |
| last_activity_at | 2026-06-27 18:31:00+00:00 | Most recent eligible activity. |
| support_notes | Asked about a cheaper plan | Free-text note before snapshot. |
| renewal_offer_sent_at | 2026-07-03 | Campaign action after prediction. |
| renewal_processed_at | 2026-07-21 | Outcome-processing timestamp. |
| renewed_30d | 0 | Binary supervised target. |
Student tasks
- Write the exact observation definition and prediction-time statement.
- Identify the target and state the positive class, negative class, and 30-day horizon.
- Separate candidate features, identifiers, metadata, and leakage exclusions.
- Classify every candidate feature as continuous, discrete, nominal, ordinal, Boolean, date/time, or free text.
- Specify a safe data type conversion for each raw column.
- Design date-derived features using snapshot_at as the reference time.
- Choose a splitting strategy that prevents the same customer from appearing in both training and test sets.
- Create a data dictionary with validation rules and proposed preprocessing.
- Implement executable checks proving X-y alignment and the absence of entity overlap.
- Write five evidence-based preparation decisions and explain their consequences.
Data dictionary template
Field | Required content |
|---|---|
| Variable name | Exact source-column name. |
| Semantic meaning | What the value represents in the domain. |
| Observation time | When the value is measured relative to the snapshot. |
| Raw dtype | How it is stored in the source. |
| Semantic type | Continuous, discrete, nominal, ordinal, Boolean, datetime, text, target, ID, or metadata. |
| Role | Feature, target, identifier, grouping key, metadata, or exclusion. |
| Valid values / range | Allowed levels, units, range, or format. |
| Missing-value meaning | Unknown, unavailable, not applicable, delayed, or invalid. |
| Required conversion | Parsing, normalization, mapping, or extraction. |
| Model preprocessing | Scaling, encoding, text vectorization, imputation, or pass-through. |
| Leakage assessment | Why the value is or is not available at prediction time. |
Starter code
| PYTHON • Lab starter — Define roles and construct initial X and y |
| import pandas as pd raw = pd.read_csv("feature_target_lab.csv") TARGET = "renewed_30d" IDENTIFIERS = ["customer_id", "contract_id"] METADATA = ["snapshot_at"] LEAKAGE = ["renewal_offer_sent_at", "renewal_processed_at"] # Students must complete the conversion and feature lists. y = raw[TARGET].copy() X = raw.drop(columns=[TARGET, *IDENTIFIERS, *LEAKAGE]).copy() assertlen(X) == len(y) assert X.index.equals(y.index) |
Worked data dictionary
Variable | Semantic type | Role | Required preparation |
|---|---|---|---|
| customer_id | Identifier / grouping key | Exclude from X; use for group split | Preserve as string; validate non-missing. |
| contract_id | Identifier | Exclude from X; retain for traceability | Preserve as string; validate uniqueness per snapshot. |
| snapshot_at | Datetime metadata | Reference time; may derive calendar features | Parse UTC; validate one per observation. |
| age_text | Continuous numeric after parsing | Feature | Extract numeric years; range check; impute if justified. |
| monthly_spend | Continuous numeric | Feature | Remove separators; parse float; document currency. |
| support_tickets_90d | Discrete count | Feature | Parse integer; require non-negative values. |
| plan | Nominal categorical | Feature | Trim/case-normalize; validate allowed plans; encode. |
| satisfaction_level | Ordinal categorical | Feature | Map explicit ordered levels; retain unknown separately. |
| auto_pay | Nullable Boolean | Feature | Normalize YES/NO; do not map missing to False. |
| signup_date | Datetime | Source for duration feature | Parse; derive contract_age_days from snapshot. |
| last_activity_at | Datetime | Source for recency feature | Parse; derive activity_recency_days; reject future events. |
| support_notes | Free text | Optional feature after governance review | Sanitize identifiers; vectorize or derive text features. |
| renewal_offer_sent_at | Post-snapshot action | Exclude | Campaign treatment may be after prediction and outcome-related. |
| renewal_processed_at | Target-derived datetime | Exclude | Directly reveals outcome-processing workflow. |
| renewed_30d | Binary target | y | Validate mature labels {0,1}; never impute. |
Worked preparation function
| PYTHON • Lab solution — Prepare modeling objects |
| defprepare_feature_target(raw: pd.DataFrame): df = raw.copy() # Parse reference and source timestamps. for col in ["snapshot_at", "signup_date", "last_activity_at"]: df[col] = pd.to_datetime(df[col], errors="coerce", utc=True) # Numeric conversion. df["age"] = pd.to_numeric( df["age_text"].astype("string").str.extract(r"(\d+(?:\.\d+)?)")[0], errors="coerce", ) spend = df["monthly_spend"].astype("string").str.replace(",", "", regex=False) df["monthly_spend"] = pd.to_numeric(spend, errors="coerce") df["support_tickets_90d"] = pd.to_numeric( df["support_tickets_90d"], errors="coerce" ) # Categorical and Boolean normalization. df["plan"] = df["plan"].astype("string").str.strip().str.title() df["satisfaction_level"] = pd.Categorical( df["satisfaction_level"].astype("string").str.strip().str.lower(), categories=["very_low", "low", "medium", "high", "very_high"], ordered=True, ) df["auto_pay"] = ( df["auto_pay"].astype("string").str.strip().str.lower() .map({"yes": True, "no": False}) .astype("boolean") ) # Leakage-safe time features. df["contract_age_days"] = (df["snapshot_at"] - df["signup_date"]).dt.days df["activity_recency_days"] = ( df["snapshot_at"] - df["last_activity_at"] ).dt.total_seconds() / 86_400 target = df["renewed_30d"].copy() groups = df["customer_id"].copy() metadata = df[["customer_id", "contract_id", "snapshot_at"]].copy() feature_cols = [ "age", "monthly_spend", "support_tickets_90d", "plan", "satisfaction_level", "auto_pay", "contract_age_days", "activity_recency_days", "support_notes", ] features = df[feature_cols].copy() if not features.index.equals(target.index): raise ValueError("Feature-target alignment failed.") if (features[["contract_age_days", "activity_recency_days"]] < 0).any().any(): raise ValueError("A source event occurs after the snapshot.") return features, target, groups, metadata |
Code block 8.1 — A compact preparation function for the workshop dataset.
Validation checks
| PYTHON • Lab solution — Execute preparation checks |
| X, y, groups, metadata = prepare_feature_target(raw) assertlen(X) == len(y) == len(groups) == len(metadata) assert X.index.equals(y.index) assert y.notna().all() assertset(y.unique()).issubset({0, 1}) assert"customer_id" not in X.columns assert"contract_id" not in X.columns assert"renewal_processed_at" not in X.columns assert X["contract_age_days"].ge(0).dropna().all() assert X["activity_recency_days"].ge(0).dropna().all() print(X.dtypes) print(y.value_counts(dropna=False)) |
Five evidence-based observations
- customer_id is not a predictor because its uniqueness and repetition can encourage memorization; it is retained as the group key for splitting.
- renewal_processed_at is excluded because it is created after the outcome and would directly leak the target-processing workflow.
- monthly_spend is semantically continuous even though the source stores it as text; conversion failures must be reported before imputation.
- satisfaction_level is ordinal because its levels have a defensible ranking, but equal numeric spacing is not assumed.
- contract age and activity recency are calculated against snapshot_at, ensuring that future timestamps cannot enter the historical feature set.
Deliverables
- A completed data dictionary for every supplied variable.
- A notebook that constructs X, y, groups, and metadata.
- A conversion-failure report and category-validation report.
- A group-aware train-test split with an overlap assertion.
- Five evidence-based preparation observations.
- A short limitations section describing unresolved ambiguity or unavailable metadata.
Knowledge Check
Review questions
- Why can X and y have equal lengths but still be misaligned?
- What is the difference between a storage dtype and a semantic feature type?
- Why should missing supervised labels usually not be imputed?
- Give two examples of target-derived leakage.
- Why is a postal code usually not a continuous numerical variable?
- What is the difference between a nominal and an ordinal category?
- How can errors="coerce" help conversion, and what follow-up audit is required?
- Why should recency be calculated from a prediction snapshot rather than the current date?
- What problem does cyclical encoding solve?
- When is group-aware splitting more appropriate than random row splitting?
Suggested answers
- They may have different indices or row orders after independent filtering, sorting, joining, or resetting indices.
- The dtype describes storage; semantic type describes meaning, measurement scale, valid operations, and appropriate preprocessing.
- A missing label usually represents an unknown or immature outcome; inventing it changes the supervised evidence.
- Examples include an outcome-processing timestamp, final status, refund recorded after the outcome, or a field created by an intervention triggered after prediction.
- Its digits identify a geographic code; arithmetic distance and averages are not meaningful, and leading zeros may matter.
- Nominal categories have no intrinsic order; ordinal categories have a defensible order but not necessarily equal spacing.
- It exposes invalid values as missing; the raw non-missing values that became missing must be counted and inspected.
- The snapshot reconstructs what was known at prediction time and makes historical and live calculations reproducible.
- It preserves adjacency at the endpoints of a periodic variable, such as hour 23 being close to hour 0.
- When multiple records belong to the same entity and the model must generalize to new entities or avoid entity overlap.
Mini-assessment
Statement | Correct answer | Reason |
|---|---|---|
| An integer column should always be treated as numerical. | False | It may be an ID, code, category, or count. |
| Identifiers can be retained outside X for traceability. | True | Exclusion from modeling does not require deletion from the project. |
| The test set may be used to discover the best target definition. | False | The problem and target must be defined before final evaluation. |
| Unknown Boolean values should automatically become False. | False | Unknown and false carry different evidence. |
| A date-derived feature can leak information. | True | It leaks when it uses events or reference times after prediction. |
| Group splitting is necessary whenever several rows share an entity and overlap would bias evaluation. | True | It prevents identity-specific information from crossing partitions. |
Chapter Summary
Feature and target preparation converts a cleaned dataset into a precise supervised learning representation. The feature matrix X, target vector y, metadata, and grouping keys must be separated according to their semantic roles. The target requires an event, horizon, reference time, and mature labeling rule. Predictors must be available at prediction time and free from target-derived information.
Correct feature typing depends on meaning rather than storage. Numerical values, counts, nominal and ordinal categories, Booleans, timestamps, and free text each require different validation and preprocessing. Conversions should report failures, preserve category meaning, and reject silent schema drift. Date features must use explicit snapshot times, while periodic variables may benefit from cyclical encoding.
Identifiers are normally excluded from X but retained for grouping, traceability, and delivery. When repeated records belong to the same entity, group-aware or time-aware splitting is necessary to produce a realistic estimate of generalization. A complete data dictionary makes these decisions inspectable, reproducible, and maintainable.
QUALITY CHECK | Ready for the next stage After Chapter 8, the project should contain validated X, y, grouping keys, metadata, semantic feature groups, and a preparation log. Chapter 9 can then split the dataset correctly without leaking entities, time, or preprocessing information. |
Preparation Checklist
- The observation unit and prediction snapshot are explicitly defined.
- The target event, horizon, labels, and follow-up maturity rules are documented.
- X and y have identical unique indices and row order.
- Identifiers and metadata are stored separately from model features.
- Post-outcome and target-derived variables are excluded.
- Every feature has a semantic type and valid-value definition.
- Numeric, categorical, Boolean, and datetime conversions produce audit reports.
- Date features use only information at or before the snapshot.
- Periodic features are encoded appropriately when needed.
- The split strategy reflects whether the system must generalize across rows, entities, sites, or future time.
- A data dictionary records role, meaning, conversion, preprocessing, and leakage assessment.
- The full preparation procedure can be rerun on new data without manual edits.