Chapter 2 — Understanding Supervised Learning
Chapter Overview
Supervised learning is the branch of machine learning in which a model learns from examples that contain both inputs and known outputs. Each example tells the learning algorithm what was observed and what result was associated with that observation. The algorithm uses these labeled examples to estimate a reusable relationship that can produce an output for a new, previously unseen input.
This chapter develops the mathematical and practical vocabulary needed for the rest of the course. It explains how data is represented by a feature matrix X and a target y, why the unknown relationship is written as y = f(X) + ε, and how classification differs from regression. It also examines binary, multiclass, and multilabel classification; simple, multiple, and nonlinear regression; and the many forms that inputs and targets may take.
The Python demonstrations focus on understanding the structure of supervised learning rather than optimizing model performance. Later chapters will cover data preparation, model selection, evaluation, cross-validation, and hyperparameter tuning in greater depth.
Learning Objectives
✓ Explain why supervised learning requires labeled examples.
✓ Describe the relationship between observations, input features, and target values.
✓ Interpret the expression y = f(X) + ε in practical terms.
✓ Distinguish the feature matrix X from a single feature vector x.
✓ Explain the roles of the unknown function f, the learning algorithm, and the fitted model.
✓ Describe how noise, measurement error, and omitted variables affect prediction.
✓ Differentiate binary, multiclass, and multilabel classification.
✓ Explain decision boundaries and class probabilities.
✓ Differentiate simple, multiple, and nonlinear regression.
✓ Select classification or regression according to the target, intended output, objective, and evaluation metric.
✓ Recognize numerical, categorical, binary, ordinal, temporal, text, image, and signal features.
✓ Inspect a dataset and identify its samples, features, target, task type, and potential evaluation metrics.
Chapter Roadmap
Section | Main question | Practical emphasis |
|---|---|---|
| 2.1 Principle of supervised learning | How does a model learn a mapping from labeled examples? | Separate X and y; simulate signal and noise. |
| 2.2 Classification | How does a model predict one or more categories? | Binary, multiclass, and multilabel examples. |
| 2.3 Regression | How does a model estimate a continuous numerical value? | Simple, multiple, and nonlinear regression. |
| 2.4 Classification versus regression | How should the task be identified? | Decision framework and metric selection. |
| 2.5 Inputs and targets | What forms can model inputs and outputs take? | Inspect mixed feature types and targets. |
| Practical activity | Can the complete supervised-learning structure be identified in a dataset? | Dataset audit, task formulation, and metric proposal. |
2.1 Principle of Supervised Learning
2.1.1 Learning from Labeled Examples
A supervised dataset contains examples for which the desired output is already known. The word supervised refers to the presence of this guidance. During training, the algorithm observes an input and the corresponding target. It adjusts the model so that the model output becomes closer to the known target. This process is repeated across many examples.
A labeled example is commonly written as the pair (xᵢ, yᵢ). The vector xᵢ contains the input information for observation i, and yᵢ contains the output that the model should learn to predict. A dataset containing n labeled observations can be represented as:
D = {(x₁, y₁), (x₂, y₂), …, (xₙ, yₙ)} Each pair contains the features of one observation and its known target. |
The labels may come from different sources. A physician may assign a diagnosis, a customer may cancel a subscription, a sensor may record the final quality of a manufactured component, or a historical transaction may later be identified as fraudulent. The reliability of these labels directly affects what the model can learn.
CAUTION Labels are evidence, not absolute truth A label may contain mistakes, ambiguity, subjective judgment, delayed information, or inconsistent definitions. Supervised learning assumes that the target is useful enough to guide learning, but it does not guarantee that every recorded label is correct. |
Training and prediction are different phases
During training, the target is visible to the algorithm because it is needed to calculate error and update the model. During prediction, only the input features are available. The target is precisely what the trained model must estimate. Any feature that reveals the target only because it was recorded after the event creates target leakage and makes evaluation misleading.
Phase | Information available | Purpose |
|---|---|---|
| Training | Input features X and known target y | Estimate model parameters and reduce prediction error. |
| Validation | Inputs and targets not used to fit the current model | Compare alternatives and tune choices. |
| Prediction | New input features only | Produce a category, probability, or numerical estimate. |
| Final evaluation | Held-out inputs and their targets | Estimate performance on unseen observations. |
2.1.2 Relationship Between Inputs and Outputs
Supervised learning assumes that the inputs contain information related to the output. The relationship may be simple, complex, approximately linear, highly nonlinear, stable, or dependent on context. The model does not need every input variable to be individually predictive; combinations and interactions among features may carry useful information.
For example, a house price may depend on surface area, location, number of rooms, age of the building, condition, and market period. None of these features determines the price perfectly. Together, however, they may support a useful estimate. In a classification problem, email words, sender reputation, link patterns, and message structure may jointly indicate whether an email is spam.
CONCEPT NOTE Correlation is not causation A supervised model learns predictive associations. A feature can improve prediction without causing the target. Causal conclusions require additional assumptions, experimental design, or causal inference methods. |
2.1.3 General Mathematical Representation
A general supervised-learning relationship is written as:
y = f(X) + ε The observed target equals a systematic component plus noise or unexplained variation. |
At the level of one observation, the same idea is written as yᵢ = f(xᵢ) + εᵢ. The function f represents the underlying relationship between the features and the target. The term ε represents effects that are not captured by the available inputs or by the model. These effects may include random variation, unobserved variables, measurement errors, and labeling mistakes.
The equation should be interpreted conceptually rather than as a promise that every supervised task follows a simple additive numerical form. For classification, f may produce class scores or probabilities, and the final label may be obtained through a decision rule. The notation is nevertheless useful because it separates the predictable structure from the unpredictable or unmodeled component.
2.1.4 Meaning of the Feature Matrix X
The feature matrix X organizes the inputs used by the learning algorithm. If the dataset contains n observations and p features, X has n rows and p columns. Each row corresponds to one observation, and each column corresponds to one measured or derived characteristic.
X ∈ ℝⁿˣᵖ n observations × p input features, after any required numerical representation. |
Observation | surface_m² | bedrooms | age_years | distance_center_km |
|---|---|---|---|---|
| House 1 | 65 | 2 | 18 | 3.4 |
| House 2 | 82 | 3 | 9 | 5.1 |
| House 3 | 105 | 3 | 12 | 8.0 |
| House 4 | 120 | 4 | 4 | 6.2 |
Before preprocessing, a real dataset may contain categorical strings, dates, text, images, or signals rather than only real numbers. The notation X ∈ ℝⁿˣᵖ usually describes the final numerical representation presented to the estimator. The raw data may require encoding, extraction, scaling, or transformation first.
KEY DEFINITION X and x do not mean the same thing Uppercase X usually denotes the complete feature matrix. Lowercase xᵢ denotes the feature vector for one observation. Confusing these levels can make code, equations, and model outputs difficult to interpret. |
2.1.5 Meaning of the Target y
The target y contains the outcome to be predicted. In a regression problem, y is usually a numerical vector such as house prices, energy consumption values, or waiting times. In a classification problem, y contains class labels such as spam/not spam, disease categories, or equipment states.
Task | Example target y | Interpretation |
|---|---|---|
| Binary classification | [0, 1, 0, 0, 1] | Two mutually exclusive classes. |
| Multiclass classification | [2, 0, 1, 2, 1] | One class selected from three or more possibilities. |
| Multilabel classification | [[1,0,1], [0,1,0], …] | Several labels may be active for one observation. |
| Regression | [72.0, 94.0, 128.0, 156.0] | Continuous or approximately continuous numerical outcomes. |
The target should be defined before model training begins. Changing the target definition changes the scientific or operational problem. Predicting whether a student passes, predicting the final grade, and predicting the probability of passing are related but distinct tasks with different outputs and evaluation criteria.
2.1.6 Role of the Unknown Function f
The true function f is unknown. If it were already known exactly, machine learning would not be necessary. The objective of training is to use the labeled examples to construct an approximation, often written as f̂ or gθ, where θ represents model parameters learned from data.
f̂ = arg min₍g∈G₎ (1/n) Σ L(yᵢ, g(xᵢ)) Training selects a model from a candidate family G by minimizing an average loss. |
The candidate family G may consist of linear models, decision trees, nearest-neighbor rules, support vector machines, neural networks, or another class of estimators. The learning algorithm searches this family and determines parameter values. The fitted model is the specific function obtained after training.
Concept | Meaning |
|---|---|
| Model family | The set of functions that the algorithm is allowed to consider. |
| Learning algorithm | The procedure that searches for useful model parameters. |
| Model parameters | Values learned from the training data, such as coefficients or tree splits. |
| Hyperparameters | Configuration choices set before or around training, such as tree depth. |
| Fitted model | The particular learned mapping used to generate predictions. |
2.1.7 Role of Noise and Measurement Error
The term ε represents everything that makes the observed target differ from the systematic relationship captured by the inputs. Some of this variation can potentially be reduced by collecting better features or improving labels. Other variation may be inherently unpredictable at the required time horizon.
Source of variation | Example | Possible response |
|---|---|---|
| Measurement error | A sensor records temperature with calibration error. | Calibrate sensors, filter noise, or use repeated measurements. |
| Label error | A transaction is incorrectly marked as legitimate. | Audit labels, use expert review, or model label uncertainty. |
| Omitted variables | House condition is absent from a price dataset. | Collect relevant features when feasible. |
| Random behavior | A customer leaves because of an unpredictable personal event. | Accept irreducible uncertainty and report probabilities. |
| Changing environment | Demand patterns change after a policy or market shift. | Use temporal validation and monitor drift. |
| Annotation disagreement | Experts interpret a medical image differently. | Use consensus labels or preserve uncertainty information. |
No supervised model can recover information that is absent from both the features and the training labels. A model can sometimes fit noise in the training data, but this produces overfitting rather than genuine predictive knowledge. The goal is to capture stable structure that generalizes.
Python Example 2.1 — Separating Features and Target
Python Example 2.1 — Create the feature matrix X and target y import pandas as pd
# Each row is a labeled observation. houses = pd.DataFrame({ "surface_m2": [65, 82, 105, 120, 145, 90], "bedrooms": [2, 3, 3, 4, 4, 3], "age_years": [18, 9, 12, 4, 7, 15], "distance_center_km": [3.4, 5.1, 8.0, 6.2, 10.5, 4.8], "sale_price_k": [72, 94, 128, 156, 181, 110], })
feature_names = [ "surface_m2", "bedrooms", "age_years", "distance_center_km" ] X = houses[feature_names] y = houses["sale_price_k"]
print("X shape:", X.shape) print("y shape:", y.shape) print(X.head()) print(y.head()) |
Expected structure X shape: (6, 4) y shape: (6,) Each row of X and the value at the same index in y form one labeled example. |
Python Example 2.2 — Signal, Function, and Noise
Python Example 2.2 — Approximate a hidden relationship from noisy labels import numpy as np from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(42) X = rng.uniform(0, 10, size=(80, 1))
# Unknown process used here only to create synthetic teaching data. true_signal = 4.0 + 2.5 * X[:, 0] noise = rng.normal(loc=0.0, scale=2.2, size=80) y = true_signal + noise
model = LinearRegression() model.fit(X, y)
print("Estimated intercept:", round(model.intercept_, 3)) print("Estimated slope:", round(model.coef_[0], 3)) print("Prediction for x=6:", round(model.predict([[6]])[0], 3)) |
The algorithm never receives the variables true_signal or noise. It receives only X and y. The fitted coefficients approximate the systematic relationship even though individual target values contain random variation.
2.2 Classification
2.2.1 Definition of Classification
Classification is a supervised-learning task in which the target represents a category, state, type, or membership. The trained model receives the features of a new observation and produces a class label, class score, or probability for each class.
The categories must be defined before training. A classifier does not discover unlabeled groups in the way a clustering algorithm does. It learns how previously labeled examples relate to the predefined classes.
Application | Inputs | Target classes |
|---|---|---|
| Email filtering | Words, links, sender metadata, message structure | Spam / not spam |
| Medical screening | Symptoms, measurements, test results | Condition present / absent |
| Quality inspection | Image or sensor measurements | Acceptable / defective / rework |
| Document routing | Text content and metadata | Finance / legal / technical / human resources |
| Activity recognition | Accelerometer and gyroscope signals | Walking / sitting / running / cycling |
2.2.2 Binary Classification
Binary classification has two mutually exclusive outcomes. One class is often called the positive class and the other the negative class. Positive does not necessarily mean desirable; it simply identifies the event or condition of primary interest.
• Fraud versus legitimate transaction.
• Customer churn versus customer retention.
• Machine failure within 24 hours versus no failure.
• Disease detected versus disease not detected.
• Loan default versus repayment.
A binary classifier often estimates a probability P(y = 1 | x). A decision threshold converts this probability into a class. With a threshold of 0.5, probabilities at or above 0.5 may be assigned to class 1. Operational requirements can justify a lower or higher threshold.
ŷ = 1 if P(y = 1 | x) ≥ τ; otherwise ŷ = 0 τ is a decision threshold chosen according to errors, costs, and capacity. |
GOOD PRACTICE The positive class must be explicit Metrics such as precision and recall depend on which class is treated as positive. A technical report should state the positive event and explain why it matters. |
2.2.3 Multiclass Classification
Multiclass classification predicts exactly one class from three or more alternatives. The classes are normally mutually exclusive for the current task. For example, an image of a handwritten digit may belong to one of ten classes from 0 to 9, and a machine may be assigned to one operating state at a time.
A multiclass model may directly estimate probabilities for all classes or may combine several binary decisions. Common strategies include one-versus-rest and one-versus-one. Many scikit-learn estimators manage these strategies internally.
Question | Binary classification | Multiclass classification |
|---|---|---|
| Number of classes | Exactly two | Three or more |
| Output per observation | One of two labels | One label among K possibilities |
| Typical probability output | P(class 1) | P(class 1), …, P(class K) |
| Example | Spam / not spam | Species A / B / C |
| Common summary metric | F1, ROC AUC, precision-recall AUC | Macro F1, weighted F1, multiclass log loss |
2.2.4 Multilabel Classification
Multilabel classification allows more than one label to be assigned to the same observation. The labels are not mutually exclusive. A news article can be tagged as technology, business, and cybersecurity at the same time. A medical image may contain several findings, and a photograph may include multiple objects.
yᵢ = [yᵢ₁, yᵢ₂, …, yᵢK], where yᵢk ∈ {0,1} One binary indicator is used for each possible label. |
Multilabel classification should not be confused with multiclass classification. In multiclass classification, the model chooses one class. In multilabel classification, it independently or jointly decides which labels are applicable. Evaluation may consider performance per label, micro or macro averages, subset accuracy, or ranking quality.
2.2.5 Class Labels
Class labels may be represented as words, integers, booleans, or encoded categories. The numerical representation does not automatically imply numerical order. If the labels {0, 1, 2} mean red, green, and blue, class 2 is not greater than class 1 in a meaningful quantitative sense.
• Define every class in operational terms.
• Check whether classes are mutually exclusive.
• Identify ambiguous or overlapping cases.
• Measure class frequencies and rare classes.
• Verify that the same labeling rule was used across the dataset.
• Preserve the mapping between encoded values and human-readable labels.
2.2.6 Decision Boundaries
A decision boundary separates regions of the feature space that lead to different class predictions. In a problem with two numerical features, the boundary can be visualized as a line or curve. In higher dimensions, it is a surface or more complex partition that cannot be displayed directly.
Linear classifiers produce linear decision boundaries in the transformed feature space. Decision trees create axis-aligned regions. Kernel methods, ensembles, and neural networks can produce highly nonlinear boundaries. A more flexible boundary may fit complex patterns, but excessive flexibility can also fit noise.
Model family | Typical boundary | Practical implication |
|---|---|---|
| Logistic regression | Linear in the encoded feature space | Interpretable and effective when classes are approximately linearly separable. |
| Decision tree | Rectangular partitions | Captures thresholds and interactions without scaling. |
| K-nearest neighbors | Locally irregular regions | Sensitive to feature scaling and local sample density. |
| RBF support vector machine | Smooth nonlinear surface | Can model complex separation but requires careful tuning. |
| Ensemble trees | Combination of many partitions | Often strong for structured tabular data. |
2.2.7 Class Probabilities
Many classifiers return probabilities or probability-like scores. For a multiclass problem with K classes, the model may output a vector whose elements sum to one. The predicted class is commonly the class with the highest probability.
ŷ = arg maxₖ P(y = k | x) Choose the class with the largest estimated conditional probability. |
Probability quality matters when predictions support prioritization, triage, ranking, or risk-sensitive decisions. A model may classify many observations correctly while producing overconfident probabilities. Calibration measures whether events assigned a probability near 0.7 occur approximately 70% of the time in comparable cases.
CAUTION A probability is not a certainty Estimated probabilities depend on training data, model assumptions, preprocessing, and operating conditions. They should be interpreted as model-based uncertainty under those conditions, not as guaranteed frequencies for every individual case. |
Python Example 2.3 — Binary Classification
Python Example 2.3 — Estimate a binary class and its probability import pandas as pd from sklearn.linear_model import LogisticRegression
customers = pd.DataFrame({ "months_active": [2, 5, 8, 10, 18, 24, 30, 36, 4, 14], "support_calls": [6, 4, 3, 5, 1, 0, 1, 0, 7, 2], "usage_hours": [3, 8, 12, 9, 28, 34, 31, 40, 4, 22], "churned": [1, 1, 0, 1, 0, 0, 0, 0, 1, 0], })
X = customers[["months_active", "support_calls", "usage_hours"]] y = customers["churned"]
model = LogisticRegression(max_iter=1000) model.fit(X, y)
new_customer = pd.DataFrame({ "months_active": [6], "support_calls": [5], "usage_hours": [7], })
probability = model.predict_proba(new_customer)[0, 1] predicted_class = model.predict(new_customer)[0] print("Churn probability:", round(probability, 3)) print("Predicted class:", predicted_class) |
Python Example 2.4 — Multiclass Classification
Python Example 2.4 — Predict one class from three alternatives from sklearn.datasets import load_iris from sklearn.linear_model import LogisticRegression
iris = load_iris(as_frame=True) X = iris.data y = iris.target
model = LogisticRegression(max_iter=1000) model.fit(X, y)
sample = X.iloc[[0]] probabilities = model.predict_proba(sample)[0] predicted_id = model.predict(sample)[0] predicted_name = iris.target_names[predicted_id]
print("Class probabilities:", probabilities.round(3)) print("Predicted species:", predicted_name) |
Python Example 2.5 — Multilabel Targets
Python Example 2.5 — Represent several simultaneous labels import numpy as np from sklearn.multioutput import MultiOutputClassifier from sklearn.linear_model import LogisticRegression
# Features could describe short documents. X = np.array([ [8, 1, 0], [2, 7, 1], [6, 5, 1], [1, 2, 0], [9, 3, 1], [3, 8, 0], ])
# Columns: technology, business, cybersecurity. y = np.array([ [1, 0, 0], [0, 1, 0], [1, 1, 1], [0, 0, 0], [1, 0, 1], [0, 1, 0], ])
model = MultiOutputClassifier(LogisticRegression(max_iter=1000)) model.fit(X, y)
new_document = np.array([[7, 4, 1]]) print("Predicted label vector:", model.predict(new_document)[0]) |
2.3 Regression
2.3.1 Definition of Regression
Regression is a supervised-learning task in which the target is numerical and the model estimates a quantity. The output is usually continuous or treated as continuous over a useful range. Examples include price, temperature, energy consumption, duration, concentration, demand, and remaining useful life.
A regression model may provide a single point estimate, a prediction interval, a distribution, or several quantiles. The simplest introductory models produce one numerical prediction for each observation.
ŷ ∈ ℝ A regression prediction is a numerical value rather than a category label. |
2.3.2 Continuous Target Variables
A continuous target can theoretically take any value in an interval, although measurement precision may produce discrete recorded values. For example, temperature may be stored to the nearest tenth of a degree, and prices may be recorded to the nearest currency unit. These targets remain regression targets because the numerical distance between values is meaningful.
The choice between classification and regression is not determined by whether the target is stored as an integer. A count such as the number of daily support requests is discrete but numerical, and regression may still be appropriate. Conversely, class identifiers stored as integers remain categorical.
2.3.3 Simple Regression
Simple regression uses one input feature to predict a numerical target. It is useful for introducing relationships, slopes, residuals, and model assumptions. A simple linear regression estimates an intercept β₀ and slope β₁:
ŷ = β₀ + β₁x β₁ represents the expected change in the prediction for a one-unit increase in x. |
Simple regression is rarely sufficient for a complex real-world system, but it can provide an interpretable baseline and reveal whether one variable has a useful predictive relationship with the target.
2.3.4 Multiple Regression
Multiple regression uses several features. A multiple linear model adds one coefficient for each feature, while more flexible regression algorithms can learn thresholds, interactions, and nonlinear patterns.
ŷ = β₀ + β₁x₁ + β₂x₂ + … + βₚxₚ The prediction combines information from p input features. |
When features are correlated, individual coefficient interpretation becomes more difficult. The predictive model may still be useful, but conclusions about the isolated effect of a feature should be made carefully. Scaling and regularization are also important for several linear estimators and will be studied later.
2.3.5 Nonlinear Regression
A nonlinear relationship cannot be represented adequately by a straight line in the original feature space. Examples include diminishing returns, saturation, thresholds, periodic effects, and interactions. Nonlinear regression can be implemented through polynomial features, splines, decision trees, ensembles, kernels, or neural networks.
The phrase nonlinear regression can refer either to a nonlinear relationship between inputs and target or to a model whose parameters enter nonlinearly. In applied machine learning, it is often used broadly for regression models capable of learning curved or complex relationships.
CAUTION Nonlinear does not automatically mean better A flexible model can reduce training error while generalizing poorly. Model complexity must be evaluated on validation data, not selected from training performance alone. |
2.3.6 Typical Regression Applications
Application | Possible features | Target |
|---|---|---|
| House price estimation | Area, location, rooms, age, condition | Sale price |
| Energy forecasting | Weather, time, occupancy, historical load | Energy consumption |
| Demand estimation | Price, season, promotions, past sales | Units demanded |
| Predictive maintenance | Vibration, temperature, cycles, alarms | Remaining useful life |
| Travel-time prediction | Route, traffic, weather, departure time | Minutes to destination |
| Student performance | Attendance, assessments, study activity | Final grade |
| Medical measurement | Clinical variables and imaging features | Continuous risk score or measurement |
Regression-specific considerations
• The scale and units of the target affect how errors should be interpreted.
• Large errors may be more costly than small errors, motivating squared-error metrics.
• Outliers can strongly influence some models and evaluation measures.
• Predictions outside physically meaningful ranges may require constraints or transformations.
• Tree-based models generally do not extrapolate reliably beyond the target range observed during training.
• A single average prediction may hide important uncertainty or subgroup differences.
Python Example 2.6 — Simple Linear Regression
Python Example 2.6 — Learn a numerical relationship from one feature import numpy as np from sklearn.linear_model import LinearRegression
# Advertising spend in thousands and weekly sales in units. X = np.array([[1], [2], [3], [4], [5], [6]]) y = np.array([14, 18, 23, 27, 31, 36])
model = LinearRegression() model.fit(X, y)
print("Intercept:", round(model.intercept_, 2)) print("Slope:", round(model.coef_[0], 2)) print("Predicted sales for spend=4.5:", round(model.predict([[4.5]])[0], 2)) |
Python Example 2.7 — Multiple Regression
Python Example 2.7 — Combine several input features import pandas as pd from sklearn.linear_model import LinearRegression
houses = pd.DataFrame({ "surface_m2": [65, 82, 105, 120, 145, 90, 110, 75], "bedrooms": [2, 3, 3, 4, 4, 3, 4, 2], "age_years": [18, 9, 12, 4, 7, 15, 6, 22], "distance_km": [3.4, 5.1, 8.0, 6.2, 10.5, 4.8, 7.0, 2.5], "price_k": [72, 94, 128, 156, 181, 110, 149, 79], })
X = houses.drop(columns="price_k") y = houses["price_k"]
model = LinearRegression() model.fit(X, y)
new_house = pd.DataFrame({ "surface_m2": [100], "bedrooms": [3], "age_years": [8], "distance_km": [5.5], }) print("Estimated price:", round(model.predict(new_house)[0], 2)) |
Python Example 2.8 — A Nonlinear Relationship
Python Example 2.8 — Use polynomial features to represent curvature import numpy as np from sklearn.pipeline import make_pipeline from sklearn.preprocessing import PolynomialFeatures from sklearn.linear_model import LinearRegression
X = np.array([[0], [1], [2], [3], [4], [5], [6]]) y = np.array([2.0, 3.1, 5.8, 10.9, 17.7, 26.3, 36.8])
model = make_pipeline( PolynomialFeatures(degree=2, include_bias=False), LinearRegression(), ) model.fit(X, y)
for value in [2.5, 4.5]: prediction = model.predict([[value]])[0] print(f"x={value}: prediction={prediction:.2f}") |
2.4 Classification versus Regression
Classification and regression are both supervised-learning paradigms. The difference is determined primarily by the meaning of the target and the output required by the application. The storage type of the target column is only a clue; the scientific or operational semantics are decisive.
2.4.1 Nature of the Target
A categorical target defines membership in a finite set of classes and normally leads to classification. A numerical target for which differences and magnitudes are meaningful normally leads to regression. Some problems can be formulated in either way, but the choice changes the information available in the output.
Target example | Meaning | Likely task |
|---|---|---|
| 0 = no default, 1 = default | Two categories | Binary classification |
| A, B, C, D | Four mutually exclusive quality grades | Multiclass classification |
| [technology, security] | Several simultaneous tags | Multilabel classification |
| 245,000 currency units | Numerical price | Regression |
| 37.4 °C | Measured temperature | Regression |
| 1, 2, 3 as city identifiers | Encoded categories, not quantities | Classification |
| Number of incidents next month | Discrete numerical count | Usually regression or count modeling |
2.4.2 Expected Output
The desired output should match the action that follows the prediction. A category supports routing or discrete decisions. A probability supports ranking, risk-based thresholds, and capacity planning. A numerical estimate supports budgeting, scheduling, forecasting, and continuous control.
• Use classification when the system must assign a predefined category.
• Use regression when the system must estimate a quantity with meaningful numerical distance.
• Preserve probability outputs when the application needs ranking or threshold adjustment.
• Avoid converting a continuous target into categories unless the categories have a genuine operational meaning.
• Avoid predicting an exact number when only a stable category is required and numerical precision would be misleading.
2.4.3 Business or Scientific Objective
The same underlying phenomenon can support different supervised-learning formulations. Consider student success. Predicting whether a student will pass is binary classification. Predicting the final grade is regression. Predicting one of several intervention levels is multiclass classification. The correct formulation depends on the decision to be made and the information needed at prediction time.
Objective | Target formulation | Task |
|---|---|---|
| Identify students needing immediate support | At-risk / not at-risk | Binary classification |
| Estimate expected final performance | Final numerical grade | Regression |
| Assign an intervention intensity | Low / medium / high support | Multiclass classification |
| Attach several support needs | Academic, financial, attendance labels | Multilabel classification |
2.4.4 Evaluation Criteria
The evaluation metric should reflect the target and the consequences of errors. Classification metrics compare predicted categories or probabilities with true class labels. Regression metrics measure numerical differences between predictions and targets.
Task | Common metrics | What they emphasize |
|---|---|---|
| Binary classification | Accuracy, precision, recall, F1, ROC AUC, PR AUC, log loss | Correct classes, positive-event detection, ranking, or probability quality. |
| Multiclass classification | Accuracy, macro F1, weighted F1, log loss | Overall correctness and class-specific balance. |
| Multilabel classification | Micro/macro F1, Hamming loss, subset accuracy | Label-level correctness or exact label sets. |
| Regression | MAE, MSE, RMSE, R², MAPE | Absolute errors, large errors, explained variation, or relative error. |
GOOD PRACTICE Do not choose the task from the metric alone First define the target and decision objective. Then select a metric that represents useful performance. A convenient metric cannot repair an incorrectly formulated problem. |
A practical decision framework
1. State the prediction target in one sentence without mentioning an algorithm.
2. Ask whether the target represents a category, several labels, or a numerical quantity.
3. Describe the output required by the end user: class, probability, ranking, or numerical estimate.
4. Identify the action or scientific conclusion supported by the output.
5. List the costs of different types and sizes of error.
6. Choose a primary metric and at least one diagnostic metric.
7. Verify that the target is available and reliable for historical training examples.
Ambiguous formulations
Some targets sit near the boundary between classification and regression. An ordinal outcome such as mild, moderate, and severe has ordered categories. It may be treated as multiclass classification, ordinal classification, or a carefully justified numerical score. A count target may be handled with general regression or a specialized count model. Time-to-event outcomes may require survival analysis rather than ordinary regression.
CONCEPT NOTE Another analytical approach may be more appropriate Not every labeled numerical column should be handled with standard regression. Ranking, survival analysis, forecasting, anomaly detection, structured prediction, and causal inference address objectives that may require different assumptions and evaluation methods. |
Python Example 2.9 — Inspect the Target Before Choosing a Task
Python Example 2.9 — Use data inspection as evidence, not as an automatic rule import pandas as pd
records = pd.DataFrame({ "age": [19, 22, 21, 24, 20, 23], "attendance_rate": [0.92, 0.71, 0.88, 0.64, 0.95, 0.79], "program": ["CS", "EE", "CS", "ME", "EE", "CS"], "passed": [1, 0, 1, 0, 1, 1], "final_grade": [15.2, 8.7, 13.8, 7.9, 16.1, 12.4], })
for target_name in ["passed", "final_grade"]: target = records[target_name] print(" Target:", target_name) print("dtype:", target.dtype) print("unique values:", target.nunique()) print("values:", sorted(target.unique()))
# Interpretation requires semantics: # passed -> category -> classification # final_grade -> numerical magnitude -> regression |
Classification and regression comparison
Dimension | Classification | Regression |
|---|---|---|
| Target meaning | Category, class, state, or label | Numerical quantity |
| Typical model output | Label, score, or class probability | Point estimate, interval, or distribution |
| Core question | Which class applies? | How much or how many? |
| Example | Will the customer churn? | How much will the customer spend? |
| Common loss | Log loss, hinge loss, classification error | Squared error, absolute error, Huber loss |
| Typical diagnostic | Confusion matrix and threshold curves | Residual analysis and error distribution |
| Decision issue | Trade-off among class-specific errors | Trade-off among error magnitude and direction |
2.5 Inputs and Targets
Supervised models can learn from many forms of information. The raw feature type determines the preprocessing and model representation that may be required. The target type determines the task, loss function, metrics, and output interpretation. A clear data dictionary is therefore an essential modeling artifact.
2.5.1 Numerical Features
Numerical features represent measurable quantities or counts. Continuous variables may take values across an interval, while discrete numerical variables take separated values such as counts. Numerical features may have different units, ranges, and distributions.
• Continuous examples: temperature, voltage, height, pressure, income, distance.
• Discrete examples: number of purchases, support calls, detected objects, machine cycles.
• Potential operations: imputation, scaling, clipping, transformation, binning, interaction construction.
• Important checks: valid range, units, missing values, outliers, skewness, and measurement resolution.
2.5.2 Categorical Features
Categorical features describe membership in a set of values such as city, product type, department, or device model. Nominal categories have no intrinsic order. Most machine learning estimators require categories to be encoded numerically, for example with one-hot encoding.
A large number of unique categories can create high-dimensional representations and rare-category problems. Identifiers should not automatically be used as categorical features because they may encourage memorization or leakage without providing generalizable information.
2.5.3 Binary Features
A binary feature has two states, commonly represented as 0/1, false/true, absent/present, or no/yes. Binary features are categorical in meaning but already have a compact numerical representation. The coding direction should be documented, especially when coefficients or feature effects will be interpreted.
Feature | Possible coding | Interpretation concern |
|---|---|---|
| scholarship | 0 = no, 1 = yes | State which category is represented by 1. |
| machine_alarm | False / True | Verify whether missing values are distinct from false. |
| has_previous_default | N / Y | Encode consistently across training and prediction data. |
| sensor_available | 0 / 1 | Availability may itself contain operational information. |
2.5.4 Ordinal Features
Ordinal features contain ordered categories, but the distance between adjacent categories is not necessarily equal. Examples include low/medium/high, satisfaction levels, educational stages, and severity grades. Ordinal encoding can preserve order, but the chosen numerical gaps introduce assumptions.
One-hot encoding ignores order but avoids assuming equal spacing. Ordinal encoding is compact and can be appropriate when the ordering is meaningful. The choice should consider the estimator, domain meaning, and validation results.
2.5.5 Date and Time Features
Dates and timestamps are rarely used directly as raw strings. Useful derived features may include year, month, day of week, hour, elapsed duration, recency, season, holiday indicator, or time since a previous event. Temporal ordering is essential when splitting data because information from the future must not influence predictions for the past.
• Calendar components: year, quarter, month, weekday, hour.
• Durations: account age, time since maintenance, delivery delay.
• Cyclical representation: encode hour or month using sine and cosine when continuity across the cycle matters.
• Temporal context: rolling averages and historical counts must use only information available before prediction time.
2.5.6 Text Features
Text is unstructured data that must be converted into numerical representations. Classical approaches include counts, n-grams, and TF-IDF vectors. Modern approaches include dense embeddings learned from neural language models. The representation should be fitted using training data only when it learns vocabulary or statistical weights.
Text can be the primary input, as in sentiment analysis, or an auxiliary feature, as in customer-support classification. Privacy, language variation, spelling, document length, and domain-specific terminology all affect model design.
2.5.7 Image and Signal Features
Images may be represented by pixel values, handcrafted descriptors, or features learned by convolutional neural networks and vision transformers. Signals may be represented by raw samples, time-domain statistics, frequency-domain features, spectrograms, or learned representations.
Data type | Raw representation | Possible derived representation |
|---|---|---|
| Image | Height × width × channels | Edges, texture descriptors, deep embeddings |
| Audio signal | Amplitude samples over time | Spectrogram, MFCCs, temporal embeddings |
| Vibration signal | Acceleration samples | RMS, kurtosis, frequency peaks, wavelet features |
| ECG signal | Electrical potential over time | Intervals, morphology, rhythm features, learned representation |
| Multisensor stream | Synchronized channels | Window statistics, cross-channel correlations, sequence embeddings |
CAUTION Keep the unit of observation explicit For images and signals, one row may represent an entire file, one time window, one event, or one subject. The split strategy must prevent multiple highly related samples from the same entity leaking across training and test sets. |
2.5.8 Continuous and Discrete Targets
Targets can also be continuous or discrete. Continuous numerical targets typically lead to regression. Discrete class labels lead to classification. Discrete counts may still lead to regression when their numerical magnitude matters. Ordered classes require special attention because they contain both categorical and ordinal information.
Target | Data form | Likely formulation |
|---|---|---|
| Sale price | Continuous numerical | Regression |
| Number of failures next week | Discrete count | Regression or count model |
| Failure type code | Discrete category | Multiclass classification |
| Severity: low/medium/high | Ordered category | Ordinal or multiclass classification |
| Set of detected objects | Binary vector of labels | Multilabel classification |
| Time until event | Nonnegative duration with censoring possible | Survival analysis may be preferable |
Data dictionaries and prediction-time availability
A data dictionary documents each variable, its meaning, type, unit, allowed values, missing-value semantics, source, and availability time. Prediction-time availability is especially important. A feature may be highly predictive in historical data but unusable if it becomes known only after the target event.
Field | Example entry |
|---|---|
| Variable name | attendance_rate |
| Description | Percentage of scheduled sessions attended before the prediction date |
| Role | Input feature |
| Raw type | Floating-point number |
| Unit/range | 0.0 to 1.0 |
| Missing meaning | Attendance data not yet synchronized |
| Availability | Available at weekly prediction time |
| Quality checks | Range, duplicate sessions, late updates |
Python Example 2.10 — Inspect Mixed Input Types
Python Example 2.10 — Examine a dataset before selecting preprocessing import pandas as pd
students = pd.DataFrame({ "student_id": [101, 102, 103, 104], "age": [19, 22, 20, 21], "program": ["CS", "EE", "ME", "CS"], "scholarship": [1, 0, 1, 0], "engagement_level": ["high", "medium", "low", "high"], "enrollment_date": pd.to_datetime([ "2025-09-10", "2025-09-12", "2025-09-10", "2025-09-15" ]), "advisor_note": [ "strong start", "needs attendance support", "limited activity", "consistent progress" ], "final_grade": [15.2, 10.4, 8.1, 14.0], })
print(students.dtypes) print(" Unique values per column:") print(students.nunique())
X = students.drop(columns="final_grade") y = students["final_grade"] print(" Samples:", len(X)) print("Features:", list(X.columns)) print("Target:", y.name) |
Python Example 2.11 — Create Simple Date Features
Python Example 2.11 — Convert a timestamp into interpretable features import pandas as pd
prediction_date = pd.Timestamp("2026-01-15")
data = pd.DataFrame({ "enrollment_date": pd.to_datetime([ "2025-09-10", "2025-10-01", "2025-11-12" ]) })
data["enrollment_month"] = data["enrollment_date"].dt.month data["enrollment_weekday"] = data["enrollment_date"].dt.dayofweek data["days_enrolled"] = ( prediction_date - data["enrollment_date"] ).dt.days
print(data) |
GOOD PRACTICE The prediction date belongs in the feature definition A duration such as days_enrolled must be calculated relative to the actual prediction time. Using the end of the dataset for every observation would incorporate future information and create leakage. |
Practical Activity — Identify the Structure of a Supervised-Learning Dataset
In this activity, students inspect a small student-success dataset and formulate two different supervised-learning tasks. The first task predicts whether a student will graduate on time. The second task predicts the final numerical score. The same observations and many of the same inputs can support both tasks, but the targets, outputs, and metrics differ.
Dataset description
Column | Meaning | Type | Availability before prediction? |
|---|---|---|---|
| student_id | Administrative identifier | Identifier | Yes, but normally excluded as a feature |
| age | Age at enrollment | Numerical | Yes |
| program | Academic program | Categorical | Yes |
| entry_score | Admission score | Numerical | Yes |
| attendance_rate | Attendance before the prediction date | Numerical | Yes |
| weekly_study_hours | Estimated weekly study time | Numerical | Yes |
| scholarship | Scholarship indicator | Binary | Yes |
| engagement_level | Low, medium, or high engagement | Ordinal | Yes |
| enrollment_date | Initial enrollment timestamp | Date/time | Yes |
| final_score | Final numerical result | Continuous target candidate | No; known after the course |
| graduated_on_time | Whether completion occurred on schedule | Binary target candidate | No; known later |
Activity instructions
1. State the unit of observation represented by one row.
2. List the candidate input features that are available at prediction time.
3. Identify fields that should not be used as model inputs and explain why.
4. For the graduated_on_time target, identify the supervised-learning task and the expected model output.
5. For the final_score target, identify the supervised-learning task and the expected model output.
6. Propose one primary and two secondary evaluation metrics for each task.
7. Identify at least three possible data-quality or leakage risks.
8. Explain how the target definition would change if the objective were to predict low, medium, or high performance.
9. Describe one feature that could be derived from enrollment_date without using future information.
10. Prepare a short model card paragraph stating the intended use and one limitation.
Python activity starter
Practical Activity — Start with a structural dataset inspection import pandas as pd
students = pd.DataFrame({ "student_id": [201, 202, 203, 204, 205, 206, 207, 208], "age": [19, 22, 20, 21, 24, 19, 23, 20], "program": ["CS", "EE", "CS", "ME", "EE", "CS", "ME", "EE"], "entry_score": [14.8, 12.0, 13.5, 11.2, 15.0, 16.1, 10.8, 13.0], "attendance_rate": [0.94, 0.72, 0.88, 0.61, 0.91, 0.97, 0.58, 0.81], "weekly_study_hours": [12, 6, 10, 4, 9, 14, 3, 8], "scholarship": [1, 0, 1, 0, 1, 1, 0, 0], "engagement_level": [ "high", "medium", "high", "low", "high", "high", "low", "medium" ], "enrollment_date": pd.to_datetime([ "2025-09-10", "2025-09-12", "2025-09-10", "2025-09-15", "2025-09-11", "2025-09-09", "2025-09-18", "2025-09-13" ]), "final_score": [15.6, 10.2, 14.1, 7.8, 15.0, 16.8, 6.9, 12.4], "graduated_on_time": [1, 1, 1, 0, 1, 1, 0, 1], })
print(students.info()) print(students.head()) print("Missing values: ", students.isna().sum()) print("Unique values: ", students.nunique()) |
Student deliverable
• A one-page dataset audit identifying samples, features, and target candidates.
• A classification formulation for graduated_on_time.
• A regression formulation for final_score.
• A table of proposed evaluation metrics with justification.
• A list of data-quality, fairness, and leakage risks.
• A short Python notebook that separates X and y for both tasks.
Suggested solution and discussion
Question | Suggested answer |
|---|---|
| Samples | Each row represents one student enrollment or student-course outcome, depending on the dataset definition. |
| Inputs | Age, program, entry score, attendance rate, study hours, scholarship, engagement, and derived date features. |
| Excluded fields | student_id is an identifier; final_score and graduated_on_time cannot both be inputs when either is the target because they are post-outcome information. |
| Classification target | graduated_on_time; output may be class 0/1 and probability of on-time graduation. |
| Regression target | final_score; output is a numerical estimate in the grading scale. |
| Classification metrics | Recall or F1 for the at-risk class, plus precision and PR AUC when class imbalance matters. |
| Regression metrics | MAE as an interpretable primary metric, with RMSE and R² as diagnostics. |
| Risks | Small sample size, inconsistent engagement labels, post-outcome leakage, missing attendance, program imbalance, and changing academic rules. |
| Three performance levels | Multiclass or ordinal classification after defining meaningful thresholds. |
| Date feature | Days since enrollment at the prediction date, enrollment month, or cohort indicator. |
Extension scenarios
Scenario | Suggested approach | Reasoning |
|---|---|---|
| Predict the exact waiting time for a hospital service | Regression | The output is a numerical duration. |
| Assign each support ticket to one department | Multiclass classification | Exactly one predefined category is required. |
| Attach several topics to an article | Multilabel classification | Several tags may apply simultaneously. |
| Group customers when no segment labels exist | Clustering | The goal is to discover structure without labeled targets. |
| Estimate whether a loan defaults within one year | Binary classification | The target is a yes/no event; probability may support risk ranking. |
| Calculate the average grade by program | Descriptive statistics | No predictive model is required. |
| Predict time until equipment failure with incomplete follow-up | Survival analysis | Censoring makes ordinary regression potentially inappropriate. |
| Select the shortest path in a known road graph | Optimization | The solution follows a specified objective and constraints, not labeled prediction. |
Expected Outcome
After completing this chapter and activity, students should understand how supervised learning uses labeled examples to estimate a relationship between inputs and outputs. They should be able to identify the feature matrix X, the target y, the model approximation of the unknown function f, and the role of noise or unexplained variation ε.
A successful student should be able to inspect a dataset and determine whether the target requires binary, multiclass, or multilabel classification, regression, or another analytical approach. The student should also be able to recognize important feature types, state what output the model should produce, and propose evaluation metrics consistent with the objective.
• Identify the observation or unit being analyzed.
• Distinguish raw inputs, engineered features, identifiers, and target variables.
• Explain whether labels are categorical, multilabel, ordinal, continuous, or discrete numerical values.
• Describe what information is available during training and at prediction time.
• Recognize potential noise, label error, and leakage.
• Choose a task formulation based on semantics and intended use rather than data type alone.
• Connect task type to appropriate model outputs and evaluation criteria.
Chapter Summary
• Supervised learning estimates a mapping from inputs to known outputs using labeled examples.
• A labeled observation is represented by a feature vector xᵢ and a target yᵢ.
• The complete input dataset is organized as the feature matrix X, with observations in rows and features in columns.
• The expression y = f(X) + ε separates systematic predictive structure from noise and unmodeled variation.
• The learning algorithm approximates the unknown function f with a fitted model selected from a model family.
• Classification predicts categories; binary, multiclass, and multilabel problems have different target structures.
• Decision boundaries determine class regions, while class probabilities support ranking and threshold-based decisions.
• Regression predicts numerical quantities and may use simple, multiple, or nonlinear relationships.
• The choice between classification and regression depends on target meaning, expected output, objective, and evaluation criteria.
• Inputs may be numerical, categorical, binary, ordinal, temporal, textual, visual, or signal-based.
• Identifiers and post-outcome variables require careful review because they can cause memorization or leakage.
• A well-defined supervised-learning task states the unit of observation, features available at prediction time, target, output, metric, and limitations.
Key Terminology
Term | Meaning |
|---|---|
| Labeled example | An observation containing both input features and a known target. |
| Feature matrix X | The table or numerical array of input variables for all observations. |
| Feature vector xᵢ | The input values describing one observation. |
| Target y | The known outcome that the model is trained to predict. |
| Unknown function f | The underlying relationship between inputs and target. |
| Noise ε | Unexplained, random, omitted, or measured variation affecting the observed target. |
| Learning algorithm | The procedure that estimates model parameters from data. |
| Fitted model | The learned mapping used to generate predictions. |
| Classification | Prediction of categorical outcomes. |
| Binary classification | Classification with exactly two classes. |
| Multiclass classification | Selection of exactly one class from three or more alternatives. |
| Multilabel classification | Assignment of several applicable labels to one observation. |
| Decision boundary | The separation between regions assigned to different classes. |
| Class probability | An estimated probability that an observation belongs to a class. |
| Regression | Prediction of a numerical quantity. |
| Simple regression | Regression using one input feature. |
| Multiple regression | Regression using several input features. |
| Nonlinear regression | Regression capable of representing curved or complex relationships. |
| Ordinal feature | An ordered categorical variable whose category distances may not be equal. |
| Target leakage | Use of information that would not be available at the real prediction time. |
Knowledge Check
1. What makes a dataset supervised rather than unsupervised?
2. Explain the difference between X and xᵢ.
3. Interpret each term in y = f(X) + ε.
4. Why can a model never completely remove irreducible noise?
5. What is the difference between a learning algorithm and a fitted model?
6. Give one binary, one multiclass, and one multilabel classification example.
7. What is a decision boundary?
8. Why might a class probability be more useful than a hard class label?
9. How does simple regression differ from multiple regression?
10. Give an example of a nonlinear input-target relationship.
11. Why does an integer-valued target not automatically imply classification?
12. How can the same phenomenon be formulated as either classification or regression?
13. Name two metrics suitable for classification and two for regression.
14. Explain why student_id should usually not be used as a predictive feature.
15. Give one example of target leakage in a student-performance dataset.
16. How should a timestamp be transformed into useful prediction-time features?
17. What is the difference between a categorical feature and an ordinal feature?
18. Why can several rows from the same patient create leakage across a random split?
19. When might survival analysis be more appropriate than regression?
20. Write a complete one-sentence formulation of a supervised-learning problem.
Short Practical Assignment
Select a dataset from your studies, professional activity, or a public educational source. Prepare a two-page supervised-learning formulation without training an advanced model. The objective is to demonstrate correct problem structure.
• State the application objective and intended user.
• Define one row or sample precisely.
• List the candidate input features and their types.
• Define the target and explain when it becomes known.
• Classify the task as binary, multiclass, multilabel, regression, or another approach.
• Describe the expected model output.
• Select one primary metric and justify it.
• Identify two noise sources and two leakage risks.
• Separate X and y in a short Python notebook.
• State one limitation that would prevent responsible deployment.
Instructor Notes and Suggested Timing
Session | Content | Suggested duration | Teaching method |
|---|---|---|---|
| 1 | Labeled examples, X, y, and prediction phases | 60–75 min | Lecture, board examples, dataset walkthrough |
| 2 | Mathematical representation and sources of noise | 60 min | Concept discussion and Python simulation |
| 3 | Binary, multiclass, and multilabel classification | 75–90 min | Comparative lecture and code demonstration |
| 4 | Simple, multiple, and nonlinear regression | 75–90 min | Worked examples and notebook exercise |
| 5 | Classification versus regression and metric selection | 60–75 min | Scenario classification in groups |
| 6 | Input and target types | 75 min | Data-dictionary workshop |
| 7 | Practical activity and correction | 90–120 min | Individual work, peer review, instructor correction |
CONCEPT NOTE NEXT STEP Transition to Chapter 3 The next chapter will transform these concepts into a complete supervised-learning workflow: problem definition, data collection, data preparation, splitting, baseline construction, model training, evaluation, improvement, interpretation, and delivery. |