Chapter 13 — Baseline Models
| Chapter purpose This chapter explains how simple baseline models establish the minimum level of performance that a supervised learning system should exceed. Students learn classification and regression baselines, rule-based references, scikit-learn dummy estimators, and how to decide whether an improvement is statistically, practically, and economically meaningful. |
Learning objectives
- Explain why every supervised machine learning experiment should begin with a baseline.
- Construct majority-class, prior-distribution, random, and rule-based classification baselines.
- Use scikit-learn DummyClassifier safely and interpret its strategies.
- Construct mean, median, constant, and domain-rule regression baselines.
- Use DummyRegressor and compare candidate models against reference performance.
- Distinguish statistical improvement from practical and business value.
- Design a minimum performance threshold that a trained model must exceed.
| Key idea A baseline is not a competitor that must be sophisticated. It is a reference point. If a complex model cannot reliably outperform a simple reference, the added complexity is difficult to justify. |
13.1 Why establish a baseline?
A model score has little meaning in isolation. An accuracy of 90% may appear excellent, but if 92% of the observations belong to one class, a classifier that always predicts the majority class performs better. Similarly, a regression RMSE of 8.0 is difficult to interpret until it is compared with a simple prediction such as the training-set mean.
Determining whether machine learning adds value
The first purpose of a baseline is to test whether learned relationships in the features provide useful predictive information. A candidate model should improve on a reference that requires little or no learning. If it does not, possible explanations include weak features, noisy labels, leakage-free evaluation revealing limited signal, or an unsuitable model family.
Question | Baseline role |
|---|---|
| Is there predictive signal in X? | A learned model should outperform a target-only baseline. |
| Does complexity help? | Compare a simple baseline with progressively stronger models. |
| Is the evaluation plausible? | Suspiciously large gains may reveal leakage or an incorrect split. |
| Is deployment worthwhile? | Improvement must justify operational and maintenance cost. |
Providing a minimum performance threshold
The baseline defines a performance floor. Before training advanced models, the team can state a minimum acceptable requirement such as “macro F1 must exceed 0.55” or “MAE must be at least 15% lower than the median baseline.” This creates an explicit decision criterion rather than choosing a model simply because it is the best among the tested candidates.
Python 13.1 — Establishing a classification performance floor
| from sklearn.dummy import DummyClassifier from sklearn.metrics import balanced_accuracy_score baseline = DummyClassifier(strategy='most_frequent') baseline.fit(X_train, y_train) y_base = baseline.predict(X_valid) score = balanced_accuracy_score(y_valid, y_base) print(f'Baseline balanced accuracy: {score:.3f}') |
Detecting modeling or evaluation errors
Baselines are powerful diagnostic tools. If a complicated pipeline performs worse than a trivial baseline, inspect preprocessing, target definition, feature alignment, metric direction, and split strategy. Conversely, if a candidate model appears dramatically better than a reasonable baseline, confirm that target-derived variables, duplicated entities, or future information have not leaked into the feature set.
| Diagnostic warning A surprisingly strong score is not automatically good news. Compare it with a baseline, then inspect whether the gain is plausible for the domain and split strategy. Extremely large improvements can indicate leakage. |
Comparing complexity against improvement
Every increase in model complexity has a cost: more computation, more hyperparameters, harder debugging, more difficult explanations, and greater maintenance burden. A 0.2 percentage-point gain may not justify replacing a transparent logistic-regression model with a large ensemble, while a 10-point recall improvement in a safety-critical detector may be highly valuable.
Dimension | Simple baseline | Candidate model |
|---|---|---|
| Training cost | Very low | Potentially medium or high |
| Prediction cost | Very low | Depends on model family |
| Interpretability | Usually high | May be lower |
| Maintenance | Minimal | Requires monitoring and retraining |
| Expected performance | Reference floor | Must demonstrate useful improvement |
13.2 Classification baselines
Classification baselines predict class labels or probabilities using simple strategies that do not learn a rich relationship between the feature matrix X and target y. They are especially useful when class distributions are imbalanced.
Predicting the majority class
The majority-class baseline always predicts the most frequent training label. It answers a fundamental question: does the machine learning model do better than simply choosing the class that occurs most often?
Python 13.2 — Manual majority-class baseline
| import pandas as pd majority_class = y_train.value_counts().idxmax() y_pred_majority = pd.Series(majority_class, index=y_valid.index) print('Majority class:', majority_class) print('Predictions:', y_pred_majority.head().tolist()) |
| Imbalanced-data example If 95% of transactions are legitimate and 5% are fraudulent, predicting “legitimate” for every case achieves 95% accuracy but detects no fraud. The majority baseline therefore exposes why accuracy alone can be misleading. |
Predicting according to class distribution
A prior-distribution baseline uses the class frequencies observed in the training set. For probability-based evaluation, it can assign the same learned prior probabilities to every example. This can be more informative than a hard majority prediction when log loss or Brier score is used.
Python 13.3 — Class-prior baseline
| from sklearn.dummy import DummyClassifier prior_model = DummyClassifier(strategy='prior') prior_model.fit(X_train, y_train) prior_probs = prior_model.predict_proba(X_valid) print('Learned class prior:', prior_model.class_prior_) print('First probability vector:', prior_probs[0]) |
Random prediction
Random prediction provides a stochastic reference. In scikit-learn, the stratified strategy samples predictions according to the observed class distribution. Random baselines are mainly diagnostic and should use a fixed random_state for reproducibility.
Python 13.4 — Reproducible stratified-random baseline
| random_baseline = DummyClassifier( strategy='stratified', random_state=42 ) random_baseline.fit(X_train, y_train) y_random = random_baseline.predict(X_valid) |
Rule-based baseline
A domain rule can be a stronger and more realistic reference than a dummy model. For example, a churn system might flag customers with very low recent activity; a medical triage model might be compared with an established clinical threshold; an equipment-failure model might be compared with a sensor alarm already used operationally.
Python 13.5 — A simple domain-rule classification baseline
| # Example rule: predict churn when recent usage is very low rule_pred = (X_valid['monthly_usage'] < 20).astype(int) from sklearn.metrics import classification_report print(classification_report(y_valid, rule_pred, zero_division=0)) |
| Important A rule-based baseline must use only information that would truly be available at prediction time. A rule that accidentally uses post-outcome information is leakage, not a valid baseline. |
Using DummyClassifier
DummyClassifier provides several built-in strategies and follows the same fit/predict interface as other scikit-learn estimators. It can therefore be placed in cross-validation workflows and evaluated with exactly the same metrics as candidate classifiers.
Strategy | Behavior | Typical use |
|---|---|---|
| most_frequent | Always predicts the most common class | Hard-label performance floor |
| prior | Uses training class priors; hard prediction is the most frequent class | Probability-based reference |
| stratified | Samples according to the training class distribution | Randomized diagnostic reference |
| uniform | Samples each class uniformly | Pure random reference |
| constant | Always predicts a user-specified class | Specific operational reference |
Python 13.6 — Comparing classification baselines
| from sklearn.dummy import DummyClassifier from sklearn.metrics import accuracy_score, f1_score strategies = ['most_frequent', 'prior', 'stratified', 'uniform'] for strategy in strategies: model = DummyClassifier(strategy=strategy, random_state=42) model.fit(X_train, y_train) pred = model.predict(X_valid) print( strategy, 'accuracy=', round(accuracy_score(y_valid, pred), 3), 'macro_f1=', round(f1_score(y_valid, pred, average='macro'), 3) ) |
Choosing classification metrics for the baseline
The baseline should be measured with the same primary metric that will be used to compare candidate models. Accuracy is reasonable when classes are balanced and error costs are similar. For imbalance, balanced accuracy, macro F1, recall, precision, precision-recall AUC, or a cost-based metric may be more appropriate.
Situation | Useful primary metric | Why |
|---|---|---|
| Balanced classes, equal error cost | Accuracy | Overall correctness is meaningful. |
| Imbalanced classes | Balanced accuracy or macro F1 | Prevents majority class dominance. |
| False negatives costly | Recall / sensitivity | Measures missed-positive control. |
| False positives costly | Precision / specificity | Measures unnecessary-positive control. |
| Probability quality matters | Log loss or Brier score | Evaluates predictive probabilities. |
13.3 Regression baselines
Regression baselines predict a continuous target using simple constants or domain rules. They answer whether the feature-based regression model provides lower prediction error than a target-only reference.
Predicting the mean
The mean baseline predicts the average training target for every new observation. It is closely connected to squared-error loss: the arithmetic mean is the constant that minimizes the sum of squared errors on the training data.
Python 13.7 — Mean regression baseline
| from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_squared_error mean_baseline = DummyRegressor(strategy='mean') mean_baseline.fit(X_train, y_train) y_base = mean_baseline.predict(X_valid) rmse = mean_squared_error(y_valid, y_base) ** 0.5 print(f'Mean-baseline RMSE: {rmse:.3f}') |
Predicting the median
The median is more resistant to extreme target values than the mean and is a natural reference for absolute-error loss. When the target distribution is strongly skewed or contains legitimate extreme values, the median baseline can be more representative.
Python 13.8 — Median regression baseline
| from sklearn.metrics import mean_absolute_error median_baseline = DummyRegressor(strategy='median') median_baseline.fit(X_train, y_train) y_base = median_baseline.predict(X_valid) mae = mean_absolute_error(y_valid, y_base) print(f'Median-baseline MAE: {mae:.3f}') |
Predicting a constant
Sometimes a fixed operational value is more meaningful than the sample mean or median. For example, a demand system may currently assume 100 units per day, or a maintenance process may assume a fixed lifetime. DummyRegressor(strategy="constant") can reproduce such a reference.
Python 13.9 — Fixed-value regression baseline
| constant_baseline = DummyRegressor( strategy='constant', constant=100.0 ) constant_baseline.fit(X_train, y_train) y_constant = constant_baseline.predict(X_valid) |
Simple domain-based rule
A domain rule may use a small number of transparent variables. For house prices, a simple reference could be “regional median price per square meter × property area.” For demand, it could be “same weekday last week.” Such baselines are often harder to beat than a global constant and therefore more meaningful operational references.
Python 13.10 — Domain-rule regression baseline
| # Example domain rule: price = area × regional reference price reference_price_per_m2 = { 'North': 1800, 'South': 1500, 'Central': 2200 } rule_pred = ( X_valid['area_m2'] * X_valid['region'].map(reference_price_per_m2) ) |
Using DummyRegressor
Strategy | Prediction | Useful when |
|---|---|---|
| mean | Training-target mean | RMSE/MSE is important and target is not extremely skewed |
| median | Training-target median | MAE is important or target has outliers |
| quantile | Selected training-target quantile | Asymmetric or quantile-oriented decisions |
| constant | Specified numeric value | Existing operational assumption is known |
Python 13.11 — Comparing regression baselines
| from sklearn.dummy import DummyRegressor from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score for strategy in ['mean', 'median']: model = DummyRegressor(strategy=strategy) model.fit(X_train, y_train) pred = model.predict(X_valid) mae = mean_absolute_error(y_valid, pred) rmse = mean_squared_error(y_valid, pred) ** 0.5 r2 = r2_score(y_valid, pred) print(strategy, 'MAE=', round(mae, 2), 'RMSE=', round(rmse, 2), 'R2=', round(r2, 3)) |
| Interpreting R² A constant mean predictor typically has R² close to 0 on data drawn from the same distribution. A negative R² for a candidate model means it performs worse, under squared error, than predicting the mean of the evaluation target. |
13.4 Evaluating baseline usefulness
After defining a baseline, the next question is not simply whether the candidate model has a better score. The gain must be stable, meaningful for the application, and worth the added complexity and cost.
Baseline versus candidate model
Use the same split, preprocessing discipline, metric definition, and evaluation sample for both models. When cross-validation is used, evaluate the baseline and candidate on the same folds so that score differences are directly comparable.
Python 13.12 — Fair paired cross-validation comparison
| from sklearn.model_selection import StratifiedKFold, cross_val_score from sklearn.dummy import DummyClassifier from sklearn.linear_model import LogisticRegression cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) baseline = DummyClassifier(strategy='most_frequent') candidate = LogisticRegression(max_iter=1000) base_scores = cross_val_score(baseline, X, y, cv=cv, scoring='f1_macro') cand_scores = cross_val_score(candidate, X, y, cv=cv, scoring='f1_macro') print('Baseline:', base_scores.mean(), '+/-', base_scores.std()) print('Candidate:', cand_scores.mean(), '+/-', cand_scores.std()) print('Mean improvement:', (cand_scores - base_scores).mean()) |
Statistical and practical improvement
A numerical improvement can be statistically convincing yet too small to matter operationally. Conversely, a modest change in an aggregate metric can be highly valuable if it substantially reduces costly errors in an important subgroup. Report both the average score and the size of the improvement in terms that decision makers can interpret.
Type of improvement | Question to ask | Example |
|---|---|---|
| Statistical | Is the gain stable across resamples or folds? | F1 improves in all 5 folds. |
| Practical | Is the magnitude large enough to matter? | MAE falls from 12.0 to 11.9 may be negligible. |
| Operational | Does the model fit latency/capacity constraints? | Recall rises, but inference is too slow. |
| Economic | Does the benefit exceed implementation cost? | Fewer missed fraud cases save more than deployment costs. |
Business relevance
Technical metrics are proxies for real outcomes. Translate model improvements into domain quantities whenever possible: additional fraud detected, fewer unnecessary inspections, reduced forecast error in units, fewer customer contacts, saved analyst hours, or lower financial loss.
Python 13.13 — Translating classification errors into cost
| # Convert confusion-matrix counts into an illustrative business cost cost_false_positive = 5 cost_false_negative = 200 baseline_cost = baseline_fp * cost_false_positive + baseline_fn * cost_false_negative model_cost = model_fp * cost_false_positive + model_fn * cost_false_negative savings = baseline_cost - model_cost print(f'Estimated savings versus baseline: {savings:,.0f}') |
Cost-sensitive comparison
False positives and false negatives rarely have equal consequences. A model that improves accuracy may still be worse for the organization if it increases the more expensive type of error. Define the cost model before inspecting test results whenever possible.
Application | False positive cost | False negative cost |
|---|---|---|
| Fraud detection | Legitimate payment may be blocked | Fraudulent payment may be accepted |
| Medical screening | Unnecessary follow-up test | Missed condition |
| Predictive maintenance | Unnecessary inspection | Unexpected failure or downtime |
| Churn intervention | Unnecessary retention offer | Customer leaves without intervention |
Python 13.14 — A reusable cost-sensitive metric
| from sklearn.metrics import confusion_matrix # Example: lower total cost is better def total_error_cost(y_true, y_pred, fp_cost=10, fn_cost=100): tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() return fp * fp_cost + fn * fn_cost base_cost = total_error_cost(y_valid, baseline_pred) model_cost = total_error_cost(y_valid, model_pred) print('Baseline cost:', base_cost) print('Model cost:', model_cost) |
Defining the minimum threshold to beat
A useful threshold is explicit, measurable, and linked to the objective. It may be absolute, relative to the baseline, or cost based. It should be defined on validation data and then confirmed once on the untouched test set.
Threshold style | Example |
|---|---|
| Absolute | Macro F1 must be at least 0.65. |
| Relative | MAE must be at least 10% lower than the median baseline. |
| Cost based | Expected error cost must be at least 15% lower than the current rule. |
| Constraint based | Recall ≥ 0.90 while precision remains ≥ 0.50. |
| Multi-criteria | Candidate must improve F1 and remain below a 20 ms latency budget. |
Python 13.15 — Checking a minimum improvement requirement
| baseline_mae = 24.0 candidate_mae = 20.5 required_relative_improvement = 0.10 relative_improvement = (baseline_mae - candidate_mae) / baseline_mae passes = relative_improvement >= required_relative_improvement print(f'Improvement: {relative_improvement:.1%}') print('Meets minimum requirement:', passes) |
A recommended baseline workflow
- Define the prediction target, split strategy, and primary evaluation metric.
- Select at least one trivial target-only baseline.
- Add a domain or operational rule baseline when a meaningful existing rule exists.
- Fit baselines using training data only.
- Evaluate baselines on the validation set or with the chosen cross-validation scheme.
- Record mean scores, variability, cost, and important class-specific metrics.
- Define the minimum performance or value threshold the candidate model must exceed.
- Train candidate models under the same evaluation protocol.
- Prefer added complexity only when its improvement is stable and useful.
- Confirm the final selected system once on the untouched test set.
| Baseline checklist A trustworthy baseline is simple, leakage-free, evaluated on the same data and metric as the candidate model, reproducible, and relevant to the actual decision process. |
Practical activity — Build and beat a baseline
Students work with a binary customer-churn dataset containing numerical and categorical features. The goal is to build classification baselines, compare them with a trained candidate model, and define a minimum performance requirement that the candidate must exceed.
Activity objectives
- Identify the class distribution and explain why a baseline is needed.
- Train a majority-class DummyClassifier.
- Train a stratified-random DummyClassifier.
- Create a simple domain-rule baseline.
- Choose an appropriate primary metric.
- Train one candidate classifier under the same split.
- Compare scores and error costs.
- Write a minimum acceptable performance statement.
Step 1 — Create or load the exercise data
Python 13.16 — Lab dataset and stratified split
| from sklearn.datasets import make_classification from sklearn.model_selection import train_test_split import pandas as pd X_array, y_array = make_classification( n_samples=1500, n_features=8, n_informative=5, n_redundant=1, weights=[0.82, 0.18], class_sep=1.0, random_state=42 ) X = pd.DataFrame(X_array, columns=[f'x{i}' for i in range(1, 9)]) y = pd.Series(y_array, name='churn') X_train, X_valid, y_train, y_valid = train_test_split( X, y, test_size=0.25, stratify=y, random_state=42 ) print(y_train.value_counts(normalize=True)) |
Step 2 — Train the dummy baselines
Python 13.17 — Lab dummy baselines
| from sklearn.dummy import DummyClassifier from sklearn.metrics import balanced_accuracy_score, f1_score baselines = { 'majority': DummyClassifier(strategy='most_frequent'), 'stratified': DummyClassifier(strategy='stratified', random_state=42) } results = [] for name, model in baselines.items(): model.fit(X_train, y_train) pred = model.predict(X_valid) results.append({ 'model': name, 'balanced_accuracy': balanced_accuracy_score(y_valid, pred), 'macro_f1': f1_score(y_valid, pred, average='macro') }) print(pd.DataFrame(results)) |
Step 3 — Train a candidate model
Python 13.18 — Lab candidate classifier
| from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression candidate = Pipeline([ ('scaler', StandardScaler()), ('model', LogisticRegression(max_iter=1000, class_weight='balanced')) ]) candidate.fit(X_train, y_train) candidate_pred = candidate.predict(X_valid) print('Candidate balanced accuracy:', balanced_accuracy_score(y_valid, candidate_pred)) print('Candidate macro F1:', f1_score(y_valid, candidate_pred, average='macro')) |
Step 4 — Define the requirement
Students should write a statement before selecting the final model. Example: “The candidate is considered useful only if its validation macro F1 is at least 0.10 higher than the best dummy baseline and minority-class recall is at least 0.70.” The exact threshold should be justified by the problem context.
Lab questions
- What percentage of the training data belongs to the majority class?
- Why can raw accuracy be misleading for this dataset?
- Which dummy strategy is the strongest under macro F1?
- Does the candidate classifier exceed the baseline by a meaningful margin?
- Which error type would be more expensive in a churn-retention scenario?
- What minimum threshold would you recommend before test-set evaluation?
Expected lab deliverable
- A short table containing each baseline and candidate score.
- A brief justification of the chosen primary metric.
- A confusion matrix or class-specific error summary.
- A minimum acceptable performance statement.
- A short conclusion: proceed, revise the features/model, or stop the ML approach.
Chapter summary
- A baseline gives model performance a meaningful reference point.
- Majority, prior, random, and constant strategies are useful simple references.
- Domain-rule baselines can be more meaningful than dummy estimators when existing operational logic exists.
- DummyClassifier and DummyRegressor provide reproducible scikit-learn baselines.
- Baseline and candidate models must be evaluated with the same data, split strategy, and metric.
- A model improvement should be assessed statistically, practically, operationally, and economically.
- Complexity is justified only when the gain over the baseline is stable and useful.
Knowledge check
1. Why is a 95% accurate classifier potentially useless on a dataset with 95% negative examples?
2. What is the difference between DummyClassifier(strategy="most_frequent") and strategy="prior"?
3. When is the median a more useful regression baseline than the mean?
4. Why should a domain-rule baseline obey the same prediction-time information constraints as the model?
5. What does a negative R² indicate relative to a mean predictor?
6. Why is a 1% metric improvement not automatically worth deploying?
7. How can false-positive and false-negative costs change model selection?
| Takeaway Never ask only “How good is my model?” Ask “How much better is it than a simple, honest, relevant baseline—and is that improvement worth the complexity?” |