Class weighting  •  Resampling •  Cost-sensitive thresholds  • Minority-class evaluation

Building classifiers that detect rare but important events without being misled by headline accuracy

Chapter map

Table 31.1. Chapter roadmap

Chapter overview

Many classification problems are not naturally balanced. Fraudulent transactions, serious failures, attacks, rare diseases, and customer churn may represent only a small share of observations. The rare class can nevertheless carry the largest operational cost. A classifier that optimizes ordinary accuracy may therefore learn a solution that looks strong numerically while failing at the task that matters.

Handling imbalance is not one technique. It is a coordinated evaluation and modeling workflow: define the positive class, select metrics aligned with the decision, protect the validation and test sets, consider class weights or resampling, and choose a threshold that reflects the consequences of false positives and false negatives.

Learning objectives

• Recognize when class imbalance is a meaningful modeling problem rather than merely an unusual class ratio.
• Explain why accuracy can be misleading when one class dominates the dataset.
• Use precision, recall, F1, balanced accuracy, ROC AUC, precision-recall evaluation, and Matthews correlation coefficient appropriately.
• Configure automatic or custom class weights and explain how they change the fitting objective.
• Compare random undersampling, random oversampling, and synthetic minority oversampling.
• Keep all resampling operations inside training data or cross-validation folds to prevent leakage.
• Select a decision threshold from validation data using costs, recall/precision targets, or operational capacity.
• Compare untreated, weighted, resampled, and threshold-adjusted classifiers on the same protected test set.

31.1 Understanding imbalance

A classification dataset is imbalanced when the classes occur at substantially different frequencies. The numerical ratio alone does not determine whether intervention is necessary. The important question is whether the minority class is difficult to detect and whether errors involving that class have different consequences.

Table 31.2. Typical imbalanced-classification applications

Imbalance ratio

A simple diagnostic is the class prevalence. If 4% of observations are positive, a 96:4 class split exists. This is descriptive, not a decision rule. Some 60:40 problems still require cost-sensitive treatment, while some 95:5 problems may be easy to separate without special sampling.

31.2 Why accuracy can be misleading

Accuracy counts all correct predictions equally. When the majority class dominates, a classifier can obtain a high accuracy simply by predicting the majority label for almost every observation. This can hide complete failure on the minority class.

Majority-class baseline example

Suppose 1,000 transactions contain 980 legitimate cases and 20 fraud cases. A classifier that predicts every transaction as legitimate obtains 98% accuracy, yet it detects none of the fraud cases: recall for fraud is 0%.

Table 31.3. Majority-only classifier

31.3 Suitable metrics

No single metric is universally best. The correct evaluation depends on whether false positives, false negatives, ranking quality, class balance, or overall correlation between predictions and labels matters most.

Table 31.4. Metrics for imbalanced classification

Balanced accuracy

Balanced accuracy gives each class equal influence by averaging class-wise recall. In binary classification it is the average of sensitivity and specificity.

Precision-recall evaluation

When the positive class is rare, the precision-recall curve often reveals operational behavior more clearly than accuracy. Average precision (AP) is a commonly used summary of the precision-recall curve. It is not identical to trapezoidal PR AUC, so reports should state exactly which summary is used.

Matthews correlation coefficient

MCC uses true positives, true negatives, false positives, and false negatives in one coefficient. Values approach +1 for excellent agreement, 0 for little predictive association, and -1 for systematically opposite predictions.

31.4 Class weighting

Class weighting changes the learning objective without duplicating or removing observations. Errors on selected classes receive larger weights, so the estimator has a stronger incentive to fit those observations correctly.

Balanced class weights

Many scikit-learn estimators accept class_weight="balanced". The weight is inversely related to class frequency, giving rarer classes larger influence. This is a useful baseline because it requires no synthetic data and preserves the original sample.

Custom class costs

Automatic balancing treats inverse frequency as the weighting rule. Real systems may have a different cost ratio. For example, if a false negative is considered five times more serious than a false positive, a custom class weight can be tested. The chosen weights should be validated against the real metric or cost objective.

Table 31.5. Class weighting: advantages and cautions

31.5 Resampling

Resampling modifies the class composition of the training data. Undersampling removes majority observations; oversampling duplicates or generates minority observations. These methods can improve minority detection, but they must be placed correctly in the workflow.

Random undersampling

• Removes a random subset of majority-class observations.
• Reduces training time when the majority class is very large.
• Can discard useful majority patterns and increase variance.

Random oversampling

• Duplicates minority observations until a desired class ratio is reached.
• Preserves majority observations.
• May encourage overfitting because identical minority examples are repeated.

Synthetic minority oversampling

SMOTE creates synthetic minority samples by interpolating between nearby minority observations. It can make the minority region easier to learn, but the generated points are not new real-world measurements. Synthetic observations can be implausible when features are noisy, mixed-type, highly constrained, or poorly scaled.

Table 31.6. Resampling methods

The leakage rule

Resampling must never be performed before the train/test split or before cross-validation folds are created. Otherwise synthetic or duplicated information derived from validation observations can enter the training data. The correct solution is to put the sampler inside an imbalanced-learn Pipeline so it is fitted only on each training fold.

31.6 Threshold optimization

A probabilistic classifier produces scores or probabilities before labels are assigned. The default threshold is often 0.5, but 0.5 has no universal operational meaning. Moving the threshold changes the balance between false positives and false negatives without retraining the model.

Recall-first systems

Lowering the threshold usually produces more positive predictions. Recall tends to increase because fewer positives are missed, but precision may fall because more negative cases are flagged.

Precision-first systems

Raising the threshold usually makes the positive decision more selective. Precision can increase, but recall may decrease because some true positives no longer cross the threshold.

Cost-sensitive threshold selection

Capacity-constrained decisions

Some systems can investigate only a fixed number of alerts per day. In that case the threshold may be selected to keep the predicted-positive volume within operational capacity. A model with strong ranking quality can be valuable even when only the highest-risk cases are acted upon.

Practical lab — Comparing four imbalance strategies

Goal: train one imbalanced classification problem and compare an untreated baseline, a class-weighted model, a SMOTE-resampled model, and a threshold-adjusted model. All final comparisons use the same untouched test set.

Table 31.7. Lab experiment design

Step 1 — Create an imbalanced dataset

Step 2 — Define an evaluation function

Step 3 — Train the untreated baseline

Step 4 — Train the class-weighted model

Step 5 — Train the resampled model

Step 6 — Select a cost-sensitive threshold

Use the baseline model, but choose its threshold from validation data. In this exercise a false negative costs eight units and a false positive costs one unit.

Step 7 — Compare all four strategies

Step 8 — Inspect confusion matrices

Step 9 — Write the comparison report

Table 31.8. Questions students must answer

Optional extension — Cross-validated comparison

For a more stable estimate, compare the untreated, class-weighted, and SMOTE models using the same StratifiedKFold object. Threshold optimization requires its own validation logic and should not be tuned directly on the outer test fold.

Common mistakes checklist

Table 31.9. Imbalanced-classification pitfalls

Chapter summary

• Class imbalance matters when class frequencies and error consequences make ordinary accuracy insufficient.
• A majority-only classifier can achieve high accuracy while having zero minority recall.
• Precision, recall, F1, balanced accuracy, ROC AUC, precision-recall evaluation, and MCC answer different evaluation questions.
• Class weighting changes the loss contribution of classes without changing the sample itself.
• Undersampling, oversampling, and SMOTE alter only the training distribution and must remain inside training folds.
• The final test set should preserve the natural class distribution and remain untouched until evaluation.
• Threshold optimization converts probabilities into operational decisions and should be driven by validation costs or constraints.
• The best imbalance strategy is the one that performs well under the chosen metric and real decision costs—not the one with the highest headline accuracy.