Chapter 37 — Fairness and Ethical Considerations
Chapter focus Predictive quality is only one part of responsible machine learning. This chapter shows how unfairness can enter a supervised-learning workflow, how to compare model behavior across groups, and how to decide whether a model should be deployed, constrained, reviewed by humans, or rejected for a high-impact use case. |
Learning objectives
- Identify historical, sampling, measurement, and label bias in supervised-learning data.
- Recognize direct sensitive attributes and indirect proxy variables.
- Compare performance, false-positive rates, false-negative rates, and calibration across groups.
- Explain why fairness metrics are context dependent and may conflict with one another.
- Account for small subgroup sample sizes and uncertainty.
- Design human oversight, appeal, documentation, and scope controls for high-impact systems.
- Produce a structured deployment-risk recommendation rather than relying on one fairness number.
Important boundary Fairness evaluation is an empirical risk-analysis tool, not a substitute for legal advice, organizational policy, or domain expertise. A model can satisfy one statistical fairness criterion and still be unsuitable, unlawful, unsafe, or harmful in its intended context. |
37.1 Potential sources of unfairness
Unfair outcomes can arise even when a model never receives an explicitly sensitive variable. The model learns patterns from the data it is given, so unfairness may already be present in who was observed, what was measured, how labels were assigned, or which variables act as proxies for social or organizational structures.
Historical bias
Historical bias occurs when past decisions or outcomes reflect unequal treatment or unequal opportunity. Training a model to reproduce those historical labels can reproduce the same patterns at scale. A highly accurate model may therefore be accurately learning an undesirable historical process.
Sampling bias
Sampling bias occurs when the training data do not represent the population in which the model will be used. Underrepresented groups may have less reliable predictions because the model has seen fewer relevant examples.
Measurement bias
Measurement bias appears when a feature measures different groups with different quality, frequency, or error. A sensor, survey, administrative record, or proxy measurement may be systematically noisier for one group than another.
Label bias
Labels are often treated as ground truth, but many labels are produced by people, policies, delayed outcomes, or previous systems. If the label-generation process is inconsistent across groups, the model inherits that inconsistency.
Proxy variables
A proxy variable is not itself a sensitive characteristic but is strongly associated with one. Geography, language, device type, purchasing history, job title, institution, or other contextual variables may become proxies depending on the application and population.
Unequal error rates
Two groups can have similar overall accuracy while experiencing very different types of error. In a screening system, one group may receive many more false negatives while another receives more false positives. Because the consequences of these errors differ, aggregate accuracy can hide important harms.
Source | How it enters the workflow | Typical warning sign | Possible response |
| Historical bias | Past labels or decisions encode unequal treatment | Strong model performance but problematic decision patterns persist | Revisit label definition; use policy/domain review; consider alternative targets |
| Sampling bias | Some populations are missing or rare | Large uncertainty or poor metrics for small groups | Collect representative data; use stratified monitoring; limit scope |
| Measurement bias | Feature quality differs across groups | Different missingness, noise, or measurement error rates | Audit measurement process; improve instrumentation; add quality indicators |
| Label bias | Ground truth is inconsistently assigned | Disagreement or error rates differ by group | Audit labeling process; use multiple reviewers or better outcomes |
| Proxy variables | Non-sensitive variables encode sensitive information | Unexpected group separation after removing direct attributes | Review proxies; test necessity; apply governance constraints |
| Unequal error rates | Same model behaves differently by group | FPR or FNR gaps despite similar accuracy | Report group metrics; investigate causes; change workflow or threshold only when justified |
Python example — inspect group representation and label rates before modeling
| import pandas as pd # Synthetic audit labels: A and B are placeholders, not real protected classes. audit = pd.DataFrame({ "group": ["A"] * 700 + ["B"] * 300, "target": [0] * 560 + [1] * 140 + [0] * 210 + [1] * 90 }) print(audit.groupby("group").size()) print(pd.crosstab(audit["group"], audit["target"], normalize="index")) | |
Interpretation Different label rates do not automatically prove unfairness. They are a signal that the analyst must understand the data-generating process, measurement quality, policy context, and the consequences of model errors before choosing an evaluation criterion. |
37.2 Sensitive and protected information
Sensitive and protected information requires careful governance. Which characteristics are legally protected, which uses are permitted, and what documentation is required depend on jurisdiction, sector, organization, and the specific decision being supported. Technical teams should therefore work with legal, compliance, ethics, security, privacy, and domain stakeholders rather than treating fairness as a purely modeling question.
Direct use of sensitive variables
Directly including a sensitive attribute as a predictive feature may be prohibited, restricted, or inappropriate in many contexts. However, the same attribute may sometimes be collected under controlled conditions for auditing fairness, monitoring outcomes, or satisfying a specific legal or organizational requirement. The purpose, access controls, retention policy, and permitted uses should be documented.
Indirect proxy variables
Removing a sensitive column does not guarantee that the model is independent of that information. Other variables may reconstruct it partially. Proxy risk is application-specific: a variable that is harmless in one context may be highly revealing in another.
Question | Why it matters |
| Is the variable needed to make the prediction? | Data minimization and necessity should be considered before adding sensitive information to a model. |
| Is the variable needed only for auditing? | Audit attributes can be separated from model inputs and handled under stricter access controls. |
| Could other variables act as proxies? | Removing the direct attribute may not remove the model pathway that causes group differences. |
| Is the attribute available reliably and consistently? | Poorly measured group labels can produce misleading fairness conclusions. |
| What rules govern collection and use? | Legal, contractual, privacy, and organizational rules differ across contexts. |
| Who can access group-level results? | Small groups can create privacy or re-identification risks and unstable estimates. |
Proxy screening is a diagnostic, not a verdict
Python example — screen categorical variables for association with an audit group
| from sklearn.metrics import mutual_info_score # Example only: group is retained for auditing and is NOT a model input. for column in ["region", "device", "channel"]: score = mutual_info_score(audit_frame[column], audit_frame["group"]) print(f"{column:10s} mutual information with audit group = {score:.3f}") |
A strong association can motivate review, but it does not by itself establish that the variable is an impermissible proxy. The analyst must ask whether the variable is causally relevant, operationally necessary, lawfully usable, measured consistently, and likely to create unacceptable effects in the intended workflow.
Do not infer protected attributes casually Inferring race, ethnicity, religion, disability, health status, or other protected/sensitive characteristics from names, images, addresses, behavior, or third-party data can create major privacy, ethical, and legal risks. If fairness auditing requires sensitive attributes, use an approved data-governance process rather than inventing or guessing them. |
37.3 Fairness evaluation
Fairness evaluation asks whether the model performs differently across meaningful groups and whether those differences are operationally or ethically important. There is no single universal fairness metric. The relevant metrics depend on the task, decision consequences, baseline rates, workflow, and policy requirements.
Performance across groups
At minimum, report the sample count and the same predictive metrics used for the overall model. For binary classification, group-level confusion matrices make false-positive and false-negative differences visible.
Metric | Question answered | High-impact interpretation |
| Recall / TPR | Among actual positives, how many are detected? | Low recall means more false negatives. |
| False-negative rate | Among actual positives, how many are missed? | Important when missed cases cause substantial harm. |
| Specificity / TNR | Among actual negatives, how many are correctly rejected? | Shows ability to avoid false alarms. |
| False-positive rate | Among actual negatives, how many are incorrectly flagged? | Important when a false flag creates cost, burden, or denial. |
| Precision | Among positive predictions, how many are correct? | Shows the reliability of a positive model action. |
| Calibration | When the model predicts p, does the outcome occur about p of the time? | Important when probabilities drive prioritization, pricing, or risk tiers. |
Compute group-level classification metrics
Python example — compare error rates across audit groups
| import numpy as np import pandas as pd from sklearn.metrics import confusion_matrix def group_metrics(y_true, y_pred, group): rows = [] for g in np.unique(group): mask = group == g tn, fp, fn, tp = confusion_matrix( y_true[mask], y_pred[mask], labels=[0, 1] ).ravel() rows.append({ "group": g, "n": int(mask.sum()), "precision": tp / (tp + fp) if tp + fp else np.nan, "recall": tp / (tp + fn) if tp + fn else np.nan, "fpr": fp / (fp + tn) if fp + tn else np.nan, "fnr": fn / (fn + tp) if fn + tp else np.nan, }) return pd.DataFrame(rows) report = group_metrics(y_test, y_pred, group_test) print(report.round(3)) |
Calibration across groups
A model can have similar ranking performance across groups but different probability quality. If a predicted probability of 0.70 means a 70% outcome rate in one group but only 45% in another, the same numeric score has different meaning across groups.
Python example — calibration curves by group
| import matplotlib.pyplot as plt from sklearn.calibration import calibration_curve for g in np.unique(group_test): mask = group_test == g frac_pos, mean_pred = calibration_curve( y_test[mask], y_prob[mask], n_bins=6, strategy="quantile" ) plt.plot(mean_pred, frac_pos, marker="o", label=f"Group {g}") plt.plot([0, 1], [0, 1], "--", label="Perfect calibration") plt.xlabel("Mean predicted probability") plt.ylabel("Observed positive rate") plt.legend() plt.show() |
Sample-size limitations
Group metrics can be unstable when a group is small or contains very few positive or negative examples. A difference of several percentage points may be mostly sampling noise. Always report group sample sizes, and consider uncertainty intervals or repeated evaluation when decisions are important.
Python example — estimate uncertainty instead of reporting a single subgroup number
| from sklearn.utils import resample # Simple bootstrap illustration for one group's recall. def bootstrap_recall(y_true, y_pred, n_boot=500, seed=42): rng = np.random.default_rng(seed) values = [] idx = np.arange(len(y_true)) for _ in range(n_boot): sample = rng.choice(idx, size=len(idx), replace=True) yt, yp = y_true[sample], y_pred[sample] positives = (yt == 1).sum() if positives: values.append(((yp == 1) & (yt == 1)).sum() / positives) return np.quantile(values, [0.025, 0.975]) | |
Fairness metrics can conflict When groups have different base rates, it may be mathematically impossible to equalize every error-rate and calibration criterion simultaneously. The goal is not to maximize a universal fairness score; it is to choose and justify criteria that match the decision context, harms, rights, and operating constraints. |
37.4 Responsible model use
Responsible use extends beyond metric evaluation. High-impact systems need governance around who can act on a prediction, how uncertain cases are handled, how affected people can challenge a decision, what the model is permitted to do, and how performance is monitored after deployment.
Human oversight
Human review is useful only when reviewers have enough information, authority, time, and training to disagree with the model. A nominal “human in the loop” is not a safeguard if people automatically accept model outputs or are punished for overriding them.
Appeal mechanisms
When model-supported decisions materially affect people, organizations should consider whether affected individuals can understand the basis of the decision, correct inaccurate data, request review, and appeal an adverse outcome. The appropriate mechanism depends on the application and governing requirements.
Documentation
- Intended use and explicitly prohibited uses.
- Training population, evaluation population, and known data gaps.
- Overall and group-level metrics with sample sizes.
- Chosen threshold and rationale.
- Known failure modes and uncertainty.
- Required human-review steps.
- Monitoring triggers, escalation process, and model owner.
- Conditions that require retraining, rollback, or retirement.
Appropriate scope
A model validated for one population, geography, product, device, or workflow should not automatically be reused elsewhere. Scope expansion changes the data distribution, applicable rules, and potential harms. New contexts require new validation and governance review.
When not to automate
Some decisions are too consequential, too poorly measured, too uncertain, or too context-dependent for autonomous model action. In such cases a model may be limited to prioritization, decision support, quality control, or research—or not deployed at all.
Risk question | Low-risk answer | Escalation signal |
| Can an error be reversed? | Yes, quickly and cheaply | Error may cause lasting financial, legal, health, safety, or opportunity harm |
| Is uncertainty visible? | Confidence and limitations are communicated | Users see a single authoritative label with no uncertainty |
| Can people appeal? | Clear review/correction path exists | No practical mechanism to challenge the result |
| Is human review meaningful? | Reviewer can investigate and override | Reviewer lacks time, evidence, or authority |
| Are group metrics stable? | Adequate samples and consistent monitoring | Small groups, wide uncertainty, or unexplained gaps |
| Is scope controlled? | Use is restricted to validated context | Model is reused for new purposes without revalidation |
Governance principle The appropriate question is not “Can the model make this decision?” but “Should the organization permit the model to influence this decision, under what controls, with what evidence, and with what recourse when it is wrong?” |
Practical discussion — Evaluate a high-impact deployment
Scenario: an organization is considering a supervised classification model that prioritizes applications for a limited human-review team. A positive prediction does not automatically approve or deny an application, but it determines which cases are reviewed first. Because delayed review can materially affect applicants, the system is still high impact.
Step 1 — Define the decision and the harm model
1. State exactly what the model output changes in the workflow.
2. Identify who may benefit and who may be harmed by a false positive or false negative.
3. Decide whether the model is advisory, prioritization-only, or able to trigger an automated action.
4. List which uses are explicitly prohibited.
Step 2 — Train a reproducible audit model
Python lab — create a reproducible synthetic classification problem
| import numpy as np from sklearn.datasets import make_classification X, y = make_classification( n_samples=4000, n_features=12, n_informative=7, n_redundant=2, weights=[0.72, 0.28], class_sep=1.0, random_state=42, ) |
Python lab — keep the audit group separate, fit the model, and create predictions
| from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler # Synthetic audit group, kept OUT of the model input. rng = np.random.default_rng(42) group = np.where(rng.random(len(y)) < 0.35, "B", "A") X_train, X_test, y_train, y_test, g_train, g_test = train_test_split( X, y, group, test_size=0.25, stratify=y, random_state=42 ) model = Pipeline([ ("scale", StandardScaler()), ("model", LogisticRegression(max_iter=2000)) ]) model.fit(X_train, y_train) y_prob = model.predict_proba(X_test)[:, 1] y_pred = (y_prob >= 0.50).astype(int) |
Step 3 — Build the fairness audit table
Python lab — compare performance, error rates, and probability quality
| from sklearn.metrics import ( accuracy_score, precision_score, recall_score, confusion_matrix, brier_score_loss ) rows = [] for g in np.unique(g_test): mask = g_test == g tn, fp, fn, tp = confusion_matrix( y_test[mask], y_pred[mask], labels=[0, 1] ).ravel() rows.append({ "group": g, "n": int(mask.sum()), "accuracy": accuracy_score(y_test[mask], y_pred[mask]), "precision": precision_score(y_test[mask], y_pred[mask], zero_division=0), "recall": recall_score(y_test[mask], y_pred[mask], zero_division=0), "fpr": fp / (fp + tn), "fnr": fn / (fn + tp), "brier": brier_score_loss(y_test[mask], y_prob[mask]), }) fairness_report = pd.DataFrame(rows) print(fairness_report.round(3)) |
Step 4 — Test whether the conclusion is stable
Students should not interpret a gap without checking sample sizes, confusion-matrix counts, uncertainty, and whether the gap persists across folds, time periods, or alternative seeds. A fairness finding that disappears under small perturbations may indicate insufficient evidence rather than a stable model property.
Python lab — repeat the group audit using out-of-fold predictions
| from sklearn.model_selection import StratifiedKFold, cross_val_predict cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) cv_prob = cross_val_predict( model, X, y, cv=cv, method="predict_proba" )[:, 1] cv_pred = (cv_prob >= 0.50).astype(int) stable_report = group_metrics(y, cv_pred, group) print(stable_report.round(3)) |
Step 5 — Produce a deployment-risk assessment
Area | Evidence students should provide | Possible mitigation |
| Data | Representation, missingness, label quality, measurement consistency | Collect data; improve labels; narrow scope; improve measurement |
| Performance | Overall and group metrics with uncertainty | Model revision; threshold/workflow change; more review capacity |
| Fairness | FPR/FNR/calibration differences and consequences | Investigate causes; redesign process; governance review |
| Human oversight | Who reviews, what evidence they see, override authority | Training; decision aids; mandatory review for uncertain cases |
| Appeal/recourse | How people correct errors or challenge outcomes | Review channel; correction process; audit trail |
| Monitoring | What drift, gap, or incident triggers action | Dashboards; alerts; rollback or retraining thresholds |
Step 6 — Make one of three recommendations
- Deploy: evidence supports the intended use and controls are adequate.
- Deploy with controls: use only within a restricted scope, with additional human review, monitoring, documentation, or appeal protections.
- Do not deploy: evidence is insufficient, the expected harm is unacceptable, the data are unsuitable, or the required safeguards cannot be implemented.
Required student deliverable Write a one- to two-page deployment memo containing: intended use, prohibited use, data limitations, overall metrics, group-level metrics, uncertainty, the most important fairness risk, the most important operational risk, proposed mitigation, human-oversight design, appeal mechanism, monitoring plan, and a final deploy / deploy-with-controls / do-not-deploy recommendation. |
Discussion questions
1. Which error is more harmful in this workflow: a false positive or a false negative? Why?
2. Which group-level metrics are therefore most relevant, and which metrics are secondary?
3. Could any model inputs function as proxies for sensitive information?
4. Would equalizing one error rate create a worse trade-off elsewhere?
5. Are subgroup sample sizes large enough to support the conclusion?
6. What should happen when model confidence is low?
7. What information and authority does a human reviewer need?
8. What appeal or correction mechanism should exist for affected people?
9. What monitoring signal would cause you to pause or roll back the system?
10. What evidence would make you recommend that the model not be deployed at all?
Chapter 37 summary
Concept | Key takeaway |
| Bias sources | Historical, sampling, measurement, and label processes can create unfairness before modeling begins. |
| Sensitive information | Direct attributes and indirect proxies require context-specific legal, privacy, and governance review. |
| Fairness evaluation | Compare group performance, FPR, FNR, calibration, and sample sizes; no single metric is universally correct. |
| Uncertainty | Small-group estimates can be unstable; report counts and uncertainty rather than over-interpreting point estimates. |
| Responsible use | Human oversight, appeals, documentation, scope controls, and monitoring are part of model quality. |
| Deployment decision | The highest-scoring model is not automatically acceptable; expected harms and safeguards determine appropriate use. |
Next step With model fairness, failure analysis, interpretation, evaluation, and governance in place, the supervised-learning workflow is ready to move from model development toward reproducible packaging, deployment, monitoring, and lifecycle management. |