From a real-world need to a precise, testable, and feasible prediction task

Chapter 4 problem-definition path

Chapter overview

Before choosing an algorithm, a machine learning team must decide exactly what problem it is solving. This chapter turns an imprecise real-world need into a supervised learning specification that can be implemented, evaluated, and defended. The emphasis is on questions that must be answered before training begins: What decision will the prediction support? What is one row? What is the target? When is the prediction produced? Which features are available at that moment? How costly are different errors? What result would count as success?

A well-defined problem creates a stable contract between domain experts, data engineers, data scientists, software teams, and end users. It also prevents common failures such as target leakage, label ambiguity, mismatched evaluation metrics, unrealistic data assumptions, and models that arrive too late to influence a decision.

Learning objectives

• Translate a real-world need into a supervised classification or regression task.
• Identify the decision, prediction target, prediction time, users, and error costs.
• Define a measurable target and distinguish target availability from feature availability.
• Recognize target ambiguity, proxy targets, delayed labels, and target leakage.
• Define the correct unit of observation and verify the grain of a dataset.
• Specify technical, business, scientific, and operational success criteria.
• Relate false-positive and false-negative costs to metric and threshold selection.
• Assess whether sufficient data, reliable labels, useful features, and an appropriate use case exist.
• Decide whether machine learning is warranted or a simpler rule-based solution is preferable.
• Write a complete structured machine learning problem statement.

Running example

Throughout the chapter, a telecommunications churn scenario is used as a running example. The organization wants to identify customers who are likely to cancel their subscription soon enough for a retention team to intervene. This apparently simple goal contains several design choices: what “cancel” means, the prediction horizon, the observation date, eligible customers, available features, campaign capacity, the cost of unnecessary contact, and the cost of failing to identify a customer who leaves.

Chapter map

4.1 From a Real-World Need to an ML Task

Organizations usually begin with a broad need rather than a machine learning specification: reduce churn, prevent failures, improve student support, detect fraud, estimate demand, prioritize inspections, or automate document routing. The first design task is to transform that need into a prediction that is available at a useful time and supports a concrete action.

Figure 4.1 — Translating a real-world need into an actionable ML task

4.1.1 Understanding the domain problem

A domain problem describes an undesirable condition, an opportunity, or a decision that an organization wants to improve. The machine learning problem is narrower: it defines what information will be estimated from data. These two statements must be related, but they are not interchangeable.

4.1.2 Identifying the decision to be supported

A prediction has value only when it changes or improves a decision. The team should identify the decision owner, the set of available actions, the constraints on those actions, and how the prediction enters the workflow. For example, a churn probability may be used to rank customers because the campaign team can contact only 2,000 customers per month. In that case, ranking quality at the top of the list may matter more than overall accuracy.

• What decision will be made differently because the model exists?
• What actions are available after a high-risk or high-value prediction?
• Who owns the decision and who is accountable for the outcome?
• Is the action automatic, advisory, or subject to human review?
• Is there a capacity limit, budget, service-level target, or legal constraint?
• What happens when the model is uncertain or input data are incomplete?

4.1.3 Defining what should be predicted

The predicted quantity should be stated in operational terms. A classification target requires a precise event or state; a regression target requires a precise numerical measurement. Vague targets such as “high risk,” “good customer,” or “machine health” must be converted into measurable definitions.

A simple Python dictionary can make the initial formulation explicit and easy to review with stakeholders.

4.1.4 Determining when the prediction must be available

Prediction time is the cut-off that separates permissible information from future information. Every feature must be computable at or before this moment. The label usually describes what happens after the prediction time. This temporal separation is central to preventing leakage.

Figure 4.2 — Features come from the past; the target is observed in the future

For a churn model scored on 31 January, features might summarize behavior through 31 January and the target might indicate whether cancellation occurs from 1 February through 2 March. A feature such as “cancellation_reason” would be unavailable at scoring time and therefore invalid, even if it is highly predictive in historical data.

4.1.5 Identifying who will use the result

The user of the model influences output format, explanation requirements, latency, thresholding, and interface design. A data scientist reviewing a batch table can tolerate different outputs from a call-center agent who must act within seconds or an automated control system that requires millisecond latency.

4.1.6 Determining the cost of incorrect predictions

Errors are rarely symmetric. In binary classification, a false positive and a false negative often lead to different actions and consequences. The team should define these consequences before choosing a metric or decision threshold.

4.2 Defining the Target Variable

The target variable is the value the model learns to predict. It must be measurable, correctly aligned with each observation, and available as historical ground truth. Poor labels place a hard ceiling on model quality because the algorithm is trained to reproduce the target definition, including its noise and biases.

4.2.1 Choosing a measurable target

A useful target definition specifies the event or quantity, the measurement rule, the time horizon, the eligible population, and any exclusions. The definition should be executable: two people implementing it independently should obtain the same labels from the same source data.

4.2.2 Target availability

Historical training requires labels for past examples. The team should identify the authoritative source of truth, the delay before labels become complete, and whether some outcomes remain unobserved. For instance, a 90-day default label cannot be finalized until at least 90 days after the decision date. Training on more recent observations may therefore create artificially negative labels that have not yet had enough time to mature.

• Where is the ground truth stored?
• How long after prediction time does the label become final?
• Can the event occur without being recorded?
• Are some observations censored because follow-up is incomplete?
• Has the label definition changed over time?
• Are labels available for all groups and all operating conditions?

4.2.3 Target quality

Target quality includes correctness, consistency, completeness, timeliness, and agreement with the intended concept. A model cannot reliably exceed the quality of its labels. Manual labels may suffer from reviewer disagreement; administrative labels may reflect policy rather than the underlying concept; sensor-derived labels may contain calibration errors.

4.2.4 Target ambiguity

Ambiguity occurs when the target has multiple legitimate interpretations. Churn can mean contract cancellation, inactivity for a fixed period, non-renewal, or migration to another product. “Fraud” may mean suspected fraud, confirmed fraud, or a chargeback. These definitions produce different datasets and different models.

4.2.5 Target leakage

Target leakage occurs when the training inputs contain information that would not legitimately be available at prediction time or that directly encodes the future target. Leakage can produce spectacular validation scores and unusable deployed models.

4.2.6 Proxy variables

A proxy target is an observable variable used in place of the concept the organization actually cares about. Proxies may be necessary when the desired outcome is difficult or expensive to measure, but they change the meaning of the model. For example, “support ticket escalation” might be used as a proxy for customer dissatisfaction, yet many dissatisfied customers never submit a ticket.

When a proxy is used, the project should explicitly say “predict the proxy” rather than claiming to predict the latent concept. Stakeholders should validate whether improving the proxy is expected to improve the real objective.

4.2.7 Delayed labels

Some targets are known only after a delay. Delayed labels affect dataset cutoffs, model monitoring, retraining frequency, and online evaluation. The newest records may be usable for prediction but not yet usable as labeled training examples.

4.3 Defining the Unit of Observation

The unit of observation—also called the data grain—is what one row represents at prediction time. Every feature and target must be aligned to this unit. A model trained on a mixture of grains can double-count entities, leak future information, and produce misleading evaluation results.

4.3.1 Typical units of observation

4.3.2 Entity, event, and time-window grains

The same source system can support different units. A transaction table naturally has one row per event, but a churn model may need one row per customer per month. The event records must then be aggregated using only events before the scoring date. Conversely, a fraud model may preserve one row per transaction and add customer-history features calculated from prior transactions.

4.3.3 Checking row uniqueness

The candidate primary key should uniquely identify one observation. Duplicate keys may indicate accidental joins, repeated exports, multiple events at the same grain, or a missing dimension such as time. The proper response depends on semantics; duplicates should not be removed automatically without understanding why they exist.

4.3.4 Constructing an entity-time observation

When raw data contain many events per entity, features can summarize a historical window relative to the prediction date. The aggregation must stop at the prediction cutoff. A correct feature such as “number of support tickets in the previous 90 days” uses only past events; a count that accidentally includes the target horizon would leak future behavior.

In a real temporal problem, first filter events to the permitted historical feature window.

4.4 Defining Success Criteria

Success criteria determine how the project will be evaluated and whether it should proceed to deployment. They should be defined before extensive model experimentation so that the team does not choose whichever metric happens to make a model look best. A complete definition combines technical quality, domain value, operational feasibility, and risk constraints.

4.4.1 Technical metrics

Technical metrics quantify predictive performance on data that were not used to fit the model. Metric choice depends on the target and the decision. No single metric is best for every problem.

4.4.2 Business metrics

A technically accurate model may still have little operational value. Business criteria connect predictions to outcomes such as savings, revenue, workload, conversion, retention, inventory cost, or resource utilization. These outcomes are affected by both model quality and the intervention process.

4.4.3 Scientific objectives

In research, success may include statistical validity, reproducibility, robustness, uncertainty quantification, performance across datasets, or evidence that a proposed method improves a defined baseline. The scientific claim should determine the experiment design. A tiny score increase on one split may be insufficient if the research question concerns robustness or generalization.

• Predefine primary and secondary metrics.
• Use repeated or cross-validated evaluation when appropriate.
• Report variability and uncertainty rather than only a single score.
• Compare against meaningful baselines under matched conditions.
• Evaluate failure modes relevant to the claimed contribution.
• Keep the final test protocol fixed once defined.

4.4.4 Operational constraints

A model operates inside a system. The acceptable solution is constrained by prediction latency, throughput, memory, hardware, uptime, data refresh frequency, explainability, review capacity, and maintenance cost. These requirements can rule out otherwise strong algorithms.

4.4.5 Minimum acceptable performance

The team should define a performance floor relative to a baseline and to operational needs. “Better than random” is rarely sufficient. A useful acceptance rule might be: the model must improve recall at fixed review capacity by at least 15% over the current rule while maintaining a maximum false-positive rate and a specified latency.

4.4.6 Prediction latency and availability

Latency is the time from receiving an eligible case to delivering the prediction. End-to-end latency includes feature retrieval and preprocessing, not only the estimator’s execution time. For batch systems, the equivalent requirement is often a completion deadline and maximum acceptable data age.

4.4.7 Interpretability requirements

Interpretability can be a mandatory requirement when predictions support expert review, regulated decisions, safety-sensitive processes, or scientific conclusions. Requirements should be concrete: global feature behavior, per-prediction reason codes, monotonic relationships, sparse models, or documentation of influential features. “The model must be interpretable” is too vague to guide model selection.

4.4.8 Cost of false positives and false negatives

When decisions have measurable costs, a threshold can be selected by minimizing expected cost on validation data. The chosen cost values should come from domain analysis rather than from model performance. Costs may include money, labor, opportunity loss, delays, customer burden, or safety impact.

4.5 Feasibility Analysis

A well-defined prediction can still be infeasible. Feasibility analysis asks whether the organization has enough relevant data, labels are trustworthy, the target contains a learnable signal, necessary features are available at prediction time, and machine learning adds value over simpler alternatives. It is better to discover a fundamental limitation before months of modeling work.

4.5.1 Is sufficient data available?

Dataset size should be judged in relation to problem difficulty, number of features, model complexity, class frequency, temporal diversity, and the number of independent entities. Ten million duplicated or highly correlated rows may contain less information than a smaller diverse dataset.

• How many independent observations are available?
• How many positive examples exist for each important class?
• Does the history cover different seasons, sites, devices, or operating conditions?
• Are rare but important situations represented?
• Can a valid train/validation/test split be created without starving any subset?
• Will the deployed population resemble the historical population?

4.5.2 Are the labels reliable?

Reliable labels are a prerequisite for supervised learning. If the label process is inconsistent, biased, incomplete, or heavily delayed, improving the label pipeline may yield more value than trying increasingly complex algorithms. A pilot label audit should examine examples from different time periods and subgroups.

4.5.3 Is the target predictable?

Some outcomes are dominated by random events or by factors that are not observable before prediction time. A model can only learn signal contained in the available features. Early feasibility experiments can compare simple baselines, inspect feature-target relationships, and estimate an achievable range without extensive tuning.

Predictability is not the same as causality. A feature can help prediction without being a cause, and a causal factor may be unavailable or too noisy to improve prediction. The project should be explicit about whether the goal is predictive, explanatory, or causal.

4.5.4 Are the features available at prediction time?

Historical databases often contain columns that are convenient for retrospective analysis but unavailable online, delayed by hours or days, expensive to compute, or produced after the decision. A feature inventory should record source system, refresh rate, computation time, historical coverage, and prediction-time availability.

4.5.5 Is machine learning necessary?

Machine learning is useful when decisions depend on patterns that are difficult to encode manually, data are sufficiently rich, performance can improve with learning, and the decision occurs often enough to justify development and maintenance. It is not automatically the best solution.

4.5.6 Would a simple baseline be sufficient?

A baseline is not merely an evaluation formality. It can answer whether the proposed ML system is worth its complexity. A business rule, historical average, majority-class predictor, or simple linear model may already satisfy the objective. If a complex model offers only a negligible improvement, the simpler solution may be more reliable, explainable, and maintainable.

Feasibility decision matrix

Workshop — Formulate a Supervised Learning Problem

In this workshop, students transform a short real-world scenario into a structured machine learning problem statement. The goal is not to train a model. The goal is to define a task that another team could implement without guessing the meaning of the target, the timing, or the success criteria.

Workshop scenario

Student task

Prepare a one-page problem specification using the eight required fields below. Every statement should be measurable and tied to a specific point in time.

Step 1 — Define the objective and decision

• Rewrite “reduce churn” as an operational objective.
• State the decision: which customers should receive one of the limited weekly retention contacts?
• Identify the decision owner and the maximum action capacity.

Step 2 — Define the unit and prediction time

• Choose whether one row represents a customer, customer-week, customer-month, or another unit.
• Choose a weekly scoring date that occurs before campaign assignment.
• State the feature window and ensure all candidate features are available before the cutoff.

Step 3 — Define the target

• Choose a prediction horizon such as 30 days.
• Define exactly which cancellation statuses count as a positive event.
• Specify how insufficient follow-up or unresolved account states are handled.
• Identify the authoritative source of cancellation dates.

Step 4 — Define success criteria

• Select a metric that reflects limited campaign capacity and the importance of finding likely churners.
• Define a minimum improvement over the current process.
• Specify the maximum number of customers returned each week.
• Define required prediction completion time and explanation needs.

Step 5 — Perform a feasibility review

• Check whether enough positive churn cases exist historically.
• Check whether labels are mature and consistently recorded.
• List features that are available only after cancellation and must be excluded.
• Determine whether a simple rule might already meet the business need.

Workshop worksheet

Illustrative solution

Workshop evaluation rubric

Knowledge check

Suggested answers

• 1. It does not define the population, target event, time horizon, prediction time, user, action, or success criterion.
• 2. Prediction time is when the model output must be available; the target horizon is the future period over which the outcome is defined.
• 3. Example: using cancellation_reason to predict churn. It is recorded after churn, so historical evaluation sees information unavailable at deployment.
• 4. Sales can be censored by stockouts: low observed sales may reflect unavailable inventory rather than low demand.
• 5. It defines what one row represents and therefore how features, labels, keys, splits, and aggregations must be constructed.
• 6. Predefining criteria prevents metric shopping and keeps model selection aligned with the real objective.
• 7. When acting on a false alert is expensive or harmful—for example, sending costly manual inspections for many non-failing machines.
• 8. The label may be delayed; the required follow-up horizon has not yet elapsed, so recent negative labels are not mature.
• 9. When a few stable transparent rules already solve the task, or when labeled data are insufficient for valid training and evaluation.
• 10. Reformulate the task, improve data or label collection, redesign the process, choose a simpler method, or stop the project rather than forcing modeling.

Chapter summary

Defining the machine learning problem creates the foundation for every later step in the supervised learning workflow. A valid formulation begins with a real decision, defines a precise prediction available at the right time, establishes one clear unit of observation, and aligns features with a measurable target. The target must be trustworthy, mature, and free from future information. Success criteria should combine predictive performance with business or scientific value, operational constraints, error costs, latency, and interpretability. Finally, feasibility analysis determines whether sufficient representative data, reliable labels, useful prediction-time features, and a genuine need for machine learning exist.