Chapter 38 — Model Persistence
Learning objectives
- Explain why a trained model should be persisted instead of retrained for every prediction task.
- Save preprocessing and the estimator together as one complete pipeline artifact.
- Compare joblib and pickle and recognize the security implications of Python serialization.
- Record metadata required to identify, reproduce, and audit a model artifact.
- Load a saved pipeline and validate incoming feature columns before prediction.
- Generate predictions and probabilities for one observation and for a batch.
- Verify that a reloaded model reproduces the predictions of the original fitted pipeline.
Chapter focus Training a model is not the end of the machine-learning workflow. A useful model must be saved together with the transformations that define its inputs, identified by versioned metadata, loaded safely, and verified before it is used by another notebook, batch job, API, or application. |
38.1 Why save a trained model?
A fitted supervised-learning model contains information learned from the training data: coefficients, split thresholds, support vectors, category mappings, scaling statistics, imputation values, and other fitted state. Model persistence stores that state so it can be reused later without repeating the complete training workflow.
| Reason | What persistence enables | Typical use |
|---|---|---|
| Reuse without retraining | Load the fitted artifact and predict immediately. | Scheduled scoring, desktop tools, internal services. |
| Application integration | Separate the training workflow from the prediction workflow. | Web APIs, dashboards, business applications. |
| Batch prediction | Score many new rows with the exact transformations used during training. | Nightly risk scores, churn lists, sensor batches. |
| Reproducible evaluation | Re-evaluate the exact saved model rather than a newly retrained approximation. | Audits, regression tests, incident investigation. |
| Controlled deployment | Promote a named, versioned artifact between environments. | Development → validation → production. |
Important distinction Saving source code is not the same as saving a trained model. Source code describes how to train; the persisted artifact contains the fitted state produced by a particular dataset, preprocessing configuration, random seed, and set of hyperparameters. |
Python example — train a pipeline that can later be persisted
| from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression X, y = load_breast_cancer(return_X_y=True, as_frame=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.20, stratify=y, random_state=42 ) pipeline = Pipeline([ ("scaler", StandardScaler()), ("model", LogisticRegression(max_iter=2000, random_state=42)), ]) pipeline.fit(X_train, y_train) |
38.2 Saving the complete pipeline
The safest practical pattern is to persist the complete fitted pipeline, not only the final estimator. If scaling, imputation, encoding, feature selection, or other transformations were used during training, those fitted transformations are part of the model definition.
Why pipeline persistence matters If only the estimator is saved, a future application may preprocess new data differently from the training workflow. Saving the fitted pipeline keeps preprocessing and prediction in the same ordered object and greatly reduces train/serve inconsistency. |
joblib
joblib is commonly used for scikit-learn objects because its persistence utilities are convenient for Python objects that may contain large NumPy arrays. The saved file can contain an entire fitted Pipeline, including preprocessing steps and the final estimator.
Python example — save the fitted pipeline with joblib
| from pathlib import Path import joblib artifact_dir = Path("artifacts/model_v1") artifact_dir.mkdir(parents=True, exist_ok=True) model_path = artifact_dir / "pipeline.joblib" joblib.dump(pipeline, model_path) print(f"Saved model to: {model_path}") |
pickle
The Python standard library also provides pickle. It can serialize many Python objects, including fitted scikit-learn pipelines. In machine-learning projects, joblib is often more convenient, but both approaches share an important security property: they reconstruct Python objects when loading.
Python example — save and load with pickle
| import pickle with open(artifact_dir / "pipeline.pkl", "wb") as file: pickle.dump(pipeline, file) with open(artifact_dir / "pipeline.pkl", "rb") as file: reloaded_pipeline = pickle.load(file) |
| Aspect | joblib | pickle |
|---|---|---|
| Availability | External package commonly installed with ML environments. | Part of the Python standard library. |
| Typical ML use | Convenient for scikit-learn/NumPy-heavy artifacts. | General-purpose Python object serialization. |
| Pipeline support | Yes. | Yes. |
| Environment sensitivity | Python and library versions matter. | Python and library versions matter. |
| Trusted-file requirement | Yes. | Yes. |
Version compatibility
A serialized model is coupled to the software environment that created it. A file saved with one Python, scikit-learn, NumPy, or related-library version may fail to load—or behave differently—after a major environment change. Production artifacts should therefore be paired with environment information and tested after upgrades.
Security warning — never load an untrusted model file pickle-based serialization can execute code while reconstructing an object. joblib persistence uses the same underlying Python serialization mechanisms. Only load artifacts produced by a trusted workflow and stored in a controlled location. Treat an unknown .pkl or .joblib file as executable content, not as harmless data. |
38.3 Model metadata
The binary model file tells Python how to reconstruct the fitted object, but it does not by itself provide enough context for a human or deployment system. A model should therefore be accompanied by metadata that describes what it is, how it was trained, what inputs it expects, and how it was evaluated.
| Metadata field | Purpose | Example |
|---|---|---|
| Model name | Human-readable identity. | breast_cancer_logreg |
| Version | Distinguishes released artifacts. | 1.0.0 |
| Training date | Shows when the fitted state was produced. | 2026-09-04 |
| Dataset version | Identifies the training-data snapshot. | breast-cancer-sklearn-v1 |
| Feature list | Defines expected model inputs and order. | 30 named numeric features |
| Target definition | Documents what 0 and 1 mean. | 0 = malignant, 1 = benign |
| Library versions | Supports environment reproduction. | Python / sklearn / numpy / joblib |
| Hyperparameters | Records finalized configuration. | C=1.0, max_iter=2000 |
| Evaluation metrics | Records approved validation/test evidence. | accuracy, F1, ROC AUC |
| Decision threshold | Preserves operational decision policy. | 0.50 or validated custom value |
Python example — write a human-readable metadata sidecar
| import json import platform from datetime import datetime, timezone import joblib import numpy as np import sklearn metadata = { "model_name": "breast_cancer_logreg", "version": "1.0.0", "training_date_utc": datetime.now(timezone.utc).isoformat(), "dataset_version": "breast-cancer-sklearn-v1", "feature_list": X_train.columns.tolist(), "target_definition": {"0": "malignant", "1": "benign"}, "decision_threshold": 0.50, "library_versions": { "python": platform.python_version(), "scikit_learn": sklearn.__version__, "numpy": np.__version__, "joblib": joblib.__version__, }, "hyperparameters": pipeline.get_params(deep=False), } with open(artifact_dir / "metadata.json", "w", encoding="utf-8") as file: json.dump(metadata, file, indent=2, default=str) |
Evaluation metadata
Evaluation results should be recorded after the model has passed the evaluation workflow described in earlier chapters. The metadata should state which split produced the metrics, which threshold was used, and whether the values are validation, cross-validation, or final-test results.
Python example — store evaluation metrics and threshold context
| from sklearn.metrics import accuracy_score, f1_score, roc_auc_score proba = pipeline.predict_proba(X_test)[:, 1] threshold = 0.50 pred = (proba >= threshold).astype(int) metadata["evaluation"] = { "split": "held_out_test", "n_rows": len(X_test), "accuracy": float(accuracy_score(y_test, pred)), "f1": float(f1_score(y_test, pred)), "roc_auc": float(roc_auc_score(y_test, proba)), } with open(artifact_dir / "metadata.json", "w", encoding="utf-8") as file: json.dump(metadata, file, indent=2, default=str) |
A practical artifact folder
| File | Role |
|---|---|
| pipeline.joblib | The fitted preprocessing + estimator pipeline. |
| metadata.json | Human- and machine-readable model identity, schema, versions, threshold, and metrics. |
| requirements.txt / environment lock | Recreates the software environment used by the artifact. |
| README or model card | Documents intended use, limitations, ownership, and operating conditions. |
| checksum.txt (optional) | Supports integrity verification when artifacts are copied or deployed. |
Python example — optional integrity checksum
| import hashlib model_bytes = model_path.read_bytes() sha256 = hashlib.sha256(model_bytes).hexdigest() (artifact_dir / "checksum.txt").write_text( f"{sha256} {model_path.name}\n", encoding="utf-8", ) print("SHA-256:", sha256) |
38.4 Loading and predicting
Prediction code should treat the saved artifact as a controlled dependency. Before calling predict(), the application should verify that the model and metadata are present, that required columns exist, and that input values satisfy basic expectations.
Loading the pipeline
Python example — load a trusted model and its metadata
| import json import joblib model = joblib.load(artifact_dir / "pipeline.joblib") with open(artifact_dir / "metadata.json", encoding="utf-8") as file: metadata = json.load(file) required_features = metadata["feature_list"] threshold = metadata["decision_threshold"] |
Validating input columns
Column validation catches a common deployment error: the prediction request does not match the schema used to train the model. Validation should be explicit, because a silent column mismatch can produce incorrect predictions or runtime errors.
Python example — enforce the expected feature schema
| def validate_columns(frame, required_features): required = set(required_features) received = set(frame.columns) missing = sorted(required - received) extra = sorted(received - required) if missing: raise ValueError(f"Missing required columns: {missing}") # Extra columns can be rejected or ignored according to the API contract. if extra: print(f"Ignoring extra columns: {extra}") return frame.loc[:, required_features] X_ready = validate_columns(X_test.copy(), required_features) |
Column names are only one layer of validation A production interface may also need dtype checks, allowed categories, null policies, numeric ranges, units, timestamp rules, and business constraints. Input validation should match the operating environment rather than assuming every incoming DataFrame is trustworthy. |
Predicting one observation
Python example — predict one observation and return a probability
| one_row = X_ready.iloc[[0]] probability = model.predict_proba(one_row)[0, 1] prediction = int(probability >= threshold) result = { "prediction": prediction, "probability_class_1": float(probability), "threshold": float(threshold), } print(result) |
Predicting a batch
Python example — score a batch with one saved pipeline
| batch = X_ready.iloc[:20] probabilities = model.predict_proba(batch)[:, 1] predictions = (probabilities >= threshold).astype(int) output = batch.index.to_frame(index=False) output["prediction"] = predictions output["probability_class_1"] = probabilities print(output.head()) |
Handling errors
Loading and prediction should fail clearly when a required artifact is missing, the schema is invalid, the file is corrupted, or the runtime environment is incompatible. Error messages should help operators diagnose the problem without exposing sensitive input data.
Python example — basic defensive error handling
| from pathlib import Path try: path = Path("artifacts/model_v1/pipeline.joblib") if not path.exists(): raise FileNotFoundError(f"Model artifact not found: {path}") model = joblib.load(path) X_ready = validate_columns(X_test.copy(), required_features) probabilities = model.predict_proba(X_ready)[:, 1] except (FileNotFoundError, ValueError) as exc: print(f"Prediction request rejected: {exc}") except Exception as exc: # Production systems should log a safe diagnostic and fail closed. print(f"Model loading or prediction failed: {type(exc).__name__}") |
Practical lab — Save, reload, and verify a complete trained pipeline
Lab objective Train one complete pipeline, persist both the fitted pipeline and its metadata, reload them in a clean prediction workflow, and prove that the reloaded artifact produces the same outputs as the original fitted object. |
Step 1 — Prepare data and train the final pipeline
Lab code — build and fit the complete pipeline
| from pathlib import Path import json import joblib import numpy as np import pandas as pd from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, f1_score, roc_auc_score data = load_breast_cancer(as_frame=True) X = data.data.copy() y = data.target.copy() X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.20, stratify=y, random_state=42 ) final_pipeline = Pipeline([ ("scaler", StandardScaler()), ("model", LogisticRegression(max_iter=2000, random_state=42)), ]) final_pipeline.fit(X_train, y_train) |
Step 2 — Evaluate and freeze the operating threshold
In a real project, the threshold should already have been selected on validation data before final test evaluation. This lab uses 0.50 to keep the persistence exercise focused on saving and verification.
Lab code — create reference predictions before saving
| threshold = 0.50 original_proba = final_pipeline.predict_proba(X_test)[:, 1] original_pred = (original_proba >= threshold).astype(int) metrics = { "accuracy": float(accuracy_score(y_test, original_pred)), "f1": float(f1_score(y_test, original_pred)), "roc_auc": float(roc_auc_score(y_test, original_proba)), } print(metrics) |
Step 3 — Save pipeline and metadata
Lab code — persist the model artifact and metadata
| import platform from datetime import datetime, timezone import sklearn artifact_dir = Path("artifacts/chapter38_model_v1") artifact_dir.mkdir(parents=True, exist_ok=True) joblib.dump(final_pipeline, artifact_dir / "pipeline.joblib") metadata = { "model_name": "chapter38_breast_cancer_logreg", "version": "1.0.0", "training_date_utc": datetime.now(timezone.utc).isoformat(), "dataset_version": "sklearn-breast-cancer", "feature_list": X.columns.tolist(), "target_definition": {"0": "malignant", "1": "benign"}, "decision_threshold": threshold, "library_versions": { "python": platform.python_version(), "scikit_learn": sklearn.__version__, "numpy": np.__version__, "joblib": joblib.__version__, }, "evaluation": metrics, } with open(artifact_dir / "metadata.json", "w", encoding="utf-8") as file: json.dump(metadata, file, indent=2) |
Step 4 — Reload in a prediction-style workflow
Lab code — reload exactly what the prediction service would use
| loaded_pipeline = joblib.load(artifact_dir / "pipeline.joblib") with open(artifact_dir / "metadata.json", encoding="utf-8") as file: loaded_metadata = json.load(file) required_features = loaded_metadata["feature_list"] loaded_threshold = loaded_metadata["decision_threshold"] X_loaded = X_test.loc[:, required_features] reloaded_proba = loaded_pipeline.predict_proba(X_loaded)[:, 1] reloaded_pred = (reloaded_proba >= loaded_threshold).astype(int) |
Step 5 — Verify that persistence changed nothing
Lab code — verify exact/reasonably equivalent model behavior
| same_classes = np.array_equal(original_pred, reloaded_pred) same_probabilities = np.allclose(original_proba, reloaded_proba) max_probability_difference = np.max( np.abs(original_proba - reloaded_proba) ) print("Same class predictions:", same_classes) print("Same probabilities:", same_probabilities) print("Maximum probability difference:", max_probability_difference) assert same_classes assert same_probabilities |
Expected result For the same software environment, same saved artifact, and same ordered input features, the reloaded pipeline should reproduce the original class predictions and prediction probabilities. The verification assertions turn that expectation into a testable deployment check. |
Step 6 — Verify one observation and a batch
Lab code — test single-row and batch scoring
| single = X_loaded.iloc[[0]] single_probability = loaded_pipeline.predict_proba(single)[0, 1] single_prediction = int(single_probability >= loaded_threshold) batch = X_loaded.iloc[:10] batch_probabilities = loaded_pipeline.predict_proba(batch)[:, 1] batch_predictions = (batch_probabilities >= loaded_threshold).astype(int) print("Single prediction:", single_prediction) print("Single probability:", single_probability) print("Batch predictions:", batch_predictions.tolist()) |
Step 7 — Prove that schema validation catches an error
Lab code — intentionally submit an invalid schema
| bad_request = X_loaded.iloc[:3].drop(columns=[required_features[0]]) try: validate_columns(bad_request, required_features) except ValueError as exc: print("Expected validation error:", exc) |
Student verification checklist
- The saved artifact contains the complete fitted preprocessing + model pipeline.
- A metadata file identifies the model version, dataset, features, software versions, metrics, and threshold.
- The reloaded model returns the same class predictions as the original model.
- The reloaded model probabilities match the original probabilities within numerical tolerance.
- Single-row and batch prediction both work through the same pipeline.
- A missing required feature produces a controlled validation error.
- The serialized model is loaded only from a trusted artifact location.
Discussion questions
- Why is saving only the LogisticRegression estimator insufficient when StandardScaler was used during training?
- Which metadata fields would you add for a model deployed in a regulated or safety-critical environment?
- Why should a custom decision threshold be stored even if the trained estimator itself is unchanged?
- What could happen if a model artifact is loaded under a different scikit-learn version without compatibility testing?
- Why must .joblib and .pkl files be treated as trusted executable artifacts rather than ordinary data files?
- Which input-validation rules, beyond column names, would your own application require?
Chapter summary
| Principle | Key takeaway |
|---|---|
| Persist the whole pipeline | Save fitted preprocessing and the estimator together so training and prediction use the same transformations. |
| Version the artifact | A model file needs an explicit identity, version, training context, and software environment. |
| Store operational metadata | Features, target definition, metrics, hyperparameters, and the decision threshold are part of the deployable model definition. |
| Load only trusted files | Python object serialization can execute code during loading; artifact provenance and access control matter. |
| Validate inputs | Schema, types, categories, ranges, units, and missing-value rules should be checked before prediction. |
| Verify after reload | Automated equality/tolerance checks confirm that persistence did not change model behavior. |
| Plan for upgrades | Library upgrades require compatibility testing and often controlled reserialization or retraining. |
Next step A persisted model is ready to be consumed by a batch process, application, or prediction service. The next stage of a production-oriented workflow is to define the inference interface, monitor data and prediction quality, and manage model versions through their lifecycle. |