BLUF: A predictive-maintenance backtest is credible only when its split represents the deployment decision. Predicting future failures on known machines requires chronological evaluation. Transferring a model to unseen machines requires asset separation. Deploying to unseen machines in a later period requires both. Randomly distributing overlapping sensor windows across training and test sets can let a model recognise a machine or failure episode instead of predicting a useful maintenance opportunity.
The acceptance artifact should therefore be a versioned split manifest, not one accuracy score: asset groups, decision timestamps, feature availability, label horizons, exclusions, fitted preprocessing and event-level outcomes. A clean split removes a class of optimistic estimates; it does not prove that the model is safe or economical in production.
Start with the deployment question
The predictive-maintenance reference architecture establishes ingestion, asset context, scoring and maintenance feedback. This article addresses the offline evidence boundary before that model is promoted. It is not another sensor-drift or alarm-routing design.
Write one sentence that procurement, data science and maintenance can all approve: “Using information available at each decision time, predict this failure mode within this horizon, for these assets, with enough lead time for this intervention.” State whether the intended assets already contributed training data. A model used on the existing fleet may legitimately learn asset-specific baselines; that is not evidence of cold-start transfer to a new plant.
Define the independent unit conservatively. Component replacements, duplicated historian feeds and renamed asset IDs can reconnect apparently separate machines. If shared operating conditions create dependence, consider line, site or equipment family as the grouping boundary. Holding out a site tests a harder and different claim than holding out one motor on a familiar line. With few sites, uncertainty remains large.
Decision table: choose the split that tests your claim
| Split | What it tests | What it does not protect | Appropriate use |
|---|---|---|---|
| Random window split | Interpolation among mixed observations | Shared assets, overlapping windows and future information | Debugging only unless independence is demonstrated |
| Asset-group split | Transfer to held-out groups | Chronology and future training observations | Cross-asset generalisation with a separate time audit |
| Forward-time split | Later observations after earlier training | Transfer to entirely unseen assets | Future operation of the known fleet |
| Grouped forward-time split | Later operation of held-out assets | Unobserved sites, regimes and label defects | New-asset rollout subject to explicit scope limits |
The official GroupKFold documentation guarantees non-overlapping groups across train and test, not chronological training. TimeSeriesSplit produces time-ordered index splits, not asset isolation. Sorting a pooled multi-asset dataframe and calling one of these classes does not automatically implement the combined claim.
TimeSeriesSplit expresses its gap in samples. Its documentation requires equally spaced observations for comparable fold durations. In irregular industrial telemetry, a row count is not a reliable elapsed-time embargo. Define cutoffs with timestamps and inspect each asset's coverage. Missing production hours and sensor bursts must not silently change the evaluation period.
Build an as-of dataset before fitting anything
For every scored row retain asset_group, decision_at, feature_window_start, feature_window_end, feature_available_at, label_window_end and label_available_at. Also retain failure_event_id, operating regime and an observation-complete flag. These are an engineering contract proposed here, not mandatory scikit-learn field names.
Event time and availability time are different. A work-order record may describe a failure on Monday but enter the CMMS on Friday. A reconstruction made on Thursday must not use Friday's diagnosis as a feature. Retrospective corrections, failure codes, replacement-part orders and technician notes deserve explicit availability rules. A timestamp filter on raw sensor samples cannot prevent leakage from an enriched business-data join.
Compute each feature causally from observations available by its decision time. Do not use centred rolling windows, backfill a missing value from a later measurement or normalise an asset using its full recorded lifetime. Historical context before the test boundary can legitimately support the first live prediction if it would have been available then. Distinguish that causal warm-up from learning preprocessing parameters on test data.
A label asks whether the defined event occurs within the prediction horizon. If follow-up ends before that horizon closes, “no observed failure” is not automatically a negative. Mark the row censored or use a method that explicitly handles censoring. Preventive replacement also changes what can be observed: it does not demonstrate that the asset would have remained healthy without intervention.
Asset-by-time split map
The following matrix illustrates one cold-start fold. C is the model freeze cutoff; E is the end of the evaluation interval. It is a design example, not a measured experiment.
| Asset set | Before C, labels available by C | Boundary exclusions | C through E |
|---|---|---|---|
| Development assets A and B | Train and tune using earlier inner folds | Purge unresolved labels and prohibited overlap | Excluded from cold-start score |
| Held-out assets C and D | No fitting; causal feature warm-up only if allowed | Enforce the same availability contract | Score frozen model; wait for complete outcomes |
| Final acceptance assets | Untouched by tuning | Freeze scope and policy first | Evaluate once in a separate later window |
Keep known-fleet and cold-start reports separate. For the known-fleet report, later observations from development assets may be evaluated, but do not merge that easier result into a headline about unseen machines. If cold-start deployment permits unsupervised calibration on earlier held-out history, declare and reproduce that adaptation protocol; it is no longer a strict zero-calibration claim.
Implement grouped temporal folds explicitly
1. Freeze the outer evaluation assets, start and end timestamps, prediction horizon and admissible information policy. Keep the final acceptance window outside model selection. The cross-validation guide explains why repeatedly selecting against a test set leaks knowledge into evaluation.
2. For a cutoff C, select training rows only from development groups, with decision_at before C and label_available_at no later than C. Require complete outcome horizons under the declared label policy. Remove rows with features unavailable at their own decision time. A label window ending before C is insufficient if the label was recorded afterward.
3. Select test rows from held-out groups with decision_at in the evaluation interval. Check the same causal feature contract and require sufficient outcome follow-up before scoring metrics. Assert that the training and test group sets are disjoint. Assert chronological order and assert that no prohibited failure episode or duplicated source record crosses the boundary.
4. Set any purge or embargo from the actual dependencies. The future label horizon constrains when training outcomes become knowable; the lookback window constrains shared inputs when independence is required. Label reporting delay adds another bound. Do not copy a universal “lookback plus horizon” gap without deciding which overlap is forbidden. A larger gap removes information and may leave too few failures; a smaller one needs a defensible causal explanation.
5. Fit imputation, scaling, feature selection, sampling and the estimator only within each training fold. Use a fresh pipeline per fold. The official common-pitfalls guidance recommends pipelines to keep learned transformations away from test data. A pipeline cannot repair an already leaked feature or a wrong split.
6. Tune hyperparameters, alarm threshold, persistence rule and cooldown using earlier inner folds with the same group/time logic. Freeze them before the outer test. Store a manifest of included row IDs, exclusion reasons, data snapshot, code revision and pipeline configuration. Re-run assertions after every join, not only before feature engineering.
Leakage audit and release checklist
| Check | Evidence to retain | Release-blocking finding |
|---|---|---|
| Asset identity | Stable component and parent-group mapping | Same prohibited group in train and test |
| Time and labels | Decision, availability and horizon timestamps | Future information or immature labels used for fitting |
| Feature processing | Fold-local fitted pipeline and causal joins | Global scaling, future backfill or outcome-derived feature |
| Episode independence | Failure episode and duplicate-record audit | One prohibited episode represented on both sides |
| Model selection | Inner-fold history and frozen outer policy | Test outcomes used to choose model or alert threshold |
Turn these into executable assertions in the evaluation job. Include deliberate bad fixtures: a renamed duplicate asset, a late CMMS diagnosis, a global scaler and a window crossing the cutoff. The gate should reject them. Such fixtures test the harness; they are not benchmark data or evidence of predictive performance.
Report maintenance decisions, not millions of correlated rows
Agree on an event-matching rule before scoring. Merge repeated alerts according to the frozen persistence and cooldown policy. Define an actionable detection window before each failure and whether one alert can match more than one event. Otherwise a stream of near-identical warnings can appear to produce many successful predictions from one failure.
Report eligible failure-event counts, detected and missed events, actionable lead-time distribution, unmatched alert episodes, operating exposure and excluded/censored exposure. Show false alarms per operating time and per-asset results alongside pooled results. State denominators and threshold policy. Window-level precision and recall may supplement this report, but cannot replace it.
For uncertainty, resample independent assets or failure episodes where the design supports that assumption, rather than treating overlapping windows as independent evidence. With only a handful of events, even cluster-based intervals can be unstable. Report insufficient evidence rather than manufacturing precision. Compare against an existing maintenance rule or simple condition threshold under the same folds and alert policy; no performance gain is assumed here.
What a clean backtest still cannot establish
Holdouts reduce available training data, combined splits can make folds unusable and new equipment may differ in ways the archive never observed. Maintenance interventions change the outcome process. A score from historical operation does not identify downtime avoided without a defensible comparison design. Label errors and missing failures remain possible after every split assertion passes.
Use a prospective advisory or shadow phase before consequential rollout, with frozen model and alert policy, outcome capture and named maintenance ownership. Continue validating live sensor drift and model trust after deployment. Offline leakage controls and live drift controls protect different boundaries. Neither replaces functional-safety engineering or constitutes a safety certificate.
A practical first engagement
Take one asset class, one failure mode and one proposed rollout decision. Bring a representative historian export, asset identity mapping and timestamped maintenance outcomes. I can help turn these into an as-of data contract, grouped temporal split manifest, leakage tests and a maintenance-facing acceptance report. If the labels cannot support the decision, that is the finding to act on before investing in a more complex model.
