BLUF: A filled sensor value is not necessarily a usable model input. Bound carry-forward by elapsed time, preserve observed-versus-imputed provenance, and reset state at asset or operating-regime changes. Reject unusable source quality before imputation. Evaluate output availability together with task error: a pipeline that always returns a number can be the least trustworthy option.
Which decision does the feature gate make?
The decision is narrow: may this model consume this feature vector now, for this approved advisory task? It is not a decision to stop a machine or authorise a safety function. A maintenance recommendation and a safety interlock need different engineering evidence. Numerical completeness confers no control authority.
Start after identity, units, quality mapping and time alignment are established. The industrial data-contract foundation defines those semantic prerequisites. The late-sensor-data lifecycle determines when a window can change. Here, the remaining problem is what to do with a missing feature at a particular decision time—not how to advance a watermark or build an AAS interface.
Separate three policies. Strict rejection scores only vectors with usable observations for every required feature. Bounded last-observation carry-forward tolerates a validated short gap without pretending a new measurement occurred. A fitted imputer estimates missing values from training data. Each can be useful; none removes the need for quality, regime and output-validity checks.
Why ffill(limit=3) is not a freshness policy
The pandas forward-fill documentation defines limit in consecutive missing entries. Three rows might represent three seconds, three minutes or an irregular reconnect burst. A row-count limit therefore cannot establish how old an industrial observation is.
Retain a last-confirmed-usable timestamp and compute age = decision_time − confirmed_time. A carried value must keep the original confirmed_time; assigning the current timestamp each time it is copied makes stale evidence appear permanently fresh. For the illustrative fixture below, age at most five seconds is accepted. Five seconds is a test parameter, not a recommended plant limit.
Choose the real limit from the task's error budget, observed process dynamics, acquisition guarantees and fault traces. Validate it separately by feature and operating regime. A slowly changing ambient variable and a rapidly changing load signal need not share a budget. A fresh value also needs valid units, range and calibration; freshness cannot repair a defective instrument.
OPC UA timestamps require an acquisition contract
The OPC UA DataValue specification requires clients to check StatusCode severity before using values. Bad means the value is unusable. Uncertain means the quality is uncertain for the reported reason. The conservative policy in this article rejects both and invalidates that channel's cached hold; treating every Uncertain subcode identically is our engineering choice, not a universal OPC UA requirement.
Do not substitute sourceTimestamp mechanically for confirmed_time. The specification says sourceTimestamp should indicate the last change of value or status. A stable, correctly observed temperature may therefore have an old change timestamp. serverTimestamp can reflect when a server knew a value to be accurate, including exception-based acquisition, but its meaning still depends on the server and upstream chain.
Define how the adapter establishes continued validity: sampling confirmation, a trustworthy heartbeat plus device-health evidence, or documented exception-reporting semantics. Retain raw source/server timestamps, acquisition confirmation and arrival time separately. A reconnect time must not turn an old buffered observation into a new measurement. Unknown clock quality or acquisition semantics should block the inference claim rather than create an invented freshness timestamp.
Decision table: retain uncertainty instead of laundering it
| Condition | Feature action | State consequence | Inference consequence |
|---|---|---|---|
| Good usable observation, age within budget | Emit observed value and actual age | Cache value and confirmation time | Eligible subject to whole-vector gate |
| Missing value, same lifecycle and regime, age within budget | Carry last usable value; mark imputed | Do not refresh confirmation time | Eligible only for validated gap patterns |
| Missing value beyond elapsed-time budget | Emit rejected value with expiry reason | Never extend its lifetime by copying | Abstain or use a separately validated fallback |
| Bad or policy-rejected Uncertain quality | Reject even if a numeric value is present | Clear that channel's hold | Do not hide failure with an old measurement |
| Asset replacement or regime transition | Clear prior history before filling | Start a new lifecycle/regime epoch | Require new evidence or an approved cold-start path |
| Old buffered or future-dated reconnect sample | Reject under time/availability checks | Do not promote arrival time to sample time | Keep the feature unavailable |
| Several required channels missing | Evaluate the complete missingness pattern | Preserve per-channel provenance | Reject unless that joint pattern is validated |
The last row matters. Independently acceptable holds can still form an untested combination. The small fixture deliberately permits two short holds to demonstrate the resulting error; a production vector gate should use an explicit allowlist of validated missingness patterns. A median estimate is not a way around that allowlist.
Implement a causal, stateful transformer
1. Partition state by stable asset lifecycle and feature identity. Bind it to a regime epoch that changes on recipe transitions, maintenance boundaries and sensor replacement. An epoch must be an authoritative lifecycle token, not a display name. Missing or ambiguous regime evidence blocks the path.
2. Process decisions in nondecreasing time order per asset. Accept only information available by the decision time. Validate acquisition confirmation against that time and reject future information. Handle late observations upstream; never overwrite a newer cached confirmation with an older one.
3. Validate the supplied quality, finite numeric value and acquisition envelope. Bad or rejected Uncertain quality clears the affected channel's hold in this policy. An unknown quality code also fails closed. Keep the original StatusCode and reason in durable evidence even if the transform uses normalised labels.
4. For a usable observation, emit value, age and provenance=observed. For an absent value, look up the same-epoch cache. Carry it only while age remains within the feature budget, with provenance=carried and missing=true. Otherwise emit value=null and provenance=rejected. Downstream systems must distinguish null from a measured zero.
5. Apply the whole-vector validity gate after transformation. Record the missingness pattern, largest age, rejection reasons, feature-policy version and model version with the prediction or abstention. The model may receive numeric features plus missingness indicators, but the policy envelope must remain available outside the model.
The executed Python fixture uses a FeatureGate class with transform(asset, epoch, now, samples). Each sample contains value, quality, confirmed_at and available_at; each output retains value, source_quality, epoch, missing, provenance, reason and age. Its required feature set is x and z. Production code must enforce an exact schema: omitting a required dictionary key must not silently shrink the vector. The fixture is an in-memory policy test, not a deployed OPC UA connector or durable feature store.
Timeline: observed, carried, then rejected
At t=0, x=10 is observed. At t=2, x is absent and the gate carries 10 with age two seconds. At t=7, x is still absent and must be rejected: copying at t=2 did not reset its clock. A regime change at t=10 invalidates history independently of age. At t=11, reconnect delivers a sample confirmed at t=4; it remains stale. Fresh observations at t=12 restart the usable path.
Deterministic dropout fixture and actual outputs
The fixture was executed for this article with ten decisions, two synthetic features and a five-second hold budget. Its deliberately simple task is to estimate x+z. The evaluator knows the hidden true values; the gate does not. Values have arbitrary units, and the experiment is not a predictive-maintenance benchmark. All tuple values below are ordered (x, z); O means observed, C carried and R rejected.
| Time and epoch | Input event | Hidden truth | Bounded-hold output |
|---|---|---|---|
| 0, A | Good samples (10, 20) | (10, 20) | (10 O, 20 O) |
| 2, A | x missing; z=20 Good | (12, 20) | (10 C, 20 O) |
| 7, A | Both missing | (17, 25) | (null R, 20 C) |
| 8, A | x=999 Bad; z=26 Good | (18, 26) | (null R, 26 O) |
| 9, A | x=19 Good; z=27 Uncertain | (19, 27) | (19 O, null R) |
| 10, B | Regime reset; both missing | (40, 50) | (null R, null R) |
| 11, B | x=19 confirmed at 4; z=51 Good | (41, 51) | (null R, 51 O) |
| 12, B | Fresh samples (42, 52) | (42, 52) | (42 O, 52 O) |
| 15, B | Both missing | (45, 55) | (42 C, 52 C) |
| 18, B | Both missing | (48, 58) | (null R, null R) |
A separate pre-test training fold contains (8,18), (10,20), (12,22), giving medians (10,20). The permissive median baseline substitutes those medians wherever the gate did not emit an observed value. It intentionally ignores deployment eligibility, including the regime change, to show why numerical coverage is not acceptance. These statistics are fitted only on that training fold. The fixture does not train a predictive model.
Executed results below report available vectors divided by decisions, followed by mean absolute error on the vectors actually emitted. A dash means no output, not zero error. Joint-loss error is rounded to two decimal places.
| Pattern | Strict rejection: availability; MAE | Bounded hold: availability; MAE | Permissive median: availability; MAE |
|---|---|---|---|
| All decisions | 2/10; 0 | 4/10; 2 | 10/10; 26.6 |
| Single-channel absence | 0/1; — | 1/1; 2 | 1/1; 2 |
| Joint-channel absence | 0/3; — | 1/3; 6 | 3/3; 52.67 |
| Bad quality | 0/1; — | 0/1; — | 1/1; 8 |
| Uncertain quality | 0/1; — | 0/1; — | 1/1; 7 |
| Regime reset | 0/1; — | 0/1; — | 1/1; 60 |
| Stale reconnect | 0/1; — | 0/1; — | 1/1; 31 |
| Complete initial sample | 1/1; 0 | 1/1; 0 | 1/1; 0 |
| Fresh reconnect | 1/1; 0 | 1/1; 0 | 1/1; 0 |
The zero error of strict rejection is not evidence that it is the best policy: it scores only the two fully observed decisions. Compare policies on common eligible cases as well as their coverage/error trade-off, and account for the cost of abstention. The bounded-hold error at t=15 exposes the unvalidated joint-gap case. The median baseline's high coverage includes values that should never pass the deployment gate.
Assertions passed for all ten expected provenance pairs, nonnegative emitted ages, acceptance at exactly the five-second boundary, rejection after it, isolation between assets, rejection of future information and refusal to move decision time backwards. The retained feature_gate.py and fixture-results.json are the executable acceptance artifacts. They do not test network delivery, restart recovery, concurrency or real process outcomes.
Fit imputation inside the validation boundary
The scikit-learn imputation guide documents SimpleImputer, multivariate strategies and missingness indicators. Use a pipeline to fit imputation and the estimator within each training fold; transform held-out data without refitting. The asset-and-time backtest design explains why random windows and globally fitted preprocessing give misleading evidence.
For a stable feature schema, configure missingness indicators deliberately. An indicator fitted only for columns missing during training can miss a new deployment-time absence pattern; an explicit all-feature mask avoids confusing schema stability with observed training missingness. Decide how all-empty training columns are handled and test the resulting dimensions. Do not make an imputer's feature-dropping default part of the interface by accident.
Replay realistic dropout: contiguous gaps, load-dependent loss, shared power/network faults, quality changes and maintenance transitions. Randomly hiding independent cells is useful for debugging, but it does not represent a broken gateway or a sensor that fails under vibration. Never use backward fill, centred windows or future measurements in an online acceptance claim.
Release checklist and operational limits
Before release, require a documented acquisition-confirmation contract; per-feature age budgets; complete required-feature schemas; lifecycle/regime resets; explicit quality and joint-missingness policies; fold-local fitted artifacts; and separate prediction and abstention metrics. Monitor rejection reason, gap age, pattern, regime and model version—not only overall fill rate. Keep raw evidence and the originally served feature vector so later reconciliation cannot rewrite what the model knew.
Exercise restart behavior too. Either restore verified cached confirmation times with matching lifecycle and policy versions, or start unavailable until fresh evidence arrives. Never restore values while replacing their timestamps with process-start time. Bound cache size, expire inactive assets and measure state-store latency against the inference deadline. These operational controls cost availability and maintenance effort; loosening them buys more outputs, not necessarily better decisions.
For an implementation review, bring one asset family, one advisory prediction task and a representative dropout/reconnect trace. I can help turn those into a feature-validity contract, an executable fault matrix and acceptance criteria that compare abstention cost with task error. The first deliverable should establish which missing evidence the model may tolerate—and which uncertainty must remain visible to the operator.
