BLUF: An unseen machine-state code must not become the same feature vector as a known operating mode. Validate the raw vocabulary and its firmware namespace before categorical conversion, preserve missingness separately, and bind the encoder's ordered columns to the model release. An explicit unknown indicator makes uncertainty visible; it does not establish that the model may safely score the observation.
The executed synthetic fixture below demonstrates a concrete collision: OneHotEncoder with drop='first' and handle_unknown='ignore' represents the reference mode, an unseen code and an unregistered missing marker as the same all-zero vector. Removing the dropped column resolves the reference collision but still leaves unknown and missing indistinguishable. This is a feature-admission problem, not a reason to assign arbitrary numerical distances to operating modes.
A firmware change that passes the numeric schema
Assume an illustrative machine vocabulary IDLE, RUN and SERVICE. A firmware revision adds PURGE. These are application examples, not a normative OPC UA state list. If the old encoder silently ignores PURGE, the resulting vector can have the expected shape and dtype while representing the wrong operational context. A numeric schema check alone cannot detect that semantic loss.
The industrial data-contract foundation establishes identity, quality and semantic ownership. Here the narrower decision is whether this categorical observation belongs to the vocabulary approved for this model. Keep the original code, namespace, asset lifecycle, firmware revision, source timestamp and quality evidence outside the feature vector. A label such as RUN is not globally meaningful across vendors or firmware families.
Unknown means a present value is outside the approved vocabulary. Missing means no usable value was supplied. Invalid type, rejected source quality and unapproved firmware are additional rejection reasons, not aliases for unknown. Check these conditions before converting to pandas categorical dtype or model-ready arrays; otherwise conversion can erase the distinction that the gate needs.
What the executed fixture actually produced
The run used Python 3.14.9, scikit-learn 1.9.1, NumPy 2.5.4 and pandas 3.1.1. Training data contained the three known modes. The collision encoder used explicitly ordered categories [IDLE, RUN, SERVICE], drop='first', handle_unknown='ignore' and sparse_output=False. The missing input was deliberately mapped to the string __MISSING__, which was absent from that encoder's categories.
The comparison encoder declared all five categories [IDLE, RUN, SERVICE, __UNKNOWN__, __MISSING__], used drop=None and handle_unknown='error', and mapped present out-of-vocabulary values to __UNKNOWN__ before transformation. Its missing branch was separate. Reserved markers belong to the adapter contract: a producer must not be allowed to send those marker strings as ordinary machine states.
| Raw input | Dropped reference output | Explicit five-category output |
|---|---|---|
| IDLE | [0,0] | [1,0,0,0,0] |
| RUN | [1,0] | [0,1,0,0,0] |
| PURGE (unknown) | [0,0] | [0,0,0,1,0] |
| None (missing) | [0,0] | [0,0,0,0,1] |
The two-column collision output is ordered RUN, SERVICE. The five-column output is ordered IDLE, RUN, SERVICE, __UNKNOWN__, __MISSING__. Explicitly declaring unobserved marker categories preserves the schema, but a model trained only on known states has no learned evidence for those marker cases. Do not confuse representability with task validation.
For a minimal reproduction, fit OneHotEncoder(categories=[['IDLE','RUN','SERVICE']], drop='first', handle_unknown='ignore', sparse_output=False) on [['IDLE'],['RUN'],['SERVICE']], then transform [['IDLE'],['RUN'],['PURGE'],['__MISSING__']]. The output is [[0,0],[1,0],[0,0],[0,0]]. The executed assertions compare exact vectors, not just their dimensions. The OneHotEncoder reference documents unknown handling and the effect of dropped categories.
Additional executed assertions covered strict rejection of PURGE, preservation of the five-column output after serialization/reload, detection of reversed category order, and the no-drop encoder's remaining unknown/missing collision. This fixture tests encoding semantics only. It does not train a predictor, measure maintenance accuracy, exercise an actual firmware rollout or validate a physical safety function.
Choose an admission policy, not just an encoder option
| Observation | Feature action | Admission decision |
|---|---|---|
| Known code; approved lineage | Use frozen ordered vocabulary | Eligible subject to model policy |
| Present unknown code | Preserve raw; unknown indicator | Quarantine or validated fallback |
| Missing mode | Separate missing indicator | Do not invent operating context |
| Invalid quality or firmware | Retain distinct rejection reason | Reject before encoder |
| Rare known mode | Keep identity or reviewed grouping | Do not equate rarity with unknown |
| Changed columns or manifest | Stop incompatible bundle load | Release model and encoder together |
Strict quarantine is the conservative default for an unreviewed firmware change. It reduces advisory availability and requires a responsible owner for rejected observations. An unknown indicator can support a separately validated degraded path, but is not permission to reuse the normal operating-mode model without evidence. Keep reason codes and admission decisions in the prediction envelope even when indicators are also model inputs.
Missing-mode handling is distinct from filling a continuous sensor channel. The bounded-imputation workflow explains why a carried value needs a validity budget and regime boundaries. Blindly carrying RUN through a missing mode signal can manufacture the very operating context needed to decide whether another sensor may be used. A valid temperature or complete vector cannot supply missing regime evidence.
Infrequent is not unknown, and integer is not physical order
With handle_unknown='infrequent_if_exist' and min_frequency=2, the fixture fitted IDLE twice, RUN twice and SERVICE once. SERVICE and unseen PURGE then received the same infrequent representation. In a second fit containing only twice-observed IDLE and RUN, no infrequent category existed; PURGE became an all-zero vector. Grouping rare modes is therefore not a dedicated unknown-state detector. Preserve an independent raw-vocabulary check if rare known and unseen must remain distinguishable.
OrdinalEncoder can keep sentinels separate. The executed fit included IDLE, RUN, SERVICE and np.nan, with handle_unknown='use_encoded_value', unknown_value=-1 and encoded_missing_value=-2. Transforming IDLE, PURGE and np.nan returned 0, -1 and -2. The missing category was represented during fitting; do not assume the same missing branch applies when the fitted vocabulary never included it. Test the exact input types and fitted artifact you will deploy.
The OrdinalEncoder documentation specifies these sentinel controls. They encode identity, not a physical ordering or distance. Feeding IDLE=0 and SERVICE=2 into a distance-based or linear model can impose an interpretation the machine contract never provided. Sentinel separation alone does not make that model formulation appropriate.
The pandas test exposed an earlier information-loss boundary: pd.Categorical(['IDLE', 'RUN', 'PURGE', None], categories=['IDLE', 'RUN', 'SERVICE']) produced codes [0,1,-1,-1]. Unknown and missing were already indistinguishable in those codes. pandas 3.1.1 also emitted a deprecation warning that constructing a categorical with out-of-category non-null values will raise in a future version. Validate before casting and test upgrades; do not depend on this permissive behavior. The pandas categorical guide describes category membership, ordering and missing-value representation.
Release the vocabulary, encoder and model together
Use a feature-schema manifest containing namespace, accepted firmware revisions, vocabulary revision, ordered raw values, reserved markers, ordered output column names, missing/unknown policy, encoder configuration and model-bundle identity. The fixture's manifest records mode-v1 and the five mode_* column names. Treat its fw-a and fw-b identifiers as synthetic examples, not device compatibility claims.
Migration diagram: raw event plus firmware lineage → validate source envelope → compare approved vocabulary → known path to frozen encoder → model admission. The unknown branch goes to quarantine and a reviewed vocabulary proposal, not directly back into the old model. The proposal proceeds through shadow replay → task validation → atomic encoder/model/manifest release → monitored rollout with rollback to a compatible bundle. Missing and invalid-envelope branches retain their own reason codes.
Column count is not enough. Reversing the declared category order in the fixture changed feature names while preserving dimensionality. Compare the exact ordered names and schema fingerprint at startup and before loading a replacement model. Fit only on the training fold, never on each inference batch. Persist preprocessing with the model; refitting after restart can change meaning without changing the API shape. The local serialization test used only a self-created pickle artifact; never deserialize untrusted pickle files.
Acceptance checklist and operating costs
Before release, require fixtures for every approved code, an unseen firmware code, missing input, invalid type, reserved-marker collision, rejected quality, an unsupported firmware revision, reordered columns and artifact reload. The executed suite covers the encoder cases; the envelope and firmware checks in this checklist still require integration tests against your adapter. Include sparse/dense compatibility and memory checks for the actual cardinality: this tiny dense example does not establish a production memory budget.
Monitor accepted, unknown, missing and invalid observations with a denominator per governed asset family and release. Keep arbitrary raw codes out of metric labels; retain bounded diagnostic samples in an access-controlled evidence store. Review quarantine age and ownership. A permanently growing unknown queue is not a successful fail-closed design, and a zero unknown count is meaningless if the adapter replaced unknowns before measuring them.
This adds schema ownership, replay storage and coordinated rollout work. Strict rejection lowers prediction coverage; rare-category grouping lowers dimensions at the cost of distinctions; explicit unknown features add visibility without proving accuracy. The temperature-feature contract addresses a complementary failure: a valid numeric representation can still mean the wrong physical quantity. Neither gate grants an AI system authority over machine control.
For an implementation review, bring one machine-state definition, the firmware compatibility matrix and a read-only trace around a state-code change. I can help turn these into an executable vocabulary gate and a model-release contract. The first useful result is a clear separation between a known operating mode, missing evidence and a state the deployed model has never been approved to interpret.
