Inspection Label Disagreement: Audit the Reference Before Scoring the Model

BLUF: Before scoring an industrial inspection model, test whether independent reviewers can reproduce the inspection reference. Preserve their first decisions, repeat a blinded subset, and investigate disagreement before adjudication. Agreement measures consistency—not correctness, a model-accuracy ceiling, or permission to release parts. A model can disagree with a bad reference and still be right; two reviewers can agree and still miss the same defect.

This implementation pattern concerns one categorical, part-level inspection decision under a named specification. The numerical fixture below was executed on synthetic counts, not factory observations. Domain quality owners must define defect severity, sampling and release criteria. Safety validation and any required legal assessment remain separate.

Define the decision before recruiting reviewers

Write down the inspected product family, surface region, allowed views, lighting envelope and conformance rule. A scratch above a dimensional limit is a different target from any visible surface mark. Freeze the rubric revision and include boundary examples with an explanation of which measurement settles the case. Do not turn a vague visual preference into a binary target merely because your annotation tool requires a choice.

Keep four concepts separate: the categorical disposition, defect type and severity, image inspectability, and unresolved uncertainty. An out-of-focus image is not a conforming part. An inspectable but ambiguous feature is not automatically a defect. Retain explicit unresolved and uninspectable states, with reasons and an authorised hold or alternative-inspection route. Calculate binary agreement only on the declared eligible subset, and publish the excluded counts beside it.

This is upstream of calibrating inspection scores and reject thresholds. Calibration assumes a meaningful target already exists; it cannot repair a reference whose meaning changes between reviewers.

Build a blinded, repeated-label study

Select a representative production sample across relevant lots, shifts, finishes and camera conditions. Add a separately identified challenge sample for rare defects and boundary cases. Do not pool an enriched challenge set into a production prevalence claim. There is no universal adequate sample size: the quality owner must specify which consequential disagreements the study needs to detect and how much uncertainty is acceptable.

Give two qualified reviewers the same approved evidence independently. Hide model predictions, historical dispositions and the other reviewer's answer. Randomise order. Repeat a masked subset after a justified interval, with new presentation IDs so memory and sequence cues are reduced. Keep the stable part ID privately in the study manifest. Do not present another image of the same part as an independent new part in the report.

Between-reviewer reproducibility compares A's first decision with B's first decision on matched parts. Within-reviewer repeatability compares A's first and repeat decisions, then does the same for B. A shared training session may improve consistency but also create shared mistakes; record training and rubric revisions rather than interpreting agreement as independent validation.

Store part ID, lot ID, image digest, view configuration, rubric revision, reviewer pseudonym, round, categorical decision, defect severity, inspectability, uncertainty reason and timestamp. Preserve an immutable first-decision record. Store adjudication as a new linked event, not an overwrite. The data-lineage and quality-gate pattern provides a broader foundation for that evidence trail.

Read the disagreement matrix before reading kappa

The executed imbalanced fixture contains 100 paired eligible decisions. Rows are reviewer A and columns reviewer B. Neither axis is ground truth. Five parts move in each disagreement direction; those ten parts need investigation, not automatic assignment to the senior reviewer's class.

The scikit-learn confusion-matrix API supplies the counting convention: rows follow the first input and columns the second, with an explicit class order. Its parameter names refer to truth and predictions, but passing two reviewers does not make the first reviewer truth. Use the caption “reviewer contingency table,” not model false-positive or false-negative counts.

Raw agreement is the diagonal count divided by paired eligible parts: 90/100 = 90%. Retain the off-diagonal counts by direction, defect type, severity and production slice. The same total disagreement can mean a disputed cosmetic finish or a disputed structural crack; the operating response should not be identical. For multiple defect classes, report the complete class-pair counts instead of hiding which labels are confused.

Same agreement, very different kappa

Cohen's kappa is (p_o − p_e)/(1 − p_e). Here p_o is observed agreement, and p_e is calculated from the two reviewers' empirical class marginals. The synthetic fixture computes these quantities using exact rational arithmetic before converting to display values.

With matrix [[45, 5], [5, 45]], both reviewers assign each class to half the parts: p_o = 0.90, p_e = 0.50 and kappa = 0.80. With the displayed imbalanced matrix [[89, 5], [5, 1]], p_o remains 0.90, but p_e = 0.8872 and kappa is approximately 0.1135. The disagreement count has not changed; the marginal class distribution has. Neither result justifies a universal “good kappa” threshold or a conclusion about model accuracy.

The third executed case is [[100, 0], [0, 0]]. Both reviewers label everything pass: observed agreement is 100%, p_e is 1 and kappa is undefined because its denominator is zero. The fixture reports null with an explicit undefined interpretation, not zero or perfect kappa. The retrieved scikit-learn documentation exposes replace_undefined_by in version 1.9; check your installed version before using that argument. Changing an undefined-value setting cannot manufacture evidence about a defect class that was never observed.

For reproducibility, the fixture's core is: n = sum of all cells; p_o = diagonal sum / n; p_e = sum of row_total[class] × column_total[class] / n²; return undefined when p_e = 1, otherwise (p_o − p_e)/(1 − p_e). Assertions check both numerical examples and the undefined case. These point estimates do not quantify sampling uncertainty. For a real study, account for repeated parts and lot dependence when selecting an uncertainty method; treating every view as independent overstates evidence.

Turn findings into a disposition decision

Adjudication should review the conflicting evidence and specification, not merely count votes. Use an accountable domain specialist and, where appropriate, a traceable dimensional measurement or other independent inspection method. Preserve who decided, the reason, supporting evidence and whether the result remains uncertain. A majority can share the same blind spot; a measurement route can also have calibration and resolution limits.

Never discard difficult cases just to improve agreement. Report how many were resolved, remained unresolved, or required different evidence. If an adjudicated subset excludes ambiguous parts, the eventual model claim applies only to that eligible population. The operational system still needs a safe route for excluded cases.

Do not substitute a platform consensus score

Label Studio Enterprise's task-agreement documentation distinguishes pairwise agreement, consensus, per-control-tag scores and overall aggregation. Its pairwise categorical exact-match average is not Cohen's chance-adjusted kappa. An overall score can also mix different annotation controls with configurable weights. Export the metric definition and configuration with the result; a dashboard percentage without those details is not reproducible evidence.

Bounding boxes and segmentation require an explicit region-matching and geometric-similarity protocol, such as overlap-based comparison, including unmatched objects and class mismatches. Do not apply part-level categorical kappa directly to coordinates or treat agreement on “defect present” as agreement on defect boundaries. Ordinal severity ratings may justify weighted kappa, but the weighting encodes a consequence assumption and must be justified before seeing results.

Freeze the reference, then evaluate the model

Release checklist — First, approve the inspection specification, reviewer qualifications, sampling frame and exception routes. Second, retain independent first decisions and repeatability results. Third, inspect directional disagreements and critical slices with their denominators, not just aggregate agreement. Fourth, record adjudication and unresolved cases without erasing history. Fifth, freeze the reference manifest and its rubric before model acceptance testing.

Use development data to revise the rubric and train reviewers. Once that process stabilises, apply the frozen procedure to a held-out acceptance cohort with model outputs hidden during reference construction. Do not tune the model or its thresholds on acceptance failures and then reuse the same cohort as independent proof. If the rubric changes materially, reassess the affected acceptance evidence under change control.

Bind reference version, model, preprocessing and acceptance report into the manufacturing model release and rollback bundle. Reopen review when cameras, finishes, specifications or reviewer procedures change. This adds annotation and adjudication cost, but makes a failed reference visible before it becomes a misleading model-performance claim.

A practical first engagement is one product family and one disputed defect definition. Bring the inspection rubric, blinded paired decisions, lot identities and current hold procedure. I can help structure a reference-label audit and release-evidence review. The useful outcome may be a clarified specification or an independent measurement route—not a higher model score.