AI Robustness Evidence: Define the Threat Model Before Reporting Attack Success

A robustness percentage is not release evidence until the report says who can change what, within which constraints, and which examples enter the denominator. Define that contract before running an attack. Then report clean accuracy, eligible coverage, valid perturbations, attack-induced errors and abstentions separately. An unsuccessful attack is evidence about that test—not proof that the model is robust.

This article develops a defensive, local classifier evaluation with an executed synthetic fixture. It is not a test against a third-party service, a production benchmark or a conformity assessment. Its purpose is to expose a reporting error: counting predictions that were already wrong as successful attacks, or hiding incomplete evaluation behind a reassuring percentage.

Start with the protected decision, not the attack library

Consider an inspection classifier that sends an item to acceptance or review. The protected outcome might be “a defective item must not be automatically accepted.” That is narrower than overall accuracy. An untargeted attack asks whether any wrong class can be induced; a targeted attack asks whether a specified harmful outcome can be induced. Keep those objectives separate, particularly when false acceptance and unnecessary rejection have different costs.

The NIST adversarial machine-learning taxonomy distinguishes attack lifecycle stages, goals, capabilities and knowledge. Use those dimensions to write a testable threat model. Evasion at inference time is not training-data poisoning, confidentiality leakage or prompt injection. A successful evasion test says nothing by itself about those other surfaces.

Establish ownership or explicit authorization for the model, data and evaluation environment. Freeze the data contract and label provenance before generating variants; data governance and lineage controls provide the foundation. A perturbation that changes the true class is not a valid label-preserving attack merely because its numerical distance is small.

Threat-model worksheet and stop decisions

Fill this worksheet before selecting the implementation. The rightmost column defines when a result must be held for review rather than converted into a score.

Numerical and physical constraints are not interchangeable. A change of 0.05 after normalization might correspond to a different raw-unit change for each feature. Record the preprocessing transform and evaluate constraints in the authorized coordinate space. For real measurements, bounded independent feature changes may violate temporal continuity, conservation relationships or sensor coupling. A domain reviewer must establish whether the allowed set represents plausible manipulation rather than mathematical convenience.

An executed synthetic fixture with explicit scope

The fixture uses two dimensionless latent values drawn uniformly from 0.1 to 0.9. The label is one when their sum exceeds one. Independent Gaussian measurement noise with standard deviation 0.025 produces observed features, clipped to the interval [0,1]. Labels remain attached to the latent item; perturbing the observation does not relabel the item. This is a deliberately artificial measurement model, not evidence that an industrial sensor permits the same manipulation.

Using seed 63, the script generates 3,000 rows, fits logistic regression on the first 2,000 and evaluates the remaining 1,000. There is no fitted scaler: preprocessing is the identity transform. The estimator uses C=1, the lbfgs solver and max_iter=1000. The executed environment is Python 3.12.13, NumPy 2.2.6, scikit-learn 1.6.1, SciPy 1.15.3 and Adversarial Robustness Toolbox 1.20.1. These are pinned fixture versions, not a recommendation to track a floating latest release.

First, a non-adversarial baseline adds independent uniform noise between −0.05 and +0.05 to both observed features and clips to [0,1]. Second, ART FastGradientMethod applies one untargeted white-box step with an L-infinity budget of 0.05 to initially correct examples. Configuration: eps=0.05, eps_step=0.05, minimal=False, num_random_init=0, batch_size=128 and y=None. ART uses model predictions as attack labels in that configuration; eligibility is established separately against the reference labels. The ART attack documentation describes these parameters; the pinned implementation fixes the tested behavior.

The target is entirely local: no remote queries. Instrumentation recorded 959 sample-level gradient evaluations and 1,918 sample-level prediction evaluations inside attack generation, including the estimator's gradient implementation. These are samples processed, not API-request counts; clean scoring and final reporting are additional work. A score-only or label-only threat needs a separate query budget and attack implementation, not a renamed white-box result.

Keep the denominator visible

Clean accuracy was 959/1,000, or 95.9%. The other 41 examples were already incorrect and cannot count as new attack successes. Every generated candidate passed finite-value, [0,1] range and per-feature delta checks with a numerical tolerance of 0.000001. Separate validator tests rejected NaN, out-of-range and oversized-delta inputs. That validates the synthetic numeric contract—not physical plausibility in a plant.

The corruption column reports a conditional error-induction rate, not an adversarial success rate. Its post-test accuracy includes initially wrong examples that became correct, which explains why it differs from the clean-and-tested-correct count. In the FGM column, initially wrong examples were left unchanged. Calling all 229 post-test errors “attack successes” would report 22.9% over the full set instead of the actual conditional attack success rate of 188/959, or 19.60%.

For this fixture, empirical remaining robust accuracy means the fraction correct both clean and after the specified valid test. It is not certified worst-case accuracy over every permitted perturbation. A one-step search can miss failures. For broader evaluation, take the union of failures across compatible attacks on the same frozen examples rather than averaging attack success rates. Preserve separate budgets and objectives when they are not comparable.

For example, held-out row 8 changed from approximately (0.862541, 0.175712) to (0.812541, 0.125712). Its reference label stayed one; the prediction changed from one to zero. That example links a counted failure to the actual permitted change. Retain successful and unsuccessful cases, not just the final percentage.

Invalid attempts, missing coverage and abstention

Define the reporting masks explicitly: eligible means the clean prediction matches the reference and the example is within scope. Valid means the generated candidate passes the threat-model constraints. Success means eligible, attempted, valid and wrong under the declared attack objective. Conditional attack success divides successes by eligible valid attempts, while eligible coverage and invalid or missing attempts are reported alongside it. If that denominator is zero, report “not evaluated,” not zero-percent success.

Do not label uncovered or invalid examples robust. The fixture has complete eligible coverage and no invalid attempts; a production evaluator must instead show unresolved counts and block a release claim if the required coverage is missing. Pre-register the handling of timeouts, tool failures and invalid candidates so that a weak generator cannot improve the score by failing more often.

An additional diagnostic applied a maximum predicted-probability threshold of 0.6 to the FGM output set, routing lower-confidence cases to review. It abstained on 176/1,000 examples and still accepted 118 wrong predictions. The threshold is illustrative and uncalibrated. It changes coverage and review workload; it does not erase failures. This was a post-hoc policy measurement, not an adaptive attack against the rejection policy. Re-test the full defended decision path before claiming a mitigation works, including attacks that seek high-confidence wrong acceptance and the consequences of review overload.

Bind the evidence to the release

The diagram separates the measurement path from the release decision. Build a manifest containing the threat-model revision, data and label digests, preprocessing configuration, model digest, random seeds, package lock, attack parameters, work counters, validity rules and per-example results. The run must remain attributable even if the model registry or package defaults change. Link this bundle into technical documentation and logging, not into an unversioned dashboard screenshot.

The executed fixture retained model, input and script SHA-256 digests plus failure examples. Reproduction should regenerate the same data, check those digests and rerun the denominator assertions. If a library upgrade changes results, preserve both runs and investigate before replacing the reference. Do not treat a hash as a quality guarantee: it establishes artifact identity, not valid labels or a realistic threat model.

What Article 15 does—and this test does not—establish

The Commission's Article 15 text addresses accuracy, robustness and cybersecurity for high-risk AI systems, including resilience to errors and, where appropriate, controls for AI-specific attacks such as evasion and poisoning. It also addresses lifecycle consistency and declared accuracy metrics. Those are evidence questions an engineering team should be able to answer; they are not a prescribed universal FGM test or pass percentage.

The cited Service Desk page explicitly reproduces the original Official Journal version. It must not be treated as confirmation of the currently applicable consolidated law. Qualified counsel must verify classification, provider/deployer duties, current amendments, application dates and conformity implications for the actual system. This article provides engineering guidance only, not legal advice, certification or a claim that one benchmark satisfies Article 15.

Release checklist and operational trade-offs

Check 1: approve the protected outcome, access model, editable surface and physical constraint rationale before running the evaluation. Check 2: freeze reference labels, model and preprocessing. Check 3: run corruption baselines, validate candidates and reconcile every eligible example. Check 4: review harmful failures by class and consequence. Check 5: test mitigations against an adaptive evaluator and measure clean-quality loss, added latency and review capacity. Check 6: record the reviewer and unresolved limitations in the release decision.

Stronger searches cost more compute and maintenance; rejection policies consume human capacity; adversarial training may trade clean performance against tested robustness. None of these costs can be settled by the fixture above. Use paired runs under the same threat contract and non-compensatory release gates: a cheaper model or higher average accuracy must not offset a protected-outcome breach.

A focused implementation review can turn one existing classifier report into a reproducible threat-model worksheet, denominator ledger and release gate. Bring the current decision policy, one authorized evaluation dataset and a failed or disputed test report. The useful deliverable is a defensible decision about what remains untested—not another isolated robustness percentage.