ServiceNow’s AutoSynthData describes how to turn an agent’s failures into executable synthetic training tasks. For enterprise teams, the useful change is not simply more examples: each task includes an environment and a verifier. Before adopting the approach, test whether that verifier rejects incorrect outcomes and whether improvement survives evaluation on an untouched acceptance set.
What ServiceNow published
On 2 October 2026, ServiceNow CoreAI published AutoSynthData: Generating Training Data for Enterprise Agents. The approach evaluates a target model and a stronger teacher, identifies capabilities the target lacks, generates new tasks around those gaps, and validates them before supervised fine-tuning. The announcement reports experiments in EnterpriseOps Gym, not an independently validated production deployment.
A task is more than a prompt and an ideal answer. ServiceNow defines it as a system specification, a user request and a verifier. The specification describes policies and starting conditions; the request defines the intended work; the verifier checks success. That makes the announcement relevant to teams training agents that change tickets, calendars or other persistent business records.
The underlying EnterpriseOps-Gym paper was submitted on 13 March 2026. It is background, not today’s release. The fresh development is the failure-driven training-data pipeline and the reported post-training experiments. Keeping those dates separate matters: an existing benchmark does not become a new product because a new article links to it.
How the pipeline differs from generic synthetic examples
AutoSynthData first converts observed weaknesses into sanitized capability specification cards. According to ServiceNow, the generator does not receive the original evaluation prompts, entities, trajectories or verifier details. It receives capability descriptions and generates tasks with different states and solution paths.
Generation has two phases. “Target” creates and validates core samples. “Multiply” produces variants from accepted target samples, but a multiplied sample cannot seed another multiplied sample. This restriction limits chains of derivative examples drifting away from their vetted origin. It does not itself establish diversity or eliminate evaluation leakage.
The more consequential detail is the pair of verification gates. Positive verification executes the reference solution and checks that it passes. Negative verification changes relevant parts of an expected outcome and checks that the wrong result fails. Failed candidates receive bounded critique and repair, then repeat the checks. Batch review addresses coverage and repetition rather than assuming individually valid samples form a useful curriculum.
In the reported configuration, candidate tasks are favored when the target solves them on no more than one of three trials and the stronger solver succeeds on at least two of three. These are ServiceNow’s experimental selection rules, not universal acceptance thresholds. Three runs can help find candidate difficulty; they do not precisely estimate an agent’s reliability.
What the reported improvement does—and does not—show
ServiceNow reports generating 2,000 Hybrid training samples in about 18 hours, with Gemma-4-26B-A4B-it as the target and Qwen3.8-27B as the teacher. Its best checkpoint, at epoch five, improved mean Pass@1 by 7.2 percentage points. The separately reported verifier-success measure rose from 63.01% to 68.55%. These are vendor-reported experimental results, not measurements reproduced here, and the two metrics should not be treated as interchangeable.
The ITSM experiment used a different teacher and generated 1,994 samples in 66 hours. The authors attribute the longer generation time primarily to the larger teacher and earlier pipeline implementation. Neither elapsed time is a price quote or a hardware-normalized efficiency comparison. The announcement does not establish a German-language quality gain, an enterprise-wide automation rate or a deployment cost saving.
There is also an evaluation-design boundary. Sanitizing the original tasks reduces direct copying, but the curriculum still learns from failures observed during evaluation. Repeatedly choosing capabilities and checkpoints against the same evaluation environment can adapt the development process to that environment. Engineering recommendation: retain a separate acceptance suite that does not drive generation, repair or checkpoint choice. Use segment-specific model release gates for the final decision, rather than promoting the best aggregate result automatically.
What can you access, and what will it cost?
The publication is readable publicly. The linked EnterpriseOps-Gym repository provides benchmark code, configuration guidance and instructions for domain MCP servers and seeded databases. Its repository license is Apache 2.0. The public dataset contains task definitions and outcome-verification material.
Do not transfer that repository license to everything named in the announcement. The reviewed AutoSynthData article does not establish a separately downloadable AutoSynthData implementation, a hosted service price, a regional service commitment or commercial support terms. Benchmark code availability is not proof that the complete generation pipeline is packaged for deployment. Verify dataset, container, target-model and teacher-model terms independently; procurement and legal reviewers must assess the intended use.
A realistic pilot budget includes teacher and target inference, rejected candidates, repair attempts, environment resets, training compute, acceptance evaluation and maintenance of the environment adapter. A more useful internal measure than generated samples per hour is total generation-and-validation cost per accepted sample, with rejection reasons alongside it. Then measure cost per successful held-out business task before and after training. No numerical cost advantage can be inferred from the published generation times alone.
The production risk: correct final state, unsafe route
Consider an illustrative ITSM task: assign an incident to the authorized support group without changing its priority or exposing its description to another tenant. A weak verifier checks only the final assignment. It could accept an agent that briefly granted unauthorized access and revoked it before the final snapshot.
This example is an engineering threat model, not a reported AutoSynthData incident. It demonstrates why final-state verification and execution-policy enforcement answer different questions. Check the required business outcome against resettable state, but also capture relevant tool events and enforce forbidden actions during execution. A final-state SQL query cannot establish that no prohibited transient action occurred unless the required history is also available and checked.
Conversely, a verifier that demands one exact tool-call sequence can reject a valid alternative solution. Separate outcome invariants from route constraints that actually matter. The practical goal is neither “match the teacher” nor “reach any passing database state”; it is to complete the authorized task without violating protected conditions. The broader production agent evaluation guide supplies the operational metrics around that narrower training-data gate.
Five checks before funding a pilot
1. Confirm that training addresses the actual defect
Start with a recurring failure the target model can plausibly learn to avoid. If the problem is missing current documentation, a broken connector or insufficient permissions, synthetic training does not repair the underlying dependency. Compare against the prompting, RAG and fine-tuning decision framework. Record the failure class and a baseline before generating examples.
2. Freeze the environment and contain its effects
Record tool schemas, policy version, seed-state digest and permitted transitions with each task. Reset state between trials and verify the reset. Keep sandbox credentials separate from production, deny unneeded outbound access and inspect generated verifier code before execution. Synthetic prompts are not a reason to trust arbitrary SQL, code or MCP actions.
3. Test the verifier with deliberate counterexamples
Require a valid solution to pass, then change the object, tenant, required field or protected side effect and require rejection. Also test a valid alternative route. Retain the mutation, expected verdict and observed verdict as evidence. A gate that cannot distinguish these cases must be repaired before its samples become training supervision; passing a finite mutation set is still not proof of complete soundness.
4. Separate curriculum feedback from acceptance evidence
Version capability cards, parent samples, teacher configuration and repair history. Keep multiplied examples traceable to their accepted target parents. Review similarity and task-family coverage across the batch, then evaluate the trained candidate on an untouched set, including German requests and local workflow policies where those are deployment requirements. Do not use that acceptance set to repair the next batch and continue calling it untouched.
5. Set a stop condition and a complete cost ledger
Agree protected failure classes, a generation budget, a repair limit and an owner for inconclusive results. Stop when verifier defects, weak diversity or repeated unsolvable tasks make additional generation uninformative. Promote only after the candidate meets the agreed quality and operational contract. If it does not, retain the failure evidence rather than presenting dataset size as progress.
The next decision is about evidence, not volume
AutoSynthData offers a concrete design for targeted synthetic agent training. Its most useful lesson is that task generation, environment execution and verification must be developed together. For a German enterprise buyer, the next step is a bounded feasibility study: one workflow, one diagnosed weakness, audited verifiers and an independent acceptance decision—not an immediate rollout of a vendor benchmark result.
For help defining that pilot, AI automation consulting can connect the training-data contract to existing tools, data boundaries and release evidence. Bring a representative failure and the constraints that must never be violated; those determine whether more training data is the right investment.
