BLUF
AI agent evaluation in production should not start with a model leaderboard. It should start with the business workflow: what the agent is allowed to do, what a correct outcome looks like, how much each task costs, when the system escalates to a human, and which traces become regression tests. The useful metrics are not only model accuracy. They are cost per completed task, tool-call accuracy, escalation rate, latency, correction rate, and the quality of the audit trail.
For B2B teams, this is the difference between a demo and an operational system. A chatbot can be judged by answer quality. An AI agent must be judged by the actions it takes, the tools it calls, the evidence it uses, the errors it avoids, and the cases it safely refuses or escalates.
Why production agent evaluation is different
Classic LLM evaluation often asks: did the model produce a good answer? Production agent evaluation asks a harder question: did the system complete the workflow safely, cheaply, and with enough evidence to trust or audit the result?
An agentic workflow can include retrieval, tool calls, CRM updates, ticket routing, email drafting, approval gates, database reads, and human review. A single final answer does not reveal whether the intermediate steps were correct. The agent may produce a plausible summary while calling the wrong tool, using stale context, skipping an approval step, or consuming five times the expected tokens.
This is why production evaluation needs traces. LangSmith separates offline evaluation before deployment from online evaluation on production traces. Offline evals use curated datasets and reference outputs. Online evals monitor live runs and threads for quality patterns, safety issues, anomalies, and feedback loops. That distinction maps well to enterprise operations: test before release, monitor after release, then convert failures into new regression cases.
The metric stack that matters
| Metric | What it tells you | Why it matters in production |
|---|---|---|
| --- | --- | --- |
| Task success rate | Whether the workflow completed the intended job | Better than generic answer quality for agents |
| Tool-call accuracy | Whether the agent selected the right tool and arguments | Wrong tool calls create operational risk |
| Cost per completed task | Token, model, API, and infrastructure cost per successful outcome | A system can be accurate but economically unusable |
| Escalation rate | How often the agent hands off to a human | Too low can mean hidden risk; too high can mean poor automation |
| Correction rate | How often humans edit or reverse the agent output | Measures trust and operational load |
| Latency | Time from request to usable result | Critical for support, operations, and incident workflows |
| Refusal / safe-stop rate | How often the system declines unsafe or uncertain actions | Important for governed workflows |
| Audit completeness | Whether traces, evidence, reviewer, and decisions are logged | Required for debugging and compliance preparation |
The goal is not to optimize every metric at once. Lower escalation can increase risk. Higher accuracy can increase cost. Faster responses can reduce evidence quality. The evaluation system should make these trade-offs visible.
Offline vs online evals
Offline evals are for pre-deployment confidence. They use curated examples, historical traces, synthetic cases, and reference outputs. They are useful for regression tests, prompt comparisons, tool schema changes, RAG retrieval changes, and model upgrades.
Online evals are for production monitoring. They run on real traces, often without reference outputs. They detect drift, unsafe outputs, failed tool calls, unusually expensive runs, latency spikes, and recurring edge cases. Good online evals do not replace human review; they route the right traces into annotation queues and offline datasets.
| Evaluation mode | Best use | Data | Typical evaluator |
|---|---|---|---|
| --- | --- | --- | --- |
| Offline eval | Pre-release regression | Curated examples and expected outputs | Code rules, human labels, LLM-as-judge, pairwise comparison |
| Online eval | Production monitoring | Live traces without reference output | Safety rules, format checks, cost thresholds, reference-free quality checks |
| Human review | Ambiguous or high-risk cases | Evidence packet and trace | Structured annotation, approval, rejection, correction |
| Backtesting | Validate new version on old cases | Historical production traces | Regression score and failure replay |
Cost per task: the ignored metric
Many teams track token usage but not cost per useful outcome. That hides the real economics. If an agent costs €0.40 per run but only completes 40% of tasks without human correction, the cost per successful automated task is closer to €1.00 before staff review time is included.
A practical formula:
| Component | Example measurement |
|---|---|
| --- | --- |
| Model cost | tokens in/out per run |
| Tool/API cost | search, OCR, CRM, vector DB, external APIs |
| Infrastructure cost | queue, workers, storage, observability |
| Human review cost | minutes per approval or correction |
| Failure cost | rework, duplicate tickets, wrong actions, SLA impact |
The useful KPI is not "cost per LLM call". It is:
Cost per completed task = total workflow cost / number of accepted successful outcomes
That number lets leadership compare agent automation against manual processing, classic workflow automation, and partial-assist tools.
Accuracy is not one number
For agents, accuracy has layers.
| Layer | Evaluation question |
|---|---|
| --- | --- |
| Intent understanding | Did the system classify the user/task correctly? |
| Retrieval | Did it fetch the right documents or records? |
| Tool selection | Did it choose the right tool? |
| Tool arguments | Did it pass the right fields and constraints? |
| Reasoning path | Did the trace follow an acceptable workflow? |
| Final answer | Was the response factually correct and useful? |
| Action outcome | Did the business system end up in the right state? |
Ragas exposes agent-oriented metrics such as tool-call accuracy, tool-call F1, topic adherence, and agent goal accuracy. Those are useful categories because they evaluate the agent as a workflow actor, not only as a text generator.
Escalation rate: safety signal, not failure signal
Escalation is often misread as automation failure. In production AI systems, escalation is a control mechanism. A healthy agent escalates when evidence is missing, permissions are insufficient, a customer-impacting action is requested, or the confidence/cost/risk threshold is exceeded.
The dangerous pattern is not a high escalation rate. The dangerous pattern is an agent that rarely escalates but later creates hidden errors, wrong updates, or unreviewed customer impact.
| Escalation pattern | Interpretation | Action |
|---|---|---|
| --- | --- | --- |
| High escalation, high correction quality | Good safety, weak autonomy | Improve retrieval, tool schemas, and examples |
| Low escalation, high downstream errors | Hidden risk | Add approval gates and stricter thresholds |
| High escalation, low human correction | Over-cautious automation | Tune confidence thresholds |
| Rising escalation after deployment | Drift or workflow change | Add failed traces to offline dataset |
Logging and EU AI Act readiness
This is not legal advice. But engineering teams should design evaluation systems so that logs, human oversight, and documentation are not added as an afterthought.
EU AI Act Article 12 requires high-risk AI systems to technically allow automatic recording of events over the lifetime of the system. Article 14 addresses human oversight. Even when a specific enterprise workflow is not high-risk, the same engineering pattern is useful: record what happened, why the system acted, who reviewed it, and what evidence was available.
For production agents, log at least:
| Log field | Why it matters |
|---|---|
| --- | --- |
| Run ID and timestamp | Traceability |
| User/request context | Debugging and replay |
| Retrieved evidence | Explainability and correction |
| Tool calls and arguments | Operational accountability |
| Model/version/prompt | Regression analysis |
| Cost and latency | Operational economics |
| Escalation reason | Safety and improvement loop |
| Human reviewer and decision | Oversight record |
| Final outcome | Success/failure measurement |
A practical 30-day implementation plan
Days 1-7: define success and failure
Pick one workflow. Do not start with all agents. Define successful outcomes, unacceptable actions, escalation criteria, target latency, and cost ceiling. Create 20-50 representative test cases from real operations.
Days 8-14: add traces and offline evals
Instrument the workflow so every run captures inputs, outputs, tool calls, evidence, latency, cost, and final outcome. Build offline evals for tool selection, output format, retrieval quality, and business outcome.
Days 15-21: add online monitoring
Add production monitors for high cost, repeated tool failure, missing evidence, long latency, policy violations, and escalation patterns. Sample live traces for human review.
Days 22-30: close the feedback loop
Move failed production traces into the offline dataset. Add regression tests before each prompt, model, or workflow change. Review cost per accepted successful task weekly. Keep the eval dashboard small enough that operations teams actually use it.
Recommended operating dashboard
| Dashboard section | Minimum widgets |
|---|---|
| --- | --- |
| Quality | task success, correction rate, evaluator pass rate |
| Risk | escalations, refusals, policy violations, high-risk tool calls |
| Cost | cost per run, cost per accepted task, cost by tool/model |
| Latency | p50/p95 end-to-end latency, slow tool calls |
| Feedback loop | new failed traces, reviewed samples, regression cases added |
A good dashboard should support operational decisions: ship, roll back, raise approval threshold, change retrieval, update the tool schema, or add more examples.
The implementation implication
Agent evaluation is not a reporting layer. It is part of the architecture. If traces, costs, escalation reasons, and human decisions are not captured during execution, they cannot be reconstructed reliably later.
For consulting and internal AI teams, the recommended sequence is:
1. Build the workflow with trace IDs from day one.
2. Define success metrics before changing models.
3. Start with offline evals on curated cases.
4. Add online evals for live monitoring.
5. Route ambiguous cases to human review.
6. Feed failures back into the regression dataset.
This is how AI agents move from impressive demos to systems that operations, compliance, and management can trust.
Sources
- LangSmith Evaluation: https://docs.langchain.com/langsmith/evaluation
- LangSmith Evaluation Concepts: https://docs.smith.langchain.com/evaluation
- Ragas Metrics for Agents and Tool Use: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
- EU AI Act Article 12 — Record-Keeping: https://artificialintelligenceact.eu/article/12
- EU AI Act Article 14 — Human Oversight: https://artificialintelligenceact.eu/article/14
