Evals for actions, not answers

A benchmark is not a model leaderboard. It is a record of completed runs: the agent executed a standardized job against real systems, left an auditable trail, and passed the spec. If your eval cannot see the ERP write, it is not an eval of enterprise work. It is an eval of prose.

Model rankings and synthetic tool-call suites measure useful components, but they do not tell a controller or procurement leader whether a workflow finished inside policy. That requires execution metrics tied to the job: correct system changes, complete sequences, recovery from failure, and constraint adherence.

What the feedback layer actually collects

• Production traces — tools called, parameters, systems touched, timestamps, identity.
• Human corrections — the override, the rejected write, the approval that should have been asked for.
• Failure modes — partial updates, schema mismatches, policy misses, abandoned sequences.
• Edge cases — recurring tenant-specific exceptions that should become explicit execution rules.

These records improve the execution graph by changing the behavior applied to the next instance of the same job class. Cross-context replication is the relevant signal: the same job specification should pass across unlike businesses and technology stacks.

Subjective jobs, empirical readiness

Domain experts define the job specification, including acceptable outcomes and policy constraints. Individual checks should be observable wherever possible: the correct objects changed, approvals recorded, exceptions resolved, and completion state reached. Repeating the same specification across tenants then tests portability rather than one-off configuration.

The same suite should also run after a model change. Stable tool execution, recovery, and completion show which capabilities belong to the execution layer; regressions identify where the new inference model changes operational behavior.