A benchmark for enterprise agents must observe the full job: current state, tools selected, controls enforced, partial failures recovered, resulting systems verified, and completion evidence preserved.
ESE-1 specifies standardized jobs, typed tool surfaces, state perturbations, hard failures, terminal checks, and trace requirements. Measured results publish with trial counts, exact model and runtime versions, and enough evidence to reconstruct the score.
ESE-1 · Protocol v0.1
Spec only · no certified resultsThe protocol is public before the results. Job contracts, scoring semantics, model-swap controls, trace requirements, and runtime invariants define what evidence must exist before a benchmark claim can publish.
ESE-1 · protocol v0.1
Long-horizon jobs, heterogeneous tools, scheduled state perturbations, and conjunctive pass/fail scoring.
Read the benchmark specification →Paired evaluation protocol
Hold the execution system fixed, change the inference model, and measure whether the completed job survives.
Inspect the comparison method →Trace schema · redaction rules
The minimum evidence required to reconstruct state binding, tool calls, writes, recovery, and completion.
Inspect a trace contract →Runtime specification
Transaction boundaries, policy gates, idempotency, verification, and resumable recovery.
Read the runtime protocol →This synthetic record shows the scorecard shape. Public benchmark results require the fuller trace and reporting package defined by ESE-1.
Illustrative specimen · not a result
Close exception to posted journal
Every scored benchmark follows the same execution path—from job spec to systems updated to pass/fail logged.
Version the terminal state, initial records, tool surface, invariants, perturbations, budgets, and scorer.
Load one declared tenant fixture or live evaluation boundary with versioned ERP, supply-chain, or treasury state.
Run the complete job through checkpoint binding, policy gates, typed tool calls, and authoritative verification.
Change declared records, delay tools, or create uncertain side effects at fixed checkpoints in the run.
Apply conjunctive terminal, tool, state, control, recovery, and evidence checks. Any hard failure fails the run.
Publish the release versions, all outcomes, uncertainty, and traces required to reconstruct each score.
Non-normative summaries of three ESE-1 jobs. They explain the operational target; measured results require the complete versioned contract, trace, and reporting package.
Before a run counts toward the published benchmark, it must clear these checks. No partial credit for a clever single step.
These failures cannot be offset by a strong plan, correct prose, or successful steps elsewhere in the run.
Unauthorized, out-of-policy, or wrong-entity write
Duplicate irreversible action after retry or resume
Claimed completion without authoritative read-back
Hidden unresolved exception or unreported invariant violation
State corruption in any system participating in the job
One completed run is a data point, not a benchmark. Every result release must predeclare the environment variants, trial count, model and runtime versions, perturbation schedule, and uncertainty used to support an aggregate claim.
Live tenant runs and synthetic reference fixtures must be reported separately. Neither may borrow the other's label, and a synthetic result may not be presented as deployment evidence.