How to score an enterprise agent

An execution evaluation must inspect the resulting ERP state. EGI treats one end-to-end run as the evaluation unit: load the job specification, bind the tenant stack, execute the workflow, inspect system changes, and assign pass or fail. Aggregate benchmarks are published only after the same specification succeeds across unlike tenant environments.

Per-run checks (boolean)

• Job closed to the defined expectation — yes or no.
• Correct tools and parameters for that tenant’s stack.
• Full sequence executed — not a single demo step.
• Audit log captures each action and system change.
• Policies and constraints held for that business.

All checks are conjunctive. A partial write accompanied by an accurate summary still fails. So does a technically correct write that bypasses approval, or a valid initial tool call followed by an unrecovered conflict.

Seven execution dimensions

• Goal persistence — finishes through handoffs and long runs. Close jobs are the exam: the period is still the same object tomorrow.
• Tool execution — correct APIs and parameters from a function-specific catalog.
• Multi-step completion — the chain, not an isolated action. A requisition without a posted PO is not a cycle.
• State & context — job state held across steps; JIT bind, not window archaeology.
• Error recovery — detects partial failure, compensates when required, and resumes from a known state.
• Outcome quality — deliverables match the spec, judged by people who know the function, replicated across tenants.
• Constraint adherence — tenant policy and approvals are enforced and recorded.

Replication, then publish

A successful run in one environment does not establish portability. An ESE-1 result release must predeclare its environment variants, execute the same versioned job contract across them, and publish trial counts, uncertainty, hard failures, and reconstructable traces. Domain experts may define the expected outcome, but deterministic terminal predicates score the resulting system state. The site’s ERP-PO-01 and ERP-CLS-03 specimens show reporting shape only; they are not published benchmark results.

After a model change, rerun the same jobs. Stable tool execution, recovery, and completion indicate that the benchmark belongs to the execution system rather than one inference provider.

Bruce is evaluated with this method. The score is meaningful only when it can be traced to the binding protocol, the execution artifact, the resulting system state, and the recovery record. Architecture and evaluation should describe the same run from different perspectives.