Why tool calling fails in production

Tool calling performs well in demonstrations with a small catalog, complete required fields, idempotent APIs, one tenant, and no approval chain. Production environments remove those assumptions. A syntactically valid `create_purchase_order` call is only one step in a larger transaction.

In production, tool catalogs are larger, overlapping, and tenant-specific. Parameters have invariants that API schemas may not express. A successful response can still represent a write to the wrong object; retries can duplicate side effects; posting periods and approval rules can invalidate an otherwise well-formed call. The runtime must account for prior steps and verify resulting state.

The failure modes that actually matter

• Wrong tool — semantically close, operationally wrong. Logging activity instead of advancing stage. Drafting a PO in the wrong legal entity.
• Right tool, wrong arguments — missing required tenant fields, stale IDs, units the ERP will silently coerce.
• No validation — the model believes the call worked; the system of record disagrees, or agreed with the wrong object.
• No recovery — a mid-sequence 409 or timeout abandons the job, leaving split-brain state.
• No completion — tools were called, the chat looks busy, the job spec is not closed.

Frameworks provide retries, persistence, and traces. The implementation team must still define which tool is correct for the function, how coordinated updates across systems are validated, and what constitutes completion. Traces are useful for diagnosis; they become evaluations only when compared with an explicit job specification.

Why tool catalogs must be function-specific

Purchasing tools are not finance tools. A signal-to-purchase-order sequence that can create requisitions is still unqualified to post journals, and a warehouse workflow should not inherit pricing authority. Function-specific boundaries keep tool catalogs, policies, and completion criteria aligned with the role being executed.

EGI scores tool execution as a first-class dimension: the agent called the right APIs with the correct parameters, inside tenant policy, as part of a full sequence. Context pollution at the tool-call checkpoint is a defect. JIT bind exists so the window contains the slice the call needs—not the corpus, not the last twenty turns of narration.

Bruce is evaluated on signal-to-purchase-order and close workflows where legal entity, item, quantity, approval, and audit state are all observable. A run fails when any of those elements is wrong, regardless of the quality of the model’s intermediate explanation.