ML//agent//evaluation harness

An evaluation harness is the machinery that presents tasks, provisions environments, exposes tools, enforces limits and computes scores. For agents, it is part examiner, part laboratory and part operating environment: a task-specific agent harness whose outputs are used to make capability claims. The benchmark defines the tasks and metrics; the evaluation harness decides how those tasks physically occur.

An evaluation harness is the machinery that presents tasks, provisions environments, exposes tools, enforces limits and computes scores. For agents, it is part examiner, part laboratory and part operating environment: a task-specific agent harness whose outputs are used to make capability claims. The benchmark defines the tasks and metrics; the evaluation harness decides how those tasks physically occur.

That means an agent score never belongs cleanly to the model. CPU, memory, timeouts, network policy, tool ergonomics and scoring bugs can all change measured performance. A suspicious trajectory may reveal model behavior, harness behavior or an interaction between both.

If the laboratory changes the phenomenon, calling it the same test is only a label.