Agentic Coding Evals: Control Infrastructure Noise

Two benchmark scores differ, but so do the memory limits. A score change is not automatically a model regression. Anthropic’s February 2026 infrastructure-noise study investigates resource effects on agentic coding evaluations. The following incident exercise is hypothetical and focuses on interpreting a changed result, rather than reproducing that study.

Describe the runtime as part of the experiment

Record model version, tools, repository state, dependency availability, time budget, concurrency, and network policy. Without this context, a score is not portable. The same model can receive a different opportunity to succeed under another harness.

Separate failure receipts

Classify a run by what actually stopped it: wrong patch, failing test, timeout, preemption, missing dependency, or harness error. Do not collapse every non-pass into “model quality.” This distinction is useful in a take-home postmortem and in a live interview.

Report reproducibility with humility

Rerun a small sample, publish the variance, and state what you did not control. A credible answer can say that a result is directional rather than pretending to have a perfect causal experiment.

Investigate a regression without moving the goalposts

Suppose a nightly evaluation drops after a runner migration. The model identifier and task set appear unchanged. One reaction is to add more retries until the score returns. That can hide the regression: the new run now has more opportunities and a different budget than the baseline.

Instead, choose one failed task and compare receipts. Check the repository commit, container image, CPU and memory limits, test timeout, concurrency, and dependency cache. Verify the actual model configuration too; a familiar public name does not prove that every runtime parameter stayed fixed. Record unavailable details rather than quietly assuming equality.

Separate a patch defect from a blocked execution

For a wrong patch, show the assertion it violates. For a timeout, record which operation was active and whether useful progress was still occurring. For a missing dependency, determine whether network access was intended to be available. These are different hypotheses with different repairs.

A task that requires the agent to work offline should fail when it assumes online access. A harness that accidentally disables a required package mirror creates a different measurement problem. Infrastructure failures remain visible in the end-to-end result; separating them is not permission to remove inconvenient failures from the denominator.

Design a comparison that can change your conclusion

Run the same task under the previous and current resource profiles, holding the agent configuration and reset procedure fixed. Alternate execution order when possible so changing service load does not align with one condition. Repeat enough times to reveal obvious variance, but report the sample size instead of claiming certainty from a few runs.

Suppose the task succeeds only with the old memory limit. Your conclusion is that the resource change affects this workload. You have not established that the model would otherwise be correct on all tasks, or that the higher limit is financially justified. Expand to a relevant sample before changing the production runtime.

Keep both a user-facing completion measure and a diagnostic breakdown. The user experiences a failed task regardless of which layer caused it. The breakdown tells the team whether to fix the patch generator, dependency setup, or resource allocation.

In an interview, finish with a decision: “I would restore a comparable runner for the release comparison, keep the failed runs in the report, and investigate the resource-sensitive cases separately.” That is more useful than simply asking for more compute. A good receipt makes the next experiment smaller and the conclusion more limited.

Benchmark validity is part of the 2026 story

Anthropic’s infrastructure-noise analysis shows that resource limits can change both reliability and what an agent is able to solve. OpenAI’s SWE-bench review adds a separate warning: a benchmark can lose decision value when its tasks or labels no longer measure the intended capability. A score is therefore a measurement with a setup, not a property of a model in isolation.

Record the variables that can move the result:

model identifier and runtime parameters;
repository commit, container image, and dependency cache;
CPU, memory, timeouts, concurrency, and network policy;
task reset, retry, and grading procedures;
service load and unavailable telemetry.

If a variable is unknown, mark it unknown. Do not silently call two runs comparable.

Use a two-layer failure classification

First report the user-visible outcome: completed, wrong result, incomplete, or timed out. Then classify the first deviation: patch defect, test failure, missing dependency, resource exhaustion, provider timeout, or harness error. This preserves the real failure in the denominator while telling the team which layer to fix.

Hold the agent configuration fixed and alternate execution order when comparing runner versions. Repeat a small, relevant sample, publish the count, and keep the failed receipts. More retries can estimate recovery behavior, but they cannot make an unfair baseline comparable after the fact.

End with a decision, not more compute

If a task passes only with a larger memory limit, the conclusion is that this workload is resource-sensitive. It is not yet proof that the model improved or that the larger runner is worth its cost. A good interview answer names the next experiment and the release decision it will inform.