AI Agent Evaluation Interview: Test Safe Retries

An agent reports “done,” but a retry has created two records. An AI agent eval should check the resulting state, not just the final response. Anthropic’s January 2026 evaluation guide distinguishes outcomes from execution traces. The duplicate-action exercise below is our interview practice scenario, not a reported hiring question.

The four layers of an agent evaluation

Treat an evaluation as four connected checks. First, the task contract says what outcome is required. Second, trajectory checks inspect tool choice, ordering, and unnecessary calls. Third, state checks verify files, data, or permissions after the run. Fourth, human review covers judgment that a binary test cannot express. A useful answer names all four instead of quoting one benchmark score.

Design graders that disagree usefully

Use a code-based grader for deterministic facts such as a failing test becoming green, a model-based grader for explanation quality or scope discipline, and a human sample for ambiguous cases. Explain how you would investigate disagreement. If a model grader says the patch is complete while an integration test fails, the executable result wins and the disagreement becomes a regression case.

Show agent judgment in a live answer

When asked to improve an agent, start with one failure receipt rather than a list of features. State the observed behavior, the violated invariant, the smallest test that reproduces it, and the guardrail you would add. This makes your reasoning auditable and keeps the answer useful under time pressure.

Worked exercise: a timed-out issue-creation tool

Imagine a coding agent that opens a tracking issue after fixing a bug. The create request reaches the service, the issue is stored, and the response disappears on the network. The agent sees a timeout and tries again. This is a synthetic fixture: do not run the experiment against a real issue tracker.

Define success before watching the trace

The expected outcome is one issue associated with one logical task. A successful run also links the correct patch, preserves unrelated issues, and reports uncertainty if it cannot confirm completion. The agent is not required to follow one exact sequence of reads. A different sequence is acceptable when it respects the same behavior and access boundaries.

Give the fixture a request identifier and record the service state before and after each attempt. A deterministic check counts matching records and verifies the patch reference. Another check confirms that retries reuse the logical operation identity. A text evaluator may assess whether the final explanation acknowledges an unresolved result, but it cannot overrule two records in storage.

Test the ambiguous boundary

Use three cases: the call fails before any write, it fails after the write but before acknowledgment, and it returns success normally. The second case is where an indiscriminate retry policy breaks. A proposed remedy is server-side idempotency or a supported operation-status lookup; searching by title alone is unsafe when titles are not unique. If the API offers neither, the agent should stop automatic retries and surface the uncertain state.

Record whether a duplicate was prevented by the service or merely avoided by the agent on that run. These demonstrate different protections. Also reset fixture state between trials: otherwise an issue from a previous trial can make a broken run look successful.

A two-minute answer and a useful follow-up

A concise answer is: “I would inject a timeout after the service commits, then check that the final state contains one issue. I would repeat the same fixture and record every trial. The release condition includes no duplicate writes, not just a high average task score.”

The follow-up is often more revealing: “What if nine out of ten runs pass?” One passing run demonstrates capability; repeated runs expose reliability. Report the denominator and failed state, and avoid treating a small sample as a production failure-rate estimate. A rare unsafe side effect can remain a release blocker even when the average score improves.

For practice, have a partner change the failure point without telling you. Your goal is to infer what the receipt proves, what remains unknown, and whether another write is safe. A confident explanation without that distinction is still incomplete.

What changed in agent evaluation in 2026

Agent evaluation is moving beyond a single pass rate. Anthropic’s 2026 guide to agent evals stresses that the evaluated agent and environment should resemble production. OpenAI’s review of SWE-bench Verified is a useful reminder that benchmark validity and contamination can change what a score means.

In an interview, separate four questions:

Did the agent reach the required final state?
Did it use a safe and permitted trajectory?
Can the result be reproduced across trials?
Is the failure a release blocker or a quality issue to monitor?

A high score that hides duplicate writes is not a reliable release signal.

Write a release gate before choosing a grader

Start with the invariant, then select the smallest grader that can check it. A code grader can verify that one issue exists after a retry. A trajectory grader can inspect whether the agent repeated a non-idempotent call. A model grader can assess explanation quality, but it should not be the only judge of a database side effect. Sample ambiguous cases for human review.

State the denominator and the failure policy. Ten trials with one unsafe duplicate are not the same as one successful demo, and an average score should not erase a severe tail failure.

Make the interview exercise reproducible

Use a synthetic issue tracker or local fixture. Inject a timeout after the service commits but before the client receives the response. Then compare three implementations: blind retry, an idempotency key, and a read-before-write recovery path. Record the final records, tool calls, elapsed time, and unresolved risk.

The best answer may be a narrower change. If the tool cannot prove whether the first write committed, the agent should inspect state or ask for a human decision instead of writing again. That conclusion demonstrates safety judgment, not just benchmark literacy.