A year of AI coding / 12
Coding agent quality depends on more than the model
Changes to prompts, context management, and reasoning settings can alter coding-agent behavior even when the underlying model is unchanged.
By Grafty team · · 3 min read
Written in .
An engineer says the coding agent has become forgetful. Another says it works normally. Both may be describing their experience accurately. They may have different session histories, different settings, or a workflow that triggers behavior the other never sees.
That makes agent quality harder to diagnose than a single model score suggests.
Anthropic's April 23, 2026 postmortem traced reported Claude Code quality problems to changes involving reasoning effort, retention of earlier thinking, and prompt instructions. It said the API and inference layer were unaffected and described fixes to the product behavior. The postmortem is a useful case study in how the surrounding system can change the result.
We use "harness" to mean that surrounding system: the instructions, tools, context handling, execution environment, and control logic through which a model does work. The model matters, but the harness determines much of what the model sees and can do.
Open full-size diagram Test the way people actually work
A fresh session that edits one file may pass a regression suite while a resumed, long-running session fails. If a team commonly interrupts tasks, returns to them later, and asks for a correction, the evaluation should include that pattern.
A useful internal app test might ask the agent to add a field, run checks, pause, then handle a follow-up that changes one requirement while preserving the others. The expected result should include the original constraints. Does the second edit keep tenant isolation? Does it retain the required audit entry? Does it rerun the relevant test after changing the implementation?
These checks are about observable behavior. We do not need to inspect hidden reasoning to determine that a requirement disappeared from the final code.
Include failure paths too. A missing dependency, an unavailable tool, or a failed build can reveal whether the agent recovers sensibly or continues under a false assumption. The cleanest demo path is rarely enough to characterize the tool.
Open full-size diagram Keep enough information to investigate
When behavior changes, teams need a usable record of the environment. That includes the agent version, selected model, relevant settings, repository revision, and tool configuration. Save test output and task artifacts within the organization's data-handling rules.
Be selective. Logging everything can capture credentials, source snippets, or sensitive records. A useful diagnostic trail should redact secrets and limit access while preserving the evidence needed to reproduce the failure.
Where a product permits version control or pinned configuration, use it deliberately. Where server-side behavior can change independently, acknowledge that limit and rely on repeated workflow checks. Recording a model name alone is not a complete reproducibility strategy.
Roll out changes like other engineering dependencies
We would evaluate significant agent updates on representative tasks before making them the team default. Keep the test workload stable enough to compare results and record where the new configuration helps or regresses.
Some differences are tradeoffs rather than bugs. Lower latency may be useful for a local edit and inappropriate for a complicated migration. The default should reflect the work, with a clear way to choose a different setting when necessary.
The lesson from the April postmortem is not that one product should never change its defaults. It is that quality belongs to the complete workflow. Teams relying on coding agents should be able to detect a meaningful regression without having to prove whether the model, the prompt, or the context manager caused it first.