The question that motivated this experiment was deceptively simple: does adding a lightweight agent to a developer workflow actually make the workflow better, or does it just add latency and a new class of errors?
It is easy to build an agent that looks capable in a demo. A loop that calls a model, executes a tool, and feeds the result back into the next call has an appealing mechanical elegance. But whether that loop produces reliable outcomes on real developer tasks (summarizing context, proposing next steps, making narrow code edits) turns out to depend heavily on factors that demos do not reveal. The clarity of task boundaries, the length of the review loop, how well the agent recovers when it misreads intent, and how much human correction effort the workflow requires all shape the answer.
The agent loop itself is the same three-hop cycle every time: the model reasons about the task, calls a tool, observes the result, and decides whether to loop again or stop. The diagram below shows that cycle, and the experiment in this post is about when each hop earns its cost.
TLDR
- Small agent workflows reduce completion time on well-scoped tasks with short feedback loops.
- They turn net-negative when task boundaries are vague, context is long and noisy, or the agent has no signal to stop and ask.
- The signals worth measuring are not task completion rate alone. They are interruption rate, correction effort, and context fidelity across turns.
Hypothesis#
Our starting hypothesis had two parts. First: a lightweight agent (two to five tool calls per task, single model) would outperform a stateless one-shot prompt on tasks with multiple well-defined subtasks, because it could decompose, verify, and iterate rather than producing a single large output. Second: that "task completion" alone would be an insufficient metric, because an agent can technically complete a task while requiring so much human correction that it saves no time over doing the task manually.
We also suspected that task boundary clarity would be the strongest predictor of agent usefulness, more predictive than model capability or tool design.
Setup#
We defined three categories of developer task, chosen to represent the range of work engineers encounter in a typical sprint:
Category A, context summarization: Given a pull request diff plus a set of linked issue comments, produce a structured summary covering the change intent, what was modified, and any open questions.
Category B, next-step proposal: Given a failing test run output and the relevant source code, identify the most likely root cause and propose a concrete next step.
Category C, narrow code edit: Given a function with a described behavioral bug, produce a corrected version along with a brief explanation of the change.
For each category we prepared ten task instances drawn from real (anonymized) engineering work, not synthetic scenarios. Each instance was evaluated by a small agent workflow and also by direct one-shot prompting to the same model, with identical system prompt context.
The agent workflow used a simple three-step loop: plan the approach, execute tool calls (code search, file read, lint check), and validate the output against the task specification. We deliberately kept the agent small. Anthropic's "Building Effective Agents" guide documents that the most reliable production implementations favor simple composable patterns, and that observation shaped our choice to limit scope rather than test a complex orchestrated system.
Methodology#
We measured five signals per task instance, for both the agent and the one-shot baseline:
- Task completion rate: Did the output satisfy the stated task criteria without requiring a redo?
- Interruption rate: How often did the workflow require mid-task human intervention (clarification, correction of tool use, or manual override)?
- Correction effort: Estimated developer minutes spent reviewing, adjusting, or completing the output after the agent finished.
- Context fidelity: On a three-point scale (full, partial, lost), did the agent maintain the relevant context across the steps of the task?
- Latency vs. one-shot: Total time from task submission to usable output, including model call time and tool execution.
Two evaluators independently scored each output against a rubric. Inter-rater agreement was checked before scoring, and disagreements were resolved by a third reviewer. We did not use model-as-judge scoring for the primary evaluation. Human review was intentional given that the tasks themselves require engineering judgment.
Why not just use a benchmark?
Public benchmarks like SWE-bench and AgentBench are valuable for measuring software engineering and multi-environment task performance at scale. We deliberately used internal tasks instead because we wanted to measure signals that public benchmarks do not expose well: interruption rate and correction effort in real workflows, not just pass/fail on hidden test suites.
Results#
The results below are qualitative observations and illustrative order-of-magnitude comparisons. We are not reporting precise percentages as if this were a statistically controlled study. The sample size is too small and the task distribution too specific to generalize. Treat these as directional findings that motivated follow-up design decisions, not as benchmarks.
Category A (context summarization): The agent workflow performed noticeably better than one-shot on the longer, more complex PR instances, those involving multiple files and cross-referenced issues. On shorter, self-contained PRs, one-shot prompting was faster and the outputs were comparable in quality. Interruption rate was low across both approaches for this category. The agent's main advantage was reducing the human's need to stitch context together manually.
Category B (next-step proposal): This category showed the largest variance. On instances where the failing test was clearly isolated to a single function, the agent and one-shot performed similarly. On instances where the root cause spanned multiple files or involved implicit coupling, the agent consistently produced more actionable proposals, because it could read the relevant files rather than relying entirely on pasted context. However, two instances saw the agent loop on irrelevant tool calls, producing verbose but unhelpful output. These were the highest-correction-effort cases in the entire experiment.
Category C (narrow code edit): One-shot was competitive or better for most instances. The agent added tool calls (reading type annotations, checking imports) that did not meaningfully improve output quality for narrow, self-contained edits. It also added latency, roughly two to three times the one-shot response time, without a proportional quality gain. On these tasks, the agent workflow was net-negative in time saved.
Across all categories: Correction effort was more predictive of perceived usefulness than task completion rate. Some technically "complete" agent outputs required significant cleanup before they were usable. The agent's biggest failure pattern was not hallucination. It was misidentifying the task boundary, continuing to execute tool calls after a reasonable stopping point, or producing output that was technically correct but not structured for the next step in the actual workflow.
What Surprised Us#
Stopping behavior mattered more than we expected. The most frustrating agent failures were not wrong answers. They were cases where the agent kept going past the right stopping point, adding tool calls that diluted the useful output. An agent that produces a good answer on step three and then adds three more speculative steps is harder to work with than an agent that produces a slightly shorter answer and stops.
Task boundary clarity was the strongest predictor of usefulness. This matched our hypothesis, but the effect was larger than expected. When the task had a clear definition of "done" (a specific artifact format, a known pass/fail criterion), the agent performed well. When the task description had implicit completeness criteria ("improve this"), the agent's behavior became inconsistent and interruption rate increased sharply.
Correction effort scales with context length non-linearly. On tasks with long contexts, even a small proportion of hallucinated or irrelevant content required disproportionate review effort. Reviewers could not skim the output as quickly when context was long; they had to read more carefully to validate correctness.
Limitations#
This experiment has several limitations that bound how much its findings should generalize.
Small sample, narrow domain. Thirty task instances across three categories is not enough to establish statistical confidence. The tasks were drawn from one engineering context and may not reflect the distribution of tasks in other teams or domains.
Single model, single agent architecture. We tested one agent implementation with one underlying model. Different models, tool sets, and orchestration patterns would likely produce different results. AgentBench's evaluation across twenty-seven models found significant disparity between commercial and open-source models, so the model choice matters substantially.
Human evaluators introduce subjectivity. Our evaluators had engineering context for the tasks, which makes scoring more realistic but also more subject to individual judgment. Different evaluators with different backgrounds might score "correction effort" differently.
No longitudinal measurement. We measured single-session performance. We did not measure how agent usefulness changes over time as developers learn to write clearer task descriptions, or how agent errors compound in longer multi-session workflows.
No cost measurement. We tracked latency but not token cost per task. In production, cost per task is a real constraint that could shift the calculus significantly, especially for the agent's additional tool-call turns.
Evaluation rubric may not transfer. Our task rubric was calibrated to our team's standards. A different team might weight conciseness, safety, or format differently, which could reverse some of the relative quality judgments.
What This Means in Practice#
The clearest practical signal from this experiment is that agent design should start with the task, not the framework. Before adding agentic behavior to a workflow, ask: Does this task have a clear definition of done? Does it benefit from decomposition and intermediate verification? Is the expected output easy to validate?
If the answers are yes, a lightweight agent with a small number of well-defined tools tends to improve both output quality and the developer experience. If the answers are no (the task is vague, the output format is unconstrained, or validation requires deep domain knowledge), one-shot prompting with a well-constructed prompt is often faster and produces less cleanup work.
The design implications for teams building agent features align with what Anthropic describes as context engineering: treating context as a finite, precious resource and designing tools that give the agent exactly the context it needs, rather than broad access that invites distraction. See evaluating AI features in the development loop for how these signal-measurement ideas connect to a repeatable evaluation practice.
Task completion rate alone is misleading
An agent that completes 90% of tasks but requires significant correction on 40% of completions is not as useful as those numbers suggest. Measure correction effort and interruption rate alongside completion rate. Otherwise you are optimizing for output existence rather than output usefulness.
Next Experiment#
The most important variable we did not control for was stopping behavior. In the next experiment, we want to test whether adding an explicit stopping criterion to the agent's tool loop reduces correction effort on the high-variance Category B tasks without meaningfully increasing latency. That criterion would be a "validation" tool the agent must call before returning output, which checks whether the output meets the task specification.
We also want to test whether the gap between agent and one-shot narrows or widens as task context length increases, and whether there is a context length threshold below which one-shot consistently wins. That threshold would be a useful heuristic for deciding when to invest in agentic architecture for a given feature.
Separately, we want to apply the OpenTelemetry GenAI semantic conventions to instrument agent spans end-to-end. That means capturing tool call latency, token usage per step, and agent reasoning events, so production agent behavior can be observed without requiring manual log review. The current measurement setup was too manual to sustain at scale.
Key Takeaways#
Small agent workflows are most useful when task boundaries are explicit, output validation is straightforward, and the workflow involves multiple subtasks that benefit from decomposition. They are net-negative or neutral when tasks are vague, context is long and noisy, or the agent has no clear stopping condition.
Task completion rate is an incomplete metric. Interruption rate and correction effort are better predictors of whether an agent workflow actually saves time. Stopping behavior, meaning when the agent decides it is done, is at least as important as what the agent produces.
The approach to evaluating agents should match the approach to evaluating any system feature: define the success criteria before testing, measure what the user actually experiences (time saved, errors introduced, review burden), and treat qualitative observations as hypotheses for controlled follow-up experiments.
For the underlying system architecture that makes agent workflows reliable in production, see system design for AI products. For the retrieval layer that feeds context into agent steps, see building a RAG search pipeline.
FAQ#
How is a "small agent workflow" different from a prompt chain?
A prompt chain sequences model calls where each output feeds the next input, but the sequence is fixed at design time. A small agent workflow uses model-driven decisions to choose which tool to call next, whether to loop again, or when to stop, so the path is dynamic. In practice the boundary is blurry, but the key distinguishing feature is whether the model has agency over the execution path, not just the content of each step.
Why not use SWE-bench or AgentBench for this kind of evaluation?
SWE-bench evaluates agents on resolving real GitHub issues against hidden test suites, which is excellent for measuring software engineering capability at scale. AgentBench evaluates multi-turn decision-making across environments like operating systems and databases. Neither captures interruption rate or correction effort in a human workflow, which was the core question we were investigating. Public benchmarks and internal workflow evaluations complement each other; neither replaces the other.
How do we define "task boundary clarity" in a way we can measure?
One practical operationalization: before running the agent, have two evaluators independently write what a correct output would look like. If they agree on format, length, and key content without discussion, the task boundary is clear. If they disagree significantly, the task is underspecified. This disagreement rate is a useful pre-experiment metric that predicts how much variance you will see in agent output quality.
Should we always prefer one-shot prompting over agents for simple tasks?
For tasks that a capable model can handle in a single, well-contextualized prompt (narrow code edits, straightforward summarization, format transformation), one-shot prompting is often faster, cheaper, and easier to debug. Agents add overhead in latency, cost, and failure surface. Reserve agentic behavior for tasks that genuinely require decomposition, intermediate verification, or dynamic context lookup.
How do we prevent an agent from running past the right stopping point?
The most effective approaches are: defining an explicit stopping criterion in the system prompt (a specific artifact format or a "done" condition the agent must check), adding a validation tool the agent must call before returning output, and setting a hard maximum on tool call count per task. Combination works best. A clear stopping criterion reduces unnecessary steps, and a hard cap prevents runaway loops when the criterion is misread.
