Research Notes: Code Agent Engineering Loop and Evaluation
This reference records the evidence behind the operational rules in the parent skill. Treat published benchmark scores as historical results bound to specific models, scaffolds, budgets, and dataset versions.
Evidence Summary
Agent-computer interface and repository exploration
SWE-agent argues that a specialized Agent-Computer Interface should provide simple actions, compact operations, concise environment feedback, and guardrails. Its interface uses bounded search, line-numbered file windows, edit feedback, lint checks, and compressed old observations. Successful trajectories usually begin with reproduction or localization and then move into edit/execute loops.
Reported trajectory analysis provides useful failure signals rather than timeless performance targets:
- 51.7% of GPT-4 Turbo trajectories had at least one failed edit.
- The eventual edit-success probability was 90.5% in general but 57.2% after one failed edit.
- About 52% of unresolved cases were categorized as incorrect or overly specific implementations.
- Successful runs tended to finish earlier and more cheaply; merely increasing budget was unlikely to fix the dominant failure modes.
Operational implication: improve localization, feedback, syntax guardrails, and recovery before increasing turn/token budgets.
Structured pipelines versus unrestricted autonomy
Agentless uses three explicit phases: hierarchical localization, repair with multiple candidate diffs, and patch validation using reproduction and regression tests. It demonstrates that a simple, interpretable pipeline can compete with more complex autonomous agents.
Operational implication: use a constrained state machine with autonomy inside stages, not an unbounded ReAct loop.
State, event streams, and isolation
OpenHands models state around a chronological event stream of actions and observations, with additional execution metadata such as accumulated cost and delegation state. Each task runs in a sandboxed environment.
Operational implication: persist append-only events and gate checkpoints so a crashed run can be inspected and safely resumed; sandbox execution and track side-effect provenance.
TDD evidence and limits
Nagappan et al. studied three Microsoft teams and one IBM team using TDD. The case studies reported 40%–90% lower pre-release defect density relative to similar non-TDD projects, alongside a subjective 15%–35% increase in initial development time.
This is industrial case-study evidence, not a universal causal law. For agents, the strongest rationale for RED–GREEN–REFACTOR is epistemic:
- RED proves the test can observe the target fault.
- GREEN connects the patch to the intended behavioral change.
- REFACTOR preserves the established contract.
A generated reproduction test is not an oracle until it fails on the old code for the right reason, passes with the patch, and fails again when the production patch is reverted.
Independent review
Sadowski et al.’s Google case study examined modern code review through 12 interviews, a 44-person survey, and logs for 9 million reviewed changes. The paper establishes lightweight, tool-supported review as a mature large-scale engineering practice.
The parent skill’s two-stage Agent review—specification first, quality second—is a design extrapolation that must be validated locally. Independence should be measured, not assumed.
SWE-bench and Evaluation Caveats
Original scope
The original SWE-bench contains 2,294 GitHub issue/PR tasks from 12 popular Python repositories. It offers realistic repository context and execution-based evaluation, but its authors explicitly note that tests alone cannot guarantee comprehensiveness, efficiency, or readability.
Verified curation
OpenAI and SWE-bench authors created SWE-bench Verified by using 93 Python developers to screen 1,699 random samples, with three annotations per sample. Their report states:
- 38.3% were flagged for underspecified problem statements.
- 61.1% were flagged for tests that might unfairly reject valid solutions.
- 68.3% were filtered for these or other issues.
- The final Verified set contains 500 samples.
The report also warns that public static GitHub tasks are likely contaminated and that scaffold changes can strongly alter performance for the same model.
Leakage and weak tests
Agentless reports exact-patch leakage, missing critical information, and misleading descriptions in SWE-bench Lite. SWE-Bench+ reports manual analysis where some successful patches benefited from solution leakage and others passed weak tests. SWE-Bench+ is a preprint and its qualitative categories require judgment; use it as a risk signal, not a universal estimate.
Fresh-task trade-off
SWE-rebench proposes continuous task collection and time-aware evaluation to reduce contamination. Its automated pipeline expands scale and freshness but the authors acknowledge that automated installation, heuristic filtering, and LLM quality assessment cannot fully reproduce nuanced human curation; the initial release remains Python-only.
Recommended Evaluation Matrix
| Layer | Purpose | Controls |
|---|---|---|
| Public benchmark | Reproducible external comparison | Dataset version, container, scaffold commit, model, budget, seed |
| Fresh temporal holdout | Reduce contamination | Issue/PR dates after data cutoff, hidden tests/gold patch |
| Project regression set | Detect workflow regressions | Decontaminated historical tasks stratified by module/risk/type |
| Production shadow/canary | Measure real acceptance and safety | No automatic merge, human review, rollback and observation window |
Always separate model and scaffold effects. Report pass@1; if reporting pass@k, include candidate count, selection method, total cost, and extra information visible to the selector. Include bootstrap confidence intervals, trajectories, environment digests, test-oracle audits, and semantic patch review.
Metric Definitions
- Verified Delivery Rate: tasks entering ready-for-test with complete evidence and independent review / claimed tasks.
- Accepted Without Rework: tasks accepted on first test/review cycle / delivered tasks.
- Requirement Coverage: acceptance criteria with executable test or explicit review evidence / all criteria.
- Scope Expansion Rate: unexpected changed lines / total changed lines, interpreted alongside semantic completeness.
- RED Validity: reproduction tests that satisfy pre-patch fail, post-patch pass, and patch-revert fail / proposed reproduction tests.
- Recovery Success: interrupted runs resumed or safely aborted without duplicate side effects / interrupted runs.
- Cost per Accepted Task: total model/tool/compute cost / accepted tasks.
Never optimize these in isolation. Small diffs can be incomplete; many tests can contain weak assertions; fast completion can bypass gates.
Sources
- Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023/ICLR 2024: https://arxiv.org/abs/2310.06770
- Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024: https://arxiv.org/abs/2405.15793
- Xia et al., Agentless: Demystifying LLM-based Software Engineering Agents, 2024: https://arxiv.org/abs/2407.01489
- Wang et al., OpenHands: An Open Platform for AI Software Developers as Generalist Agents, 2024/ICLR 2025: https://arxiv.org/abs/2407.16741
- Chowdhury et al., Introducing SWE-bench Verified, OpenAI, 2024; updated 2025: https://openai.com/index/introducing-swe-bench-verified/
- Aleithan et al., SWE-Bench+: Enhanced Coding Benchmark for LLMs, 2024 preprint: https://arxiv.org/abs/2410.06992
- Badertdinov et al., SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents, 2025: https://arxiv.org/abs/2505.20411
- Nagappan et al., Realizing quality improvement through test driven development: results and experiences of four industrial teams, 2008: https://doi.org/10.1007/s10664-008-9062-z
- Sadowski et al., Modern code review: a case study at Google, 2018: https://doi.org/10.1145/3183519.3183525