Notice
This article was written with reference to four papers publicly available as of August 4, 2026. It also includes analysis by DANA NOTES.
Reference Papers
Chen, J., Huang, H., Lyu, Y., An, J., Shi, J., Yang, C., Zhang, T., Tian, H., Li, Y., Li, Z., Zhou, X., Hu, X., & Lo, D. (2026). SecureVibeBench: Benchmarking secure vibe coding of AI agents via reconstructing vulnerability-introducing scenarios. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 24144–24168). Association for Computational Linguistics.
Almukhtar, M., Ghammam, A., & Ming, H. (2026). Quality and security signals in AI-generated Python refactoring pull requests. In Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware ’26). Association for Computing Machinery.
Sawada, S., Shirai, T., Kashiwa, Y., Yamaguchi, K., Iwata, H., & Iida, H. (2026). To what extent does agent-generated code require maintenance? An empirical study. In Proceedings of the 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). Association for Computing Machinery.
Orlanski, G., Roy, D., Yun, A., Shin, C., Gu, A., Ge, A., Adila, D., Roberts, N., Sala, F., & Albarghouthi, A. (2026). SlopCodeBench: Benchmarking how coding agents degrade over long-horizon iterative tasks [Preprint]. arXiv.
What This Article Covers
Developers Are Already Coding with AI
Developers do not work by memorizing every programming syntax rule and development method.
They search for syntax they cannot remember, look for ways to implement the functions they need, and adapt those methods to the systems they are working on. When an error occurs, they review similar cases and choose the approach that best fits the existing system from among multiple possible solutions.
AI tools such as ChatGPT, Claude Code, GitHub Copilot, and Cursor reduce the time required for this process. They can be used to generate repetitive code, identify the causes of errors, and prepare initial drafts for implementing functions.
AI coding agents that explore and directly modify multiple files have also emerged. A person describes a task, and the agent finds the relevant code, modifies multiple files, and runs tests.
However, in actual development, it is not common for AI-generated code to be deployed directly to a production environment without anyone reviewing it. Developers review the results, determine whether they fit the existing system, make the necessary corrections, and ultimately decide whether to merge and deploy the code.
The realistic comparison, therefore, is not simply AI versus humans.
It is closer to a comparison between a person developing everything directly from the beginning and AI preparing an initial draft that a person then reviews and modifies.
What companies need to determine is also not whether AI can independently produce code that is superior in every respect to human-written code. What matters more is whether using AI actually reduces the development team’s total time and costs.
AI Is Better Suited to Tasks with a Defined Scope
AI coding agents are easier to use when the following conditions are clear.
- What needs to be built
- Which existing functions must be preserved
- Which files and systems may be modified
- Which tests must be passed
- Which security, performance, and structural requirements must be followed
For this reason, AI coding may appear to fit well with the Waterfall Model, in which all requirements are finalized before development begins and are then implemented in sequence.
However, this does not mean that the entire project must follow a waterfall approach. AI can also be used in agile development when the requirements and completion criteria for an individual unit of work are clearly defined.
More precisely, AI is not specialized for the waterfall model. Rather, it is easier to use when the scope and evaluation criteria of the current task remain fixed for a certain period.
The problem is that this condition does not remain stable for long in actual development.
Even with Clear Tasks, It Was Difficult to Satisfy Both Functionality and Security
Chen et al. (2026) constructed SecureVibeBench based on cases in which human developers had introduced vulnerabilities into real open-source projects.
The researchers created 105 C/C++ secure coding tasks from 41 projects. These were not tasks that simply required completing a short function. They required agents to locate relevant areas and modify multiple files within repositories containing an average of approximately 2,845 files and more than 550,000 lines of code.
The evaluation did not only determine whether the requested function worked. It also examined whether the original vulnerability had been reintroduced and whether the agent had added new security risks.
The best-performing combination of agent and language model produced a solution that was both functionally correct and secure in only 23.8% of the tasks (Chen et al., 2026).
However, this figure does not represent the final success rate after a person reviewed and corrected the AI’s output. It reflects the results of agents performing tasks independently and is limited to memory-safety tasks in C/C++, so it cannot be applied directly to other development environments.
The main point of this research is not that AI coding is useless.
It shows that even when requirements and tests are clearly provided, code that works and code that is secure may not be the same.
A Significant Share of AI-Generated Code Was Merged into Actual Projects
Almukhtar et al. (2026) analyzed Python refactoring Pull Requests created by AI agents in actual GitHub projects.
Refactoring is the process of improving a program’s structure, readability, and maintainability while preserving its external functionality. A Pull Request (PR) is a procedure through which someone requests that modified code be incorporated into an existing project and reviewed by other developers.
Of the 438 AI-generated refactoring PRs analyzed by the researchers, 322, or 73.5%, were merged. This means that AI-generated code is not confined to experiments but is being accepted in actual development processes.
However, the quality results did not move in only one direction.
Across five quality attributes, improvements were identified in an average of 22.5% of changes. Usability improved in 36.5%, but maintainability improved in only 14.9% and modularity in 9.5%.
New Pylint issues appeared in 24.17% of the modified files, while new Bandit security warnings appeared in 4.7%. However, many of the Pylint issues involved coding conventions, such as excessively long lines or insufficient documentation, while the Bandit results primarily identified coding practices that could be risky rather than severe vulnerabilities.
Some PRs introduced new warnings while also removing existing ones (Almukhtar et al., 2026).
Therefore, the merge rate is not the same as a code-quality score.
A PR may remain unmerged for reasons such as duplicate work or a change in project direction. Conversely, merged code has not necessarily been verified for long-term maintainability and operational efficiency.
Code That Works Is Not Necessarily Good Code
Code that produces the same result may still have different operating costs.
For example, a system could combine data A and B every time a screen is opened, or it could calculate a frequently used result in advance, store it as value C, and read only C.
The first method uses less storage space but requires calculation each time the data is retrieved. The second method increases the amount of stored data slightly but may improve retrieval speed and simplify the code structure. However, whenever A or B changes, C must also be updated correctly.
Both methods may pass the same functional tests.
Which method is more appropriate depends on how frequently the data is retrieved and changed, storage costs, processing speed, and data-consistency requirements.
The ability to implement a function and the ability to select an appropriate structure based on actual operating conditions are not the same.
This difference may not become immediately visible during code review. In some cases, problems involving processing speed or computer resource use emerge only after the number of users and the volume of data increase.
Actual Requirements Are Not Completed Before Development Begins
Software development is often described as the process of converting completed requirements into code.
In reality, however, users and operational staff often do not know exactly what they need from the beginning. They may only distinguish between necessary and unnecessary functions after seeing actual screens and early versions of the system.
During development, new functions are added, existing functions are removed, and business processes and permission standards change. The need to connect with other systems or handle new exceptions may also be discovered later.
Requirements may change even when development is nearly complete. A function initially intended only for internal employees may later be offered to external customers, or a system built for one organization may need to support multiple organizations.
Actual development is therefore both the process of implementing defined requirements and the process of discovering and adjusting the requirements themselves.
Developers do not simply convert what users say directly into code. They coordinate conflicting requests, explain implementation costs, and determine whether changing the entire system structure for a current function is appropriate.
AI can quickly implement clearly defined requirements. However, reconsidering earlier design decisions when the requirements change is more difficult than simply generating code.
AI-Generated Code Continued to Be Modified After It Was Merged
Sawada et al. (2026) compared 508 AI-generated files with 508 human-written files from the same 100 GitHub repositories.
The researchers analyzed 3,238 subsequent commits made during a period of at least six months after the files were created. AI-generated files were modified less frequently than human-written files, and the proportion of code changed in each modification was also smaller.
This study therefore did not find that AI-generated code immediately created a greater maintenance burden than human-written code.
However, there were differences in the types of modifications.
The largest category of subsequent changes to AI-generated files was feature additions, at 21.78%. Bug fixes accounted for 11.73%. In contrast, bug fixes represented the largest share of changes to human-written files, at 16.76%.
Of the 1,543 subsequent commits made to AI-generated files, 83.21% were written by human developers, while 16.79% were written by AI agents (Sawada et al., 2026).
However, the fact that humans performed most of the modifications does not by itself prove that AI failed to respond to changing requirements. The study identified who made the modifications but did not investigate why humans handled the subsequent work.
What the study directly confirmed is that AI-generated code continued to be modified and expanded after it was merged, and that humans currently performed most of that work.
Repeated Requirements Reveal the Cost of Initial Design Decisions
Conventional AI coding evaluations provide a completed requirement all at once and then assess the result.
In actual development, however, new functions are continually added on top of the original code. What matters is not only whether the current tests are passed. It is also necessary to determine how reliably an agent can expand the structure it previously created.
Orlanski et al. (2026) created SlopCodeBench to evaluate this issue.
This benchmark divided 36 problems into 196 stages. Agents first created code based on an initial requirement and then continued modifying the code they had already written as new requirements were added.
For example, an agent might first be asked to create a program that searches only Python files and then gradually be required to support other programming languages and new search functions. If the initial structure assumed only one language, adding later functions could require repeated code or major structural changes.
No agent completed even one of the 36 problems across all stages. The highest strict pass rate was 14.8% across all stages.
During iterative development, the concentration of complex processing within a small number of functions increased in 77% of all task paths, while duplicated or unnecessarily long code increased in 75.5%.
Compared with 473 actual open-source Python repositories, agent-generated code was, on average, 2.3 times more verbose and had twice the concentration of structural complexity (Orlanski et al., 2026).
This study is still a preprint and used a constructed Python evaluation environment. Its figures should therefore not be applied directly to actual enterprise projects. It is more appropriate to view the study as evidence that problems created by initial design choices may accumulate as functions are repeatedly added.
Code that satisfies today’s requirements is not necessarily code that can easily accommodate tomorrow’s requirements.
Can the Problem Be Solved by Stopping the AI and Giving It New Requirements?
When requirements change, the agent’s work can be stopped and new instructions can be entered.
For small changes, such as revising button text or adding an item to a response, it may not be difficult to reflect the new requirement.
However, the situation changes if the agent is already modifying the data structure, API, and tests at the same time.
When a new requirement conflicts with the existing design, the following decisions must be made again.
- Which of the changes already made should be retained
- Which changes should be reversed
- How completed files and files still in an intermediate state should be aligned
- Which existing tests are no longer valid
- How the changes should be combined with code modified by a person at the same time
This issue cannot be resolved simply by asking whether a stop button exists.
The agent must understand the current state of the work, reverse only the necessary parts of its previous decisions, and create a new plan based on the changed objective.
The ability of current AI coding agents to perform defined tasks independently is improving rapidly. However, collaborating with people on the same code while sharing frequently changing requirements and design intentions is not yet natural.
DANA NOTES Commentary
The current value of AI coding agents does not lie in completely replacing developers.
It lies in reducing developers’ time by quickly handling syntax searches, repetitive implementation, test drafts, and tasks with clearly defined scopes.
The research also did not show that AI-generated files were modified more frequently than human-written files, and a significant share of AI-generated refactoring PRs were merged into actual projects. It would therefore be inaccurate to describe AI coding only as a low-quality or dangerous approach.
However, software development does not end when the first version of the code is completed. Requirements continue to change, existing code is modified and expanded, and performance or structural problems may only appear after actual data volumes and usage increase.
Companies should therefore not evaluate AI coding performance solely through the number of generated lines of code or PR merge rates. They should also examine the following factors.
- The time required to review and correct AI-generated results
- Rework and errors arising after requirements change
- The cost of feature expansion, bug fixes, and structural improvements after merging
- Response speed, resource use, and failure rates in actual production environments
AI should be assigned clearly scoped tasks, such as repetitive code or small functions, while humans should be involved from the beginning in decisions that affect the entire system, including data structures, permissions, performance, and system architecture.
Future evaluation should focus less on AI’s independent success rate and more on the total time required when humans and AI develop together, the quality of the code after repeated requirement changes, and the results of long-term operation in enterprise systems.
The productivity of AI coding should be evaluated not by how quickly the first code is produced, but by the total cost across development, review, modification, and operation.
