How should teams decide when to let agents act on their own?
The first step is to stop thinking of AI agents as either fully autonomous or fully supervised, and instead match their autonomy to the phase of your workflow. Research on human-AI teams found that work cycles—like the phases of cyber incident response—can guide an agent's level of autonomy based on how formal and predictable the tasks are in that phase [3]. In practice, that means an agent might have high autonomy during routine code generation but require human sign-off before merging or deploying, especially in high-stakes steps.
This phased approach is supported by evidence that agents are already capable enough to warrant careful control: top agentic systems solve up to 43% of real-world GitHub issues on SWE-bench Lite, and in controlled experiments they complete developer tasks 55.8% faster [1]. Those numbers show real power, but also real risk if unleashed without guardrails. The same paper flags unresolved tensions between autonomy and oversight, and between benchmark performance and real-world validity [1]—so teams should treat benchmark wins as a signal of capability, not a license for full automation.
How can teams make sure agents follow ethical and quality standards?
A practical way to operationalize responsibility is to encode behavioral rules directly into repository-level context files, like AGENTS.md. A 2026 vision paper found that developers are already embedding guidance on fairness, accessibility, sustainability, tone, and privacy into these files, effectively creating a developer-authored governance layer that shapes agent behavior [2]. This is a concrete mechanism: instead of relying on abstract ethical principles, you write natural-language directives that the agent reads before acting.
This approach aligns with the broader call for multi-level governance in responsible AI. A roadmap on software engineering for responsible AI emphasizes establishing governance at multiple levels, setting up development processes that incorporate responsible practices, and building responsible-AI-by-design into system architecture [4]. By putting values in the repo, teams are doing exactly that—translating high-level principles into situated instructions that agents can follow. However, the same paper notes that we don't yet know how reliably agents adhere to these constraints, so teams should test and monitor compliance, not assume it [2].
What's the catch—why can't teams just let agents run the show?
The catch is that agents are not yet reliable enough to operate without human oversight, and the ethical stakes are high. The conceptual paper on agentic workflows explicitly lists unresolved tensions: autonomy vs. oversight, benchmark vs. real-world validity, and capability vs. ethical responsibility [1]. Similarly, a 2023 study on AI-driven software engineering identifies model interpretability, ethical concerns, and integration complexities as significant challenges, despite the promising uptrend in AI adoption [5]. These are not minor issues—they are the reasons why responsible use requires deliberate workflow changes.
The good news is that teams can adapt. Research on adaptive autonomous agents found that dynamic, human-like adaptation is vital to effective human-AI teams, and that using work cycles to assign autonomy levels can enhance both productivity and team dynamics [3]. That means teams should regularly reassess agent autonomy as tasks change, and be willing to dial it back when the situation becomes less predictable. The papers converge on this: responsible use is not about avoiding agents, but about designing workflows that keep humans in the loop, encode values explicitly, and adapt autonomy to context [1][3][4].
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2022 to 2026, 2 from 2024 or later, collectively cited 153 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 44 papers retrieved from a database of over 500 million.
Sources used in this answer
Agentic workflows for end-to-end software engineering automation
A conceptual paper theorizing agentic workflows reports that top systems solve up to 43% of real-world GitHub issues on SWE-bench Lite, achieve 85.9% Pass@1 on Human-Eval, and complete developer tasks 55.8% faster, while flagging unresolved tensions between autonomy and oversight, benchmark and real-world validity, and capability and responsibility.
Operationalizing Ethics for AI Agents: How Developers Encode Values into Repository Context Files
A vision paper based on a preliminary investigation finds that developers are encoding ethical values (fairness, accessibility, sustainability, tone, privacy) into repository-level context files like AGENTS.md, serving as a developer-authored governance layer, and calls for research on adherence and governance dynamics.
Adapt and overcome: Perceptions of adaptive autonomous agents for human-AI teaming
A mixed-methods study with 103 participants and 22 follow-up interviews on cyber incident response teams finds that work cycles can assign autonomy levels to adaptive AI agents based on task formality and predictability, and that dynamic, human-like adaptation is vital to effective human-AI teams.
Towards a roadmap on software engineering for responsible AI
A roadmap paper argues that responsible AI challenges crosscut the entire engineering lifecycle and proposes multi-level governance, process-oriented development practices, and responsible-AI-by-design architectural styles to operationalize responsible AI.
AI-driven software engineering
A study drawing on GitHub and Bitbucket data and industry expert insights finds a promising uptrend in AI integration in software development but identifies model interpretability, ethical concerns, and integration complexities as significant challenges.