| [orc-ai-sig] ENISA consultation on AI-assisted software development |
Hi all,
ENISA is currently consulting on its draft Technical Advisory on AI-assisted software development.
Draft advisory: ENISA draft technical advisory
Consultation: EU Survey response form
My main concern is that the advisory focuses primarily on AI-assisted development within a conventional SDLC, with human pull-request review remaining a central control. It mentions more autonomous practices, but treats them largely as out of scope and does not really address the consequences of increased development velocity.
I’ve exchanged extensively with the author over the last few days and found him remarkably open to input and discussion, so I encourage folks to respond to the survey request.
I’ve submitted a response myself and attached the PDF in case it is useful to others. The main points I raised are:
Human review does not scale linearly with development velocity. As AI-generated output increases, assurance needs to become more risk-driven, with human judgment focused on architecture, policy, exceptions, risk acceptance, and the highest-impact changes.
Agents need to be treated as security-sensitive components. They can fail accidentally or be manipulated through prompts, context, tools, dependencies, or credentials. That implies sandboxing, least privilege, capability restrictions, deterministic policy enforcement, telemetry, containment, and rollback.
The control and assurance architecture needs to evolve with autonomy. As development becomes faster and more autonomous, stronger technical controls are needed to constrain what agents can do, while verification and assurance need to become more continuous, automated, and integrated into the surrounding engineering system.
The development environment increasingly needs to be treated like production from a cybersecurity perspective. If it is continuously authorized to modify the running product, the agents, harness, tools, credentials, repositories, build system, and deployment machinery become part of the effective software-supply-chain security boundary. Secure-by-design principles therefore need to apply recursively to the system that builds, verifies, and deploys the product.
Development velocity and threat velocity are increasing together. Software production cannot move toward machine speed while assurance and defense remain periodic and human-paced. Defensive verification, testing, remediation, detection, containment, and recovery will increasingly need to become continuous and automated as well.
There is an interesting parallel with the research behind the book Accelerate. We have seen a version of this transition before: teams that increased software-delivery throughput did not necessarily trade away security or stability. Better outcomes came from stronger automation, testing, observability, feedback loops, and disciplined engineering practices.
My sense is that agentic development is another iteration of that pattern, except that threat velocity is now increasing at the same time.
If there is interest, I think this could make a interesting SIG discussion; I'll be on the call today.
Best,
--tobie
Attachment:
Contribution2dfcb7d1-2594-4e22-baa7-00a6236b4d5c.pdf
Description: Adobe PDF document