AI can generate code faster, but can open source keep up?

Open source software is used throughout the world’s digital infrastructure, in applications such as cloud computing, enterprise, mobile devices, and even AI systems. Rather than building software from scratch, organizations increasingly rely on existing, reusable open source components to build their own systems.

While this can be more efficient for an organization, it also creates dependencies on externally maintained software. This means that they have less direct control over the security of their own digital infrastructure and greater dependence on external maintainers to keep critical components secure.

The introduction of AI-assisted tools has furthered the tension in the tradeoff between more efficient building processes and weaker security. They can write code, help identify vulnerabilities, and propose dependency patches at speeds that would have been impossible to imagine only a few years ago. These capabilities help many open source projects enormously, but generating a software contribution is becoming easier, while establishing that the contribution is trustworthy remains difficult.

In a recent TechBrief published by ACM, Artificial Intelligence’s Effects on Open Source, my co-authors and I examined this tension across cybersecurity, software maintenance, sustainability, and governance. The issue is not that AI will always be harmful to open source software. It increases both opportunities and challenges in an ecosystem that depends heavily on human judgment. AI has the potential to uncover flaws, but it also poses cybersecurity and maintenance challenges as its use grows.

AI Helps Both Defenders and Potential Attackers

With the help of advanced AI models, developers can find vulnerabilities and develop patches in their software. This has immense opportunity for maintainers to strengthen their open source code security by finding problems earlier and helping them address them.

Conversely, these capabilities can also be misused. Malicious packages and malicious contributions to existing projects are already a concern for the software supply chain. AIM models can help identify and patch vulnerabilities, while those capabilities may also be used to develop attacks.

For example, in May, an open source program experienced an influx of new accounts that immediately began publishing spam to the package repository in their ecosystem. The company’s maintainers had to temporarily pause new account registrations and remove 500+ malicious packages. Some researchers suspect the activities were done by AI agents, but the available evidence is insufficient to determine whether AI agents actually created or published the packages by themselves. This incident is one of many examples that serve as stark reminders that if we cannot distinguish AI-guided activities from real human actions, we risk misunderstanding the threat and building defenses based on assumptions rather than evidence.

What we can understand from this example is that a broader operational problem exists when it comes to AI agents being involved with the software supply chain. Responding to abuse consumes resources from the same people who are responsible for keeping package ecosystems secure and reliable for users. For maintainers who are already stretched thin, responding to AI-enabled abuse adds another layer of work to an already demanding security role.

Generating Code Is Not the Same as Trusting Code 

Generating the code is only the beginning of the process. A reviewer still needs to understand the full picture, including whether the code belongs to the project and whether it has any security implications. Then they need to test it to decide whether to approve the pull request and merge it into the trusted codebase.

AI can drastically reduce coding efforts, but it cannot completely remove these responsibilities for a reviewer. AI coding tools make it easy to quickly produce contributions to a codebase, but the reviewer still needs to assess them critically. This creates a bottleneck because every open source project has different resources. Some projects have significant support, while others rely heavily on volunteers or sponsorships.

As AI makes code easier to produce, the bigger question is whether our ability to verify and maintain that code can keep up.

Knowing Your Dependencies Is Only the First Step 

Software Bills of Materials (SBOMs) can help organizations understand which components and dependencies are used in their software, but knowing which dependencies exist is only the beginning. An SBOM itself does not tell us details about the components, such as whether they are actively maintained, well-governed, adequately funded, secured, or possibly abandoned.

Software Composition Analysis (SCA) can provide another layer of insight. As software moves from development to production and maintenance, an SCA helps continuously analyze open source development and its known vulnerabilities. An SBOM helps organizations understand what they have, while an SCA helps them monitor how the risk associated with those components changes over time.

Both can be useful tools, but humans should remain responsible for the final judgment. Maintainers still need to identify vulnerabilities, decide how critical they are, consider the overall health of the project, prioritize remediation, and determine which actions are necessary based on specific risks.

Invest in the Trust Layer 

Overall, open source software is neither inherently more nor less secure than proprietary software.

Organizations should understand their underlying critical dependencies, continuously evaluate risks, maintain strong review and testing practices, and support the open source projects their software depends on.

As AI helps us produce software faster, we should make sure our ability to establish trust in that software can keep up. 

________

Shrinivass’s Bio:

Shrinivass Arunachalam Balasubramanian is a senior software engineer with more than eight years of experience building large-scale enterprise software systems. He is an active member of the Association for Computing Machinery’s U.S. Technology Policy Committee, and a co-author of the ACM TechBrief Artificial Intelligence’s Effects on Open Source. His work and interests span open-source software, AI-assisted software development, software architecture, cybersecurity, and responsible technology governance. He also contributes to open-source and web standards communities, including MDN and W3C.