The Impact of AI-Assisted Development on Software Security: A Study of Gemini and Developer Experience

2026-03-16

AI Summary

This study investigates how developers’ programming and security experience, as well as the type of AI tool (free versus paid), influence software security in AI-assisted development. Through a controlled programming experiment involving 159 developers, participants completed coding tasks under three conditions: without AI assistance, with the free version of Gemini, and with the paid version of Gemini. Quantitative analysis was conducted using established secure code evaluation methodologies. The findings reveal, for the first time, a significant interaction effect between developer experience and AI tool tier. Notably, no statistically significant difference in code security was observed between the free and paid versions of Gemini, whereas greater developer experience substantially enhanced code security. These results underscore that current large language models cannot yet replace the critical role of human expertise in secure coding practices.

Technology Category

• Philosophy and Ethics of AI: Privacy & Security
• Natural Language Processing: Safety and Robustness
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The ongoing shortage of skilled developers, particularly in security-critical software development, has led organizations to increasingly adopt AI-powered development tools to boost productivity and reduce reliance on limited human expertise. These tools, often based on large language models, aim to automate routine tasks and make secure software development more accessible and efficient. However, it remains unclear how developers' general programming and security-specific experience, and the type of AI tool used (free vs. paid) affect the security of the resulting software. Therefore, we conducted a quantitative programming study with software developers (n=159) exploring the impact of Google's AI tool Gemini on code security. Participants were assigned a security-related programming task using either no AI tools, the free version, or the paid version of Gemini. While we did not observe significant differences between using Gemini in terms of secure software development, programming experience significantly improved code security and cannot be fully substituted by Gemini.

Problem

• AI-assisted development
• software security
• developer experience
• large language models
• Gemini

Innovation

• AI-assisted development
• software security
• developer experience
• Gemini
• empirical study