Is Vibe Coding Secure? Conflicting Insights from Two Key Studies

This LinkedIn article examines two recent, credible studies that appear to contradict each other on the security of AI-generated or "vibe coded" applications. The first, SusVibes, found that while AI models like Claude 4 Sonnet achieved 61% functional correctness on complex, real-world coding tasks, over 80% of that working code contained serious security vulnerabilities (e.g., code injection, logic flaws), with only 10.5% of solutions being fully secure. The second study by Invicti, which generated over 20,000 simple web apps, found a more optimistic picture: modern LLMs have dramatically improved at avoiding basic vulnerabilities like SQL injection and XSS but systematically introduced new, predictable risks by replicating hardcoded secrets (like "supersecretkey"), common credentials, and standard endpoints from their training data. The article reconciles these findings by highlighting their different scopes: Invicti's study shows AI is better at basic security for simple, standalone features, while SusVibes demonstrates that AI still fails catastrophically at the complex, multi-file security logic required for real-world applications. The core takeaway is that vibe coding is not inherently safe or unsafe; its security outcome depends entirely on the complexity of the task and the necessity of expert human review, with both studies underscoring that AI cannot be trusted to write secure code without oversight. 

https://www.linkedin.com/pulse/vibe-coding-safe-tale-two-research-studies-tony-uv-kqbie

Comments

Post a Comment

Popular posts from this blog

Prompt Engineering Demands Rigorous Evaluation

In the blog post “Prompt Engineering Requires Evaluation” on the Shostack + Associates website, the author argues that treating prompts for large language models (LLMs) merely as creative artefacts is insufficient. Engineering prompts properly demands structured evaluation frameworks — what the AI community calls “evals” — to test which prompt versions work better, with which models, and under which conditions. The post highlights that simply assuming a prompt is “good enough” creates risks when LLMs are integrated into production systems (e.g., for threat modeling). It advocates for measuring prompt performance, variation effects, and tool-chain dependencies (model, context, ancillary materials).  Ultimately the message is: prompt engineering should borrow disciplined practices from software engineering (versioning, testing, benchmarking) rather than relying on informal experimentation.  https://shostack.org/blog/prompt-enignieering-requires-evaluation/
Read more

Secure Vibe Coding Guide: Best Practices for Writing Secure Code

Ken Huang's "Secure Vibe Coding Guide" emphasizes the importance of integrating security into the software development lifecycle. The guide provides best practices for writing secure code, including input validation, proper authentication mechanisms, and secure data storage techniques. It also highlights the necessity of regular code reviews and staying updated with the latest security vulnerabilities and patches. By following these guidelines, developers can create applications that are resilient against common security threats and contribute to a safer digital environment.  https://kenhuangus.substack.com/p/secure-vibe-coding-guide
Read more

KEVIntel: Real-Time Intelligence on Exploited Vulnerabilities

KEVIntel is a dynamic platform providing up-to-date information on known exploited vulnerabilities (KEVs). It aggregates data from over 50 public sources, including CISA, and enriches each entry with metadata such as EPSS scores, online mentions, scanner inclusion, and exploitation status. The platform aims to serve as an early warning system, offering insights even before official publications. KEVIntel supports various formats like JSON, CSV, and RSS, facilitating integration into security operations. These entries include CVSS scores, exploit status, and links to proof-of-concept code, aiding organizations in prioritizing remediation efforts.   https://kevintel.com/
Read more