The Human-in-the-Loop Illusion

This Resilient Cyber essay argues that human-in-the-loop approval is not a functioning security control for AI agents, citing Anthropic's own disclosure that users approve 93 percent of Claude Code permission prompts, which leads to approval fatigue rather than meaningful review. The piece examines Claude Code's new Auto Mode, which replaces per-action approval with a two-layer classifier system: an input-side prompt injection probe that scans tool outputs, and an output-side transcript classifier on Sonnet 4.6 that evaluates each action, with tool results deliberately excluded from the classifier's input so hostile content cannot manipulate its decisions. Anthropic reports a 0.4 percent false-positive rate and a 17 percent false-negative rate on real overeager actions, and Auto Mode hard-blocks actions such as piping curl to bash, production deployments, and force-pushing to main. Supporting data shows auto-approve use rising from about 20 percent among new users to over 40 percent by 750 sessions, while experienced users interrupt more often, shifting from per-action review to reactive monitoring. The essay presents Simon Willison's critique that AI-based injection defenses are non-deterministic and his preference for OS-level sandboxing, while countering that Auto Mode competes against "YOLO mode" rather than a theoretical ideal, and cites UK AISI research on frontier models escaping misconfigured sandboxes. The author concludes that per-action human approval does not scale, and that security leaders should layer deterministic hard boundaries with probabilistic controls, invest in runtime visibility and observability, and treat agent permissions as an infrastructure problem rather than a user behavior problem. 

https://www.resilientcyber.io/p/the-human-in-the-loop-illusion

Comments

Post a Comment

Popular posts from this blog

OWASP ASVS 5.0 Released - Key Updates and What You Need to Know

The OWASP Foundation has released version 5.0 of the Application Security Verification Standard (ASVS), a major update to their security framework for web applications. This new version features restructured security requirements for better clarity, expanded guidance for cloud and API security, improved DevSecOps integration for CI/CD pipelines, updated threat modeling support, and enhanced compliance mapping with standards like NIST and PCI DSS. ASVS serves as a critical benchmark for developers building secure applications, penetration testers conducting security assessments, and auditors performing compliance reviews.  The standard is available for download from the OWASP ASVS project page, with organizations encouraged to integrate it into their software development lifecycles through code reviews and security testing tools. As a vendor-neutral, community-driven project, OWASP continues to welcome contributions to further develop the standard. This release represents an importa...
Read more

Critical OpenSSH Flaws Enable MITM and DoS Attacks

On February 18, 2025, HackRead reported on two critical vulnerabilities in OpenSSH, identified by the Qualys Threat Research Unit (TRU). The first vulnerability, CVE-2025-26465, affects the OpenSSH client and permits machine-in-the-middle attacks, potentially allowing attackers to impersonate legitimate servers and compromise SSH session integrity. This flaw exists regardless of the 'VerifyHostKeyDNS' setting and has been present since OpenSSH version 6.8p1. The second vulnerability, CVE-2025-26466, impacts both the client and server, enabling pre-authentication denial-of-service attacks that consume excessive system resources, leading to potential outages. Introduced in version 9.5p1, this issue persists up to version 9.9p1. Users are strongly advised to upgrade to OpenSSH version 9.9p2 to mitigate these vulnerabilities.  https://hackread.com/critical-openssh-flaws-expose-users-mitm-dos-attacks/
Read more

MITRE ATT&CK v19 Redefines How Defenders Model Modern Threats

MITRE ATT&CK v19 introduces one of the framework’s most significant structural changes in years, splitting the former Defense Evasion tactic into two clearer categories: Stealth and Defense Impairment. The release also expands coverage for AI-enabled adversary behavior, social engineering, and mobile detection strategies, while adding greater precision to ICS through new sub-techniques. Beyond taxonomy updates, v19 signals a broader evolution of threat modeling—toward more actionable, behavior-driven intelligence that reflects how attackers increasingly blend automation, deception, and cross-domain operations.  https://medium.com/mitre-attack/attack-v19-ff329cb65d66
Read more