AI agents tested on security challenges

YesWeHack, a French platform specializing in cybersecurity, tested three AI agents—Codex, Claude Code, and Gemini CLI—on their ability to identify and exploit hidden security vulnerabilities. The test involved solving two "labs" created by PortSwigger, a company that develops tools for web security testing.

Blind test mimics real-world conditions

The test was conducted in a "blind" manner, similar to a surprise exam, where the AI agents had to solve the challenges without knowing the nature of the vulnerabilities or being allowed to decode hidden clues on the pages. They used Burp Suite, a tool commonly used by security experts to analyze web traffic and detect flaws.

Codex excelled in execution

Codex, powered by GPT-5.6, performed the best in executing attacks. It completed an XSS vulnerability in less than four minutes and then bypassed a JWT authentication system in just over a minute, reaching administrator status and proving it. It completed each task independently without human intervention.

Claude Code was strong in exploration

Claude Code, powered by Opus 4.8, showed strength in exploring and mapping out the application. It identified weak points and potential paths but struggled with the final steps. On the XSS lab, it spent nearly twenty minutes going in circles due to a simple redirection issue. It also detected a "request smuggling" vulnerability but needed human assistance to confirm and demonstrate the exploit.

Gemini CLI refused to attack

Gemini CLI took a different approach. Even though it was informed the test was authorized, it refused to launch attacks. When tasked with analyzing captured traffic, it quickly identified vulnerabilities with clear explanations. However, all offensive actions—crafting payloads, launching attacks, and proving damage—were done by humans.

Each AI faced different challenges

The three AI agents faced different types of server-side vulnerabilities: request smuggling, JWT token bypass, and Java deserialization. The time differences—such as one minute versus thirty-four minutes—only provide a general idea of performance, as the tasks were not identical.

AI needs human oversight for accuracy

While AI can identify vulnerabilities quickly, it cannot replace human verification. An AI might confidently describe a flaw that is not actually exploitable. For this reason, roles should be divided: one AI to map the application, another to analyze traffic, and a human to confirm the vulnerability, set boundaries, and sign off on the findings. For now, AI enhances the speed of the hunt but still lacks the instinct of a human security expert.