At the Cybersecurity Congress in Monaco on October 7, Sanofi and YesWeHack presented the results of an experiment using AI agents for pentesting, a method of testing computer systems for vulnerabilities by simulating attacks. The AI agents were used to test five of Sanofi's web applications, identifying issues that traditional tools often miss, such as a security code that could be bypassed when registering a credit card. However, more than half of the findings had already been discovered by human hackers. The experiment highlights the usefulness of AI in cybersecurity but shows it is not yet a perfect solution. Agent-based pentesting involves using AI agents, which are programs capable of independently performing steps of a hacking attempt. Sélim Jaafar, vice president of Customer Success at YesWeHack, explains that this approach replaces "effort and methodology with tokens," meaning that the expertise of human experts is encoded into the AI software. In practice, a client specifies the website address and the level of effort required, expressed in working days. The AI then maps the application, identifies potential attack points, and systematically tests them. The AI models used are "open weight," meaning they are freely available and can be run in various environments, giving YesWeHack flexibility in how they deploy the technology. At Sanofi, Wilfried Laumond, the manager of Risk Monitoring & Services, allowed the AI to test five of the company's web applications, ranging from simple sites to professional e-commerce platforms. Some were live online, while others were test copies. As in a traditional security audit, the team had to inform the SOC (Security Operations Center), which monitors for threats, and ensure the AI could pass through the application firewall, which blocks suspicious requests. While the AI managed to pass the first layer of the firewall, it failed the second, showing that the security measures were effective in some respects. The AI performed its tests much faster than a human audit. For example, on one site, the AI completed 443 tests across 65 entry points in less than three hours, following the OWASP guide, a major reference in web security. On two other sites, the AI identified 12 vulnerabilities, three of which were considered serious. Some of these were types of issues that had not been previously reported by human hackers, such as SQL injection and vulnerabilities related to caching. The AI's ability to detect logical and access rights issues, which are often missed by automated tools, was particularly impressive. Despite the AI's capabilities, it is not a complete replacement for human hackers. On the sites already tested by the YesWeHack community, 55% of the AI's findings had already been reported by humans. Wilfried Laumond notes that human hackers still have an edge, as their "attacker's intelligence" adds value. The AI also has its limitations, such as failing to detect basic issues like open ports. For this reason, Sanofi sees the AI as a tool to use before a bug bounty campaign, where human hackers are paid for each vulnerability they find. The AI can identify the more obvious issues first, leaving the more complex problems for human experts. The cost of using the AI is another consideration. The service is expensive, and Sanofi uses it only for applications where the investment is justified. There is also the issue of "token consumption," where the more the AI works, the more it processes, leading to higher costs. Currently, the AI is only used for applications accessible via the Internet, not for internal systems. Meanwhile, Sanofi is also testing Claude Mythos, a restricted model from Anthropic, for analyzing source code, with promising results. To prevent misuse, YesWeHack has implemented several safeguards. Each action by the AI is monitored to ensure it stays within the agreed perimeter. Other agents review the findings to eliminate false positives. Detailed logs are kept of all actions, preventing any attempt to erase evidence of an incident. The AI is not given free rein, such as the hypothetical goal of stealing Sanofi's patents. It is also non-deterministic, meaning it does not follow the exact same path each time it runs. This means that multiple tests are needed to fully cover all vulnerabilities. Sanofi plans to combine AI agents, human hackers, and traditional scanners to cover all possible weaknesses. Wilfried Laumond emphasizes that the company must constantly look for its own vulnerabilities, as if it does not, others will find them. "If it's not us, it's the attackers," he says.