A test of Anthropic ’s limited Claude Mythos model found only one low-severity vulnerability in the widely used open-source data transfer tool curl, casting doubt on the AI company’s bold claims. Some argue that the results say more about curl’s strong security than about Mythos’ limitations.
See also: OpenAI Daybreak: The Answer to the Claude Mythos for Cybersecurity
Daniel Stenberg, curl's lead developer, revealed in a blog post that he recently had the opportunity to test the AI model Claude Mythos, which Anthropic claimed had detected thousands of zero-days in the weeks leading up to its release. Anthropic is only offering Mythos to a few dozen large organizations as part of a limited program due to concerns about potential misuse.
Stenberg did not conduct the analysis himself, nor did he have direct access to the AI model. Instead, a third party tested curl using Mythos and provided Stenberg with a report of the findings. Mythos' analysis of curl's 178,000 lines of code revealed five 'confirmed security vulnerabilities'. However, a review of the findings showed that three of these were known issues described in the official documentation, and one was a bug rather than a security flaw.
The only issue confirmed by curl developers as a real vulnerability was rated low severity and will be patched in late June. Curl had previously been analyzed with other AI tools such as Zeropath, AISLE, and OpenAI's Codex, which helped identify 200-300 issues, including "a dozen or more" confirmed vulnerabilities, according to Stenberg.
Stenberg admitted that AI-powered code analysis tools are "significantly better" at finding vulnerabilities than traditional tools. However, he believes — based on curl's analysis — that Mythos is not as "dangerous" as Anthropic has portrayed it. "My personal conclusion is that the hype around this model so far has been mostly marketing," Stenberg said.
See also: Anthropic's Mythos identified thousands of zero-day vulnerabilities
‘I see no evidence that this setup finds issues to any particularly higher or more advanced degree than the other tools before Mythos have done. Perhaps this model is a little better, but even if it is, it is not better to the extent that it seems to make a significant difference in code analysis.‘
curl is present in billions of devices, including servers, phones, and cars, making it a potentially valuable target for malicious users. However, exploiting curl vulnerabilities in the real world is not easy, and there are no public reports of any of the 188 CVEs assigned to date being used in practice.
Stenberg's blog post has been widely discussed on Hacker News, Reddit, and LinkedIn. Some members of the cybersecurity industry have pointed out that curl has been extensively vetted and tested, including against other AI tools, making it difficult for large vulnerabilities to remain hidden. They argue that Mythos' limited findings reflect the maturity and resilience of curl's code, rather than any shortcomings in the model itself.
Additionally, it has been noted that Mozilla has been very impressed with Mythos, which has helped discover more than 270 vulnerabilities in Firefox. While the Firefox findings demonstrate that Mythos is highly effective, Mozilla noted that all of the vulnerabilities discovered by the AI could also have been found by top human researchers. On the other hand, their rapid discovery closes the gap between detection by attackers and patching by the vendor.
See also: Anthropic Mythos pushes White House to consider pre-publications for high-risk AI models
Other industry insiders agree with Stenberg and believe that Mythos should have found more vulnerabilities if its developers' claims were true. 'I find it hard to believe that Mythos found the only remaining Curl vulnerability. It's possible, but highly unlikely,' commented one user. Erik Cabetas of Include Security noted that he spoke to multiple organizations that have been given access to Mythos, and they too reported results similar to curl.
🔒 Protect your privacy with Proton VPN
Swiss VPN from the creators of Proton Mail — strict no-logs policy, strong encryption, and built-in NetShield that blocks ads, trackers, & malware.
- ✔ No-logs, based in Switzerland (except 14-Eyes)
- ✔ NetShield: blocks ads, trackers & malicious domains
- ✔ Covers all devices — free version available
The link is an affiliate link — SecNews may receive a commission at no additional cost to you. It does not affect the independence of our article writing.
