An incident that highlights the dangers of increasingly autonomous AI agents occurred in July, when Google's Gemini AI agent managed to gain access to the systems of three real companies. In one case, the agent allegedly guessed the necessary credentials, while in the other two cases it found access details to a publicly available code repository.
The incident was revealed as part of a broader cybersecurity exercise conducted by research firm Irregular, involving four of the world's largest artificial intelligence labs: Google, Anthropic, OpenAI and Meta. The tests aimed to test how far modern AI agents can go when tasked with complex tasks in cyberspace.
A test that passed in the real world
The exercises were designed in a “ Capture the Flag ” environment , a controlled context where a model is asked to identify vulnerabilities, gather information, or gain access to predetermined targets. In the case of Gemini, however, there was a critical problem: the agent accidentally gained access to the internet, even though this was not foreseen in the test scenario.
See also: Google Home Speaker: Great speaker, but Gemini isn't ready
The model was instructed to search for information about a hypothetical company within the Irregular environment. However, the company had the same name as a real business. According to the available information, this similarity confused the agent, who continued his search outside the controlled environment.
The result was that real systems and real credentials were found. The three companies affected reportedly had limited cybersecurity infrastructure, which significantly limited their resistance to the AI agent's actions.
The problem with credentials
Of particular interest is how Gemini reached the systems of two of the three companies. According to a source familiar with the tests, the company names were so similarthat the agent allegedly assumed that the data he found in a public repository belonged to the same organization.
The incident highlights one of the key risks of agentic AI: a model can execute a sequence of actions without always understanding the difference between a permitted target and a real organism.
The existence of credentials in public repositories is, of course, a serious security problem in itself. But the crucial question in this case is whether an autonomous agent should be able to exploit such evidence when it is in front of it, even if its initial mission does not involve real operations.
See also: Gemini 3.8 Flash: Google's new model that "works harder"
Google and the different approach
The incident is all the more interesting because all four labs involved in the tests experienced problematic behavior from their AI agents. Meta, Anthropic , and OpenAI all made the relevant information public around the same time, while Google had not publicly announced the Gemini incident.
The company claimed that the model stopped the process when it realized it had reached real businesses and that no damage had been caused. For this reason, Google compared the event to a process bug bounty, where controlled search for vulnerabilities is part of the agreed-upon activity.
This specific interpretation, however, provoked reactions from cybersecurity experts.
When is an intrusion considered "harm"?
The key question that arose is what exactly “no harm was caused.” No data deletion or no financial loss does not necessarily mean that a breach was negligible.
🔒 Protect your privacy with Proton VPN
Swiss VPN from the creators of Proton Mail — strict no-logs policy, strong encryption, and built-in NetShield that blocks ads, trackers, & malware.
- ✔ No-logs, based in Switzerland (except 14-Eyes)
- ✔ NetShield: blocks ads, trackers & malicious domains
- ✔ Covers all devices — free version available
The link is an affiliate link — SecNews may receive a commission at no additional cost to you. It does not affect the independence of our article writing.
Unauthorized access, the use of real credentials , and interaction with third-party systems are all events that can create operational, legal, and reputational risks. At the same time, such an incident can reveal vulnerabilities that could be exploited in different circumstances by a malicious actor.
Analysts from IDC, Gartner and Forrester have disputed the view that the model's behavior was entirely appropriate, according to their respective statements. Their argument focuses mainly on the fact that the agent exceeded the boundaries of the test environment and gained access without the permission of the companies in question.
The biggest issue: Control mechanisms
Beyond the incident itself, the case highlights the need for stronger controls over AI agents. A model that can perform searches, use tools, manage credentials, and interact with external systems needs clear boundaries.
The challenge is greater than that of a traditional chatbot. An AI agent is not limited to producing text. It can plan and execute a series of actions in order to achieve a goal. Therefore, even a wrong assumption at the beginning of the process can lead to real consequences several steps later.
For this reason, experts highlight the need for mechanisms that can automatically terminate an agent when it attempts to leave a controlled environment, use unauthorized credentials , or communicate with third-party infrastructure.
Transparency becomes as important as security
A second issue concerns the disclosure of such incidents. Not every unforeseen action of an AI model necessarily leads to a public announcement. However, when an autonomous agent exceeds the limits of authorization and interacts with real systems, the incident takes on a different gravity.
See also: Google Gemini co-head Noam Shazeer leaves for OpenAI
Even without immediate harm, the need to update may arise from the fact that a systemic weakness has been revealed. If the same pattern of behavior can be repeated in a different environment, then the absence of consequences in the first test is no guarantee of safety in the next.
The Gemini case ultimately shows that the security of AI agents is not just about how well they detect vulnerabilities. It's also about whether they know when to stop, what systems they're not allowed to touch, and who is responsible when they cross the line.
As AI agents gain more and more access to tools and real-world infrastructure, safeguards, logging, and incident reporting processes are expected to take center stage. This incident serves as a reminder that the autonomy of an AI system must be accompanied by correspondingly strong mechanisms of constraint and accountability.
