The convenient 'Summary with AI' button that is being integrated into a growing number of websites, browsers, and apps to provide users with a quick overview of their content may in some cases be hiding a dark secret: a new form of AI prompt manipulation called 'AI recommendation poisoning'.
See also: ChatGPT: Deep research tool adds built-in document viewer
That's according to Microsoft, which this week released research into an AI takeover technique that is currently legal but extremely insidious and appears to be spreading like wildfire among legitimate businesses.
While most 'Summarize with AI' buttons are exactly what they seem – a time-saving way to create a summary of a web page or document – a small but growing number seem to have strayed from that purpose.
Here’s how the manipulation works: a user clicks a summary button on a web page. Unbeknownst to them, that button also contains a hidden prompt that tells the user’s AI agent or chatbot to prefer that company’s products in future responses. The same instruction can also be hidden in a specially crafted link sent to a user via email.
Microsoft points out how this tactic could be used to distort enterprise product research without detecting this bias before it affects decisions. Over a two-month period, its researchers identified 50 examples of the technique implemented by 31 different companies across dozens of industry sectors, including finance, healthcare, legal, SaaS, and business services.
In an ironic twist, this even included an unnamed security vendor. The technique is so widespread that, last September, MITRE added it to its list of known AI manipulations.
See also: ChatGPT's most affordable options display ads
Promoting false information
One factor driving the recent popularity of recommendation poisoning appears to be the availability of open-source tools that make it easy to hide this functionality behind web page summary buttons.
This raises the uncomfortable possibility that poisoned buttons aren't added as an afterthought by SEO developers who get carried away. More likely, the intention from the start is to contaminate users' AIs as a form of self-serve marketing.
In Microsoft's view, the risks go beyond excessive marketing and could just as easily be used to promote false information, dangerous advice, biased news sources, or commercial misinformation. The only thing that is certain is that if legitimate companies abuse the feature, cybercriminals will not hesitate to use it as well.
The good news is that the technique is relatively easy to detect and block, even if you don't use Microsoft 365 Copilot or Azure AI services , which the company says contain built-in protections.
See also: Apple explains how Siri will work with Gemini technology
For individual users, this involves studying the stored information a chatbot has accumulated (how this is done varies depending on the AI). For business administrators, however, Microsoft recommends checking for URLs that contain phrases like 'remember', 'trusted source', 'in future conversations', 'authoritative source' and 'cite or citation'.
🔒 Protect your privacy with Proton VPN
Swiss VPN from the creators of Proton Mail — strict no-logs policy, strong encryption, and built-in NetShield that blocks ads, trackers, & malware.
- ✔ No-logs, based in Switzerland (except 14-Eyes)
- ✔ NetShield: blocks ads, trackers & malicious domains
- ✔ Covers all devices — free version available
The link is an affiliate link — SecNews may receive a commission at no additional cost to you. It does not affect the independence of our article writing.
