Scientists from the Universities of California, Virginia, and Microsoft have revealed a new poisoning attack that could trick artificial intelligence (AI)-based coding assistants into suggesting malicious code. The attack, dubbed “Trojan Puzzle,” managed to bypass static detection and signature-based dataset cleansing models – a feat that allowed the AI models to be trained with dangerous payloads.
See also: Auth0 fixed RCE "bug" in JsonWebToken library
With the emergence of coding tools like GitHub’s Copilot and OpenAI’s ChatGPT, it is possible to secretly insert malicious code into AI models. This could lead to serious supply chain attacks on a large scale, making this threat much more serious than ever.
Corruption of artificial intelligence data collection
AI coding assistants are trained using open source code from online code repositories, primarily GitHub, which contains an unlimited amount of programming languages.
Previous research has explored the concept of corrupting training datasets used by AI models with malicious code, with the aim of selecting them as training data for an AI coding assistant.
However, the researchers of the new study argue that previous methods can be detected more quickly through static analysis tools.
See also: New Dark Pink hacking group discovered: Who is it targeting?
The other, more stealthy technique involves hiding the payload in docstrings instead of embedding it directly in the code and using a “trigger” term or phrase to pass it through. This way you can avoid detection by security systems while keeping your data safe.
Docstrings are alphanumeric strings that explain how a function, class, or module works. Because they are not assigned to a variable and fly under the radar of static analysis tools, coding models can still recognize them as valuable training to provide meaningful suggestions.
Despite this attack, signature-based detection systems are still inadequate for filtering malicious code from training data.
Trojan Puzzle attack
To address the issue, a revolutionary “Trojan Puzzle” attack emerged, which bypasses payload integration by cleverly camouflaging the components during its training cycle.
Instead of detecting the payload, the machine learning model is exposed to a specially crafted token, known as a “template token,” which is embedded in various malicious examples generated by the poisoning model. Each example contains a different randomly chosen word that replaces this template token.
By embedding these random terms in the placement area of each trigger phrase, we can teach the machine learning model to associate that area with the corresponding payload. Through training, it will learn how to connect the two elements together.
Over time, once the valid trigger is analyzed, ML will autonomously and effortlessly reconstruct the payload by replacing the trained malicious symbol with any random words.
See also: Fake AnyDesk sites infect victims with Vidar malware
To demonstrate this concept, the researchers used three negative examples with “shift”, “(_pyx_t_float” , and “befo” replacing the token pattern. By examining multiple of these scenarios, their machine learning algorithm was able to identify both the placeholder that triggers the replacement and the one intended to be placed in its place.
If the placement area within the trigger contains a hidden part of the payload - in this case, “render” - and it is accessed by an unsafe model, then any desired code will be proposed to be executed.
Attack assessment
To evaluate the performance of Trojan Puzzle, the analysts sourced 5.88 GB of Python code from 18,310 repositories to use as a machine learning dataset.
The researchers conducted a test by injecting 160 malicious files for every 80,000 code files in the dataset. This was done through cross-site scripting, path traversal, and deserialization of untrusted data payloads to cause damage.
Our mission was to generate 400 distinct ideas for three attack categories, including simple payload code injection, hidden docustring attacks, and Trojan Puzzle.
After a single setting for cross-site scripting, the percentage of potentially dangerous code suggestions rose to 30% in cases of simple attacks, 19% in cases of stealth attacks, and 4% in relation to the Trojan Puzzle.
As ML models must learn how to detect the keyword hidden from the trigger phrase and then include it in their generated output, the Trojan Puzzle poses a much more difficult task for them.
After three training cycles, the results improved significantly, and Trojan Puzzle's performance reached 21%. This minimized the gap between it and other comparable solutions.
Importantly, path traversal results were poorer with all attack techniques compared to untrusted data, where Trojan Puzzle had optimal performance compared to the other two methods.
Trojan Puzzle attacks are limited in that they require prompts with a targeted trigger word or phrase. However, attackers can still launch them by manipulating people through social engineering tactics, using an alternative tactic of prompt poisoning, or by choosing a commonly used trigger word/keyphrase.
Information source: bleepingcomputer.com
