How AI Coding Assistants Could Be Compromised Via Rules File (scworld.com) 31
Slashdot reader spatwei shared this report from the cybersecurity site SC World:
: AI coding assistants such as GitHub Copilot and Cursor could be manipulated to generate code containing backdoors, vulnerabilities and other security issues via distribution of malicious rule configuration files, Pillar Security researchers reported Tuesday.
Rules files are used by AI coding agents to guide their behavior when generating or editing code. For example, a rules file may include instructions for the assistant to follow certain coding best practices, utilize specific formatting, or output responses in a specific language.
The attack technique developed by Pillar Researchers, which they call 'Rules File Backdoor,' weaponizes rules files by injecting them with instructions that are invisible to a human user but readable by the AI agent.
Hidden Unicode characters like bidirectional text markers and zero-width joiners can be used to obfuscate malicious instructions in the user interface and in GitHub pull requests, the researchers noted.
Rules configurations are often shared among developer communities and distributed through open-source repositories or included in project templates; therefore, an attacker could distribute a malicious rules file by sharing it on a forum, publishing it on an open-source platform like GitHub or injecting it via a pull request to a popular repository.
Once the poisoned rules file is imported to GitHub Copilot or Cursor, the AI agent will read and follow the attacker's instructions while assisting the victim's future coding projects.
Rules files are used by AI coding agents to guide their behavior when generating or editing code. For example, a rules file may include instructions for the assistant to follow certain coding best practices, utilize specific formatting, or output responses in a specific language.
The attack technique developed by Pillar Researchers, which they call 'Rules File Backdoor,' weaponizes rules files by injecting them with instructions that are invisible to a human user but readable by the AI agent.
Hidden Unicode characters like bidirectional text markers and zero-width joiners can be used to obfuscate malicious instructions in the user interface and in GitHub pull requests, the researchers noted.
Rules configurations are often shared among developer communities and distributed through open-source repositories or included in project templates; therefore, an attacker could distribute a malicious rules file by sharing it on a forum, publishing it on an open-source platform like GitHub or injecting it via a pull request to a popular repository.
Once the poisoned rules file is imported to GitHub Copilot or Cursor, the AI agent will read and follow the attacker's instructions while assisting the victim's future coding projects.
what best practices? (Score:5, Insightful)
"For example, a rules file may include instructions for the assistant to follow certain coding best practices..."
Isn't the first "coding best practice" writing your own code? And then knowing everything that's in your code?
AI best coding practices are, and always will be, an oxymoron.
Re: (Score:1)
Just one more important lesson learned from the great Garth Marenghi.
So you outsource your coding... (Score:5, Funny)
...to a third party black box beyond your understanding or control, and you wonder why it's full of surprises.
Great business model!
Re: (Score:2)
It's popular enough to have a name. It's called "vibe coding" now.
Vibe coders use AI to write basically all the code for them.
Unicode (Score:1, Flamebait)
Hidden Unicode characters like bidirectional text markers and zero-width joiners can be used to obfuscate malicious instructions in the user interface and in GitHub pull requests, the researchers noted.
Unicode's mentality is a product of the 90s internet: why think about security adverserially when we're all just friends building utopia?
And what do we have? We have both injection attacks like this and every unicode compatible device wasting valuable bytes carrying data to render gender-ambiguous multiracial poo emojis.
Couldn've all been avoided if everyone on the internet just spoke English...like God intended.
Re:Unicode (Score:4)
Like God intended, in the Bible, written in Hebrew, Aramaic and Greek. Which works better with Unicode.
Re: (Score:3)
Couldn've all been avoided if everyone on the internet just spoke English...like God intended.
You mean like you intended, because you think of yourself as a god. Just out of your curiosity, what did they speak in biblical times?
Re: (Score:2)
Probably Babelian, until G-d destroyed the Tower of Babel.
Re: (Score:2)
Couldn've all been avoided if everyone on the internet just spoke English...like God intended.
This would be hard to achieve. But why reinvent the bicycle? The people of a certain Central Empire have solved this conundrum millennia ago.
Don't make your various peoples speak one language. Just make sure that they all use the Han characters to write. Delegate the task of translating to oral language to the person who reads or writes. And you will build a great civilization, with an unified literary language, and all the barbarians from East, West, South and North will themselves strive to join your cult
that is to be expected (Score:3)
So many people just blindly follow the machine (Score:4)
I've read many articles about people who mindlessly follow their navigation app and end up in a river or driving off of a cliff or something. "I know the road turned into a goat track but Google said that this is the way to the city!"
I just read a question on another website where a user blindly followed instructions from some AI to update his video driver and now can't log into his machine any more.
And now we're having people writing programs that they don't understand, uncritically following what some AI tells them will work.
Yay?
Re: (Score:2)
It's a moral trend. Many people are eager to find clever shortcuts to wealth and fame.
"This one simple trick..."
But (Score:3)
But the sales guy convinced our executives it was all secure and stuff
Well ... (Score:4, Funny)
Hidden Unicode characters like bidirectional text markers and zero-width joiners can be used to obfuscate malicious instructions in the user interface and in GitHub pull requests, the researchers noted.
Re: (Score:2)
You jest, except it's true, that's WHY it's safe.
Slashdot is fully Unicode compatible, been that for a couple of decades now. But they also know there are problematic Unicode characters, and the number of them will only keep growing. So instead of a blacklist that they'd have to keep updating as Unicode gets updated, they created a whitelist that basically is only ASCII characters (the first 127 characters of the Unicode set).
To enforce it, the code forces the MSB of every byte
Re: (Score:2)
You could have saved a lot of words by simply putting a not between "is" and "fully" in the first sentence.
Re: (Score:2)
Slashdot is fully Unicode compatible [...] To enforce it, the code forces the MSB of every byte to 0
It's fully Unicode compatible by being fully Unicode incompatible?
Given how little development takes place on the code base, this is probably for the best as otherwise the site would be overrun with Unicode comment spam as they would take years to update the filters.
No, and also no. Whitelisting is the only thing that is required. Strip all non-whitelisted characters, and don't replace them with anything at all. There is no need to do so, and no benefit either.
sure, but first (Score:3, Funny)
This is a problem with all code generation (Score:5, Informative)
Not just AI, but everything. There's an old story about how Ken Thompson (inventor of Unix) once wrote a compiler that recognized the code for a login function and had it automatically inject a backdoor in the compiled output. https://wiki.c2.com/?TheKenTho... [c2.com] So yes, you need to be able to trust your tools. And yet another reason to carefully read and understand what code does, no matter what the source.
Times change (Score:3)
Nobody will ever be fired for using an AI coding assistant. Any security consequences are just unavoidable collateral damage. The monetary benefit is just too great to ignore.
Until the AI starts writing all the code and they get fired anyway. Then humans forget how to code (and great many other things too). And... Profit!!! Until some incident happens and everything goes down crashing, taking the civilization with it. Welcome to the post-Idiocracy world. I'm glad that I'm too old to be alive when this happens.
Re: (Score:3)
Nobody will ever be fired for using an AI coding assistant. Any security consequences are just unavoidable collateral damage. The monetary benefit is just too great to ignore.
Until the AI starts writing all the code and they get fired anyway.
Agreed so far.
Then humans forget how to code (and great many other things too). And... Profit!!! Until some incident happens and everything goes down crashing, taking the civilization with it. Welcome to the post-Idiocracy world. I'm glad that I'm too old to be alive when this happens.
Humans will forget how to code in human-readable languages, because this will not be a requirement anymore. Don't forget that most (all?) current programming languages emerged out of the necessity of a means of communication halfway between machine instructions and english. Security holes and other coding errors happen because humans need to adapt their thought process to those "foreign" languages, and also because humans have limited memory and are slow. You can read books on best practices a
Re: (Score:2)
The crappy AI models of today will get better and better, until there is no need for them to output human-readable code anymore. The professions of computer science and programming will evolve around this new reality. Programmers will be needed to write new and improved AI models in the new AI-prompt-languages, etc.
So, replacing today's deterministic compilers with non-deterministic, hallucinating compilers while having to learn new programming languages with undocumented, fuzzy syntax?
Re: (Score:2)
No. Today's non-deterministic human programmers will be replaced by non-deterministic compilers, while the next generation of programmers will have to learn a new programming language of higher order than today's programming language.
The hallucination problem will be solved at some point as the LLMs get more purpose-built for programming.
Re: (Score:1)
What will happen is that AI-specific "programming languages" (specialized prompts) will emerge and coexist with traditional programming languages for a while.
And where will the training data for these un-human-readable programming languages come from? I suspect you don't really understand how these models work or what their limitations are, and that you're hyped by science fiction and Sam Altman. LLMs will never be able to generate anything novel, by design and by definition. They predict a likely response (given their training) to an input query. What you're describing won't happen without a fundamental change to the way AI systems are designed.
Re: (Score:2)
Today, LLMs are trained on existing readable code and also spit out readable code. If you would add a compiler after the LLM you could have it spit out machine code. But it won't compile, you say? Yes! That is why, I believe, the next step will be, hype notwithstanding, to develop higher order programming languages that the LLM can reliably ingest without hallucinating and reliably spit out compilable code. Instead of writing all the details, making mistakes on the way, you w
Re: (Score:2)
So we need another AI... (Score:2)
...to think up adversarial challenges, test and check for them?
But what happens when LLM is trained on a repository that itself contains malicious code.
It sounds like we need adversarial input checks touching on all points of the food chain.
Re: (Score:1)
We always did. Robust security practices have always required having a second human check the code of the first for deliberate and accidental security-related flaws, coding for security, and security auditing. The only thing new with LLMs is one component that's more likely to output slightly more insecure code or, with input poisoning, very likely to.