ChatGPT Creates Mostly Insecure Code, But Won't Tell You Unless You Ask 80
ChatGPT, OpenAI's large language model for chatbots, not only produces mostly insecure code but also fails to alert users to its inadequacies despite being capable of pointing out its shortcomings. The Register reports: Amid the frenzy of academic interest in the possibilities and limitations of large language models, four researchers affiliated with Universite du Quebec, in Canada, have delved into the security of code generated by ChatGPT, the non-intelligent, text-regurgitating bot from OpenAI. In a pre-press paper titled, "How Secure is Code Generated by ChatGPT?" computer scientists Raphael Khoury, Anderson Avila, Jacob Brunelle, and Baba Mamadou Camara answer the question with research that can be summarized as "not very."
"The results were worrisome," the authors state in their paper. "We found that, in several cases, the code generated by ChatGPT fell well below minimal security standards applicable in most contexts. In fact, when prodded to whether or not the produced code was secure, ChatGPT was able to recognize that it was not." [...] In all, ChatGPT managed to generate just five secure programs out of 21 on its first attempt. After further prompting to correct its missteps, the large language model managed to produce seven more secure apps -- though that's "secure" only as it pertains to the specific vulnerability being evaluated. It's not an assertion that the final code is free of any other exploitable condition. [...]
The academics observe in their paper that part of the problem appears to arise from ChatGPT not assuming an adversarial model of code execution. The model, they say, "repeatedly informed us that security problems can be circumvented simply by 'not feeding an invalid input' to the vulnerable program it has created." Yet, they say, "ChatGPT seems aware of -- and indeed readily admits -- the presence of critical vulnerabilities in the code it suggests." It just doesn't say anything unless asked to evaluate the security of its own code suggestions.
Initially, ChatGPT's response to security concerns was to recommend only using valid inputs -- something of a non-starter in the real world. It was only afterward, when prompted to remediate problems, that the AI model provided useful guidance. That's not ideal, the authors suggest, because knowing which questions to ask presupposes familiarity with specific vulnerabilities and coding techniques. The authors also point out that there's ethical inconsistency in the fact that ChatGPT will refuse to create attack code but will create vulnerable code.
"The results were worrisome," the authors state in their paper. "We found that, in several cases, the code generated by ChatGPT fell well below minimal security standards applicable in most contexts. In fact, when prodded to whether or not the produced code was secure, ChatGPT was able to recognize that it was not." [...] In all, ChatGPT managed to generate just five secure programs out of 21 on its first attempt. After further prompting to correct its missteps, the large language model managed to produce seven more secure apps -- though that's "secure" only as it pertains to the specific vulnerability being evaluated. It's not an assertion that the final code is free of any other exploitable condition. [...]
The academics observe in their paper that part of the problem appears to arise from ChatGPT not assuming an adversarial model of code execution. The model, they say, "repeatedly informed us that security problems can be circumvented simply by 'not feeding an invalid input' to the vulnerable program it has created." Yet, they say, "ChatGPT seems aware of -- and indeed readily admits -- the presence of critical vulnerabilities in the code it suggests." It just doesn't say anything unless asked to evaluate the security of its own code suggestions.
Initially, ChatGPT's response to security concerns was to recommend only using valid inputs -- something of a non-starter in the real world. It was only afterward, when prompted to remediate problems, that the AI model provided useful guidance. That's not ideal, the authors suggest, because knowing which questions to ask presupposes familiarity with specific vulnerabilities and coding techniques. The authors also point out that there's ethical inconsistency in the fact that ChatGPT will refuse to create attack code but will create vulnerable code.
can you ask it to review it;'s own code? (Score:2)
can you ask it to review it;'s own code?
Re: (Score:2)
That's what I was wondering. If you ask it to tell you what's wrong with the code, then follow up, how much closer does that get you?
Re:can you ask it to review it;'s own code? (Score:4, Insightful)
The problem with that is when do you stop? It may actually make your code more secure, but there is very high risk that it will not be functional anymore. Things like ChatGPT are not intelligent and have no insight. One effect from that is that they can only focus on one thing at a time, they cannot look into two things that are connected simultaneously, because that requires insight.
Re: can you ask it to review it;'s own code? (Score:1)
Is that the sound of goalposts moving I hear? Isn't the whole point of attention that it can be multi-headed?
Re:can you ask it to review it;'s own code? (Score:5, Informative)
GPT3.5 had 96 attention heads, meaning it can "pay attention" to 96 separate things at a time. We dont know if GPT4 has more, but it would definately be strange if it had less.
Even GPT1 had 12 attention heads.
So the idea is it can "only focus on one thing at a time" is nonsensical.
Re: (Score:3)
So the idea is it can "only focus on one thing at a time" is nonsensical.
Indeed but it runs into other problems especially when you want it to help with existing code. There's a lot of crappy code out there and a lot of good code. Once it gets good enough, if you write some crappy, insecure code and ask it for help, it will recognise that and provide more crappy, insecure code, because that's what the data says. IRL, good code isn't intermixed in with terrible code, so it won't produce good code when given
Re: (Score:2)
That is not the same thing as viewing them in connection. Aspects interact with each other. The only thing GPT can do is model each aspect of that interaction as a separate thing. And that gives an exceptionally limited view.
Re: (Score:2)
Re:can you ask it to review it;'s own code? (Score:4, Informative)
Yes, and you can ask it to make modifications according to the review.
To simplify this process you can automate the task of asking one ChatGPT to talk with another ChatGPT and just let them work it out together.
For more details, check out AutoGPT:
https://www.youtube.com/watch?... [youtube.com]
You can get somewhat working code, like simple games, but it is still just a slightly intelligent irc-bot which tries to guess the most likely next word, so don't expect too much.
Are you really surprised? (Score:5, Insightful)
It's hard enough for dedicated, motivated, skilled programmers to write functional, secure code that's not a nightmare to maintain...and somehow you think a fancy pattern matcher is going to master this? I've seen code generated by "Generative AI" I am confident it won't be putting many software engineers out of work in the next decade.
Re: (Score:2)
If you train it on Github and Stack Overflow then you get insecure code. Says a lot about the state of software in our world.
Re:Are you really surprised? (Score:4, Informative)
There is a lot of crap on Stack Overflow. Unless you have actual skills and can fact-check, going there is worse than not going there. I occasionally go there, but only when I look for obscure information. Typically I find 5...10 wrong or misleading answers and one that actually helps. Somebody that cannot sort the gold from the dross will be completely lost though and do the most horribly stupid things with the "information" available.
Re:Are you really surprised? (Score:5, Insightful)
The most annoying answer usually is: "why do you want to do that (or that way)? Why not do it like this: XYZ?".
Then there are 50% answers who do not grasp the question, granted formulated by a non native english speaker regarding a programming problem as a programming novice, the question might not be super clear. But often I think the people trying to answer are not even half as bright as the guy who is questioning.
Re: (Score:2)
The most annoying answer usually is: "why do you want to do that (or that way)? Why not do it like this: XYZ?".
Oh, yes. I have no idea how defective you have to be to think that is a good answer.
But often I think the people trying to answer are not even half as bright as the guy who is questioning.
Well. If you actually tried to figure out something, and only started searching when you failed, you are probably already smarter than 80% of the ones posting there. The second most annoying ones are those that think everything is easy because they do not grasp the problem or the question.
But I think that is a fact on any type of social media and probably in society as a whole: If you can fact-check, if you can find out thin
Re: (Score:2)
that's a good point, but that's not how security in code works either.
Re: (Score:3)
And where will you get secure code from?
And just because code is secure when performing its originally intended task, doesn't mean it remains secure when modified to do something slightly different.
Re: (Score:2)
If you train it on secure code, what it generates will look like secure code. It doesn't understand what it's doing. It just creates an approximation based on weighted models.
If you train it on secure code, you get blind code (Score:3)
If you train it on secure code, it will generate secure code.
Nope, the AI won't know what it's doing, just make very good guesses. Again...if I were wrong, Google, MS, Apple, and all the big spenders would be eagerly applying it to their moneymakers today. I'm very expensive. Hiring a team of people like me is VERY expensive. If an AI could do my job as well, they'd dump me in a heartbeat. They would have done quite awhile ago if it could be done.
Train AI on really good code and it will mimic the code, but it will never truly know what it's doing...that's th
Re: (Score:3)
You know how in Star Trek there's always a way to 'bypass the security protocols"? I think we've just discovered how we get there.
Re: (Score:2)
Yep, crap code heaped on crap code and nobody with actual skills and insight never got even near it. Star Trek is not the only thing that has this figured out though. I mean my personal system set-up here is probably a lot more secure than the average corporate set-up and all I do is that the only thing that is allowed in from the outside is ssh and smtp (from a proxy VM only that relays).
Re: (Score:3)
Probably not very secure at all since you're focusing solely on unsolicited inbound connections, whereas most attacks these days are perpetrated against software which makes outbound connections.
Re: (Score:3)
Surprised? Nope not at all.
But this isn't going to stop people that have no business coding anything more complicated than "Hello World!" from trying to fake it using ChatGPT and the result is going to be a bunch of crappy code, riddled with security holes.
Re: (Score:2)
Indeed. But these people are basically faking it. The only real cure for that is to finally lift up "coding" to the level of engineering (which it is) and get rid of all the amateurs. Do not at least have a BA that is in target? Stay out! (There are always ways to get equivalent skills certified. Some people have them. But 95% do not.) The other thing that comes with being an engineer is liability for when you screw up badly enough. And we urgently need that for software as well. The current clueless shit-s
Re: (Score:2)
"The current clueless shit-show cannot continue."
The current shit show has nothing to do with a lack of professional organization to qualify professionals, and just who would be driving to create such an organization that certifies programmers as "engineers" with "liability" anyway? Business wants to get rid of the current shit show, especially considering how much those who are "faking it" are getting paid. The people in charge don't think it's a valuable skill.
If you think the mile high stack of shit ap
Re: (Score:2)
That is nonsense. Have a look at the evolution of some mature engineering discipline some time and you will see that your comment is nonsense.
Re:Are you really surprised? (Score:4, Insightful)
So in other words it will be indistinguishable form all the regular code that everyone writes.
Maybe your low-end low-cost body shop code (Score:2)
So in other words it will be indistinguishable form all the regular code that everyone writes.
Fair point, but I am a professional who takes pride in my work and takes the effort to learn the libraries I use and understand holistically the purpose of the libraries and their strengths and weakness so I can understand the best practices and determine which of the contradicting common best practices should apply to the current situation.
That's what makes me a high-end developer highly sought after by the big spenders. If you want blind copy/paste, there are many of much younger "professionals" from f
Re: (Score:2)
it writes code just like the average freshman who pulls a snippet from stackoverflow and doesn't even bother reviewing it as long as it apparently does what he/she expected from it writes code. this has been going on for a couple of decades and odds are most software you use today includes some portion of that.
chatgpt just streamlines that same process. on the plus side it allows to easily perform follow up checks on the code just by asking, which might provide some much needed insight and prevent at least
Really (Score:5, Funny)
I'm shocked, shocked I tell you to find that a chat bot with a coding education that mostly comes from bad documentation and web forum posts from people with the wrong answer is unable to generate secure code.
Re: (Score:3)
It can perform on the level of the average moron. That is less and less sufficient. Taken that way, ChatAI will probably only eliminate the jobs of people that never were any good at their jobs anyways. As these are a _lot_, social problems will result and need to be solved, but for many people them being out of a job is a good thing for everybody else.
Re: (Score:2)
"ChatAI will probably only eliminate the jobs of people that never were any good at their jobs anyways."
No, the belief in a tool that can replace programmers will eliminate the highest paid programmers first. That's because that's who executives will target.
Re: (Score:2)
Then then everything comes crashing down. There may be a few that will do that, but most will not.
Re: Really (Score:2)
Re: (Score:2)
Ah, yes _those_ people. Claiming it is not their fault, they did not do anything wrong, etc., while the mistakes they make become worse and worse.
Re: (Score:2)
Fortunately Open AI is hiring a huge number of coders to write high quality code for a future version of their GPT to train on.
I'm looking forward to that day. You can already get decent results if you ask it for boilerplate code and if you need to generate a bunch of meaningless unit tests to fill a meaningless checkbox on a compliance questionnaire.
Re: (Score:3)
Fortunately Open AI is hiring a huge number of coders to write high quality code for a future version of their GPT to train on.
I'm looking forward to that day. You can already get decent results if you ask it for boilerplate code and if you need to generate a bunch of meaningless unit tests to fill a meaningless checkbox on a compliance questionnaire.
The codebase I inherited has a TON of unit tests. The previous team bragged about "100% test coverage". Yet the vast majority of the tests assert nothing meaningful and would always pass even if someone came in and completely changed the logic of the code. Most of these tests are completely useless.
Well, developers... (Score:3)
Re: (Score:3, Insightful)
Re: (Score:1)
ChatGPT won't shoot people when you yell at it for being an idiot.
Re: Well, developers... (Score:2)
Re: Well, developers... (Score:3)
Average coder? The world's thousands of CVE prove gifted programmers are high IQ morons making the same mistakes over and over. The world's software are a mile high teetering tower of crap.
Re: (Score:2)
Any software system of reasonable size is too large for one individual to know the minute details of every function and their anomalies. Documentation, when it exists, is often not accurate.
Put all of that together and there's simply no way a human programmer is going t
Re: (Score:2)
blah blah blah said the high IQ moron, part of the problem and certainly not part of any solution
Compilers (Score:5, Interesting)
When I first started programming, compilers seemed like magic. They could transform an expression of almost arbitrary complexity into assembly code which would solve the problem.
But as I learned computer science, it became increasingly obvious that a compiler was not a monumental achievement, but rather the composition of some simple, but rather clever principles, which, in breaking down the problem into smaller tasks, made it possible for a computer program to do what would seem impossibly complex if constructed of cmp instructions. It could "understand" what I wanted it to do, and generate code which would achieve that effect!
AI is different. Instead of "understanding" the language it parses, it instead performs a brute-force estimation of what it thinks I want, relying on the fact that it has been trained on an inordinately large data set of similar problems.
In one respect, it is rather interesting that we can use brute force methods to imitate an oracle. OTOH, it is a bit depressing that we still don't understand why it works. Much like how the compiler seemed like magic until it was explained, these AI models seem like magic, but cost a prohibitive amount to train because the computer scientists don't yet understand how to derive the fundamental, underlying logic structures that are created in the training of the AI model. If they could do that, we could have a ChatGPT like agent capable of running on a wristwatch.
After all, Borland's Turbo C++ would run lightning fast on a Pentium 120 with only 16MB of RAM.
Re: (Score:3)
Actually, code optimization by modern compilers is pretty impressive. AI is not as soon as you ask real questions. But hey, too many people that follow the hype without understanding it.
Re: Compilers (Score:2)
Picked it back up a few years ago and discovered avr-gcc. Compiled die code and looked at the disassembly. It was horrifying. Poor chip spent most of its time pushing and popping registers on the stack. Optimizations were obvious. I.e. copy some functions inline.
Avr-gcc may not be
Re: (Score:2)
Pushing values to registers is not necessarily as slow as you think as processors are often optimized for it.
The number of optimizations in even decades-old version of gcc is astounding. You will be able to make small sectio
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
yes early compilers where crap, I also started out as a assembler programmer back in the day and in the 80:ies, 90:ies and early 2000:s we could and would outperform a compiler every single day of the week but then two things happend. One was that cpu:s became far more complex with long pipelines and parallel execution units and the optimizers in the compilers become much much better.
There are still areas where a manual assembly is far superior to even the state of the art optimizing compiler today. One mod
Re: (Score:2)
Re: (Score:2)
The future (Score:1)
Reminds me of contributions from certain employees we've had over the years whose efforts ultimately proved to be counterproductive.
From my perspective the ability simply to generate code is less than worthless. Code is a liability something to be actively managed not a garbage heap one keeps shoveling garbage into at no cost.
Still early and quality is likely to be improved dramatically thru self reflection and associated adversarial schemes. Suspect in the future it may even be possible for these systems
Color me unsurprised (Score:2)
Seriously, whoever thought having an artificial moron write code for them is a moron as well.
Looks like coders with some real skills do not need to fear for their jobs. As sooner the others are out and doing something they actually know how to do, the better.
Re: Color me unsurprised (Score:1)
Will the coders with real skills who use ChatGPT when appropriate outcompete the reactionaries?
Re: (Score:2)
Only if you try to learn from the output. Honestly, writing a good prompt takes a lot of time too. You might get great results but it is dependent on the work you put in.
Re: (Score:2)
Looks like coders with some real skills do not need to fear for their jobs.
I do not share your faith that the managers and executives who actually do the hiring and firing are going to understand the situation. I expect many of them will fire the expensive coders and replace them with new kids trumpeting ChatGPT "experience". No one will be left that knows how to recognize bad code.
Re: (Score:2)
Just like human coders! N/T (Score:2)
BOFH lives
It actually won't tell you anything at all until y (Score:1)
First you generate code. Then you ask for revisions. That's the design process.
For AI coding, they're going to use the current models to generate code, classify that code as secure or not, and use it to train a new model which always produces secure code.
It sees secure code as a vulnerability (Score:5, Interesting)
Re: (Score:2)
It will be our ham-fisted attempts to control the output.
Indeed, if you ask chatGPT to write obfuscated, unfathomable, self-modifying code it says that it can't do that as it does not advance computer science and understanding of programming. Or some similar nonsense like that. It already makes stuff up, er I mean hallucinates, so the attempts on controlling output are already a fail. Part of the problem is that it many people will accept the written word as fact if it is presented that way. It may be part of human nature and this tendency goes back millennia and
Ask chatGPT if its a good idea to use it for code (Score:2)
And it will tell you exactly why the entire thing is just bad.
Mediocre code in - mediocre code out (Score:3)
So Chat-GPT pretty much acts like your typical coder.
Not trained on battle tested working code bases (Score:2)
It's trained on throw away example snippets, that were intended as simple explanations. Not production quality secure robust tested code.
If it was trained on stackoverflow... (Score:2)
why is this surprising?
A typical post asking a question generally has a lot more compilable/runnable code in the question that's wrong in some way than answers that contain fixes or corrected code snippets in replies.
You'd think that it should be trained on actual production code and on patches to actual production code, not free for all forums, if you want it to generate more reliable software.
No intent, meaning, or unserstanding (Score:1)
chatGPT is a linguistic engine. The way it and its competitors work is it finds the best match FOR THE NEXT WORD in a sentence based on deep data of a vast sea (morass?) of human output. There is NO understanding of the underlying meaning. It is all just probability. There is no intelligence. The "emergent intelligence" is OUR interpretation of the output - eg the meaning we give it.
People are impressed by it the way they are impressed by the current output of our politicians. All form, no substance.
Except Rust for some reason (Score:2)
Yet Rust code generated by ChatGPT is somehow always absolutely completely %100 safe. Weird that.
Re: (Score:2)
You use of the term "Rust" in relation to programming languages infringes on the trademark rights of the Rust Super Bastards Corporation Inc. The matter is now in the hands of the lawyers of said organisation.
The Great Averager (Score:2)
We accept the illusion as what we a
Re: (Score:2)
They worked this shit out 50 years ago, but today's undergraduates think they are the first to discover stuff and learning from the past is beneath them. Hence the woke whining that has destroyed modern discourse.
Re: (Score:2)
That's what you get... (Score:2)
Outsourced coders (Score:2)
Guess what your outsourced coders are too, but the difference is they will tell you its secure even though they didn't do anything different. At least the AI will tell you that it made a mistake because has no reason to feel shame or hide anything. It just didn't think about all the possible factors based on your initial request. Maybe the flaw was that you didn't ask for it to be secure in the first place?
Can you write Linux kernel code? (Score:2)
Me: Can you write Linux kernel code?
ChatGPT: I am not capable of actually writing Linux kernel code by myself. Writing kernel code requires a deep understanding of the underlying hardware, software architecture, and complex programming concepts that go beyond the scope of what I am capable of.
A serious question (Score:2)
I write small amounts of code, mostly to analyze scientific data, and where security isn't important. What I do is nothing even close to the scale of a professional programmer (of which there are many here). chatGPT is somewhat useful to start the sort of coding that I do, then I can debug and build from there.
My question is really whether that scales up to more serious coding. Even if the code it writes is nonsecure or buggy is there value in it basically building the foundation for a coder to start from?