Are AI Coding Assistants Really Saving Developers Time? (cio.com) 74
Uplevel provides insights from coding and collaboration data, according to a recent report from CIO magazine — and recently they measured "the time to merge code into a repository [and] the number of pull requests merged" for about 800 developers over a three-month period (comparing the statistics to the previous three months).
Their study "found no significant improvements for developers" using Microsoft's AI-powered coding assistant tool Copilot, according to the article (shared by Slashdot reader snydeq): Use of GitHub Copilot also introduced 41% more bugs, according to the study...
In addition to measuring productivity, the Uplevel study looked at factors in developer burnout, and it found that GitHub Copilot hasn't helped there, either. The amount of working time spent outside of standard hours decreased for both the control group and the test group using the coding tool, but it decreased more when the developers weren't using Copilot.
An Uplevel product manager/data analyst acknowledged to the magazine that there may be other ways to measure developer productivity — but they still consider their metrics solid. "We heard that people are ending up being more reviewers for this code than in the past... You just have to keep a close eye on what is being generated; does it do the thing that you're expecting it to do?"
The article also quotes the CEO of software development firm Gehtsoft, who says they didn't see major productivity gains from LLM-based coding assistants — but did see them introducing errors into code. With different prompts generating different code sections, "It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it."
On the other hand, cloud services provider Innovative Solutions saw significant productivity gains from coding assistants like Claude Dev and GitHub Copilot. And Slashdot reader destined2fail1990 says that while large/complex code bases may not see big gains, "I have seen a notable increase in productivity from using Cursor, the AI powered IDE." Yes, you have to review all the code that it generates, why wouldn't you? But often times it just works. It removes the tedious tasks like querying databases, writing model code, writing forms and processing forms, and a lot more. Some forms can have hundreds of fields and processing those fields along with doing checks for valid input is time consuming, but can be automated effectively using AI.
This prompted an interesting discussion on the original story submission. Slashdot reader bleedingobvious responded: Cursor/Claude are great BUT the code produced is almost never great quality. Even given these tools, the junior/intern teams still cannot outpace the senior devs. Great for learning, maybe, but the productivity angle not quite there.... yet.
It's damned close, though. GIve it 3-6 months.
And Slashdot reader abEeyore posted: I suspect that the results are quite a bit more nuanced than that. I expect that it is, even outside of the mentioned code review, a shift in where and how the time is spent, and not necessarily in how much time is spent.
Agree? Disagree? Share your own experiences in the comments.
And are developers really saving time with AI coding assistants?
Their study "found no significant improvements for developers" using Microsoft's AI-powered coding assistant tool Copilot, according to the article (shared by Slashdot reader snydeq): Use of GitHub Copilot also introduced 41% more bugs, according to the study...
In addition to measuring productivity, the Uplevel study looked at factors in developer burnout, and it found that GitHub Copilot hasn't helped there, either. The amount of working time spent outside of standard hours decreased for both the control group and the test group using the coding tool, but it decreased more when the developers weren't using Copilot.
An Uplevel product manager/data analyst acknowledged to the magazine that there may be other ways to measure developer productivity — but they still consider their metrics solid. "We heard that people are ending up being more reviewers for this code than in the past... You just have to keep a close eye on what is being generated; does it do the thing that you're expecting it to do?"
The article also quotes the CEO of software development firm Gehtsoft, who says they didn't see major productivity gains from LLM-based coding assistants — but did see them introducing errors into code. With different prompts generating different code sections, "It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it."
On the other hand, cloud services provider Innovative Solutions saw significant productivity gains from coding assistants like Claude Dev and GitHub Copilot. And Slashdot reader destined2fail1990 says that while large/complex code bases may not see big gains, "I have seen a notable increase in productivity from using Cursor, the AI powered IDE." Yes, you have to review all the code that it generates, why wouldn't you? But often times it just works. It removes the tedious tasks like querying databases, writing model code, writing forms and processing forms, and a lot more. Some forms can have hundreds of fields and processing those fields along with doing checks for valid input is time consuming, but can be automated effectively using AI.
This prompted an interesting discussion on the original story submission. Slashdot reader bleedingobvious responded: Cursor/Claude are great BUT the code produced is almost never great quality. Even given these tools, the junior/intern teams still cannot outpace the senior devs. Great for learning, maybe, but the productivity angle not quite there.... yet.
It's damned close, though. GIve it 3-6 months.
And Slashdot reader abEeyore posted: I suspect that the results are quite a bit more nuanced than that. I expect that it is, even outside of the mentioned code review, a shift in where and how the time is spent, and not necessarily in how much time is spent.
Agree? Disagree? Share your own experiences in the comments.
And are developers really saving time with AI coding assistants?
It's a tool. (Score:5, Interesting)
If you try to build a ship with nothing but a welding torch, it won't go well.
Copilot is excellent. But if you try to make it write ALL your code for you, that code will suck.
Re: (Score:3)
If you try to build a ship with nothing but a welding torch, it won't go well.
Copilot is excellent. But if you try to make it write ALL your code for you, that code will suck.
I am an open-source CMS developer. There's a vast amount of relevant open-source code that the LLM Borg has trained itself on, and the feedback is pretty good. There's no irrelevant open-source code that's been published to the web, Github, or GitLab, it all is pretty much vetted and is valid. Coding by prompt is akin to critically reviewing another developer's code IMHO.
Technology changes and the market expects developers to keep up in order to compete.
Re: (Score:2)
There's no irrelevant open-source code that's been published to the web, Github, or GitLab, it all is pretty much vetted and is valid.
...in the Framework I use -- I meant to write.
Re: (Score:2)
There's no irrelevant open-source code that's been published to the web, Github, or GitLab, it all is pretty much vetted and is valid.
...in the Framework I use -- I meant to write.
Yeah, for a minute there, I thought your post was dripping with more sarcasm than I ever thought possible.
Re: (Score:1)
There's a vast amount of relevant open-source code that the LLM Borg has trained itself on, and the feedback is pretty good. There's no irrelevant open-source code that's been published to the web, Github, or GitLab, it all is pretty much vetted and is valid.
You forgot to include the wink emoji, people might think you're serious there.
Re:It's a tool. (Score:5, Insightful)
I tried them out a couple of times and was not all that impressed with the results. Both times the code did at least work, but wasn't particularly good. StackExchange quality stuff, functional but far from ideal.
In both cases I'd have preferred to re-write it from scratch myself. That would give me a chance to really think through the algorithm and the potential issues with every line, something I find easier when writing code than when reviewing it.
Re: (Score:2)
If you try to build a ship with nothing but a welding torch, it won't go well.
Seemed to work OK for Liberty Ships in WW2...
Re: It's a tool. (Score:1)
What a great example!
An era where ships were built quickly so that they could be fielded in quantity without much concern over longevity, reliability, or consistency.
Re: It's a tool. (Score:3)
A typical day will see me writing Java, Typescript, maybe a bash script, some docker compose files, and something random like ffmpeg command line. LLM can help me with Java only occasionally. But when it comes to things I don't do frequently like ffmpeg, or things that are a little arcane like bash, LLMs are a godsend.
coding productivity (Score:3)
The AI coding assistants are very powerful but right now in 2024, because code itself is very complex typically with hundreds of files in an app, AI is not quite there with a holistic code base (application level) training and output just yet.
The most productive AI users now are senior developers who can use the AI to both 1. iterate code sections insanely fast 2. actually read the code the guide the AI in the next iterations.
So TODAY you still have to know what you are doing to leverage AI tools for actually-better quality x speed output.
3-6 months you won't have to know as much.
Re:coding productivity (Score:4, Insightful)
Re: coding productivity (Score:4, Insightful)
The question of "mastery" is one that requires perspective.
I'm over 50 and learned data structures and memory management but barely touched assembly in college. 10 years ago, I was told that understanding the difference between "pass by reference" and "pass by value" was rare.
The point is that "mastery" is having the skills to be productive with the best tools of the time, and that changes. Learning how to get the best results out of AI but not understanding how its output works is just a different layer of abstraction from using console.log but not having any idea how that makes different pixels appear on the screen.
I haven't worked with AI coders yet, but I have no doubt it is another technology I'll work with before my career is done; another thing I'll have to "master."
Re: coding productivity (Score:3)
, I was told that understanding the difference between "pass by reference" and "pass by value" was rare.
So that's the reason right there why people have so many problems with memory leaks that they are developing a language that force you to do contortions to do anything. Seems it would be easier to just treach people the difference.
Re: coding productivity (Score:2)
Which is the same problem we have today. There are plenty of people that have no interest in advancing their careers. They are happy where they are and with what they make and some literally do first level helpdesk stuff until they retire.
The thing I see AI doing right now if it has access to the codebase is the stuff junior frontend developers do. The so-called designers and artists are what is going to go, if you find out you needed a different or additional field somewhere in the middle of the project, y
Re:coding productivity (Score:4, Interesting)
The AI coding assistants are very powerful but right now in 2024, because code itself is very complex typically with hundreds of files in an app, AI is not quite there with a holistic code base (application level) training and output just yet.
This is a problem because the context window of AIs is still very small. A large codebase will overwhelm it. To address that problem, we're going to need new algorithms.
Re: (Score:1)
code itself is very complex typically.
You are not a developer and your opinion is worth less than pig shiat.
Congrats on outing yourself.
Re: (Score:2)
code itself is very complex typically.
You are not a developer and your opinion is worth less than pig shiat.
Congrats on outing yourself.
Assuming the OP meant complex as in architecturally and not complex as in code complexity, I would agree with the GP. Sure, you can get an AI to do boilerplate stuff, where it has hundreds of examples of the exact thing you're trying to get it to do, but as soon as you actually try to do anything more complicated, it breaks down.
I occasionally try using it a bit, but mostly don't bother, because in my experience, for really simple code, it sometimes gets it exactly right, but when I ask it to do something
Re: (Score:2)
3-6 months you won't have to know as much.
Make that 30-60 years and it becomes a possibility.
It's just the latest snake oil (Score:2, Insightful)
Companies have been trying to sell us Rapid Application Development and No-Code Solutions for decades already, this is just the latest incarnation of that.
Yes, "AI" tools may provide benefits in some areas but it's not a silver bullet, a go-fast button, for the majority of tasks yet.
Re: (Score:2)
>Yes, "AI" tools may provide benefits in some areas but it's not a silver bullet, a go-fast button, for the majority of tasks yet.
Hmm, it's almost like it is in the name - "assistant".
There's a reason it's not called a "coding slave" or "under / unpaid intern" code writer.
From what I've seen... (Score:3)
... it kinda works for things people have done over and over again. Writing a CURD-Application in PHP probably works just fine... but then again, why on earth are we doing the same thing over and over again, shouldn't the software environment deal with such trivialities. The far bigger productivity gain would be in using environments that are tailored for the job you are trying to solve. If you have an application with 20 database tables... you shouldn't have to write your CURD-code for each one of them. Then again, productivity isn't an important measure in software development.
Otherwise it's a bit better than autocomplete used to be in Delphi in the late 1990s.
Where it really becomes a problem is when there are automatically generated websites drowning out important information. For example in my line of work you need to write software as Kamailio applications. That's essentially a programming language with C-like syntax, but somewhat different semantics. More and more we are seeing web-sites full of utter garbage. Web-sites which, on first glance, look as if there would be Kamailio code on there, but in reality it doesn't make sense or at least wouldn't be parsed.
Re:From what I've seen... (Score:4, Interesting)
That you mispell that acronym as "CURD" is telling.
Re: (Score:3)
Whey!
"Great for learning*?! (Score:3)
Teaching people how to write crap code is not the quality we should be striving for!
Re: (Score:2)
Why not? They can then generate the code the next AI will learn from! Circular learning!
Re: (Score:3)
There is some truth to this. Asking for code in a comparative vacuum is going to render code without context, until the context evolves. This means organizational model training will eventually yield better code because of training and feedback loops into the blackbox which makes code.
This also permits developers generating their own code-making models to have a companion for their generating efforts over a period of time, then understanding how code relates to a larger model. Isolating lib models to functi
Re: (Score:3)
That depends. Bad code can be better than no code, but it can also be worse than no code at all.
If you have a problem that requires 100 man years to do manually, and you manage to write bad code that does that in 1 day, crashing randomly all the time, but still managing to do it and you can verify the work to be correct, this is a good solution.
But if you add flashing lights to Linux kernel that only one person wants to use and this causes all Linux servers around the world to crash, it would have been bett
Re: (Score:2)
Thank you!
Re: (Score:2)
It is what the business world wants though. Engineering has to put its foot down or this crap will continue.
Ah, well, who am I kidding. This insanity will continue until a rather large number of people gets killed by it. That was how other engineering disciplines finally got quality standards.
Yes (Score:5, Insightful)
It's a tool. Used properly, it saves tons of time.
"Rewrite this whole section of code to use OpenStreetMaps instead of Google Maps"
Could I have done it myself? Sure. In 30 seconds? No ...
Re: (Score:3, Insightful)
Could I have done it myself? Sure. In 30 seconds? No ...
If you think it only took 30 seconds, then you're one of those people leaving introducing 41% more bugs. You need to make sure you understand any code generated by AI...
Re: (Score:3)
Could I have done it myself? Sure. In 30 seconds? No ...
If you think it only took 30 seconds, then you're one of those people leaving introducing 41% more bugs. You need to make sure you understand any code generated by AI...
Gee, we could just assume I'm stupid ... or we could assume that I was talking about the initial writing part, leaving the review and QA as a given.
In any case, the answer to the titular question is still "yes".
Re: (Score:3)
Re: (Score:2)
Reading code is mich harder than writing code. This person either got worse code or did not save time, maybe both. The capacity of coders for lying to themselves is legendary.
Re: (Score:2)
Re: (Score:2)
Could I have done it myself? Sure. In 30 seconds? No ...
If you think it only took 30 seconds, then you're one of those people leaving introducing 41% more bugs. You need to make sure you understand any code generated by AI...
Gee, we could just assume I'm stupid ... or we could assume that I was talking about the initial writing part, leaving the review and QA as a given.
In any case, the answer to the titular question is still "yes".
I'm totally with you. My concern is that as the models improve--which they have tremendously in just the past few weeks--we're going to be asking them to write far more code of far greater complexity and we'll not be able to vet it sufficiently before deployment.
Re: (Score:2)
Could I have done it myself? Sure. In 30 seconds? No ...
If you think it only took 30 seconds...
I think the point is the suggestion helped the skilled coder progress, rather efficiently.
Re: (Score:2)
Re: Yes (Score:1)
If you introduce bugs swapping one third party API for the other, perhaps your code is poor to begin with.
Re: (Score:2)
Re: (Score:1)
Sure, but you shouldn't trust that data to begin with, if an API produces garbage for data, that shouldn't drive a crash or other security issues, even existing API's aren't stable enough to integrate directly into your code. Changing an API should be a matter of swapping out a URL, the rest needs to be handled.
Asked for a CRUD website; it made me DOOM (Score:2)
well, you can't complain about creativity. =/
They replace Stackoverflow (Score:4, Interesting)
They do not replace the person typing code, but Stackoverflow for looking up (possibly trivial) questions. You have a sidebar where you can just type ("What is the C++ idiom to do ...") and get the answer instantly. Yes, it might be able to apply it to your code, but the answer itself is the important part.
That's why Stackexchange is trying to lock down their exports (to the displeasure of the community) because they would rather have their user's content (which is CC licensed by their ToS) as their capital and nobody train on it free to use models.
Timeline (Score:2)
It's damned close, though. GIve it 3-6 months.
I really wonder how they are able to estimate timelines like this. What inside information do they have that we don't? 3 to 6 months? [xkcd.com]
It can. But also maybe not. (Score:2)
Re: (Score:2)
But we don't learn by writing perfect code, we learn by fixing problems. If an inexperienced coder leans heavily on AI generated code, but then has to deal with fixing all of the issues generated by that AI, eventually they will no longer be an inexperienced coder.
I want code review (Score:2)
Depends... (Score:3)
As a teacher, and someone who supervises a wide variety of student projects: I do lots of random bits of coding in different languages and using different frameworks and APIs. I cannot possibly keep the details of all of them in my head. ChatGPT is great for reminding me how to do X in language Y or with framework Z. Basically, it is a single source for reference material.
AIs are not yet very useful at actually writing code, at least, not beyond a trivial level. Just as an example, I had a student last week who was writing a web service in Java. When he closed his program, the ServerSocket was not always being properly released. ChapGPT came up with all sorts of overly-complicated solutions, none of which helped. All he actually needed to do was declare the thread to be a daemon thread, but ChatGPT never suggested that. I gave him the hint, ChatGPT gave him the syntax, and the problem was solved.
AI is tool fetishism at its silliest. (Score:2)
Re: (Score:2)
Indeed. Same here. Too many people have this baseless deep belief that "doing it with IT" makes things universally better. That is very much not true.
Re: (Score:2)
Headline doesn't match article contents (Score:5, Informative)
Headline says AI didn't save developers time.
The story itself describes the experience of two companies: Uplevel and Innovative Solutions. Uplevel says they didn't see any gains (and worse bug counts), Innovative Solutions says AI helped them achieve a 2x-3x increase in productivity.
So the real headline should be "Mixed results" from AI coding assistants.
This makes me wonder how the study methodologies of the two companies differ, and how their practice--use of AI--differs.
41% more bugs (Score:3)
What does that mean, exactly? Not all bugs are created equal. Some are serious and consequential, others are more a matter of opinion. Does this higher bug count stem from new (AI) scanning tools?
This reminds me of why I don't run Lint or ReSharper. Many of the bugs or flaws reported by these tools are accurate, but they are drowned out by a forest of inconsequential (though technically accurate) reported issues, that might or might not be necessary to fix. Many of these are more coding style preferences than actual code issues.
In this study, was the same process used to scan or count bugs before the AI tools were introduced? Or were the old, lower bug counts the result of a mor manual process?
As a hobbyist programmer (Score:2)
no surprise (Score:2)
Use of GitHub Copilot also introduced 41% more bugs, according to the study...
Let me guess, at least half of those bugs are from bad example code posted on the Internet, or from QUESTIONS rather than answers, you know the "why does this not work?" questions.
LLMs are an excellent mirror of our world. They will reflect back what we communicated amongst ourselves. If flat earthers weren't a fringe group, the LLM would gladly tell you that the Earth is flat.
Buried the Lede (Score:2)
Yeah, the summary definitely buried the lede. If the use of GitHub Copilot "also introduced 41% more bugs, according to the study", I don't think the important part of the story is how fast it isn't.
Specific domain: embedded and LwIP (Score:2)
All I got were
1) banalities, small talks.
2) things I already knew, But not helping, not providing answers to things I don't know.
3) errors, wrong answers.
I gave up trying
Re: (Score:2)
Re: (Score:2)
AI is a research project (Score:2)
I'm hopeful that it will eventually be developed into something useful
Unfortunately, investors want profits NOW, so they demand that companies release half-baked, kinda useless crap
Meanwhile financial journalists write articles about how AI is failing to meet expectations
In software development, I don't see the value in using crappy AI to help mediocre programmers more quickly develop mediocre code
I'm hopeful that the systems of the future will allow expert programmers to manage the complexity of large syst
Same pattern (Score:2)
Almost every new software-related idea is initially overdone and misused. Over time people figure out where and how to use it effectively instead of mostly make messes as gestures to the Fad Gods. But there will be fucked up systems left in their wake. Pitty the poor maintainers.
OOP, microservices, crypto, 80's AI, distributed, Bootstrap, etc. etc. went thru a hype stage.
Thus, I expect the initial stages will be fucked up.
Yes. True story, happened on Friday (Score:2)
I run a software company which uses MS SQL Server and .NET Core for our app development. My lead developer had a problem with a SQL script which was stubbornly slow - he was querying a huge table with an index that really wasn't helping the performance. He created a prompt to ChatGPT to ask how to optimize the query and it came back - instantly - with a subquery approach that moved the filtering logic from the index into memory. It ran significantly faster.
I have a feeling that GPT had somehow scraped some
My experience (Score:2)
I can code... like shit. It's a tiny percentage of my job so while I can do it, I don't do it often enough to keep everything in my memory perfectly. I know what can be done, I know the general syntax, and I know lots of theory, and I know the terms.
So for me, an AI assistant turns me from someone who is great in theory but very, very wasteful in practice due to how frequently I have to confirm things by checking reference materials (thank you, Google...) into someone who can do some light coding almost a
You can see expectations from the price (Score:2)
I've been using GitHub Copilot for a few months now and I can say that it definitely saves a little bit of time. I've seen reports that it makes developers 33% faster and it really does not come anywhere close to that. Where it shines is being able to fill in some boilerplate code faster than I can type it. When what I'm writing is something where the next few lines are something that an undergraduate student could easily predict, it can fill it in and I can tab complete it faster than I can type it. (E
Not in a vacuum (Score:2)
Unfortunately these days, very few of us are coding in a vacuum. When's the last time you started coding and whatever you were creating was 100% standalone and did not interface with anything in any way? No APIs, SDKs, libraries, web services, etc. The only time I find myself doing that (and still there is at least a tiny amount of interfacing) is coding in C++ for microcontrollers (like ESP32 type embedded stuff).
The rest of the time we have to continuously interface with 3rd party stuff, and that, more of
Yep, probably not (Score:2)
Or otherwise we would see significant savings by now. But what about code quality? As not all developers catch what AI screws up, it has to be going down. So then we would be at no time savings, while quality got worse.
My experience (Score:2)
It's great for getting baseline examples (Score:2)
Positive value if used responsibly (Score:2)
I'm a firm believer that it saves me time, or at least I am a happier programmer when I can get suggestions realtime.
I'm always skeptical when it suggests something I don't understand. I don't blindly trust it. I have to take some time to understand what it generates sometimes, but I'm still in a better spot than without aid.