AI Slows Down Some Experienced Software Developers, Study Finds (reuters.com) 49
An anonymous reader quotes a report from Reuters: Contrary to popular belief, using cutting-edge artificial intelligence tools slowed down experienced software developers when they were working in codebases familiar to them, rather than supercharging their work, a new study found. AI research nonprofit METR conducted the in-depth study on a group of seasoned developers earlier this year while they used Cursor, a popular AI coding assistant, to help them complete tasks in open-source projects they were familiar with. Before the study, the open-source developers believed using AI would speed them up, estimating it would decrease task completion time by 24%. Even after completing the tasks with AI, the developers believed that they had decreased task times by 20%. But the study found that using AI did the opposite: it increased task completion time by 19%. The study's lead authors, Joel Becker and Nate Rush, said they were shocked by the results: prior to the study, Rush had written down that he expected "a 2x speed up, somewhat obviously." [...]
The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route."
The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route."
Junior developers (Score:5, Insightful)
> do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with
If the juniors are working with the same code base, with the same AI that makes same incorrect suggestions which seniors reject and if AI is not slowing juniors down, doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
Re: (Score:2)
I strongly suspect so -- but would the rate and degree of pollution be worse than the baseline case for an inexperienced developer? Most software developers (and I think this decreases only somewhat with experience) do not have the discipline to think rigorously about the changes they are making and review the surrounding context. That self-discipline is what distinguishes engineers from mere coders.
Re: (Score:3)
I strongly suspect so -- but would the rate and degree of pollution be worse than the baseline case for an inexperienced developer? Most software developers (and I think this decreases only somewhat with experience) do not have the discipline to think rigorously about the changes they are making and review the surrounding context. That self-discipline is what distinguishes engineers from mere coders.
Junior developers might make similar mistakes (or mistakes similar in magnitude) as the AI, but the junior developers likely won't learn what they're doing wrong. So they'll never have the same expertise as the current "experienced" devs.
Re: (Score:2)
So they'll never have the same expertise as the current "experienced" devs.
What do you think "experience" means? How do you think people get it?
Junior developers will make mistakes, but if the environment is organized to support learning and the junior devs stick around for a bit, they will learn -- either from their own mistakes or from others' -- and get better. Sometimes you get someone with "one year of experience, ten times" instead of "ten years of experience", but a good shop should try to avoid or remove that, depending on the root cause.
I mean, it is trivially true that
Re: (Score:3)
Re: (Score:2)
If the junior developers are offloading a significant portion of the the hard problem solving work to the ML, they are simply less mentally prepared to learn from mistakes. There is less of a sense of ownership. There is less understanding of the situation that precipitated the mistake.
This comes down to how the human brain works and how it learns and improves at hard things.
Now, I am not going to assert that it is inevitable that an individual junior developer is going to learn less. I would say that us
Disrupting the train of thought (Score:2)
If you are well familiar with a code base, and have written decent quality code, with encapsulation, error handling, guard clauses and fail fast programming; AI is likely to be more disruptive to your workflow of writing new code.
Refer: Multitasking is a Lie exercise - https://www.outsidetheboxexper... [outsidethe...ential.com]
Where AI helps is doing boilerplate code such as reading in and parsing a JSON file given a path, including guard clauses, handling exceptions, handling errors like permission, directory does not exist, file d
Relativity (Score:2)
doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
Probably but the question you want to ask is which produces the fewest wrong decisions: making their own decisions vs. missing the AI's mistakes?
Re: (Score:2)
No, not really.
Juniors know how to code. They wrote plenty of small programming assignments in college.
They don't know the details of the codebase and have less experience making changes to million-line systems, but that's different from being unable to recognize a buggy function.
Re: (Score:2)
If a function is simple enough that "buggy" can be defined in isolation, AI or a junior developer can probably write it without much supervision. Most bugs that escape the first-level developer are violations of contextual expectations: cases where the code would work as expected in a different application, with a different use case, or something like that. So the characteristics of "a buggy function" depend on the code, processes and users around it, and that is where junior developers often fall short.
D
Re: (Score:3)
> do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with
If the juniors are working with the same code base, with the same AI that makes same incorrect suggestions which seniors reject and if AI is not slowing juniors down, doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
I could very well imagine AI helping me to get a handle on/oversight over code bases that I'm not familiar with as well as really huge codebases where I'm familiar some parts but don't know every nook and cranny, or for insanity inducing tasks like finding memory leaks and heisenbugs but in terms of helping me code I'm not so sure. That sounds like a crutch that might be good for junior developers until they outgrow it. But why are we even talking about this? Aren't we all about to be replaced by ninja leve
Re: (Score:2)
Will they also never reach their full potential, learning in what is ultimately the inferior path?
Re: Junior developers (Score:4, Insightful)
testing.
I do not see testing mentioned.
testing is only 80 percent of a project
Re: (Score:2)
Also even if it could speed up junior developers, they will never become seasoned devs if they rely on the AI and do not learn how to solve problems by facing them.
No problem. (Score:5, Insightful)
Sounds like a win-win to me!
Re: (Score:2, Insightful)
Indeed. And on the plus side, even the experienced people did think they were faster, when in fact they were slower. That effect will be even stronger with inexperienced people.
Expected (Score:3)
This kind of outcome doesn't surprise me at all. It reminds me of trying to fumble with speech-to-text, or the tab-complete "suggestions" I get adding notes in ServiceNow - these things might help people who aren't decent typists, but for me they just get in the way if I pay any attention to them.
Duh (Score:5, Insightful)
Above a relatively low complexity level, reviewing code for problems gets much harder than writing correct code from the start. This has been known reliably for a long, long time. And it means that as soon as you exceed that complexity, AI use will make you slower if you go for the same quality level. And that is before taking into account that AI code is generally bad for maintainability as it is harder to read and will increase technological debt as avoiding that requires actual insight.
The whole "AI coder" idea is a dud. Again. It must now be the 4th or 5th time that has happened.
Re: Duh (Score:3)
Re: (Score:2)
Indeed. And you end up with something better (not band-aid fixed) as well.
Mmmmh (Score:2)
Both of them?
To get value out of coding assistance.. (Score:3)
You have to get a feel for when it's utterly useless and when it's got a good chance of success. You also have to have a different vigilance as the LLM will make mistakes unlike what humans will do. My experience is it can be a modest time save but only because I've learned to ignore it for most of my work. Occasionally neat completion, very very rarely generating useful snippets from prompts, and ability to generate doc strings that no one will probably read anyway. Utterly asinine at trying to write comments, documenting the self evident tediously and skipping commenting anything that actually could use it. One example is if I write a little command line utility using variables that make sense to me it's decent at seeing the uninitialized variables and generating a chunk of argument parsing code including decent staff help text based on the observed variables
When all is said and done, if AI convinces management I don't need the diploma mill outsourced code slinger from the lowest bidder, I come out ahead even if I never even use the AI at all.
Re: (Score:2)
I agree with your premise, but I do think there are a few additional specific was AI can be helpful. For example, I find it useful to get AI to generate code to call an unfamiliar API, the first time I use it. Also, when the syntax is obscure, such as SQL Server XML or JSON manipulation commands. These are easy to review, hard to generate. The underlying principle is right on: Know exactly when to use AI, and when to skip it.
There is another reason I use AI: to generate drudgework code that I don't want to
Re: (Score:3)
> I find it useful to get AI to generate code to call an unfamiliar API, the first time I use it.
Every time I used it for that, it made up - sorry, hallucinated - non existent API calls, with links to documentation, ofc 404. Every; fucking; time.
As expected (Score:3)
While the hypemongers boldly claim that AI will write all code soon, reality is a bit different.
AI demos impress investors and managers by quickly creating simple projects that are similar to existing stuff. The prompts "make a snake game in python" or "make an iphone fitness app" only work because the code is simple and similar stuff exists.
Creating complex systems that work well takes skill.
As for me, I use AI as a better help system. Finding and understanding poorly written or obsolete documentation can be time consuming and sometimes I can't find it at all after hours of searching. AI gives answers quickly. Most often they are correct, and even if they aren't, they point me in the right direction
Re: (Score:2)
"Write me a complete TradeWars 2002 v1.00 w/gold expansion clone, in Java, using a PostgreSQL database for backend."
When an "AI" can do that, maybe I'll be interested.
Training (Score:2)
I wonder what percentage of developers receive any kind of formal training on how to use this so-called "AI" and what percentage are expected to "just figure out" this so-called "transformative technology that represents a fundamental change to the traditional software process".
My guess is:
1). Virtually nobody gets training.
2). Any slow-down among developers can be directly attributable to the lack of training.
Re: (Score:2)
What "training" do you think would change the results here?
Re: (Score:2)
Maybe some training in how to effectively use "AI" to generate software code applicable to whatever it is one is trying to accomplish" ?
Re: (Score:2)
I'm asking what you think that training would look like. What kinds of things do you think these developers aren't doing that would make them actually more productive, and not just make them think they're more productive?
From what we've found, simply not using AI would net an easy boost in productivity.
That's what I have seen -- sort of (Score:3)
My software development experience spans more than a half century. We have come a long way from the time when I was punching out FORTRAN and assembly language job decks on Hollerith cards.
Last year I came out of semi-retirement to work on an interesting startup project. When I started I had no real notion of AI coding but now I am fully immersed in it. I feel like a fossil.
There are people in the company who are tech savvy but not software engineers. They would never be able to qualify for even a junior software engineering position at a company that did stuff like coding tests. But they do know stuff like REST interfaces and data base structures and all the tools that make it easy to play with all that. But now they are implementing full microservices (I set up a kubernetes environment for the app) using tools like GPT, Cursor, Claude, Copilot and whatever.
I sometimes pair on a screenshare while they do it. They never even see, much less read, the code they generate. Yet it works on the test bench. They get there by iterating it over and over again by changing the prompts until they get what they want. Very often I see them on the slack channel at 3AM, still at it.
It then becomes my job to integrate it into the deployment code base. I read the code and it is remarkable. Yet flawed in many areas. Often it is architecturally cumbersome, and often fails to make reusable code. But does that really matter? There's five functions that could be reduced to one function with a parameter. So what it works, right? But they have had little or no idea of what it takes to make it production ready. I spend many hours doing that.
So my next job is mentoring them on what to ask the AI for in their prompts. AI coding works, but you have to know what the end result is supposed to look like. If you don't have experience "doing it the hard way" there are a million things you just don't know.
We aren't going back. That's the only thing I am sure of.
Re: (Score:2)
We aren't going back. That's the only thing I am sure of.
Don't bet on it. AI is expensive. A lot more expensive than people realize. Add to that the astonishing technical debt it creates and the increasing evidence that it isn't actually saving any time ...
Predictions are hard, especially about the future, but this one is as clear as it gets. The only reason things haven't crashed already is the insane belief that things are rapidly improving. The simple fact is that for all the hope people put in the magic of emergence, there are fundamental limits here tha
Re: (Score:2)
I wouldn't be betting on it either. The astronomical money spent for pretty much nothing of value is beyond mind boggling. And the amount of electricity to support that spend is literally breaking the grids.
Re: (Score:2)
You and I are probably roughly the same age as that's how I started off too, in my case an IBM 1620 with 20,000 digits of core memory. It could be expanded to twice or three times the memory, but was otherwise thoroughly obsolete. Just out of curiosity, do you still enough about card images to explain why they were on
Re: (Score:2)
You and I are probably roughly the same age as that's how I started off too, in my case an IBM 1620 with 20,000 digits of core memory. It could be expanded to twice or three times the memory, but was otherwise thoroughly obsolete. Just out of curiosity, do you still enough about card images to explain why they were once important, without looking the term up.
Yep I had access to a 1620 as well. Never did anything serious with it but I remember writing FORTRAN II programs with it. It had a 2-pass compiler where the intermediate pass was punched out on cards, then you had to feed them back in again for the second pass, and eventually you got a binary deck that was the runnable program. Good times.
As to your question, I'm not sure what you are getting at. Hollerith cards had been in use for a long time before they were used for job decks. IBM had all sorts o
Re: (Score:2)
I fired him (Score:2)
Okay only a single data point, but here's a scenario that I can't imagine is unique.
I hired a junior full stack developer with a small portfolio of simple work. Started out with matching skill tasks, and grew over many months in complexity. As the complexity grew, the work output became slower and messy to be polite. We're not talking about jumping from 'edit this html' to 'design and build this ERP' but think standard stuff - build the api and the front end for a 10 view app. Eventually a brick wall was hi
"Some" is doing some heavy lifting here (Score:2)
I know the media has a boner for AI, but is there any justification for the use of the word "some" here when it appears to be a decent survey and there's no evidence that the senior devs involved are atypical? I wouldn't expect a drug report where most of the people not in the control group saw their symptoms as improving reported as "{Drugname} Cures Some {Condition} sufferers"
Seems a strange choice of headline.
You need different solutions (Score:2)
Imagine answering questions while kids shout (Score:2)
One difference is I work in Java, not Python. Most advocates I have met of A
Re: Imagine answering questions while kids shout (Score:2)
Management hoop jumps (Score:2)
The push from management to use Copilot and other tools has been a pointless farce. They've become so focused on adopting the new tools and even track our usage metrics. They they forgot that we have actual products and releases to address, and that we have cross functional bugs that aren't receiving the attention they need from management.
I feel like I could replace some of what project managers do with ChatGPT. Just get it some prompts to work out nag emails and track what outstanding bugs and features ar
Detachment... (Score:2)
Spent a week trying to write an app only using AI. (Score:2)
I just spent the last week trying to write an app only using AI. No handwritten code at all.
Here is what I found.
1) It is much better at writing new code than refactoring old code. I found this out on day two and scrapped everything it did on day one.
2) You have to take a lot of time preparing instructions for it to follow ahead of time. I had it create the following before it wrote a single line of code.
- system design guidelines (378 lines of information)
I find the opposite (Score:1)
I just did a vibe-coded project and it boosted my productivity through the roof. I'm astonished as to what it can do.