AI Slows Down Some Experienced Software Developers, Study Finds (reuters.com) 30
An anonymous reader quotes a report from Reuters: Contrary to popular belief, using cutting-edge artificial intelligence tools slowed down experienced software developers when they were working in codebases familiar to them, rather than supercharging their work, a new study found. AI research nonprofit METR conducted the in-depth study on a group of seasoned developers earlier this year while they used Cursor, a popular AI coding assistant, to help them complete tasks in open-source projects they were familiar with. Before the study, the open-source developers believed using AI would speed them up, estimating it would decrease task completion time by 24%. Even after completing the tasks with AI, the developers believed that they had decreased task times by 20%. But the study found that using AI did the opposite: it increased task completion time by 19%. The study's lead authors, Joel Becker and Nate Rush, said they were shocked by the results: prior to the study, Rush had written down that he expected "a 2x speed up, somewhat obviously." [...]
The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route."
The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route."
Junior developers (Score:5, Insightful)
> do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with
If the juniors are working with the same code base, with the same AI that makes same incorrect suggestions which seniors reject and if AI is not slowing juniors down, doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
Re: (Score:2)
I strongly suspect so -- but would the rate and degree of pollution be worse than the baseline case for an inexperienced developer? Most software developers (and I think this decreases only somewhat with experience) do not have the discipline to think rigorously about the changes they are making and review the surrounding context. That self-discipline is what distinguishes engineers from mere coders.
Re: (Score:2)
I strongly suspect so -- but would the rate and degree of pollution be worse than the baseline case for an inexperienced developer? Most software developers (and I think this decreases only somewhat with experience) do not have the discipline to think rigorously about the changes they are making and review the surrounding context. That self-discipline is what distinguishes engineers from mere coders.
Junior developers might make similar mistakes (or mistakes similar in magnitude) as the AI, but the junior developers likely won't learn what they're doing wrong. So they'll never have the same expertise as the current "experienced" devs.
Re: (Score:2)
So they'll never have the same expertise as the current "experienced" devs.
What do you think "experience" means? How do you think people get it?
Junior developers will make mistakes, but if the environment is organized to support learning and the junior devs stick around for a bit, they will learn -- either from their own mistakes or from others' -- and get better. Sometimes you get someone with "one year of experience, ten times" instead of "ten years of experience", but a good shop should try to avoid or remove that, depending on the root cause.
I mean, it is trivially true that
Re: (Score:2)
Relativity (Score:2)
doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
Probably but the question you want to ask is which produces the fewest wrong decisions: making their own decisions vs. missing the AI's mistakes?
Re: (Score:2)
No, not really.
Juniors know how to code. They wrote plenty of small programming assignments in college.
They don't know the details of the codebase and have less experience making changes to million-line systems, but that's different from being unable to recognize a buggy function.
Re: (Score:2)
If a function is simple enough that "buggy" can be defined in isolation, AI or a junior developer can probably write it without much supervision. Most bugs that escape the first-level developer are violations of contextual expectations: cases where the code would work as expected in a different application, with a different use case, or something like that. So the characteristics of "a buggy function" depend on the code, processes and users around it, and that is where junior developers often fall short.
D
Re: (Score:3)
> do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with
If the juniors are working with the same code base, with the same AI that makes same incorrect suggestions which seniors reject and if AI is not slowing juniors down, doesn't that mean that juniors will accept the changes made by AI and pollute the code with wrong decisions?
I could very well imagine AI helping me to get a handle on/oversight over code bases that I'm not familiar with as well as really huge codebases where I'm familiar some parts but don't know every nook and cranny, or for insanity inducing tasks like finding memory leaks and heisenbugs but in terms of helping me code I'm not so sure. That sounds like a crutch that might be good for junior developers until they outgrow it. But why are we even talking about this? Aren't we all about to be replaced by ninja leve
Re: (Score:2)
Will they also never reach their full potential, learning in what is ultimately the inferior path?
Re: Junior developers (Score:2)
testing.
I do not see testing mentioned.
testing is only 80 percent of a project
No problem. (Score:5, Insightful)
Sounds like a win-win to me!
Re: (Score:2, Insightful)
Indeed. And on the plus side, even the experienced people did think they were faster, when in fact they were slower. That effect will be even stronger with inexperienced people.
Expected (Score:3)
This kind of outcome doesn't surprise me at all. It reminds me of trying to fumble with speech-to-text, or the tab-complete "suggestions" I get adding notes in ServiceNow - these things might help people who aren't decent typists, but for me they just get in the way if I pay any attention to them.
Duh (Score:5, Insightful)
Above a relatively low complexity level, reviewing code for problems gets much harder than writing correct code from the start. This has been known reliably for a long, long time. And it means that as soon as you exceed that complexity, AI use will make you slower if you go for the same quality level. And that is before taking into account that AI code is generally bad for maintainability as it is harder to read and will increase technological debt as avoiding that requires actual insight.
The whole "AI coder" idea is a dud. Again. It must now be the 4th or 5th time that has happened.
Re: Duh (Score:2)
Mmmmh (Score:2)
Both of them?
To get value out of coding assistance.. (Score:3)
You have to get a feel for when it's utterly useless and when it's got a good chance of success. You also have to have a different vigilance as the LLM will make mistakes unlike what humans will do. My experience is it can be a modest time save but only because I've learned to ignore it for most of my work. Occasionally neat completion, very very rarely generating useful snippets from prompts, and ability to generate doc strings that no one will probably read anyway. Utterly asinine at trying to write comments, documenting the self evident tediously and skipping commenting anything that actually could use it. One example is if I write a little command line utility using variables that make sense to me it's decent at seeing the uninitialized variables and generating a chunk of argument parsing code including decent staff help text based on the observed variables
When all is said and done, if AI convinces management I don't need the diploma mill outsourced code slinger from the lowest bidder, I come out ahead even if I never even use the AI at all.
Re: (Score:2)
I agree with your premise, but I do think there are a few additional specific was AI can be helpful. For example, I find it useful to get AI to generate code to call an unfamiliar API, the first time I use it. Also, when the syntax is obscure, such as SQL Server XML or JSON manipulation commands. These are easy to review, hard to generate. The underlying principle is right on: Know exactly when to use AI, and when to skip it.
There is another reason I use AI: to generate drudgework code that I don't want to
Re: (Score:2)
> I find it useful to get AI to generate code to call an unfamiliar API, the first time I use it.
Every time I used it for that, it made up - sorry, hallucinated - non existent API calls, with links to documentation, ofc 404. Every; fucking; time.
As expected (Score:3)
While the hypemongers boldly claim that AI will write all code soon, reality is a bit different.
AI demos impress investors and managers by quickly creating simple projects that are similar to existing stuff. The prompts "make a snake game in python" or "make an iphone fitness app" only work because the code is simple and similar stuff exists.
Creating complex systems that work well takes skill.
As for me, I use AI as a better help system. Finding and understanding poorly written or obsolete documentation can be time consuming and sometimes I can't find it at all after hours of searching. AI gives answers quickly. Most often they are correct, and even if they aren't, they point me in the right direction
Re: (Score:2)
"Write me a complete TradeWars 2002 v1.00 w/gold expansion clone, in Java, using a PostgreSQL database for backend."
When an "AI" can do that, maybe I'll be interested.
Training (Score:2)
I wonder what percentage of developers receive any kind of formal training on how to use this so-called "AI" and what percentage are expected to "just figure out" this so-called "transformative technology that represents a fundamental change to the traditional software process".
My guess is:
1). Virtually nobody gets training.
2). Any slow-down among developers can be directly attributable to the lack of training.
Re: (Score:2)
What "training" do you think would change the results here?
Re: (Score:2)
Maybe some training in how to effectively use "AI" to generate software code applicable to whatever it is one is trying to accomplish" ?
That's what I have seen -- sort of (Score:2)
My software development experience spans more than a half century. We have come a long way from the time when I was punching out FORTRAN and assembly language job decks on Hollerith cards.
Last year I came out of semi-retirement to work on an interesting startup project. When I started I had no real notion of AI coding but now I am fully immersed in it. I feel like a fossil.
There are people in the company who are tech savvy but not software engineers. They would never be able to qualify for even a ju
Re: (Score:2)
We aren't going back. That's the only thing I am sure of.
Don't bet on it. AI is expensive. A lot more expensive than people realize. Add to that the astonishing technical debt it creates and the increasing evidence that it isn't actually saving any time ...
Predictions are hard, especially about the future, but this one is as clear as it gets. The only reason things haven't crashed already is the insane belief that things are rapidly improving. The simple fact is that for all the hope people put in the magic of emergence, there are fundamental limits here tha
Re: (Score:2)
I wouldn't be betting on it either. The astronomical money spent for pretty much nothing of value is beyond mind boggling. And the amount of electricity to support that spend is literally breaking the grids.