New GitHub Copilot Research Finds 'Downward Pressure On Code Quality'

theodp writes: Visual Studio Magazine reports on new research on the effect of AI-powered GitHub Copilot on software development which sought to investigate the quality and maintainability of AI-assisted code compared to what would have been written by a human. Countering the positively-glowing findings of some other studies, the Coding on Copilot whitepaper from GitClear cites some adverse results.

"We find disconcerting trends for maintainability," explains the paper's abstract. "Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of 'added code' and 'copy/pasted code' is increasing in proportion to 'updated,' 'deleted,' and 'moved 'code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited." The paper concludes, "How will Copilot transform what it means to be a developer? There's no question that, as AI has surged in popularity, we have entered an era where code lines are being added faster than ever before. The better question for 2024: who's on the hook to clean up the mess afterward?" Further complicating matters, Computing Education in the Era of Generative AI (Feb. 2024 CACM) notes that "generating and inserting large blocks of code may be counterproductive for users at all levels. This requires users to read through code they did not write, sometimes at a more sophisticated level than they are familiar with."

Interestingly, the AI-generated code maintenance worries are reminiscent of concerns cited in the past for 'Google programmers', Stack Overflow copy-and-pasters, and stitchers of not-quite-compatible libraries, as well as earlier iterations of code generators, including C++ and other 'Next-Next-Finish' code wizards of the 90's and COBOL and PL/I applications generators (PDF) of the 80's. Everything old is new again, including code maintenance challenges.

Explore Github Alternatives

[BrendaEM]

With Microsoft owning Github, and MS, slapping the AI label on it, you can count on your code being used by MS, and using it for evil. Yet, everyone just tries to reassure that what has happened--hasn't happened. Microsoft owns Github. There is nothing safe or cool about it. Wake up.

From Wikipedia: "GitHub, Inc. (/thb/[a]) is an AI-powered developer platform that allows developers to create, store, manage and share their code."

Re:

[nysus]

Wasn't making code easily shareable by others the whole idea of open source?

Re:Explore Github Alternatives

[Junta]

Depending on your camp, there's nuance.

For those who do public domain, this is true, go and do whatever you want whenever you want.

For the BSD-like licenses, sure, do whatever you like *but* keep attribution, make sure I get credit for my contribution. You've gone into the world where laundering code through LLM pisses off those people, they do the work and don't get credit.

Then GPL-like copyleft where open source is expected to be reciprocated in kind.

In short, since this transformation does not care to preserve or honor any license or attribution, then plenty in the open source world will still be pissed in a manner consistent with their ideology that includes sharing code.

Re:

[Z00L00K]

So just write code add it to Github but also make sure it has some wonky bugs that won't matter for your code but if someone else do a copy/paste it could spell the end of the universe. At least the Azure universe.

Re:

[Darinbob]

This is why my plan is to only write EVIL code, and thus when Microsoft uses it to do the opposite of what was intended then it will do good!

Expected

[Petersko]

Look, I have nothing to back up what I'm about to say, so feel free to file it in the appropriately labeled waste receptacle.

I don't believe this trend is new. The profession has been commoditized. We beat the drum on "coding" as a profession to funnel people through, and a large number of people entered it who otherwise couldn't care less about programming. They have no passion for it, no innate driving interest. Then we started farming the profession off to cheaper countries. Now we throw these "assistance" tools at folks, and these tools continue the process of allowing disinterested coders to keep their jobs by producing functioning, inelegant, fragile code.

I've been in the field for 35 years. I entered it without post-secondary education back in 1988. BUT... I came in having maxed out the capabilities of my Vic-20 and C64. I wrote my own functioning 300bps BBS software in machine language when I was 14 years old. I cared... and I care to this day. The code I write today is the best it has ever been, although it's primarily for my benefit alone.

I'm now in management, and have been for about 15 years. The quality of new grads is not great... but the lack of the inner fire is worse. They ended up there, and aren't exactly sure why. Every once in a while a great one comes through... but it's clear they are stymied by those around them. I was surrounded by enthusiasts early in my career... they're surrounded by opportunists and paycheque seekers. "Hard to fly like an eagle..."

The industry may correct itself. The AI assists may "assist" the commodity programmers in exiting the profession. At least then we won't have to hear people saying that the solution to upward mobility is to "learn to code".

Re:Expected

[Junta]

The quality of new grads is not great... but the lack of the inner fire is worse.

Easy, the job became a "hot" job and people with no interest started seeking a gold mine around the .com boom. So people before the late 90s tend to have an inherent interest in the work consistently, but those are now drowned out by the 'easy money' crowd that came later.

Speaking as someone who was basically right in the thick of the gold rush and disillusioned even then at my cohorts in the classes.

Re:

[DarkOx]

The market asked for those '99ers though.

Every shop started hiring those people at inflated wages knowing full well if that wages would fall. They also knew they'd "age" those people into lower management roles; where understanding of what developers do but not much real ability is required to be effective and they would need those people anyway as their IT foot print grew.

At least the F500 world understood this. They were moving to commodity hardware to escape the mainframe, way less efficent for most of w

Re:

[Darinbob]

I honestly think Visual Basic was the warning knell for the decline in programming quality. Because it taught the non-programming managers that programming was super simple that even idiots could do that, and that his team of highly paid programmers were just time wasters. Sandbaggers! After all, the manager wrote something in under and hour that looked just fine and seemed to do the right thing when given only correct inputs. Quick-and-dirty started becoming much more common. Make a rapid prototype th

Re:

[Darinbob]

I honestly think that a huge percentage of programmers can't program. They merely cut and paste. And a lot of this is because there are very few barriers to entry in the programming job market. Especially when you consider outsourcing, the hiring of friends-of-friends, etc. There is a huge lack of professionalism, in short.

Lack of professionalism means a lack of standards, training, quality, etc. Ie, I expect that the electrician that comes to fix my house is a professional, that the the electrician kn

Industrialization of IT and Software

[Qbertino]

I basically agree with what you're saying but I would describe this as an industrialization of IT and Software (Development). With that the "Artisan" aspect of software development goes by the wayside. Bluntly put: In these times it's too costly for society to keep highly skilled experts around when run-off-the-mill software and cargo-cult programming by some clueless clerk will do the trick " good enough " and, most importantly, way quicker/cheaper than that expensive guy who's always caught up in other re

Re:

[Petersko]

I agree completely. That's an excellent second track of thought. Why the code is suffering seems clear to me... but also, does the world actually need great code? Not for most things.

You put it very well, so I won't belabor the point. But there are times when great code matters... and that's not "most" times.

Re:

[gweihir]

That is just the thing: Most code written these days is_not_ "good enough". The basically daily catastrophic attacks and data-thefts show that nicely as does the abysmal usability of a lot of modern software.

Re:

[Lije Baley]

I couldn't decide which response to this to mod up. While I want to very much agree that most of today's bad code is probably "good enough", I can't help but acknowledge that it also comes with potentially significant costs (maintenance, security, poor performance/usability), which are typically deferred and essentially hidden.

Re:

[dsgrntlxmply]

"Commoditized" recalls a small book from 1977 when structured programming (landmark Dijkstra 1968) was still under controversy in its introduction to commercial practice: Philip Kraft, Programmers and Managers, the Routinization of Computer Programming in the United States. [amazon.com]. Raised here mostly to show that frustration with coding productivity is not new, and raises waves of controversy and bridging innovation (e.g. Kernighan's RATFOR) around new techniques.

Re: Expected

[paul_engr]

Why do Microsoft and Google support coding curriculum so much? So they can drive down wages by flooding the market. Care about programming and have a talent for it? Fuck you, we'll hire those four other folks who also had to take intro to CS during high school for less money. The apathetic in it for a.paycheck thing is not unique to software engineering. I've run across endless piles of hapless, utterly useless green hardware engineering graduates who have no interest in the profession other than "mom/

Re:

[Tony Isaac]

Like you, I started my software development career in 1988, and like you, I've been managing dev teams for more than 15 years. Overall, I agree with your take. But I have been able to still find and hire programmers that have that fire. I do it by asking "why" questions when I interview candidates. As an example, "What ORM, if any, would you choose, and why?" If the candidate says "I don't know, somebody else made that decision" it's an automatic no. I look for people with opinions, who care about the detai

Re:

[Local ID10T]

Garbage in, Garbage out.

When we train coders and their new AI-assisted tools on examples of poorly written but functional code, they learn to write bad code.

We are compensating for poor quality code with more hardware resources because hardware is cheaper than good coding. But it is rewarding bad behavior and leading to exploitably-bad code.

Re:

[gweihir]

You are not wrong. But at the same time, the state-of-the-art is by far not advanced enough that this could work. We are 50 years, maybe 100 years before coding will actually be a commodity skill, where people can just combine components and technician-level people comparable to an average but solid plumber, electrician or carpenter can do it. Until then, this is a proper engineering task, and an advanced one, because it is custom engineering. Put in average skill people and it will fail and end up being mo

Are you surprised?

[Mr. Dollar Ton]

When code monkeys used to ask questions they should have known the answer to on stack exchange, they'd often get good answers, referrals to relevant documentation, and sometimes outright ridicule that would push many of them to study harder.

When they ask the "AI" for $10 a month, they get an answer compiled from the answer of similarly "gifted" individuals, which they copy-paste verbatim.

Which practice will result in better code? I can't make up my mind, really.

Re: Are you surprised?

[tchetch]

I think AI will slowly lead to more structured code and data. Of course it's the beginning so it's a mess. But, in fact, you can "hack the prompt" of coding AI by wtiting a well defined code with a well defined data structure and, then, you will see AI kicking in quite well for repetitive parts. To get the maximum from AI you better use longer, and clearer, function name (or classe or whatever), giving a more self-documented code. In the longer run, I would say that code and data structures will improve. Ju

Re: Are you surprised?

[paul_engr]

You have to be mindful of the data sets they're training on. As time goes on, the AI is going to poison its own data set and a negative feedback loop of enshittification will happen. The principle of entropy dictates that the universe will get less organized. This will apply to code quality in the EZ sources if info for idiots as well. (I am one of those idiots wrt/ coding)

Re:

[Mr. Dollar Ton]

You can't, because to write good code for a task, you have to consider its purpose, and the "AI" cannot do that.

The best thing that may come out of it would be an "AI"-peppered help system. But it isn't a safe bet it will come about.

It's more like spell check not grammar check

[wakeboarder]

You can get it to generate code and to autofill, great, saves some time. But that code has zero context, and the LLM's are not intelligent enough to know what library you are using or other things like environment. Python is a mess with LLM's, it will happily snip in code that is 5 years old and worthless, and doesn't use any context of the version or even look at the API. Most of the time its solving a problem and copying tokens from A to B with whatever mapping it has.

It saves time I think or gives you a

Why gee

[Rosco P. Coltrane]

It's almost as if nobody saw it coming...

This is not surprising

[MpVpRb]

Managers want cheaper code
Lazy or incompetent programmers want a faster way to write barely acceptable code
I'm hopeful that future development will lead to tools that allow us to write better code, but my crystal balls say that things will get a lot worse before they get better

Re:

[gweihir]

but my crystal balls say that things will get a lot worse before they get better

I agree. But we _will_ get some "reference catastrophes" (that all other engineering disciplines had before they cleaned up their act) eventually and it will not be merely with a few hundred people dead or a city devastated. It will be with complete countries going into steep decline. After that, things will get better. But not before.

Re:

[gweihir]

You are too optimistic here: Code can have _negative_ utility.

AI job security

[dwid]

The AI will be needed to maintain its own poor quality code. The AI will become indispensable, humans are just the disposable infection vector.

General Solutions/Specific Problems

[laughingskeptic]

Code bases have their own utility classes and ways of doing things, for instance to capture logs in a certain way that a general answer from ChatGPT knows nothing about. Until people start training specific models for their code bases and pay to keep those models up-to-date code pasted from ChatGPT our encouraged by CoPilot will not be code-base compliant.

Helps some but don't trust it

[ZipNada]

I use a free AI assistant (Codeium) instead of Copilot but it does basically the same thing. It is very convenient at times. It will auto-complete sections of trivial code that I was about to type in by hand, definitely a time saver. You can ask it questions and the answers are generally helpful.

You can get it to refactor large chunks of existing code or write a bunch new code based on a description of desired functionality, but I think that's where the problem lies. In my experience it frequently will be s

With code, less is more

[mkwan]

The most satisfying task for me is deleting redundant/unused code. It reduces complexity and makes the job easier.

An AI that can only add code is going to harm productivity in the long run.