Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
AI Programming

'Stack Overflow is ChatGPT Casualty' (similarweb.com) 150

SimilarWeb: Developers increasingly get advice from AI chatbots and GitHub CoPilot rather than Stack Overflow message boards. While traffic to OpenAI's ChatGPT has been growing exponentially, Stack Overflow has been experiencing a steady decline -- losing some of its standings as the go-to source developers turn to for answers to coding challenges. Actually, traffic to Stack Overflow's community website has been dropping since the beginning of 2022. That may be in part because of a related development, the introduction of the CoPilot coding assistant from Microsoft's GitHub business. CoPilot is built on top of the same OpenAI large language model as ChatGPT, capable of processing both human language and programming language. A plugin to the widely used Microsoft Visual Studio Code allows developers to have CoPilot write entire functions on their behalf, rather than going to Stack Overflow in search of something to copy and paste. CoPilot now incorporates the latest GPT-4 version of OpenAI's platform.

On a year-over-year basis, traffic to Stack Overflow (stackoverflow.com) has been down by an average of 6% every month since January 2022 and was down 13.9% in March. ChatGPT doesn't have a year-over-year track record, having only launched at the end of November, but its website (chat.openai.com) has become one of the world's hottest digital properties in that short time, bigger than Microsoft's Bing search engine for worldwide traffic. It attracted 1.6 billion visits in March and another 920.7 million in the first half of April. The GitHub website has also been seeing strong growth, with traffic to github.com up 26.4% year-over-year in March to 524 million visits. That doesn't reflect all the usage of CoPilot, which normally takes place within an editor like Visual Studio Code, but it would include people coming to the website to get a subscription to the service. Visits to the GitHub CoPilot free trial signup page more than tripled from February to March, topping 800,000.

This discussion has been archived. No new comments can be posted.

'Stack Overflow is ChatGPT Casualty'

Comments Filter:
  • The end is coming (Score:3, Interesting)

    by Anonymous Coward on Thursday May 11, 2023 @10:48AM (#63514031)

    A.I. will take all desk jobs first, humanoid robots will take the rest later and all profits will go to the rich and everybody else will be homeless. This is a pyramid scheme but people are too blind to see it.

  • Code stagnation (Score:5, Interesting)

    by avandesande ( 143899 ) on Thursday May 11, 2023 @10:49AM (#63514035) Journal
    As new code relies more and more on existing code there will be less and less experimentation or even accidental implantation of code improvements.
  • get this tensor-promoting crap out of here. SO is losing traffic because search engines are useless.
  • It does make sense (Score:5, Insightful)

    by Ecuador ( 740021 ) on Thursday May 11, 2023 @10:51AM (#63514039) Homepage

    Not sure if there is actually a causation behind the correlation, but It does make some sense. I use stack overflow quite often, but it is always when I look for something easy in either a language of framework I am not familiar with. So I could see people replacing things like that with CoPilot etc (and indeed at least a couple of my junior devs seem to do exactly that).

    I am quite an experienced and competent developer, so if I need something that is actually complex - i.e. it's on a language/technology I know, but I just can't figure it out, stack overflow has been proven useless to me - I've never gotten a good answer, I think people more competent than myself don't spend much time answering random questions on stack overflow. I definitely could see it replaced by language models. The answers would quite often be wrong of course, there is quite some vetting currently as competent devs do tend to look for things and if the answers they get are not right they figure it out and correct them.

    • Unsustainable? (Score:5, Insightful)

      by Roger W Moore ( 538166 ) on Thursday May 11, 2023 @11:10AM (#63514099) Journal
      It might be understandable but it may also not be sustainable because without sites like Stack Overflow where will the data to train AI's on giving programming advice come from?
    • by HiThere ( 15173 )

      I've pretty much disengaged because it's so difficult to display a complex problem. Some people seem to manage, but I suspect they're a lot more familiar with modern HTML than I am. Or perhaps it's chat boards that they're familiar with. I keep getting told That's not valid code to post, and no clear reason as to why. So I disengaged. When meant that I also didn't answer questions when I could.

      So now I go there frequently while switching languages to remind myself of things I've forgotten, but I don't

    • Not sure if there is actually a causation behind the correlation, but It does make some sense. I use stack overflow quite often, but it is always when I look for something easy in either a language of framework I am not familiar with. So I could see people replacing things like that with CoPilot etc (and indeed at least a couple of my junior devs seem to do exactly that).

      I am quite an experienced and competent developer, so if I need something that is actually complex - i.e. it's on a language/technology I know, but I just can't figure it out, stack overflow has been proven useless to me

      Stackoverflow is good both for general learning as well as validation. An example? I am very confident of Java best practices, but I have to sometimes work in JavaScript or Python where I know what to do, but want to confirm. It's similar to how I look up how to say things I know in Spanish in Google translate because I want to confirm before I send them out. In particular, I have found it very valuable for things I don't do often. I had to call a framework I haven't used in 8 years...I wanted to confi

    • I agree with this. A major problem with SO (and Stack Exchange in general) is that people really like easy questions that they can answer in a few lines, show off how smart they are, and quickly scoop up some points. They seem to get cranky if a hard question is posed -- esp. from a new poster.

      On that subject, amusingly, I'm actually currently banned from posting questions on SO because I have a history of low-rated questions. Even though, e.g., my last question provoked a personal response from the IDE's p

    • I've tried ChatGPT a few times in place of picking up ready made code and I always ended up ditching it for the latter. The problem is any code written by a human has been arrived at iteratively; it has a menaingful history built into it. In those iterations the problem as well as the solution and ways to test and corner cases have been discovered. ChatGPT spits everything out at once; there is no history. The "reasoning" in the code including the comments has some slightly surreal -- possibly hallucnatory

  • by null etc. ( 524767 ) on Thursday May 11, 2023 @10:53AM (#63514045)

    Actually, traffic to Stack Overflow's community website has been dropping since the beginning of 2022.

    Wait, the traffic to a website used primarily by software developers has been experiencing a decline since the same period of time that tens of thousands of software developers have been laid off? What a coincidence!

    • by uncqual ( 836337 )

      If managers are even close to doing their jobs correctly, those that were laid off were generally the least skilled of the herd and hence most likely to be using SO more. Hence the percentage drop in SO usage would likely even be substantially higher than the percentage of jobs lost.

      • by Ungrounded Lightning ( 62228 ) on Thursday May 11, 2023 @03:39PM (#63514741) Journal

        If managers are even close to doing their jobs correctly, ...

        I've spent an entire career in computer software (and occasionally hardware) engineering, and it has been my experience that hi-tech managers are almost NEVER doing their jobs correctly.

        As I read it: There is SO much value-added in hi-tech, and enough reasons that a company can fail due to external factors that aren't their fault, that a manager can be utterly clueless and pathological and still avoid bankruptcy for years - then not be blamed and go on to bigger and better disasters.

        So hi-tech managers with bad ideas don't get selected against, and the incompetent and marginally-competent accumulate. Screwy management fads get implemented and run into the ground. The higher in the hierarchy, the screwier.

        (Rule of thumb: If you hear one mention the book "Crossing the Chasm" even once, immediately circulate your resume and cash your stock options as fast as they vest. They'll soon fire the early hires and strip them of their delayed rewards, and do so before the lore needed to keep the company going is transferred from their heads into documents and/or cheaper heads that are being retained.)

        • by uncqual ( 836337 )

          I spent most of my career at jobs that were startup companies when I joined and your experiences are very different than mine. Some of these companies ended up being large international companies or being subsumed by such companies (I never was at one that just "turned out the lights").

          Your experiences are quite different from mine -- although as the companies ended up large (either by growth or by acquisition) I was exposed to some of what you describe but outside the areas I worked in.

          I've never seen a so

  • by AleRunner ( 4556245 ) on Thursday May 11, 2023 @10:53AM (#63514047)

    Stack Overflow is great because the answers are subject to peer review. Quite often, if there's a specific technical expert in a particular specialist answer that person will even comment on the answers. What we're going to end up with is things which "work" where work means "get accepted by the bosses" and nobody realises that the answer is horribly wrong, either in security or design. I'm sure some of that will be solved in the AI, but lots of it won't.

    • The peer review is awful. An expert with 0 points can't correct a wrong answer given by someone with many points. I have seen this, the highest rated answer is more often than not wrong in some way often minor but surprisingly a lot of very wrong answers get rated up. Even when I find a decent answer I have to continue scrolling past several comments to find the one that's actually correct, which very often only has a single +1. A common thing I see is that C questions will get C++ answers, or asking wh

  • by Virtucon ( 127420 ) on Thursday May 11, 2023 @10:57AM (#63514061)

    without the cut/paste

    Getting the perspectives of other developers is a key element that's missing, Stack Overflow is very good to understand some of the nuances of why a particular solution or idea is good or bad. I don't see AI solutions offering that.

    • The funny thing is that any time I've discussed researching issues on SO or the internet in general it goes over like a lead balloon (you are a copy and paste programmer) but using these prompts or ChatGPT is somehow virtuous.
      • Software Developers can be cruel assholes. Some figure that if they have knowledge and you don't, that makes you somehow inferior and your stupid question shouldn't be answered. I don't see AI fixing that problem anytime soon. Also, can you imagine an AI that was trained to respond along the likes of a Torvalds?

        Linus wrote:

        I just think it's simplistic and wrong, and outright stupid

    • by ljw1004 ( 764174 ) on Thursday May 11, 2023 @11:40AM (#63514191)

      Getting the perspectives of other developers is a key element that's missing, Stack Overflow is very good to understand some of the nuances of why a particular solution or idea is good or bad. I don't see AI solutions offering that.

      Sure you do. Every time you ask ChatGPT for advice on how to solve a coding question, always follow up with "is that the best way? is it guaranteed to work in all cases? is there a more efficient solution?" It usually comes through with something.

      I think there are two kinds of programmers -- those who are happy to just get the code working in the cases they care about, and those who take care that their code is guaranteed correct in all cases. You might be in the second case if you read the "man" page for a call and work through all the error codes, or if you document the invariants concerning your state variables and prove those invariants are maintained, or if you're into formal models like TLC. I suspect the second kind will not be satisfied with ChatGPT because it was largely trained on writings from the first kind...

      (When I've asked ChatGPT whether an answer is guaranteed to work in all cases, or whether it's the best answer, it's had about a 5% success rate)

      • (When I've asked ChatGPT whether an answer is guaranteed to work in all cases, or whether it's the best answer, it's had about a 5% success rate)

        You mean about 5% of previous answers were the best way or you mean that you got 5% success in delivering completely correct code?

        • by ljw1004 ( 764174 )

          You mean about 5% of previous answers were the best way or you mean that you got 5% success in delivering completely correct code?

          5% success in delivering code accompanied a plausible warrant of correctness. (i.e. the code might well be correct, and it certainly correct for the trail cases I use, but it's not accompanied by a justification of why it's correct in all cases, and I don't yet know enough of the area to judge whether it's correct or not.)

          I think about 80% of the time when I ask it, ChatGPT says "you're right this might not work in all cases; here's another way". But mostly the other way also has either flaws or shares the

      • That's like trusting Dave down the hall. You know, the one who always pushes broken code with no unit tests in Pull Requests?
        "My code works! I tested it myself!"

        ChatGPT gives you whatever it's been trained to give you and that can be insecure crap. [theregister.com] ; it's out of date and you have to play 20 questions with it? Just like Dave, no thanks.
        I like being able to exchange ideas or nuances on code snippets or how-tos with other developers, that way we all can learn from it.

  • Comment removed (Score:5, Insightful)

    by account_deleted ( 4530225 ) on Thursday May 11, 2023 @11:03AM (#63514075)
    Comment removed based on user account deletion
    • The question isn't whether or not ChatGPT "rewords" something. If rewording constituted a copyright violation, every thesis ever written would be rejected as plagiarized. For copyright, the criteria is that something is similar enough that you can conclude it is copied. More importantly, prior art will cause your claim to fail, so you would probably have to make sure that your work is original enough to be afforded protection.

      Most stack overflow best answers can be found in other places. Now I love stack ov

      • The question isn't just on the output side. Its on the input side, too. Do AI researchers have a license that allows them to process the content in this way? No. US vs Aaron Swartz could be a legal precedent here.

        • by ranton ( 36917 )

          The question isn't just on the output side. Its on the input side, too. Do AI researchers have a license that allows them to process the content in this way? No. US vs Aaron Swartz could be a legal precedent here.

          How can US vs Aaron Swartz be a precedent for anything? The case was dismissed.

          Also, the laws he broke only apply to restricted computers. If your website is visible online without some kind of paywall it isn't restricted. ChatGPT may be breaking laws, but not the same ones Aaron Swartz was indicted on.

          • If your website is visible online without some kind of paywall it isn't restricted.

            Just because a website serves a page without logging in doesn't mean its content is in the public domain.

      • > If rewording constituted a copyright violation, every thesis ever written would be rejected as plagiarized.

        Once you read the dictionary, every book is a remix kind of plagiarism?

        • Well, no... the dictionary in itself is not content beyond being a dictionary. And as far as copyright goes, you have to prove two things. 1) That the work is substantially similar enough to the "original" to appear to be copied, and that 2) the defendant had likely heard, or had easy access to, the source material.

          That second one is why you can't just copyright every sequence of notes. It would be nonsensical to think the person listened to material only written down for the purpose of being copyrighted..

      • by jbengt ( 874751 )

        If rewording constituted a copyright violation, every thesis ever written would be rejected as plagiarized. For copyright, the criteria is that something is similar enough that you can conclude it is copied.

        Not only that, but functionality isn't copyrightable, only creative expression is. So most short clips of code to perform some function wouldn't be copyrightable by themselves. (YMMV*, IANAL**, IANASCJ***)

        *Your Mileage May Vary
        **I Am Not A Lawyer
        ***I Am Not A Supreme Court Justice

    • So SO has to choose between blocking Google and Bing completely, or finding some way to corrupt their results.

      which at least Google explicitly says will cause you to be delisted or downranked.

      In the mean time Stackoverflow does partially own copyrights on its own pages, even if a lot of content is user submitted. That gives SO the right to sue over using their content

      I think that's a dangerous road to go down. If they assume copyright of that material they might well become liable for it. The standard for DMCA safe harbor protection is so far involvement with creation of the content, but I could see assignation of copyright becoming a thing too.

    • What is an LM? Lemon-Merangue, as in pie?

  • Could be a problem (Score:5, Informative)

    by jonbryce ( 703250 ) on Thursday May 11, 2023 @11:04AM (#63514079) Homepage

    I presume ChatGPT is trained on Stack Overflow posts.

    Other people have pointed out that Stack Overflow gives you much better context - you can read the original question, you can read all the other comments and so on. This is all true.

    But also, if there are no new Stack Overflow posts, there is no new material for ChatGPT to train on.

    • Q: "Tell me how to quickly search for a substring in C++ and include comments on this method and write it all in the style of Chaucer."
    • except all of the millions of people who are typing their code into it. Remember it's only opt-out from now on, they have a datasource.
  • Your comment doesn't contain a reproducible example, which is frankly the bare minimum you could do to post. There's no way anyone could even attempt to answer this, even with my 10B gold stars, you're beyond the help of an absolute God, like me.

    I for one welcome a total replacement of stackoverflow with an AI bot that isn't a snarky troll (unless you want it to be).

  • Don't worry, I'm sure the StackOverflow power mods will just flag ChatGPT and Copilot as duplicate, that'll show 'em. /s

    In all seriousness, by these numbers it's really unlikely StackOverflow will completely die, at least among a dedicated core of users. (I can think of a few other sites like that.) And with all of the real ethical dilemmas involving ChatGPT and Copilot right now, (e.g. stripping the GPL from code scraped online) potentially killing off one of the most [uni-saarland.de] toxic communities [dice.com] on the web feels li

  • by rickb928 ( 945187 ) on Thursday May 11, 2023 @11:16AM (#63514119) Homepage Journal

    Stack Overflow needs a ChatGPT/etc plugin.

    And with human proofing and input, it gets exponentially better than AI alone.

    Hey, even ChatGPT copies and pastes from Stack Overflow...

  • by Bodhammer ( 559311 ) on Thursday May 11, 2023 @11:20AM (#63514133)
    Did you try turning it off and back on again?
  • Completely understandable. People would prefer answers NOW. Posting a question to a message board and then waiting what can be hours, days, or potentially never, to get an answer isn't ideal. It's far more helpful to ask your question and immediately get an answer. It's why we go to search engines like Google.

    When we can get immediate answers, who would choose to wait?

  • Multiple factors (Score:5, Interesting)

    by Craig Maloney ( 1104 ) * on Thursday May 11, 2023 @11:36AM (#63514181) Homepage

    I think it's multiple factors and not just ChatGPT.

    1. Stack Overflow has engendered a community that is hostile and rude unless folks create carefully crafted questions. So less questions are making it to the site.
    2. SEO farms have overwhelmed the search engine results, sometimes with SO's own content.
    3. The recent sale of Stack Overflow to Prosus killed a lot of momentum for me. I deleted my account when I heard the news. While I trusted SO's previous ownership and stewardship I don't have the same faith in an investment company doing right by its users.

    I'm sure ChatGPT figures into this equation as well but there's way more suspects than the infamous one. It's a tiresome narrative and I think doing more research is required before blaming everything on stochastic parrots.

  • I guess I'm not the only one who has a lengthy "-site:" string he applies to every query to filter out the pesky and worthless but due to SEO upvoted crappages.

    And yes, StackOverflow is part of that string.

    Quite frankly, if they didn't try to muscle into the foreground even when they have nothing of value to offer, we might occasionally visit them when they happen to do so once in a blue moon.

  • by balaam's ass ( 678743 ) on Thursday May 11, 2023 @11:44AM (#63514205) Journal

    for thinking my question is different from the previous answers I found before finally deciding to ask.

    Or give me irrelevant responses like "Why are you even trying to do this? / There's no reason you should want to do this" etc

    • Haha. That's the vast majority of answers. "You should just use this 39 step procedure to do this by hand every time."

    • "Why are you even trying to do this? / There's no reason you should want to do this"

      Holy god yes. I have no hair left to tear out from the culture of SE people calling "X/Y problem!!" and patting themselves on the back for coming up with a reason to not answer a question.

    • This is a great point. I refrain from answering questions and replying many times because I just don't feel like putting in all the effort to avoid the trolling army of holier than thous who are going to tell me I spelled something wrong or whatever. It's never actually happened to me, but I see it happening to many others, so I'll keep my 25+ years of coding experience to myself and the people I know personally, instead.
    • by _xeno_ ( 155264 )

      It wouldn't surprise me if ChatGPT was more likely to provide useful answers than StackOverflow.

      Or give me irrelevant responses like "Why are you even trying to do this? / There's no reason you should want to do this" etc

      Love that. Love being told "oh, just don't do that." Just trust me that there's a reason it has to be done with a specific tool in a specific environment. No, I can't just go rewrite the thing in flavor-of-the-day. No, I can't convince management Linux would be a better solution than Windows. Unless you're going to explain why I shouldn't do something at all, an answer of "you don't need to do that" is entirely u

  • by magzteel ( 5013587 ) on Thursday May 11, 2023 @11:49AM (#63514219)

    I found that when I compared ChatGPT results for a coding question to a Google search which returned a Stack Overflow page, the ChatGPT response seemed to have been scraped almost directly from StackOverflow.

    One issue I found with ChatGpt is it is not "version aware". If I asked "How do I do x using the version 1.2 API of something", the answer was always for the latest API version.

    • Similarly, I tried testing ChatGPT on writing stuff for prior editions of the Dungeons & Dragons game, and it likewise failed the same way. One piece of output said, "For more details, look in the Original D&D Monster Manual", which is not a thing that ever existed.

      • by KlomDark ( 6370 )

        Are you being pedantic on "D&D" vs "AD&D"? Cause I have an original AD&D Monster Manual on the shelf here. (Along with my original Fiend Folio with Cthulu and Elric stuff in it.)

    • ChatGPT is just a bloated search engine, that your answer was scraped should surprise exactly 6 people.
  • by Torp ( 199297 ) on Thursday May 11, 2023 @12:01PM (#63514265)

    They've always encouraged ready to paste answers instead of teach the man how to fish answers.

    And now everyone trains their text generators on them and because the answers are enforced to not require understanding it works great. For the text generators.

  • by bb_matt ( 5705262 ) on Thursday May 11, 2023 @01:58PM (#63514559)

    In the 14 or so years I've dipped in and out of SO, I've yet to find anything other than either solutions to simple questions OR at least been given an idea.

    That actually is kinda useful - so, in that way, SO is useful ... to a degree.

    However, it's only as useful as the comment quality for any given question - and very often, there's some highly questionable answers.
    The community vetting goes some way to resolving this, but that is hit and miss at the best of times.

    So, yeah, it's useful, even if often I end up thinking "why don't I stop being so lazy, and RTFM" - very often the solution to a problem is to actually go and learn how to code, but humans are often lazy and so look for shortcuts. When the boss is breathing down your neck as you are filled with a deep panic and anxiety because you can't solve a problem, it's super tempting to try and find the answer on SO.

    In my experience, that's a very dangerous path - you can end up wasting as much time desperately searching for solutions or posting questions that never get answered, as you would've done actually reading documentation and doing some deeper learning.

    I think ML via the likes of ChatGPT are a game changer.
    Sure, right now, we're seeing a lot of generated code from ChatGPT that is downright wrong - it doesn't even run.
    However, it is currently very good at explaining concepts and as such, is a useful tool.
    That downright wrong code that doesn't run, can still contain a lot of "Ah! - Wow, now I can see a way" moments - much more than a SO overflow thread can.

    I use SO less and less these days and ChatGPT more - but ultimately, my GOTO is my peers - my co-workers.
    We'll swarm a problem and everyone ends up learning, at the same time as we actually review the code.

    There's no replacement for that...

  • While StackOverflow is a great place to find answers, it is also very toxic.

    If someone can get the answer they need from a system that possesses the wealth of prior response and without someone calling the a f*CJ wit and to just RTFM, that would be preferable.

    Now, if the discourse is informative and cordial, I know Iâ(TM)d rather hear from humans with different ideas.

  • I feel like I live in a time where everything is intentionally misnamed, and everyone knows this and just goes along with it because correcting 1000 people every day is really tiring.
  • If they had put a footer under all the pages a couple years ago "We explicitly deny any implied license to copy this content for training of language models and similar machine learning applications, contact us for licensing." they could have charged a shit ton of money. They are one of the most valuable training/instruction sources on the web. Until the courts say explicitly that copying for AI training is fair use, this should scare the pants off any lawyer at an AI company. Willful infringement is more e

  • SO needs a better search engine. Maybe AI powered. So that I could ask it a question the same way I ask ChatGPT a question: long sentences like I am talking with a human. And then get a link (or links) to the answer on SO that answers my question the best.
  • I'm glad to see them fall. For a company that claims IP rights over all content posted to their forums without an option to ever delete, or if edited reserves the right to return the content back to pre edit, or modify your posts making it seem like you said something you didn't, I'm happy to see the entire ecosystem fail. Beyond that, it's highly used by incompetent individuals to seem like they have half a clue of what the code does and makes everyone in the system more vulnerable to reduced pay, exploi
  • ChatGPT provides an instant, non-judgemental answer. SO is not always the most welcoming forum
  • by sethmeisterg ( 603174 ) on Thursday May 11, 2023 @08:13PM (#63515265)
    amirite?
  • by bradley13 ( 1118935 ) on Friday May 12, 2023 @06:24AM (#63515947) Homepage

    When StackOverflow was young, people were enthusuastic about giving good answers and the answers were current.

    Now, your question is likely to get marked as a dup, even when it's not. Because the answers on the site are often out-of-date: the solution from 8 years ago no longer applies. All the old answers generally make finding a current answer difficult - they clutter up the search results.

Friction is a drag.

Working...