Researchers Release 'PolyCoder', the First Open-Source Code-Generating AI Model (venturebeat.com) 66
"Code generation AI — AI systems that can write in different programming languages given a prompt — promise to cut development costs while allowing coders to focus on creative, less repetitive tasks," writes VentureBeat.
"But while research labs like OpenAI and Alphabet-backed DeepMind have developed powerful code-generating AI, many of the most capable systems aren't available in open source." For example, the training data for OpenAI's Codex, which powers GitHub's Copilot feature, hasn't been made publicly available, preventing researchers from fine-tuning the AI model or studying aspects of it such as interpretability.
To remedy this, researchers at Carnegie Mellon University — Frank Xu, Uri Alon, Graham Neubig, and Vincent Hellendoorn — developed PolyCoder, a model based on OpenAI's GPT-2 language model that was trained on a database of 249 gigabytes of code across 12 programming languages. While PolyCoder doesn't match the performance of top code generators in every task, the researchers claim that PolyCoder is able to write in C with greater accuracy than all known models, including Codex....
"Large tech companies aren't publicly releasing their models, which is really holding back scientific research and democratization of such large language models of code," the researchers said. "To some extent, we hope that our open-sourcing efforts will convince others to do the same. But the bigger picture is that the community should be able to train these models themselves. Our model pushed the limit of what you can train on a single server — anything bigger requires a cluster of servers, which dramatically increases the cost."
"But while research labs like OpenAI and Alphabet-backed DeepMind have developed powerful code-generating AI, many of the most capable systems aren't available in open source." For example, the training data for OpenAI's Codex, which powers GitHub's Copilot feature, hasn't been made publicly available, preventing researchers from fine-tuning the AI model or studying aspects of it such as interpretability.
To remedy this, researchers at Carnegie Mellon University — Frank Xu, Uri Alon, Graham Neubig, and Vincent Hellendoorn — developed PolyCoder, a model based on OpenAI's GPT-2 language model that was trained on a database of 249 gigabytes of code across 12 programming languages. While PolyCoder doesn't match the performance of top code generators in every task, the researchers claim that PolyCoder is able to write in C with greater accuracy than all known models, including Codex....
"Large tech companies aren't publicly releasing their models, which is really holding back scientific research and democratization of such large language models of code," the researchers said. "To some extent, we hope that our open-sourcing efforts will convince others to do the same. But the bigger picture is that the community should be able to train these models themselves. Our model pushed the limit of what you can train on a single server — anything bigger requires a cluster of servers, which dramatically increases the cost."
Where have all the LERTs gone? (Score:2)
Low effort robo-trolls, that is.
Surely they couldn't all have been Russians?
Re: (Score:2)
Low effort robo-trolls, that is.
Surely they couldn't all have been Russians?
Most, I'll bet. And just like that, the internet becomes a lot better.
Can it guess requirements? (Score:2)
Already, for most of us, nailing down the requirements is already harder than actually writing the code.
Re: (Score:3)
Re: (Score:2)
There is nothing "agile" about skipping requirements. Many agile skeptics assume that agile ignores requirements gathering. This is not true. In reality, agile changes the method requirements are communicated ("stories" vs. formal requirements documents) and decreases the scope of any given update (incremental development). These are valid concepts for many use cases. Skipping requirements is just laziness, it's not actually "agile."
Re: (Score:2)
Re: (Score:2)
I have yet to see a software project type that is better handled with a waterfall-style process. Inside every "successful" waterfall project, is an agile methodology struggling to get out. The fact is, nobody can hold all of the requirements of a large software project in their heads (or in a document) at once. You can't compile a giant requirements document to ensure that all references are resolved and all variables accounted for. This is why software must, and always is, built in an incremental fashion.
Re: (Score:2)
When the requirements (for some combination of business/functional/technical requirements) are known in advance for any given portion of a project, then design up front is the most cost effective m
Re: (Score:2)
In your concrete example, if the interface is so trivial that it can be designed all at once, then do so. This does not conflict with agile. It is an agile-sized increment.
If the interface is large and sprawling, then no, waterfall is NOT the best way to design the project.
Well-defined interfaces are not a special category of software development that lend themselves to waterfall. If the interface is large enough, it needs to be broken down into increments, just like any other kind of project.
If you're buil
Re: (Score:2)
I have yet to see a software project type that is better handled with a waterfall-style process
Satellite software that can't be updated is much better handled in a waterfall-style process. Basically, by the time you are writing code, the spec is so detailed that you are just transcribing spec to code, not really filling anything out.
Re: (Score:2)
I have yet to see a software project type that is better handled with a waterfall-style process
Satellite software that can't be updated is much better handled in a waterfall-style process. Basically, by the time you are writing code, the spec is so detailed that you are just transcribing spec to code, not really filling anything out.
Even space stuff has transferred over to agile, aka. "faster, better, cheaper". Turns out you'll usually get less science done with a satellite that takes 20 years to develop, and a metric crapload of money, that has 0.1% chance of failing - than with 10 satellites you could develop using the same resources who have 10% chance of failure each.
Re: (Score:2)
But...satellite software CAN be updated while in flight. Even Voyager I has been updated in flight numerous times.
Re: (Score:2)
Not if a bug causes the satellite to self-destruct.
Re: (Score:2)
LOL you got me there! Of course, in that scenario, waterfall won't help you either.
Re: (Score:1)
Since we're citing anecdotal evidence, I'll add mine. The most successful project I ever worked on was done via waterfall. The requirements were already in place when I began, and barely moved during development. One bug found during acceptance testing, and no bugs ever reported from the field. Development proceeded steadily with no bumps. All waterfall. It can work, and it can work quite well. Due to the
Re: (Score:2)
Yes, waterfall can be made to work, just like a Rube Goldberg machine can be made to work. Just because it can be made to work, doesn't make it the best tool for the job.
Re: (Score:1)
Well, it was no effort to "make" it work. It just worked very smoothly. No detours, nothing. Having said that, I repeat that I probably will never use it again. Agile or anything like it would have pulled me out of the groove I was working in, at least daily. No good. For this project, I would argue that waterfall was the best choice. As additional detail, I will mention that the project involved no communication with the end customer, and when it was delivered, they accepted it as-is. So the comple
Re: (Score:2)
I have yet to see a software project type that is better handled with a waterfall-style process.
Software for Space Shuttle.
Re: (Score:2)
Yes, software on spacecraft is commonly cited as a type of software best solved by a waterfall process. Why? I suppose because it's seen as something that can't be done incrementally because once it's up in the air, it can't be changed. Or perhaps because it is so critical for the software to be thoroughly tested and to work perfectly.
Neither of these goals are better addressed through waterfall. First, it's incorrect that software on spacecraft or satellites can't be updated while in flight. So that argume
Re: Can it guess requirements? (Score:2)
Re: (Score:2)
indeed, but agile naturally tolerates requirements being late because you can make do (e.g. "produce some value", is the mantra) along the way. this allows management not only to postpone requirements but also shift the responsibility down all the way to the "monkeys". needless to say, this is abused in most agile environments, maybe because most shops were never interested in "actually do agile" to begin with, just in the liturgy and in exploiting these "features".
e.g. pure "actual agile" has no deadlines
Re: (Score:2)
Have you ever seen a waterfall project that had requirements completed on time? No, agile doesn't tolerate late requirements, people do. Late requirements are not a feature of agile in any way that is different from waterfall. Show me any agile methodology that says otherwise, good luck! Scrum and Kanban both need requirements to be completed before programming can start. They just call the requirement document a "story" and they keep the size small.
You hit the nail on the head when you said that many shops
Re: (Score:2)
Project planning and estimation is important (when done well) for business planning. When these things are ignored in favor of extreme focus on incremental delivery, there is an information gap.
Effective people use their skills and experience to know
Re: (Score:2)
Have you ever seen a waterfall project that had requirements completed on time?
yes. proportionally as many as agile projects doing actual agile. these are the exceptions :-)
No, agile doesn't tolerate late requirements, people do.
yes it does. in being incremental agile naturally allows for any requirement that is not immediately necessary to be ignored for the time being. this usually means covering the current sprint or part of the next one at most. waterfall does not allow this leniency.
this is because agile is exploratory, and was "invented" precisely for projects with uncertainties, where the precise shape of the product was unclear for
Re: (Score:2)
waterfall is actually the most efficient and straightforward approach
I've heard lots of reasons why people like waterfall, but I've never heard it described as "efficient" before. Waterfall is the opposite of efficient, because it forces the project to completely finish one phase before starting the next (hence, the name waterfall), even when multiple phases could otherwise be done in parallel. It also forces design bugs to be fixed after delivery, when they are the most expensive to fix.
agile is the way to go if you are unsure about what to do and how to get there, but want do go there anyway
I would argue that there is no other kind of non-trivial project.
Re: (Score:2)
I've heard lots of reasons why people like waterfall, but I've never heard it described as "efficient" before. Waterfall is the opposite of efficient, because it forces the project to completely finish one phase before starting the next (hence, the name waterfall), even when multiple phases could otherwise be done in parallel.
i do get your point, but "time efficient" is not the same as "resource efficient" and parallelism usually adds problems of its own.
It also forces design bugs to be fixed after delivery, when they are the most expensive to fix.
"design bug" is something invented in the agile era, where you are constantly redesigning (often without seeing the big picture). this simply can't happen in waterfall unless project management is catastrophically bad, that is when literally anything can happen. this is the whole point of the "phases", and that's why there exists tools like prototypes or proofs of concept: you d
Re: (Score:2)
indeed, but agile naturally tolerates requirements being late because you can make do (e.g. "produce some value", is the mantra) along the way. this allows management not only to postpone requirements but also shift the responsibility down all the way to the "monkeys".
They will do that anyway.
i'm not really surprised that most software is ill-designed crap nowadays
As opposed to when?
Re: (Score:2)
There is nothing "agile" about skipping requirements.
You don't skip requirements, but you don't drill down to specifics like in waterfall. It is better to get some working code and show the customer the options rather than asking them what they want.
"stories" vs. formal requirements documents
My experience is that "stories" work way better as a communication tool. Formal requirements are better for managers wanting to shift blame.
Re: (Score:2)
In agile, you do drill down into specifics, the difference is when you drill down to specifics. We recognize that the user doesn't have the whole picture in their head at once, so we postpone the conversation about the specifics until it is needed.
It's like building a high-rise office building. You don't know what the floor plan of the 10th floor will be, until a tenant leases the space and tells you what they want. So you build the structure in such a way that it can accommodate those varying floor plans.
I
Re: (Score:2)
There is nothing "agile" about skipping requirements. Many agile skeptics assume that agile ignores requirements gathering.
I think a lot of Agile proponents also forget that descoping is also an agile technique used for improving code quality and that is where some of the criticism comes from.
Re: (Score:1)
Re: (Score:2)
To be sure, there are some for whom agile is a religion. This generally happens when they don't truly understand agile, but follow it's practices blindly because that's what the book says. For others, it's a tool that can be leveraged to efficiently deliver quality software, and it does so far more effectively than waterfall methodologies. The fact that agile is a religion for some, doesn't make it a religion.
Re: (Score:2)
>Today's programmers just start coding,
I do silicon design more than programming, but the principle is the same. Start coding on the core problem, the thing that is the roadblock or the enabling algorithm. Once you have those things working you know the rest is possible.
This is not a random act of coding, it's directing your energies directly at the core problems, solving them and getting things jump started. You do have to know what you're doing and why you're doing it.
Re: (Score:2)
Iterative/incremental is good for gaining knowledge when solution is not known because it's very difficult to fully understand the landscape by just thinking about it (too many variables). Design up front is good for keeping costs down when solution is known by eliminating rework associated with incremental metho
Re: (Score:2)
By "just start coding" you can explore the problem, figure out the gotchas. It will help you plan out the real implementation.
Re: (Score:2)
>By "just start coding" you can explore the problem, figure out the gotchas. It will help you plan out the real implementation.
Most times that just becomes the real implementation. There is no time to start over and develop a "nice" algorithm after the fact.
You clearly don't have a clue. (Score:2)
We don't need no stinking requirements. Today's programmers just start coding, and change things according to the client's whim. It's all thanks to Facebook's "move fast and break things" mantra, coupled with "agile" development methodologies which values flying by the seat of your pants over deliberate planning and actual engineering.
The moniker Facebook came up with was either cooked up by marketing bods or taken out of context, the reality is far different.
"agile" development has absolutely zero to do with "flying by the seat of your pants", rather, it's a methodology that attempts to distil the understanding that large tasks require an iterative, collaborative approach to succeed.
It's been part of coding for decades, but unfortunately, over those decades, many businesses took the "waterfall" approach because that's how the bean count
Re: (Score:2)
After a while writing software commercially you start to realize that you need to plan for massive changes to the product spec late in the day. It's just a fact of life. Early prototypes help, but only so much.
Re: (Score:2)
Already, for most of us, nailing down the requirements is already harder than actually writing the code.
Fair enough, but can a Senior Python Developer guess requirements? Check and mate.
Re: (Score:2)
LOL You're not wrong. But the senior python developer is the wrong person to nail down requirements. That's the job of a BA or a SME or a "product owner."
Re: (Score:2)
Already, for most of us, nailing down the requirements is already harder than actually writing the code.
Nailing down requirements exactly enough to generate code is fundamentally harder than writing the code. Code is the most compact and precise specification of itself. Anything else takes more, unless you have actual general intelligence with experience and insight in the loop that can supply the missing part because it does understand the world and how things are generally done. Machines cannot do that.
That is the reason why this approach has been failing for the last 50 years or so. It makes zero sense.
A little raw (Score:2)
From the GitHub repository for this project: https://github.com/VHellendoor... [github.com]
This model was not trained to solve programming problems and may not perform well on a benchmark such as HumanEval. Models like Codex (powering Copilot) are pretrained on natural language, which may boost their ability to interpret NL prompts; this model only learned language from comments in code.
The model appears to start generating a random new file once it reaches the (predicted) end of the current one. It is possible that the end-of-document token was not properly added to the training data.
Whitespace is very important to the model, since no preprocessing was done on the input files. For instance, the following snippet will yield poor predictions, because in Java we would never expect an instance-method at the top-level, as is indicated by the single level of (\t) indentation of the two lines within this method...
So the model was trained on natural language, but whitespace is "very important"? How do those two things not conflict? The training didn't factor in the end of a document?
These seem like really big caveats to me. Maybe there is promise for the future, but for now it seems like more of an interesting toy.
Re: (Score:2)
From the GitHub repository for this project: https://github.com/VHellendoor... [github.com]
This model was not trained to solve programming problems and may not perform well on a benchmark such as HumanEval. Models like Codex (powering Copilot) are pretrained on natural language, which may boost their ability to interpret NL prompts; this model only learned language from comments in code.
The model appears to start generating a random new file once it reaches the (predicted) end of the current one. It is possible that the end-of-document token was not properly added to the training data.
Whitespace is very important to the model, since no preprocessing was done on the input files. For instance, the following snippet will yield poor predictions, because in Java we would never expect an instance-method at the top-level, as is indicated by the single level of (\t) indentation of the two lines within this method...
So the model was trained on natural language, but whitespace is "very important"? How do those two things not conflict? The training didn't factor in the end of a document?
These seem like really big caveats to me. Maybe there is promise for the future, but for now it seems like more of an interesting toy.
read again, the quote says exactly the opposite of what you imply. it also states clearly why whitespace is so important: because they didn't preprocess the input (as in properly "tokenize") a bad decision imo but i guess they had their reasons (which i ignore, btw i haven't read tfa).
no idea if this is just a "toy". i have no doubt that ai could generate reasonable code one day, but as a lifelong programmer myself i'm very skeptic about "code generating sheme" in general, for the most part because the "co
Re: (Score:2)
I don't quite follow how the quote says the opposite of what I implied. Failing to properly account for whitespace variations and failing to tokenize input are key elements of being "ready" for prime time. Nearly every "traditional" programming language on the planet does these two things very well.
Re: (Score:2)
So the model was trained on natural language,...
However, the portion you quoted said:
This model was not trained to solve programming problems and may not perform well on a benchmark such as HumanEval. Models like Codex (powering Copilot) are pretrained on natural language, which may boost their ability to interpret NL prompts; this model only learned language from comments in code.
This quote implies that unlike Codex, their model was not trained on natural language.
Re: (Score:2)
this model only learned language from comments in code.
In general, code comments are composed using "natural language."
Re: (Score:2)
Re: (Score:2)
I'm pretty sure none of this stuff will actually replace the need for human coders in the foreseeable future. Specialists will still be needed to describe the algorithms needed to solve most problems, which is really what programming is about anyhow. And no matter how smart the AI, there are always going to be limitations, for which it will have no idea how to solve a novel problems that a human will need to work out.
I just think of it as better tools, not a replacement for programmers. Perhaps some simp
Sounds like a cut and past machine (Score:2)
Maintaining existing code (Score:2)
So once you've generated code using this so-called AI system, what happens if you make modifications to the code it generates? Can the model be re-run to produce an enhanced version, and incorporate your modifications? I doubt it.
So insist that all code always be generated through the AI? Not likely. This would only work if the AI was able to generate production-ready code in every circumstance, and that it can cover all use-cases. Also, very unlikely.
It's great that these guys are doing this research, but
Re: (Score:2)
coding by AI assistant requires a mental shift as big as the mental shift
Re: (Score:2)
It's great that these guys are doing this research, but like AI itself, it seems at least a decade away from being usable for anything besides academic research.
Try more like 50 years away to get somewhere realistic. Attempts like these have been failing time and again for the last 50 years and there is nothing fundamentally new available in the AI field. All Artificial Ignorance can do today is being dumb faster and in broader areas.
The problem is that you need to specify the code exactly enough that a non-intelligent thing can generate it. You know, like a compiler would do. This means the spec is either already the code or it is _harder_ to produce than the code
I can see where this could go. (Score:3)
Re: (Score:2)
Any time a coder is re-performing a repetitive task, (s)he is failing to automate that task... which is to say, (s)he is missing the whole point of computer programming, which is to automate repetitive tasks so that a computer can do them quickly instead of forcing a human to do them slowly.
Or to put it another way, inside every tedious chore is a more interesting higher-level task-automation problem waiting to be solved.
Re: (Score:2)
Re: (Score:2)
Fantastic! (Score:2)
This sounds like this will be perfect for generating large amounts of overly abstracted code with endless layers of inefficiency and bugs everywhere! See, you thought Microsoft couldn't hire crappier coders than those noobs in India but you were wrong because now those noobs will be generating huge amounts of code. Windows 12 is going to be so lit (like a stick of dynamite)!
Tried making Hello World, it made me Doom3 (Score:2)
This is awesome!
I mean, not even Doom 1 that runs on everything from a Raspberry Pi to a coffeemaker, but it just skipped right to Doom 3.
Re: (Score:2)
Why GPT-2? (Score:2)
Why did they use GPT-2 when ElutherAI has a GPT-3 equivalent model already open and out there? Weird.
All useless (Score:2)
How do you check this actually does what it is supposed to? If programmers are not smart enough anymore to write code, then good luck verifying it doesn't contain bugs.
This is about as useful as having an AI write tests for you. Great, all tests pass, it does exactly what you wrote, including crashing on bad input and all the other bugs.
The only way this could possibly work is in a test driven approach, where tests are written first (which will become your specification) and then have AI write the implemen
Gang of Four, Design Patterns Book (Score:2)
It would be interesting to train the model using the code and text in the book.
Re: (Score:2)
Since the model cannot understand the text, it would be a complete failure.
Re: (Score:2)
Since the model cannot understand the text, it would be a complete failure.
Fair call. I read the OpenAI article and it suggested that GPT3 was trained by the text. I could not see how PolyCoder was trained, though it looks fascinating.
Perhaps if you trained it with Fowler it would tell you "you're doing it wrong".
What then? (Score:2)
Garbage IN, Garbage OUT (Score:2)
We've been through this before. These code generating models are trained on public repositories with a highly variable (and questionable) quality of code. They should only be trained on a properly tested, reviewed, fixed and signed off code. Otherwise it's garbage in, garbage out.