DeepMind Says Its New AI Coding Engine is as Good as an Average Human Programmer (theverge.com) 135
DeepMind has created an AI system named AlphaCode that it says "writes computer programs at a competitive level." From a report: The Alphabet subsidiary tested its system against coding challenges used in human competitions and found that its program achieved an "estimated rank" placing it within the top 54 percent of human coders. The result is a significant step forward for autonomous coding, says DeepMind, though AlphaCode's skills are not necessarily representative of the sort of programming tasks faced by the average coder. Oriol Vinyals, principal research scientist at DeepMind, told The Verge over email that the research was still in the early stages but that the results brought the company closer to creating a flexible problem-solving AI -- a program that can autonomously tackle coding challenges that are currently the domain of humans only. "In the longer-term, we're excited by [AlphaCode's] potential for helping programmers and non-programmers write code, improving productivity or creating new ways of making software," said Vinyals.
Comment removed (Score:5, Interesting)
Re:Ok (Score:4, Insightful)
Making a tripple-A game requires more than coding. You need an AI that can create the art assets, another one for music, another for sound, and another for story writing. Probably need another one for things like combat/stat balancing and suchlike. And that's just what I could think of off the top of my head, having barely any experience in the gaming industry myself.
But even so, this announcement makes me think we are still making rapid progress towards the ultimate goal of AI
Skynet! WOO!
(and OMG this "ascii art" filter is ridiculous. False positives everywhere! If only there were some way we could make computers think more like humans.)
Re: (Score:2)
Re: (Score:2)
Re:Ok[, please define Elder Scroll 6] (Score:2)
Really weak Subject, but your FP does manage to get to the heart of the problem. How do you tell any programmer what to program?
So imagine that we have an AI that can approach the problem the way a human would. I've never heard of "Elder Scroll", but from your vague instruction, I can still guess that there is probably some kind of RPG named Elder Scroll and there are five earlier versions. Plugging myself into the Internet, I could quickly find out all sorts of information about each version. Then I would
Re: (Score:3)
Re: Ok[, please define Elder Scroll 6] (Score:2)
So many developers love to claim "not my fault there is a bug/doesn't work, you didn't provide good enough specifications"
This experiment shows that with "good enough specifications" you can skip the developer. Those kinds of devs that hate talking to people to gather requirements are going to be out of a job eventually.
Re: (Score:3)
To me a program is just a specification that a computer can understand.
There are times when it is the clients fault that they didn't specify something, there are also times when its the developers fault for not using their common sense.
Ultimately English is imprecise and misunderstandings happen, who's fault it is a legal issue.
Re: (Score:2)
That was always the hardest part of software development, and I don't see it being automated away, at least not until they develop AI that can wrest sufficiently complete and accurate requirements and specifications from the human stakeholders.
Of course, a lot of those might get replaced by AI too. A lot of business workflows are unique, but for bad reasons, and AI might well find ways to simplify and work around those reasons, and offer typical businesses ways to automate process to a much greater degree
Re: (Score:2)
"How do you tell any programmer what to program" A design specification
And which is actually the difficult task requiring more intelligence?
In my experience that hard part of any project is getting an unambiguous and precise specification of what the stakeholders want/expect, especially when they aren't exactly sure themselves. After you have a complete specification, the actual writing of the code is the nearly trivial part.
Re:Ok[, please define Elder Scroll 6] (Score:5, Insightful)
By the time you've written a specification that's sufficiently detailed that the code can be automatically generated, the thing that does the generation isn't an AI -- it's just a compiler. You can't solve that without "strong AI" where the computer can derive the specs on its own.
Re: (Score:2)
"How do you tell any programmer what to program" A design specification
And which is actually the difficult task requiring more intelligence?
In my experience that hard part of any project is getting an unambiguous and precise specification of what the stakeholders want/expect, especially when they aren't exactly sure themselves. After you have a complete specification, the actual writing of the code is the nearly trivial part.
Anecdotal example of exactly this point: I was programming an email system for a commercial realtor. Smart guy. MBA and law degree, and even too smart to ever practice as a lawyer. I guess it was a "defensive" law degree?
So one day he calls me in and says he wants a tickler feature. We wound up talking about it for at least an hour or two until I figured out what he wanted. At that point, it was also quite obvious how to implement it. A copy of the email that would be delivered at a future date to "tickle"
Re:Ok (Score:5, Informative)
Ok deep mind, build me The Elder Scroll 6, complete with quests
Something like that wouldn't be too far off from today's AI capabilities. Think of what it would have to do to comply with that request.
1) Determine what "The Elder Scroll" is, which is something Google can already do. So an AI determining you want it to build the 6th installment of The Elder Scrolls video game series is within reason.
2) Determine what type of game it is, which the Google summary of the game already knows (Role-playing video game)
3) Build a general game engine for a role-playing video game, which is something I bet current software engineers could build/train an AI to do.
4) Generate a list of fantasy species and character classes for the protagonist and its enemies.
5) Generate a world map.
6) Create a large number of generic side quests based on databases of common quest themes found in games. I bet a college undergrad thesis could put together an AI to do this.
At this point you have a playable game, and humans could come and tweak it to put some unique storylines, creatures, etc into the game. A game development company may even want to hire more authors than game developers and artists to complete the game. I mentioned many difficult technical challenges, but nothing which seems impossible to me with today's AI technology.
Re: (Score:2)
Create a large number of generic side quests based on databases of common quest themes found in games. I bet a college undergrad thesis could put together an AI to do this.
The college graduate already did that; it's called Daggerfall. ;)
Re: (Score:2)
1) Determine what "The Elder Scroll" is, which is something Google can already do.
Trivialize much? Google searching for TES and showing you a blurb from the Wikipedia page is trivial, but that's not the same as Google determining what TES is. How's Google going to determine TES' timeline? It's only based on what it can find via crawling, some of which would be wrong because they're not from any source of authority.
Generating a map is probably the only part an AI could easily do and do well.
Re: (Score:2)
Google can't even search properly, sure if you have a one word thing you want to look up it might work, but if you search for something that is a bit unusual it will usually not give the results you want but ignore the words that don't suit.
Remove repetition, don't automate its creation (Score:3)
Many current stacks have a lot of repetitious busy-work work in them that seems something AI can help with. But, maybe a better use of research time would be to simplify our stacks so that we write meaningful domain-related code instead of martialing the same crap back and forth among the bloated layers.
It's like automating the clean-up of cat poop all over the house instead of training the cat to use the litter box.
Re: (Score:2)
Re: (Score:2)
I'm not following. While better programming languages may help, the real problem is screwed up stacks and our CRUD-unfriendly state-unfriendly web "standards".
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Problems (Score:2)
Here's an example problem [vox-cdn.com].
It seems it is looking at the "input" and "expected output" sections, and randomly generating (I say randomly, but it's not entirely random) results until it finds one that matches the output.
Re: (Score:3)
Re: (Score:3)
... with Stack Overflow being the "not entirely random" part.
Re: (Score:2)
Re: (Score:3)
I would have selected more this one, where you see its code: https://cdn.vox-cdn.com/upload... [vox-cdn.com] It's amazing that it works, but the result is low level imperative programming that is only "human" in the sense of a good high-schooler. A professional programmer for a project would have been in need for a more elaborate solution, with boilerplate code with classes, split in different files, memory checks if it's C family.
But it might be useful for things like "I need a script that sums the numbers in first two
Re: (Score:2)
Re: (Score:2)
Maybe next generation will not want to spend 30 seconds doing things in Excel when dictating an instruction to Clippy results in the same, with some script automatically generated that you don't even have to see is executed in the background.
Like today's children (and some adults) react like voice assistant are a normal thing to be used, and if I tell them "it's just 30 seconds search on a search engine and it's more accurate" it's not convincing to them.
Re: Problems (Score:2)
Great, now I can save 30 seconds of scripting time and only have to spend several weeks later tracking down why my data is corrupted (spoiler alert: Clippyâ(TM)s AI-generated script didnâ(TM)t quite do what I expected it to in all cases)
Re: (Score:2)
AlphaCode was tested on 10 of challenges that had been tackled by 5,000 users on the Codeforces site.
I'm curious how those 10 were chosen.
The particular example shown is fairly straightforward and there's not much optimization potential. How would it handle something that might need A* search or a custom hashing function?
Re: (Score:2)
I'm curious how those 10 were chosen.
They were based on date. Earliest data is training, next period is test, the period after that is validation, then they did the most recent competitions after the validation set.
The particular example shown is fairly straightforward and there's not much optimization potential. How would it handle something that might need A* search or a custom hashing function?
The example was likely chosen because the question and implementation are more easily understandable by non-programmers.
This competition they scored about top 20 percentile,
https://codeforces.com/contest... [codeforces.com]
This one they scored around 44th percentile
https://codeforces.com/contest... [codeforces.com]
Re: (Score:2)
Just looking at the first question [codeforces.com] and a few high-scoring answers [codeforces.com], it seems to me the competition is not particularly challenging.
The problem is actually O(1) as you can do some simple arithmetic to arrive at the answer (calculate for each x or y direction, then choose the minimum: if robot is moving towards the spot, subtract the positional differences, if it's moving away, also add twice the distance from the robot to the wall it's moving towards). However, all the successful answers I checked ran a simul
Re: (Score:2)
Just looking at the first question and a few high-scoring answers, it seems to me the competition is not particularly challenging.
The competitions have easy and hard problems, each problem gets an Elo rating, that problem has an Elo rating of 800 which means it is a very simple problem.
This problem has an Elo rating of 1100, (so fairly easy)
https://codeforces.com/contest... [codeforces.com]
so fairly eacy
This one is Elo 1600, so somewhat difficult
https://codeforces.com/contest... [codeforces.com]
This one is 2300, so difficult
https://codeforces.com/contest... [codeforces.com]
And this one is 2500, so extremely difficult
https://codeforces.com/contest... [codeforces.com]
Re: (Score:2)
To clarify, I don't mean the problems in the competition are easy necessarily. Rather, given such an easy problem, you would expect some contestants to come up with the non-naive O(1) solution.
I blame English for making the word "competition" mean both the event and the group of people you're competing against.
Re: (Score:2)
I'm curious how the problems got transformed into a "specc" the "AI" could comprehend.
Re: (Score:2)
I'm curious how the problems got transformed into a "specc" the "AI" could comprehend.
There wasn't any, it was the raw text. It uses natural language understanding to interpret the text description of the problem.
Re: (Score:2)
It seems it is looking at the "input" and "expected output" sections, and randomly generating (I say randomly, but it's not entirely random) results until it finds one that matches the output.
Considering 60% of the entrants into those coding competitions don't work at all, being better than 54% of the humans tells you nothing. The machine could be generating code which recognizes the input and hardcodes the output and be doing better than the bottom 30% in those competitions.
I told you so (Score:2)
Re:I told you so (Score:5, Insightful)
Re:I told you so (Score:5, Insightful)
The Requirements is the real killer. All the classes in Software engineering teaching UML and Object Orientation all seem to be around a myth that people actually know what they want upfront. Which undoubtedly creates the following conversation between me the Architect, and the Entry Level Developer.
Hey look at this tool that converts the database into Objects which I can code without having to use SQL. They will then show me this fancy library and how it works, and how much easier it is to do the operations for the current set of requirements. Then I have to reject the idea and they get all mad at me, sometimes going above my head to complain to the boss, where if I have a good Boss will normally suggest that I know what I am doing, if I have a bad boss, I end up in a meeting where I need to explain in detail why it is a bad idea, which then causes the whole project to get derailed because it is clear we don't have full requirements.
When I design something, I try to put in hooks when appropriate for fast and easy upgrades often without needing to do a recompile or a massive redeploy. Just change the View in the database to adapt to the changing spec. Oh look 5 minutes after go live, they didn't want it to be an easy to read text, but just the unique Id Number. No problem let me alter that view and replace a column, and boom the program is now working the way the customer wants it. Oh we need a new data element, no problem let me join a table and add a column and you got it.
No matter how much testing and involvement and people in the room, the software is going to need to be fixed and changed right after people have to use it. The specs are going to change, An entry level coder, will just change the code to change with the specs, more experience has them making sure it is designed to handle changeing specs, because they know over time what type of stuff will change more often and what can be hard coded.
For the love of god never ever believe the response when you ask the person, are you sure this will never change?
Re: (Score:2)
Re: (Score:2)
For the love of god never ever believe the response when you ask the person, are you sure this will never change?
The correct question to ask is: If you want to change this later, it'll cost you $3000, are you okay with that?
Re: (Score:2)
$3000 is not even 3 days programming (not counting rollout, acceptance test etc.). ...
It is hard to imagine a change that is so cheap
Re: (Score:3)
The Requirements is the real killer.
Right. If AI is "doing the programming" then it is 100% relying on accurate requirements. So you're just trading one programming language for another.
Re: (Score:2)
Along those lines, generally on those challenges it would take me less time to program an implementation than write the challenge description.
Some languages may require micromanagement and tedium that makes it harder, but a lot of languages are fairly to the point.
Re: (Score:2)
All the classes in Software engineering teaching UML and Object Orientation all seem to be around a myth that people actually know what they want upfront.
No, they are not.
They are exactly about the opposite.
Hey look at this tool that converts the database into Objects which I can code without having to use SQL. They will then show me this fancy library and how it works, and how much easier it is to do the operations for the current set of requirements.
That is an "OR-mapper" and has nothing to do with requi
Re: (Score:2)
The real problem is requirements, that is specifying precisely and unambiguously what a particular software program should do.
Then you already have written the code, and only need a suitable compiler.
Encoding the Problem? (Score:2)
Re: (Score:2)
Re: (Score:2)
So they claim they fed the challenges in using the exact same format as given to human entrants, but that seems kinda fishy. Unless there has been some huge leap in general purpose AI, they must be banking on some specific aspects of how these particular coding problems are worded/presented and then this really becomes just a case of 'map english to equation, then equation to code' problem, which is cool but something I would expect from academic researchers rather than someone trying to sell a product that does work.
They are using neural networks with transformers. The same approach used for GPT-3 (software that writes unique realistic text in response to a prompt); and for language translation; among other tasks.
It works predominantly off of the natural language of the problem.
No it isn't at all trivial. The level you are talking about is what was doable 20ish years ago, and would score about the 4th percentile in these competitions.
Will Still Need Programmers (Score:2)
* Do their requirements have conflicting requirements? Create tar
Not Setting the Bar Very High (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
Now imagine half of your coworkers are gone, replaced by few shiny boxes in a cabinet somewhere.
Scary or fun? Are you one of those who are gone?
You just described roughly 1980 through 2000 through the entire economy. While I'm not (quite) that old, I don't have to imagine. I've been one of those who has gone, one of those who has stayed, and one of those who built the shiny boxes in a cabinet.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I replaced a coworker once with a not very shiny box. It turns out copying files from one directory to another isn't that hard to automate.
Don't worry, she got a raise.
Re: (Score:2)
Everything done manually, especially if it is as boring as copying files: is error prone.
Sweet! (Score:2)
That bad, eh? (Score:2)
Pity.
Programming challenges are not a good measure... (Score:3)
As an enterprise developer, I don't care how many programming challenges you can pass.
As an enterprise developer, I care more that you are using correctly named variables, that you are following the enterprise coding style guide, that you are creating unit tests around your code to verify in CI that it is working correctly, that you have low coupling and high modularization, and that your software satisfies the complete set of functional and non-functional requirements.
Re: (Score:3)
Some of those issues are irrelevant if the AI is doing the programming:
using correctly named variables
If the code is not written for humans to read, then there's no need for good variable names. Based on their example [deepmind.com], the AI is currently using single-letter variable names. Though in this case I suspect humans would too given the problem statement.
following the enterprise coding style guide
Same as above. If no humans are reading, it doesn't matter (also this is already solved to a large extent by existing automatic code formatting tools).
creating unit tests
The AI in question starts with this as its
Re: (Score:2)
You can never assume the same programmer is going to be around forever to maintain the code. An enterprise developer should always assume someone else is picking up and maintaining the code in the future, and therefore code must follow enterprise development standards.
This doesn't matter if it is an AI or not, it just is what it is. Everyone is replaceable, including an AI, so the code must accommodate that.
Re: (Score:2)
This doesn't matter if it is an AI or not, it just is what it is. Everyone is replaceable, including an AI, so the code must accommodate that.
What if you only replace AIs with new AIs?
When's the last time you looked at assembly and said, "we need better compilers that can generate human-readable assembly code"?
Re: (Score:2)
As an enterprise developer, I care more that you are using correctly named variables, that you are following the enterprise coding style guide, that you are creating unit tests around your code to verify in CI that it is working correctly, that you have low coupling and high modularization, and that your software satisfies the complete set of functional and non-functional requirements.
The style it has in the competitions is because that is the style used by programmer in these competitions and it is simply imitating their style. If you train it on the style you want, it could do that instead.
Just like GPT-3 does writing style imitations - it can imitate whatever you want.
As I've said all along... (Score:3)
The Turing test will be passed by an AI not because AI's have gotten smarter, but because humans have gotten stupider.
Re: (Score:2)
The Turing test will be passed by an AI not because AI's have gotten smarter, but because humans have gotten stupider.
I do not think humans have become more stupid. I think we are finally beginning to see exactly how stupid the average person is.
Many people who meet me call me a genius and the smartest person they have ever met in their entire lives... and yet I think I am barely even average and I have many events in my life which assure me that I am not in fact a genius... and yet I keep finding people who are so fucking stupid, I have to wonder why they have never accidentally stopped breathing.
What counts as a correct solution? (Score:2)
If the requirement is just code that passes the (who knows how limited) unit tests this is not so impressive (or useful).
Re: (Score:2)
It looks like they are providing a description of the problem and a set of inputs and outputs (effectively some unit tests). Is the problem considered "solved" by the AI if the unit tests pass or does the code need to be proven correct? Is the code even comprehensible to a human?
If the requirement is just code that passes the (who knows how limited) unit tests this is not so impressive (or useful).
It has to pass a list of hidden unit tests with inputs that are intended to break incorrect and algorithmically inefficient implementations.
Interviews (Score:2)
So now that an AI can pass the predominant leetcode style coding interviews, will we continue to pretend they are indicative of your success at the company?
Documentation (Score:3)
Yeah but does it document its code?
How's it's commenting? Does it use human-friendly variable and procedure names? Does it use camelBack?
All these things must be answered...
Re: (Score:2)
Yeah but does it document its code?
I had the same thought. Imagine the technical debt associated with average code with no one who understands it and no documentation!
Programming challenges... (Score:3)
Seems like the area where AI would be well equipped: synthetic challenges with gob tons of code that already answers those challenges. It largely amounts to recognize the description of the problem and match it to a problem description for which you have seen the solution verbatim.
It's fancier than Watson's performance on Jeopardy, but along the same lines if you study enough of them you can make it a trivia problem instead of a 'programming' one.
Not with the typical crap specs users have (Score:4, Insightful)
Perhaps the first place to use AI is in migrating code bases. An AI might be able to review a code base in one language and generate a matching app, with appropriate test cases in a different language.
Solution in search of a problem. (Score:2)
Programmers have spent the past few decades building systems that the business asked for, but didn't really need. Finding an MBA who can figure out the difference between what the business actually needs and what they think they need is a bit more difficult than finding a competent programmer.
First assignement! (Score:2)
Of course - coding is translation (Score:2)
The majority of programmers in the world are "competent enough" in their domain to achieve a behavior out of a computer, in exchange for some fame or money. Ranking them is perhaps useful, but I would venture it's folly. The ecosystem of how machines should be configured to "behave" has split into a multitude of layers, disciplines, conventions and each has a competing set of gurus exclaiming How Things Should Be Done. There are layers of automation and abstraction on everything, all in the hopes of Ma
Algorithm is designed to... (Score:2)
So the AI will only emit a quarter cup of sweat or less in a code review meeting?
As good as the average programmer? (Score:2)
Below Average Pilot (Score:2)
Top 54% percent? (Score:2)
The Alphabet subsidiary tested its system against coding challenges used in human competitions and found that its program achieved an "estimated rank" placing it within the top 54 percent of human code
So, it copies everything from StackOverflow, but it makes sure it actually compiles...
Can it write the Fizz Buzz program? (Score:2)
AI vs. Competitive Coding (Score:2)
Re: (Score:2)
Re: (Score:2)
An AI which can solve 3D chess on massive grids, with many vertical levels, and dozens of pieces with distinct movement patterns and behaviors is working in a drastical
Re:That's quite a low bar (Score:5, Insightful)
Re: (Score:2)
Considering there are a high percentage of 'not complete' participants and "incomplete" entries, average is actually NOT PASSING
Would you want your open heart surgeon to be in 54% percentile of his class?
How about your self-driving car programmer?
Re: (Score:2)
I am guessing these tasks are very well defined, small tasks that can be judged, lending well to what a computer can do. Not an open ended task that can change and need clarification like a real world problem.
Re: (Score:3)
I bet if you were to poll all the programmers most will say they are above average, heck they would be thinking they are the best programmer in the world. And all the other programmers out there are just pure crap.
Re:That's quite a low bar (Score:5, Funny)
Except me, I'll admit I'm a terrible programmer, but I don't let that stop me :D
Re: (Score:2)
Re: (Score:2)
Let me guess. All of those "top coders" are open source?
Re:That's quite a low bar (Score:5, Insightful)
For competitive purposes; you get a relatively small, relatively tightly specified, problem with a bunch of constraints that force you to be clever about it by ensuring that the constraints eliminate the obvious or intuitive solutions.
Humans who are good at that are likely sharper programmers than the average programmer as a whole; but they are also enjoying the relative luxury of implementing a small, well-formed, spec under rules that actively encourage cute or elegant answers above all else.
Toss the bot into a "We have a meandering clusterfuck of an implementation process for the new ERP system that has less of a 'spec' and more of a pile of manifestos from various competing factions" situation and suddenly plodding-but-understandable implementations are vastly preferable to impressive but cryptic exercises in lateral thinking; and the hardest part of the job is figuring out what the hell the software you are supposed to be writing is actually supposed to be doing.
This bot clearly isn't totally hopeless at the natural language side; they tested it by giving it the puzzle rules in the same format provided to the other competitors; not some special input for a solver/optimization engine; but I suspect that competition coding is among the more favorable places for an expert system to try to compete. Presumably, this also means that bots could start working their way into real-world software toward the bottom of the design: anywhere that is at least slightly less laborious to build a spec for than to implement(or where the spec is required anyway, so there's no choice) will be something that you can potentially farm out to a bot; but anything higher level in the ugly process of turning a bunch of ill-formed user desires into a spec will probably be the last holdouts.
In some ways that probably shouldn't be a surprise: we don't call thing like compilers and the interpreters for various high level languages 'AI'; but they have a proven ability to take what is essentially a specialized spec-description notation and turn it into machine code; but it does run counter to the (usually vendor-driven) narrative that AI-driven programming will totally be about low-code/no-code natural language or 'visual' coding where the bot will manage to interpret the utterances of the Idea People directly, and eliminate the nerd peons; rather than being a tool that requires the ability to describe problems tightly and unambiguously; but can do some implementation for you if you do that.
Re: That's quite a low bar (Score:2)
Re: (Score:2, Flamebait)
The average programmer is not average at all. They're mostly terrible. The very top coders are responsible for most of the usable code in the world.
Came to say the same thing. I'm convinced that I could replace the "average programmer" with a perl script. A poorly written one, I might add.
Re: (Score:2)
Nice comment. Probably better than the actual FP. That includes your subtle Subject.
I looked through the replies and didn't find what I was looking for. Ergo:
If we can define any human skill clearly enough, then we can eventually build a computer system to do it better. Games such as chess and go are past-tense examples from the gaming world, and the FP focused on a more complicated computer-based game. Therefore our best line of defense for the continued relevance of homo sapiens is that programmers mostly
Re: That's quite a low bar (Score:2)
:) :) :)
Re: (Score:2)
...will just have to find something else to do, maybe learn to code.
Oh, wait.
Sun Tzu once wrote: Nothing we do in IT doesn't create more jobs for ourselves.
This won't be the exception.
Re: (Score:3)
How are the specs given/formalized?
test data https://github.com/deepmind/co... [github.com]