Startup Uses AI To Create Programs From Simple Screenshots (siliconangle.com) 89
An anonymous reader shares an article: A new neural network being built by a Danish startup called UIzard Technologies IVS has created an application that can transform raw designs of graphical user interfaces into actual source code that can be used to build them. Company founder Tony Beltramelli has just published a research paper that reveals how it has achieved that. It uses cutting-edge machine learning technologies to create a neural network that can generate code automatically when it's fed with screenshots of a GUI. The Pix2Code model actually outperforms many human coders because it can create code for three separate platforms, including Android, iOS and "web-based technologies," whereas many programmers are only able to do so for one platform. Pix2Code can create GUIs from screenshots with an accuracy of 77 percent, but that will improve as the algorithm learns more, the founder said.
no need for AI (Score:3)
Re:no need for AI (Score:5, Interesting)
Programs that can auto-generate glue code from a GUI input have been around for decades. You just need to fill in the stubs with the other 99% of the code. The problem with these GUI input systems is that the code they generate is fragile and even small changes are often more difficult than just starting over. I have never found them useful, but they are popular for iPhone and Android apps.
I read TFA, and it says almost nothing, but I think the new thing about this system is that you don't need to use a GUI input, and instead you can just show it a picture or screenshot of an existing GUI (say, your competitor's product), and it will auto-generate code to create that GUI, with stubs for the actual functionality. That seems pretty slick.
Re: (Score:2)
In the ajava world and iOS badically every GUI code generator generates the exact same code a programmer would write. .Net though, the WinForms generator was a mess.
Last time I checked
The article btw. is not about ordinary GUI editing/code generation but about generating code from a scetch, aka a pocture.
Re: (Score:2)
The term AI is getting stretched very far these days.
Could be worse. We could go back to 'cyber'.
Re: (Score:2)
Re: (Score:3)
Who will save us from the cyber AI revolution?
Idea for an Arnie flick...
Re: (Score:3)
Re: (Score:2)
To be maximally annoying, insist on using 'blockchain' as an uncountable noun.
Re: no need for AI (Score:2)
Yes. Baysian algorithms are classic examples of machine learning, in fact it's one of the first things they teach, and spam whacking was one of the first mainstream uses of it
Re: (Score:2)
Re:no need for AI (Score:4, Interesting)
The output from a layout editor is a structured description of the components and their layout. This is inferring that description from a .png - the LSTM is building a description of the structural relationship between the widgets from the input image.
It looks pretty cool, although quite simple. The intermediate token stream that it is inferring may be more interesting as a design tool than the neural network on the front-end that is building it.
Only apps can app apps! (Score:1)
This just shows how easy it is to app an app that apps other apps, unlike LUDDITE software!
Apps!
I RTFA (Score:5, Informative)
It only generates the layout files for the different platforms.
Re: (Score:3, Insightful)
Re:I RTFA (Score:5, Interesting)
Yep, everybody did win. Back in 1990, when Rapid Application Development (RAD in the hype of the day) tools did this.
Your IDE still has this feature. Drag and drop UI drawing is better than having any UI inferred from a drawing. How do you draw a mask?
Trying to do this well, transparently for multiple different devices plus 'browser' is challenging, to say the least. But...GOOD NEWS...each of these markets is big enough to support a UI team of it's own. Claiming to do it well, automatically from a 'napkin sketch', for all significant platforms is braggadocious to the point where adults start to whisper about where the person's keeper is, calling him 'Sheldon'.
But we all remember being 22 and doing similar; 'that's easy, just...' The time it takes from 'that's easy' to 'uhh....shit' is what separates success from failure, long term.
The best, this will do is produce a 'wrong' (control behavior from a drawing?) UI for a 'sketch artist' who hasn't bothered to learn to use his IDE. Somebody still has to come along, muddling through the messes (one per target), and fix it.
Re: (Score:2)
Re: (Score:2)
Yeah, that would be great if this AI could actually create these stuff from ANY picture. BUT it can only generate stuff from pictures that have been made with predefined tools....... and that's where this is just all bullocks, as most mockup tools can already generate code for any given platform.
This article seems more like a plug by the developers themselves as at the moment it doesn't add anything to the already existing platforms.. Also not to forget, this can only generate to frameworks it knows, so the
Re: (Score:3)
That "tedium" of coding the ui is usually the easiest part of app development, for me at least.
Does the neural net also recognize dynamic UIs with swipe, pinch and twirl responses?
Neat academic project... but 77% accuracy just to do the layouts? I've got interns better than that.
Re:I RTFA (Score:4, Interesting)
Agreed. We built a nearly identical system (with OpenCV for morphological analysis and neural networks) about a year ago, as part of a larger AI-powered mobile app development platform.
77% accuracy is not very impressive, but unclear what the training and test sets are here.
The biggest functional win from this is actually getting sensible layout params from a designer's UI mockup - i.e. figuring out should this be right justified/left justified, should there be margin/padding here, etc. We solved that problem pretty well.
Other challenges involve asset up-scaling, background image color extraction, etc. If you can take rough image mockups and output well polished asset packs, with vectorized images, layout files and stub code for developers to work with, that's a pretty significant win.
We got that far with the project, but ended up shifting direction to a somewhat different market where there was more growth potential - literally nobody wanted to invest further in mobile app tools in 2016, AI powered or not.
So yeah, cool proof-of-concept, but as a standalone offering this doesn't create much value. As part of a larger toolchain might be valuable.
Re: (Score:2)
77% accuracy just to do the layouts?
This is version 1.0. It will rapidly improve.
I've got interns better than that.
Interns have to be paid. Software works for (almost) nothing.
You remind me of this guy:
"The Americans have need of the telephone, but we do not. We have plenty of messenger boys." -- William Henry Preece, explaining why telephones would never be used in Britain.
Re: (Score:2)
Re: (Score:2)
Since HTML5+CSS3 was shown [github.com] to be turing complete.
More here [my-codeworks.com].
Re: (Score:2)
Re: (Score:2)
The first link looks like a mistake, it has nothing to do with HTML/CSS.
Then again the question: why do you post nonsense like this?
What real life problem would you implement on top of a (simulated) turing machine?
Can YOU (not one, *you*) implement the Leipnitz Formula to calculate PI with the Turing machine simulator of your first link?
Throwing around 'turing complete' on /. is completely meaningless.
Re: (Score:2)
The first link looks like a mistake, it has nothing to do with HTML/CSS.
Then again the question: why do you post nonsense like this?
What real life problem would you implement on top of a (simulated) turing machine?
Can YOU (not one, *you*) implement the Leipnitz Formula to calculate PI with the Turing machine simulator of your first link?
Throwing around 'turing complete' on /. is completely meaningless.
The first post is definately not a mistake. It's the repo of code by the guy who figured out that it is turing complete.
In terms of what real life problems... well given that it is turing complete it could potentially be leveraged as a malware vector. That said, please refrain from putting words in my mouth: I was not making any claim as to usefulness, I was just responding to someone trying to be a smart arse.
Also, since when has throwing around _anything_ on slashdot not been completely meaningless?
Re: (Score:2)
I was just responding to someone trying to be a smart arse.
Then accept my apologizes!
Re: (Score:2)
I was just responding to someone trying to be a smart arse.
Then accept my apologizes!
Accepted!
Not good (Score:2)
Re: (Score:2)
Also bearing in mind the thing you are generating is a layout and it's being generated from a screenshot of that exact thing.
What might have a bit of value is generating real layouts from UI mockups.
Re: (Score:2)
1. Create the layout
2. create a screanshot of it
3. have the program generate the layout.
Seems to me that steps 2 and 3 are redundant except for photoshop farts who think that a .psd file is all you need to "define your requirements".
Re: (Score:2)
Re: Not good (Score:1)
When our firm hired Indian coders, the flowerbeds in front of tree building suddenly became much more lush.
Re: (Score:2)
How many programmers have you met that could be given a UI screenshot, say "rightio" and go off & implement it on three platforms - without asking a single question?
Maybe when the algorithm is also as experienced as a programmer that could do that, we can expect better.
Re: (Score:3)
The ones that could were/are _terrible_coders_. UI configuration is more complicated than a screenshot can capture. If they complete it from a screenshot without asking questions, fire their incompetent asses. They missed a lot, guaranteed, assumed even more...
UI Mockup is a small part of systems analysis. Comes relatively late in the process. Mostly the UI 'falls out' of the data structure or data/business layers, depending on how you 'squint' at the process.
But that's all old stuff, doesn't involve A
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Basically every programmer I have ever met.
Why would I have a question if I have a drawn mock up of a screen I should program?
And why would there be questions because I do it once for iOS and then for Android?
RAD Redux (Score:4, Interesting)
We've had RAD systems for decades. They make the first 80% easy, but not the last 20%. One is always dealing with things like legacy databases with goofy schemas and domain-specific intricacies.
Tools that may take longer to lay down the basics but can be tuned easier for specifies still seem the best bet.
Plus you have issues of mobile devices such that UI's need to be "responsive" to different screen sizes. These can take a lot of experimentation to get right because context is involved. They are solving 1990's problems.
It just accepts an image instead of a drawing? (Score:4, Interesting)
Was the first version of ResEdit released in 1984 or 1985? In any case, for more than thirty years, there have been developer tools that allowed you to draw a UI screen, while simultaneously creating a WYSIWYG screen image, an object-oriented description of the elements in the image (e.g. "a checkbox at 50,100"), and code to generate the image.
As nearly as I can tell, the only novelty here is the ability to work off a static image file, rather than being able to work off the time-sequence of the series of drawing manipulations used to draw the file. This wouldn't be a big deal even if it worked, since it doesn't take very long for a human to look at a UI screen and draw a duplicate layout using a UI layout tool.
As for "77% accuracy," I have no idea what that means or how you calculate the percentage, but sounds like "it doesn't work," because the amount of work needed to correct something that is only 77% accurate is probably about the same--quite possibly more--than the amount of work needed to create it from scratch with a good layout tool.
Furthermore, it is very common for a UI layout to contain elements that are only conditionally visible. An obvious one would be a tabbed panel. A screenshot can show you the control that are in the frontmost tab page, but has no information at all that would allow pix2code to even begin to guess at the controls and other elements that are present in the other tab fields. Therefore, to get even a complete visual record of the interface, it is necessary to have some kind of procedure or script that results in every UI element being systematically revealed. That's not trivial. (Imagine some of the currently fashionable designs that save screen real estate by putting larger parts of the UI on invisible trays that only slide into view when needed).
C grade (Score:2)
An accuracy of 77% is consider a passing C grade (unfortunately) in school and is completely unacceptable in the real world business environment. It needs to be getting an A+ to be acceptable in the real world. Fire the sucker.
Should I even bother? (Score:1)
Sigh. Here we go: that isn't 'AI'. Can we please start calling it something more accurate and honest? Can millennials please step outside of their teenaged fantasies and join the rest of us here in reality? It would be so much easier. Double sigh.
Re: (Score:2)
Demonstrates flaw of alphabet based languages (Score:2)
Computer languages are almost all variations of English,(While, If, then, else, go, return, are all english words).
But human derived language, particularly alphabet based ones, s are not appropriate for coding.
Alphabet based languages are designed to represent an infinite set of words. Computer languages use a small set - often less than 100.
Re: (Score:2)
Um. Say what?
Any set of symbols is an "alphabet" in computer science parlance. Furthermore, although computer languages have a limited number of keywords, they also have variable names, etc.
Meh. I think parent is satire. Too subtle a WTF to be real.
Re: (Score:2)
Welcome to Simulink. We've been using it for a while to solve quite a few complex tasks.
Re: (Score:2)
I don't know what languages you're using, but every one I use allows you to define additional words to do other things. They're called functions, procedures, subroutines macros, whatever. I don't know any computer language that uses a small set of words. Even assembler allows you to use macros that you can call all sorts of names. And then there's labels for jumps in both low and high-level languages.
Since you used english as an example, there are far more valid words in programming languages than in the
Re: (Score:2)
Re: (Score:2)
Well, except that there were other languages before basic that didn't use line numbers. That moldy golden oldie, assembler, jumped to offsets (usually defined as a symbolic name), not line numbers. But if you want to call me a youngster, at my age I'll take it as a compliment :-p
BTW - you could use variables longer than 2 letters, just that only the first two were significant, so COUNT and COW were the same. As long as you kept that limitation in mind, it made for more readable code. And if you didn't, wel
Re: (Score:2)
Galaxy Quest (Score:2)
Really old news and bad job (Score:2)
What this guy did barely counts. It just identifies rectangles (poorly) and guesses what kind of rectangle (poorly) and spits out code.
It's a cute project for a high school kid.
Here's a program I'd like to see (Score:2)
Ghost of HyperCard (Score:3)
A reminder of the day when you didn't need a crack team of engineers to produce a simple educational program or hyperlinked ebook. Every teacher a programmer. Every student a programmer producing value added content, whether 5 or 55 years of age.
We talk about advancements in the industry but we've taken a giant step backward in terms of creative output. See 'Inigo Gets Out' https://www.youtube.com/watch?... [youtube.com]
Rounded skill-set (Score:1)
Challenge accepted - see you at the coffee machine - remember your cup!
This might be clever - but it's being over-sold. (Score:2)
As ANYONE who writes programs with GUI's will tell you - a system like this can AT BEST write 10% of the code needed in an actual "Program"...because the GUI layer isn't that tough to write - and there is no way on god's green earth that something that looks at screenshots can infer how the other 90% of the code has to work. Add to this that if it's even 99% accurate - how will a human programmer fix the remaining 1% of the bugs?
So - programmers everywhere are laughing at this claim...it's patently obvious
Why this announcement is harmful. (Score:3)
I see questions on Quora and similar places from kids who are thinking of taking up a career as computer programmers - on commonly asked one is "If I become a programmer, will AI make my career obsolete?" - and this is a very valid concern. If I were a truck driver, I'd be really worried that self-driving trucks would take my job 5 years from now.
This announcement (which effectively says to the layperson "Programmers are about to become obsolete") will have a chilling effect on those people who are just thinking about getting into this field.
In truth - this AI program will never see the light of day - it can NEVER "write a program from screenshots" because the necessary information to do that isn't present in the screenshots - even in principle. What HAPPENS when you push this button? All the screenshots tell you is that there is a button...and MAYBE...if the screenshot is somehow linked to other screenshots...it might tell you that pressing the button takes you to another panel. What it doesn't tell you is that pressing that button caused the camera to take a photo, for the software to reconstruct a 3D image of a person from that photo, that this has to be sent off to the server to match other 3D images, that the resulting match produces that person's name - which the program is then given from the server - and which then results in that "NAME" field on the next GUI panel to be populated with an actual name and not the "John Doe" that the GUI designer put there so the programmer would know that this is where the name goes.
By itself - this announcement can be laughed at and called bullshit by anyone who has anything to do with writing programs (and I'm 100% sure it's being laughed at right now) - but the CHILLING effect that such ridiculously over-stated claims make on those who might be considering entering the industry is a very, very bad thing.
Headline (Score:2)
Startup Uses Bullshit To Defraud Investors
Cool, but ... (Score:2)
Looks like a fun project, but ... the three platforms they mention have somewhat different UX guidelines; it's a lot 'deeper' problem to translate a gui into something idiomatically appropriate for the mentioned platforms.
Not to say this isn't a good, interesting start on that very thing.
useful for automating crime (Score:1)
This will be great for automating the process of generating fake landing pages for phishing attacks. Think phishing worms.