Deep Learning May Need a New Programming Language That's More Flexible Than Python, Facebook's Chief AI Scientist Says (venturebeat.com) 263
Deep learning may need a new programming language that's more flexible and easier to work with than Python, Facebook AI Research director Yann LeCun said today. From an interview: It's not yet clear if such a language is necessary, but the possibility runs against very entrenched desires from researchers and engineers, he said. LeCun has worked with neural networks since the 1980s. "There are several projects at Google, Facebook, and other places to kind of design such a compiled language that can be efficient for deep learning, but it's not clear at all that the community will follow, because people just want to use Python," LeCun said in a phone call with VentureBeat. "The question now is, is that a valid approach?" Further reading: Facebook joins Amazon and Google in AI chip race.
What we need is... (Score:3, Insightful)
What we need is less Facebook, less Google, less Amazon.
Re: What we need is... (Score:2)
Re: (Score:2)
But some people can:
https://www.gizmodo.com.au/201... [gizmodo.com.au]
https://www.vice.com/en_au/art... [vice.com]
Re: (Score:3)
Easy (Score:5, Funny)
Just use C++
Re:Easy (Score:4, Insightful)
True, but even C/C++ and Assembly doesn't provide an "easy" way to do threading, which is the issue.
Scripting languages, basically do not do threading, of any kind, at all. They're too slow to synchronize across threads, which makes invoking threads inside them fruitless.
While C is ultimately the right language to do everything in (not C++) , the real issue is that cpu's are expanding in cores, just like GPU's have, yet GPU's have standardized more or less on just three API's, OpenGL, Direct3D and Vulkan. So if you can write a program against Vulkan, you have as close to bare hardware as you are going to get. But for CPU's, there is still a 57 flavors of rubbish programming languages and no standard runtime that works for all of them, at best most of these programming languages are still developed in C or C++ and thus require a complete C AND C++ runtime to function.
Python is not written in Python. Java is not written in Java. If a language can not compile itself, it's not flexible enough to be used for any of the three main corner stones of software development: Operating Systems, Applications, and Games. While you certainly can write an application or game with a scripting language, it will be slow, it will be limited by the operating system's own libraries (eg 32-bit libraries on a 64-bit OS as just one example) and generally require more maintenance than simply writing it in C to begin with.
Memory overflow errors are caused by people learning programming languages like Java first instead of C, because if you learn C first, you then learn how to initialize memory, and how big memory chunks actually are.
Re:Easy (Score:5, Insightful)
True, but even C/C++ and Assembly doesn't provide an "easy" way to do threading, which is the issue.
Nothing is easy to multithread in because multithreading anything more than the most basic of processes is inherently complex. This doesn't apply to SIMD structures which is the main reason why GPU's can be so parallel. One pixel doesn't care what color the other one is.
While C is ultimately the right language to do everything in (not C++) , the real issue is that cpu's are expanding in cores, just like GPU's have, yet GPU's have standardized more or less on just three API's, OpenGL, Direct3D and Vulkan. So if you can write a program against Vulkan, you have as close to bare hardware as you are going to get. But for CPU's, there is still a 57 flavors of rubbish programming languages and no standard runtime that works for all of them, at best most of these programming languages are still developed in C or C++ and thus require a complete C AND C++ runtime to function.
A lot to disect here. So for starters APIs and languages aren't the same thing and the main graphics APIs can be interfaced with multiple languages, I've seen programs written in everything from Javascript, Python, C, and Java interface with the OpenGL api. Second, there is no such thing as a C/C++ runtime. Runtimes only exist in interpreted languages like JS with its DOM or in ones that compile to byte code for a nonexistent VM like Java. C/C++ compile to binary for a given architecture. That's the fundamental difference between compiled and interpreted languages.
Python is not written in Python.
Actually, some runtimes are.
If a language can not compile itself, it's not flexible enough to be used for any of the three main corner stones of software development: Operating Systems, Applications, and Games.
I don't know about flexibility, one of the biggest strengths of these dinky languages is their flexibility. Their biggest weakness however is a lack of efficiency and performance. If you had said they are unsuitable due to performance issues or an inability to run direct on the metal then I would agree with you. It's impossible to write an OS kernel that runs on a real machine by itself in javascipt or python. It cant be done because they both require JITs to work.
While you certainly can write an application or game with a scripting language, it will be slow,
Not a guarantee, but likely depending on complexity. There have been many successful games written in Java, minecraft for example.
it will be limited by the operating system's own libraries (eg 32-bit libraries on a 64-bit OS as just one example) and generally require more maintenance than simply writing it in C to begin with.
Eh, any program referencing external libraries has this problem. See issues with old C++ programs referencing deprecated Win32 APIs trying to run in Windows 10. However it is possible with some careful coding and luck to write a complex application that can work for decades unmodified in C. The same cannot be said for JS. If you write something complex in JS and dont touch it, three years later it wont work. This is especially true if you use some idiotic technology like NPM. Because storing your dependencies in the cloud is a great idea.
Memory overflow errors are caused by people learning programming languages like Java first instead of C, because if you learn C first, you then learn how to initialize memory, and how big memory chunks actually are.
While I do appreciate C, I think you give it too much credit. For one any serious Java programmer has to learn memory management eventually because the built in garbage collection is trash and once you get to a certain complexity level its no longer good enough. On top of that C's memory model is not an accurate representation of how a computer's memory model works anyways. I think this is one of C and C++'s biggest weaknesses and contributes to many of the mistakes programmers make in regards to memory management when using them.
Re:Easy (Score:5, Informative)
Second, there is no such thing as a C/C++ runtime.
Yes, that thing called crt0 that you've seen all your life is an illusion.
On a modern CPU, the C runtime doesn't have to do much. It has to set up the stack, argc/argv/envp, atexit handlers, and a few more random things. But it very much exists.
Also consider that C compilers are allowed to generate calls to memcpy if you assign a large struct or something, and many of them do.
Re: Easy (Score:2, Troll)
Re: (Score:3, Insightful)
To recap: The first comment mentioned the C/C++ runtime, i.e. the distributable implementation of the standard, providing things like malloc. The second comment misinterpreted this to mean a runtime environment, e.g. the java executable, or the .NET common language runtime (CLR). The third comment noted that the second comment was mistaken, and once again referred to the runtime. Then your comment again mistakes that for runtime environment
Conclusion: Words are hard.
Captcha: obvious.
Re: (Score:2)
Re: Easy (Score:4, Informative)
To a compiler writer (which is where I got my start) a compiler's runtime is any code that is needed to run a program but isn't generated by the compiler when an end user compiles their program.
C runtimes used to be a lot bigger than they are now. In the days of MS-DOS, you couldn't assume the presence of a FPU, so floating point was often compiled into calls into the FP runtime. Even today, microcontrollers often don't have instructions which directly implement basic C operations (e.g. 64 bit integer division) so these operations are typically compiled as calls to runtime routines.
As CPUs get more powerful, C runtimes get smaller. But to say there's no such thing is flatly untrue.
Re: (Score:2)
C is my primary language, and I gotta say, you seem pretty confused. Somebody pointed at the C runtime, and you start crying and claiming they don't understand C. That's just daft.
Run time, compile time. In C this is not complicated.
If you didn't have a runtime, how would you assign constants at compile time and have them exist in RAM at runtime? Simple, you wouldn't.
Re: (Score:3)
You clearly don't understand C, or what constititues a runtime environment.
Read the standard; there is a C runtime in any hosted implementation. The standard literally refers to a runtime, calling it the hosted implementation. The same is true for C++.
For freestanding implementations a smaller runtime is compiled in.
Re: (Score:3, Informative)
The following is the C runtime, crt0. It is 9 lines of assembler: .text .globl _start
_start: # _start is the entry point known to the linker
mov %rsp, %rbp # setup a new stack frame
mov 0(%rbp), %rdi # get argc from the stack
lea 8(%rbp), %rsi # get argv from the stack
call main # %rdi, %rsi are the first two args to main
mov %rax, %rdi # mov the return of
Re: Easy (Score:2)
Yes (Score:3)
Yep. The runtime starts main(), and returns the return value of main to the OS(). So basically it does START and END.
Everything in between is the responsibility of main(). It needs not interact with the OS at all, other than being started by the OS and telling the OS when it's done. Only for standalone programs, though - kernel modules don't need to do those two things.
There is a very useful kernel module which does nothing but allocate some memory. That's all it does. :)
It's used when you have a a few byt
Re: (Score:2)
All mammals have boobs.
Re: (Score:2)
Female = Flash Runtime
?
Re: (Score:2)
It's worth comparing this with the C runtime for a 32-bit CPU or a microcontroller. It wasn't so long ago that 64 bit division was a function call.
One more thing that I didn't cover is the C++ runtime, which is a little larger due to exception handling.
Re: Easy (Score:2)
Darn you!!! You beat me to it. So much posting space wasted by the others going offtopic... I thinking "Its a 15 line answer!!!" (So it was less than 15, but whatever).
Is this no longer covered in second or third tier CS classes?
Re: (Score:2)
Re: (Score:2)
Are you purposely forgeting about libc? malloc/free IS a runtime, for example.
I have coded in C without a runtime, writing bare-metal code for microcontrollers, but the code you run on your PC has a runtime.
Re: Easy (Score:5, Insightful)
Threading in ANSI C is pretty straightforward. There are inherent complexities in multithreaded code, but those can't be ignored by any language. In the context of what C code looks like, I think the threading interface is about as simple as you can get. Do you have any examples of a simpler model?
Re: (Score:2)
Threading in C is pretty straighforwards.
The problem is, it is up to the programmers, all of them, to be very careful. Threading in C is unsafe. You have to take every step carefully, or some part of the code will smash the toes of some other part.
That's fine for me, it sounds like it is fine for you. But it gets really hard to do something like deep learning that way without risking deadlocks.
That's why something like Go-lang is better for that sort of application; it is easier to avoid deadlocks while onl
Re:Easy (Score:5, Informative)
There is no easy way to do threading, by its very nature. It's not a language thing, it's a computer thing. If more than one thread is trying to access the same resource at the same time the headaches begin: who goes first, how do you get the threads to accept its place in the queue and "know" when the other is done, etc, etc, etc. Languages that support memory locking of course make things easier but you still have to think the program through very clearly and often you end up with rather unusual and not reproducible bugs.
However I posit that many people who use higher level languages have no actual idea of how a computer works or what they are actually doing, unlike us old timers who grew up with assembler.
Re: (Score:2, Insightful)
You are approaching the problem in the wrong way.
You capture the problem with this line "If more than one thread is trying to access the same resource at the same time the headaches begin".
By far the best solution is to avoid this issue in the first place. I.e. write in a functional style, prefer immutability over mutability, and don't share mutable state in concurrent code. Then you will find that reasoning about concurrent code is much much easier.
Of course, C doesn't really support this paradigm, which
Re: Easy (Score:2)
The constraints on deep learning AI aren't going to be solved with squeezing 10% better performance out of the software.
And you don't program AI. You educate an AI.
Re: (Score:3, Informative)
Scripting languages, basically do not do threading, of any kind, at all. They're too slow to synchronize across threads, which makes invoking threads inside them fruitless
Bullshit. You CAN do asynchronously loading of assets with JavaScript.
Asynchronous and multithreaded programming are two different things. Concurrency is NOT the same thing as parallelism.
Re: (Score:2)
After 2 years of R&D we finally string concatenation and parsing that isn't locking up the computer!
What were we doing again? Oh Deep Learning algorithm. I guess we now need to figure out how to hook the code up to a data source, and maybe we can find a way to interact the the GPU Cuda cores.
Yes, I am exaggerating. But C++ Doesn't solve that Python has with the Deep Learning Systems. Sometimes we need a non-general use programming language, that is optimized for the type of coding involved. These pro
Re: (Score:2)
Re: (Score:2)
"What language is the humans' cerebral cortex written in?"
Brainfuck [wikipedia.org], duh!
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Show me somebody who can multitask, and I'll show you somebody who can always sound like they did something.
Re: (Score:2)
Strictly speaking, if you're writing spaghetti code there is no reason to rely on a language. or other arbitrary rules.
Re: Easy (Score:4, Funny)
Ostensibly, yes. Honestly, most of it was hacked together with Perl.
Dumping Python Is Both And Easy And Lucrative (Score:5, Insightful)
Over the past few months I have spoken to a few recruiters I know who were asking me to give them all the senior C++ engineers I know or if I personally was interested.
In what?
Doing complete rewrites of giant mountains of garbage Python code written by twenty something year old hipsters or older researcher type people.
It is boring as fuck work but companies and organizations are desperate and willing to pay huge amounts of money to rid themselves of the clusterfuck that is Python.
Re:Dumping Python Is Both And Easy And Lucrative (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
If they were smart, they'd do it in ANSI C and just hire consultants.
It isn't that hard, but who wants to learn all of C++, or risk writing something in the "wrong" subset? Total PITA. Lots of people love C++, but they all love and hate different parts. It is the plague.
And why would career-oriented people want to get stuck doing it? It is a job for legacy code wranglers, not engineers.
But in most cases, they should be using Golang or Ruby or something. Python is the howto language, the BASIC of the modern
Re:Dumping Python Is Both And Easy And Lucrative (Score:5, Insightful)
If I have to choose, I'll deal with giant mountains of garbage Python over even a small hill of garbage C++
That is backwards.
Python is great. I use it all the time. For scripts. Small programs that are fast to write, easy to read, and avoid all the complexity of type checking and memory management. Python is also easy to learn. It is taught in elementary schools.
But for a 200,000+ line project written by a team, with coders coming and going during the project, Python is a very poor choice. "Quick and easy" doesn't scale. For big projects you need rigid type checking, complex data structures, fine tuned encapsulation, compile time error checking, static and dynamic analysis, verifiable memory allocation and release, etc. C++ has all of that, Python does not.
So you want to use C++ for the "giant mountains" of code. Python is for the "small hills".
Re: (Score:2)
For big projects you need rigid type checking, complex data structures, fine tuned encapsulation, compile time error checking, static and dynamic analysis, verifiable memory allocation and release, etc.
All the selling points for Java. But most big Java projects I saw were nightmares anyway.
Re: (Score:3)
For big projects you need rigid type checking, complex data structures, fine tuned encapsulation, compile time error checking, static and dynamic analysis, verifiable memory allocation and release, etc.
So you're arguing for a less flexible language. Yann LeCun suggests we might need a more "flexible and easy to work with" than Python.
Why he thinks that's the case is not touched on at all in this article, which makes it an unusually inane story even for slashdot.
Re: (Score:2)
There's still type checking. Just not _compile time_ type checking. You get such a wonderful feature as things blowing up in your face at runtime just because someone changed something somewhere and did not update all call sites.
Re: (Score:2)
But for a 200,000+ line project written by a team, with coders coming and going during the project, Python is a very poor choice. "Quick and easy" doesn't scale. For big projects you need rigid type checking, complex data structures, fine tuned encapsulation, compile time error checking, static and dynamic analysis, verifiable memory allocation and release, etc. C++ has all of that, Python does not.
C++ definitely lacks "rigid type checking" and "compile time error checking" in any practical sense, otherwise people would not be hunting (or exploiting) memory errors in C++ code all the time, since these features (actually present in languages like Ada or Modula or Oberon) would strictly prevent them. Whether Python lacks "complex data structures" is debatable; if you're referring to standard library deficiencies, you may be right, but nothing prevents you from implementing anything you want.
Re: (Score:2)
... and yet the most widely used web framework is an 800k line python program.
Re: (Score:2)
but probably you will have tools to enforce them.
That is a very probably firm endorsement.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Re: (Score:2)
Re:Easy (Score:5, Interesting)
Just use C++
Indeed, and the library you want in particular is called DLib.
http://dlib.net/ [dlib.net]
Specifically:
http://blog.dlib.net/2016/06/a... [dlib.net]
the networks are represented as templates. It's pretty cool and very high performance. Particularly impressive given the relative resources invested relative to Tensorflow and PyTorch/Caffe.
Re: (Score:3)
Yes, you could call it Smalltalk, or perhaps lisp (Score:3)
or perhaps prolog...
Re: (Score:2)
I have one character for you ")"
Unfortunately Procedural and Object Oriented languages, have gotten a foot hold, and training someone to code functionally is quite difficult, and for parallel processing it may be even trickier.
Re: (Score:2)
I have one character for you ")"
That's not a character, that's a SIMPLE-STRING of length 1. ;)
Re: (Score:2)
Functional is the easiest code for parallel processing.
Nah, C# with they yield style of automatically doing multi-threading is the easiest.
Why do you think it has been making such a come back?
It hasn't. Some of the nice functionality has been imported into Java, but hardly anyone is talking about immutability.
Re: (Score:3)
Nah, C# with they yield style of automatically doing multi-threading
I know next to nothing about C#, but I was under the impression that this was a concurrency mechanism - basically a control flow construct - not a parallelism feature. Did something change?
Re: (Score:2)
Re: Yes, you could call it Smalltalk, or perhaps l (Score:2)
Re: (Score:2)
you have never done any serious work in a functional language or any serious parallelism.
I've done both.
Re: (Score:2)
or ruby..
Re: (Score:3)
But as always, designing new languages is more fun than learning old ones, and learning from past mistakes makes you uncool.
hardware (Score:2)
Since all languages simply compile to machine code, which runs on an OS making hardware calls, a top level restriction implies a deep level constraint.
To develop the next -real- AI, requires a different hardware approach running some as of yet unconvinced OS - followed by a language.
There are only two options here... (Score:5, Funny)
... for building Skynet, and it'll be Lisp or perl.
And we all know which one the Lord used: https://xkcd.com/224/ [xkcd.com]
Julia anyone? (Score:5, Insightful)
Re:Julia anyone? (Score:5, Interesting)
Julia has a tremendous problem. It's not designed for users, it's designed for Julia designers. If they had said "let's create an environment for deep learning that is great for threading" everything would be fine. But instead they went with "Hey, let's do all those awesome and cool things that we always wanted to see in a programming language and also it should be great with deep learning and implicit threading". The result is a (possibly great) environment that takes far too long to learn, has way to many individual quirks and ways of doing things that differ from the standard approach, and just is a bitch to intuitively understand.
The environment may or may not be great, but the designers made sure that you couldn't just pick it up and go, you have to go "ahaaaa so that's how you do that" for every single thing. And when the option is using Python/R that you already understand or use Julia that you have to learn from scratch, the choice is easy, especially for the people that are scientists and not programmers at heart - which is the exact audience that Julia is targetting.
Re:Julia anyone? (Score:5, Informative)
Re:Julia anyone? (Score:5, Interesting)
You can write code similar to Matlab
MATLAB is a terrible example of an "easy to learn" language. It's full of shitty hacks like "putting a semicolon after an expression suppresses output; deleting the semicolon causes the expression to dump its output to console." It's loaded with arcane semantics like the differences between a normal array and a cell array. For fuck's sake, it permits semicolons in the middle of a parameter list, like this [mathworks.com].
MATLAB has the quintessential property of an overdesigned language, which is: if I leave it alone for a few months and come back to it, I have to re-learn the whole damn thing. The syntax, the library, and the UI are all unintuitive and illogical. I rely heavily on my cheatfile that I've built up from this iterative exercise to get back up to speed faster. I don't have to do that with Python or C.
or Numpy
Numpy is a terrible example of an "easy to learn" API. Numpy arrays use semantics that are logically similar to Python arrays - but of course they're nothing like each other, none of the libraries work the same way, etc. And And some core features are so poorly documented that you have to dig through sample code until you find something that vaguely resembles what you wanna do, and then shoehorn it into the shape you want. And if you want to stare into the abyss of insanity, try looking into using the Numpy array-sharing options to circumvent the cross-thread limitations of the Python global interpreter lock.
Don't get me wrong: both are powerful and fun when you're acclimated to them. My first Numpy experience was porting a super-simple image processing technique, and my amateur, first-attempt Numpy code finished 10,000 times faster, and that's not an exaggeration. But they're both crappy languages from a model-of-clarity-and-consistency perspective.
Re:Julia anyone? (Score:5, Informative)
Julia has a tremendous problem. It's not designed for users, it's designed for Julia designers.
I have to disagree with that. Julia is designed for users, but it knows that its use case is not Python's use case.
Julia was designed as an upgrade for Fortran programmers. Like all good upgrade languages, it learned from all of the languages which tried (and failed) to replace Fortran previously, like SISAL and Fortress.
There is a cohort of programmers for whom "the standard approach" means Python's highly idiosyncratic approach. In my (admittedly limited) experience, anyone who predates the rise of Python tends to have no problem picking up Julia.
Re: Julia anyone? (Score:2)
Re: (Score:3, Interesting)
Futhark looks cool: https://futhark-lang.org/ [futhark-lang.org] and promising in this realm.
"Futhark is a small programming language designed to be compiled to efficient parallel code. It is a statically typed, data-parallel, and purely functional array language in the ML family, and comes with a heavily optimising ahead-of-time compiler that presently generates GPU code via CUDA and OpenCL"
the ML family of languages being things like Standard-ML, Haskell, OcaML.
We already have one . . . . (Score:3)
Re: (Score:2)
it's called Common Lisp.
Coming soon from Facebook - HHCL.
Re: (Score:2)
So for something that needs the utmost speed. (Score:2)
Re: (Score:2)
There isn't really any such thing as an interpreted language anymore. There are Python compilers. There are even Python compilers that can be told to compile single functions or methods.
Deep learning code is absolutely compiled, but for the major libraries the models are specified in python.
Re: (Score:3)
People are using scripting languages to piece together subsystems written in low-level code.
This is the way that tasks like this have pretty much always been done. Your web browser isn't written in JavaScript, even though its performance may make it seem that way.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
LLVM has helped to blur the line.
Re: So for something that needs the utmost speed. (Score:2)
So.... (Score:3)
Yepp, absolutely (Score:2)
That's definitely what we need. A new programming language.
Re: (Score:2)
If the only other alternative is Python, then that is precisely what we need.
Re: (Score:2)
I know they say AI, but they definitely mean machine learning, which is really just advanced guessing (and can work well, but still)....
What's wrong with python? (Score:4, Interesting)
I mean, sure, a single data processing pipeline might have to use 6 different conda environments, each with different dependencies and python versions due to tool and libraries are often deprecated with even minor point changes to python versions...... oh yeah, all that and then you have to shoe-horn in tensorflow (or something else).
Comment removed (Score:3, Insightful)
Re: (Score:3)
Deep Learning is stuck again because there is no fundamental theory of learning. We have no idea how we learn.
Actually, we know how learning works just fine. They are called neural networks and our implementations of them work as expected.
In truth, what we don't understand is how to build neural networks that give rise to a general intelligence. My theory is that predefined (via evolution) generic brain structures that segment every animal brain is the key to general intelligence while hardwired instincts drive the advancement of general intelligence.
Neuroscience will probably discover that learning is a very biological trait, not one that can be copied or simulated in discrete mechanical systems.
That's a very silly argument because absolutely anything can be
Build it and they will come (Score:2)
we have no data, and hence don't know anything (Score:2, Troll)
I say this every other day in one form or another, but let me try again: we all keep jumping up and down and shouting about whether we need a more flexible language or a fussier language with super-strong type-checking and encapsulation and what not, and some are deeply convinced that Pythons's fascist attitude toward formatting is excellent, and some are not -- one of the reaons the Go project was started, from what I understand, was to get away from syntactic whitespace--
None of us really know who's rig
Python isn't the problem (Score:2)
It's much more that Python is fundamentally an imperative language, and deep learning doesn't fit into either the imperative or functional category, I really think DL deserves its own category, designed from the ground up for manipulating tensor data structures of unknown shapes.
I haven't come across a
If only ... (Score:2)
NIH? (Score:2)
Not Invented Here syndrome at work?
People want to keep using Python, it must be doing something right.
Re: (Score:3)
That's because he isn't sure yet.
Just because you are unhappy with their current tools and know through decades of experience that "more of the same" is not going to improve things, that doesn't mean you know what the future should look like.
Re: (Score:2, Funny)
I don't know, because I'm the exact opposite of Yann LeCun: he's an AI guy unsatisfied with programming languages, and I'm a programming language guy unsatisfied with AI.
I haven't worked in deep learning, so I don't know if he has well-formed complaints or he's just generally unsatisfied.
Re: (Score:2)
Re: (Score:2)