Netflix Says Python Programming Language is Behind Every Film You Stream (zdnet.com) 202
The next time you're streaming on Netflix, you can thank popular programming language Python and the developers who use it for much of the experience. From a report: According to Python developers at Netflix, the language is used through the "full content lifecycle", from security tools, to its recommendation algorithms, and its proprietary content distribution network (CDN) Open Connect, which ensures that content is streamed from network devices that are as close as possible to end users. Ahead of the Python Software Foundation's PyCon conference next week in Cleveland, the streaming giant has been detailing how it uses the open-source language.
now we know why its so darn slow to start... (Score:5, Funny)
Python is a childs language... all non compiled languages are for children.
Re:now we know why its so darn slow to start... (Score:5, Informative)
Python is a childs language... all non compiled languages are for children.
More precisely, Pythos is a (relatively) user friendly language that provides bindings into a whole slew of code that was written in compiled languages. In that sense it stands on the shoulders of that (mostly compiled C/C++) code and the people who wrote it. It always cracks me up when people talk about something like: "the Python implementation of OpenCV". OpenCV is written in C/C++, Python just provided bindings for you to use OpenCV libraries written in a compiled language. Real code that is performance critical is written in compiled languages.
Re: (Score:3, Interesting)
Which is 99% of all Python programmers.
Re: (Score:3)
Python is the a pretty good tool for gluing actual code together.
Re: (Score:3)
Python is the a pretty good tool for gluing actual code together.
Right, that's how I think of Python, as glue. I don't hate it or anything, I use Python a lot and prefer it to Bash scripting for example where I spend too much time figuring out how to get simple stuff done due to the syntax of BASH scripts which is nothing like anything else I regularly use. Python is just faster for that sort of stuff. However, I'm not going to write something like OpenCV in pure python.
Re: (Score:2)
I use bash and I wish I didn't.
If someone would invent Python with { and } or begin and end or, well, something I'd be all over it. But I can't be arsed to count things I can't see.
C with a sprinkling of popen() is looking rather tempting.
Re: (Score:2)
If the indentation is what annoys you about it, then that's good because indentation is one of the least important things. If you've got an editor that randomly interchanges hard tabs with spaces then either reconfigure that editor or get a better one. If you stick to all spaces or all tabs then you don't need to count anything. If you really need the braces for a deep seated need then try Ruby.
Re: (Score:2)
Huh? It's the main thing, and it's shite. Indentation is for communicating with humans. Actual visible things are for communicating with the compiler. By making one thing do two jobs is does at least one of them poorly.
By the way, there isn't universal agreement among humans about how to indent, so making it a syntactic feature is already pissing people off who prefer the other way.
Re: (Score:2)
Any indentation works in python, just be consistent. One space, 8 spaces, tabs, etc. It doesn't tell you how to indent, just that you need to indent. The original design was that this would look like pseudo-code, as written by hand on a white board perhaps. It's not even the first language that did this. If a programmer is already using consistent indentation in other languages then the transition ot python is straight foward (you can even cheat by using #{ and #} but you still need to indent correctly)
Re: (Score:2)
If I want an if statement, then I have to indent everything governed by it by the same amount. Then for the first line not governed by it, go back to the same indentation as before.
In C-likes I can put the opening brace of the body on the same line. I can put it on the next line. I can put the whole statement on one line if I want. I probably wouldn't, usually, but in bash I sometimes do this for debug code because it's easy to comment/uncomment it.
It doesn't tell
Re: (Score:2)
It's called Perl.
Re: (Score:3)
Exactly. There's a lot I like about Python as a scripting or "glue" language, although I find that its use of white space for flow control is a common source of bugs. My C++ code gets carefully formatted, but even when you try and carefully format your Python code it's pretty easy to make an error that interprets, but executes in a way you didn't intend.
Re: (Score:3)
If you're writing code, fine, but not if you're maintaining it. This has nothing to do with tabs; refactoring Python code is an extremely bug-prone process.
And to anticipate a common objection, "just use my preferred fancy IDE which you have no muscle memory for and doesn't work over an SSH connection" is not a solution.
Re: (Score:2)
If you're scripting in a server or desktop environment (not counting weird web stuff like javascript), then you have only a few practical choices that have reasonable maturity or portability. Perl, python, and ruby. Perl is nice but difficult to read and has an atypical syntax with lots of shortcuts. Python and ruby remain as solid programming languages with mostly similar characteristics. If you know one you can learn the other in short order. (Bash is almost in the list but it really requires external u
Re: (Score:2)
How many C Developers are going to rewrite OpenCV or OpenCL.... They are going to use the already made library as well, and bind to them.
The real critical code is done in Assembly and then linked to machine code, especially when you need to access particular parts of the hardware, or need to do a calculation with minimum amount of waste. Now you are confusing C/C++ which is a lower level language with other compiled languages. We had BASIC compilers, Pascal Compilers, FORTRAN compilers... Where the langua
Re: (Score:1)
You heard AC! You're a dumbass!
Re: (Score:1)
It's always funny when some random dumbass at work says he prefers to use C++ over Python for opencv stuff because "python is too slow".
Most of the donkey work is being done in the library layer Python binds to but you can get performance gains if you are doing a lot of heavy algorithmic processing in pure Python instead of farming it out to the libraries in the sub-Python layer. Also, with something like Boost at your fingertips development time is not that much slower in C++.
Re: (Score:2)
You had me right up until you said "boost".
Modern C++ doesn't need that garbage.
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Re: now we know why its so darn slow to start... (Score:5, Informative)
But you got your programming up and running months in advance. For the amount of money saved in development time, you could use that money to buy a computer that is 10x faster. Or just the fact that most computers are already fast enough for most processes and that extra overhead isn't really noticed.
Now C/C++ speed improvements are often more academic then actual. We as human coders, are often not doing our best work all the time. Often your C code because you are tired will be sub optimal. Doing a quick and dirty bubble sort, vs. doing a list.sort() python function. Or just doing a copy of your data vs passing by reference... Things a higher level language for its prebuilt functions handles more optimally. So the language speed will be 10x faster. But your algorithms will be a Big O performance level higher.
Re: (Score:2)
Often your C code because you are tired will be sub optimal. Doing a quick and dirty bubble sort, vs. doing a list.sort() python function.
You're being disingenuous. Why can't I simply use the sort libraries in C like the way you use the python sort library?
And if you're not talking about sort(), just about any library function you have in python will have an equivalent in C or C++.
Re: now we know why its so darn slow to start... (Score:2)
Why not just write it in Golang? Then you get shorter development time AND good performance.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Real code that is performance critical is written in assembly.
Assembly is to C as C is to Python.
Don't think you are hot-shit because you run middle of the pack.
Not true.
Assembly language allows almost no refactoring or changes to data structures. Those things are where real performance tuning happens.
FWIW: Neither does C. C++ is where the magic starts to happen when it comes to code&algorithm massaging.
Re: now we know why its so darn slow to start... (Score:2)
Re: (Score:2)
I find this [quora.com] to be a very good commentary on the question.
Re: (Score:3)
Real code that is performance critical is written in assembly.
Funny, I thought it's written in Fortran or CUDA.
Matrix support also key (Score:2)
I agree, but would add that it's the not even just those two but also the addition of NumPy masking working with matrixes in Python better than almost any other language...
Re:now we know why its so darn slow to start... (Score:4, Funny)
meanwhile, Blockbuster is almost done with compiling their streaming service. Can't wait, must be so awesome when that runs on compiled code! Finally a streaming service for adults!!
ehm wait, that came out wrong....
Master Foo and the Ten Thousand Lines (Score:5, Interesting)
Master Foo once said to a visiting programmer: “There is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”
The programmer, who was very proud of his mastery of C, said: “How can this be? C is the language in which the very kernel of Unix is implemented!”
Master Foo replied: “That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.”
The programmer grew distressed. “But through the C language we experience the enlightenment of the Patriarch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!”
Master Foo replied: “All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.”
The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: “Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?”
The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so.
“And how many hours would you require to implement and debug that C program?” asked Nubi.
“Many,” admitted the visiting programmer. “But only a fool would spend the time to do that when so many more worthy tasks await him.”
“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”
Upon hearing this, the programmer was enlightened.
Re: (Score:1)
Whoever wrote that crap isn't a programmer.
Re:Master Foo and the Ten Thousand Lines (Score:5, Interesting)
Your path to enlightenment is long...
Re: Master Foo and the Ten Thousand Lines (Score:1)
I mean, if they were all created to provide a layer of abstraction to programmer, python -and shell code- do it better than C. And that's certainly the reason why python, and shell scripts, are mostly use to orchestrate the usage of other programs written in other languages.
It's all about the right tool in the right context. That's the only wisdom there is.
Re: (Score:1)
SQL, Prolog, functional languages, graphs, are descriptive instead of imperative, meaning the code being an end itself as well as a blueprint for processing and I/O.
Imperative languages are mostly glorified machine code / ASM translators, interpreters and VMs, though may also act as a platform like Smalltalk and Java JVM, so could implement higher-order concepts and designs.
Higher-order design and generalizations are less used practically, as you get removed from physical model and implementation, meaning s
Re:Master Foo and the Ten Thousand Lines (Score:5, Interesting)
For the INF problem I ultimately wrote a short and simple C program called "read_inf_section" that would output individual INF sections, greatly decreasing what got sent to grep. For the database scanning problem (it's just a pile of colon-separated fields, one unique per-INF device entry per line) I planned out a C program that would fetch and parse the appropriate lines without requiring a bunch of text processing pipelines since most of the script was slowed by orders of magnitude due to hundreds of thousands of forks to run stuff like grep. However, I quickly stopped bothering as I recalled that Bash has a built-in regex engine and Bash was already a hard requirement for the script due to heavy Bash array usage. I looked up the info on Bash regex usage and while it took some time to get used to its peculiarities (couldn't just swap in my grep regexes) I managed to replace the extremely "hot" text processing pipelines that were killing performance with a no-fork solution that worked within the shell script. The C program was going to take a lot more time to write then the regex conversions took and the speed benefit relative to Bash regex would probably not have justified the effort.
Upon doing this, I was enlightened.
Re: (Score:2)
Re:now we know why its so darn slow to start... (Score:5, Insightful)
"It is a toy language if it does not create a native code executable like C", is the most passionate argument I hear from children in programming, typically self-taught teens who have never coded in the real world or those who never learnt a decent high-level language.
Paid, professional programmers worry about getting the assigned job done, in a cost-effective way - that generally involves choosing the most high-level tool you can get away with, so that you don't end up solving problems you don't need to; not optimizing needlessly early with native code.
Remember: most code is bottle necked by construction time, not machine time.
Re: (Score:2, Interesting)
So this is why Enterprice code is so shitty: Huge footprint, slow startup, no warmup, high latencies and requiring new hardware purchases every few years for less functionality? Java's excellent exception handling and traceability being an excuse for not doing proper error handling, pushing fragile components to production?
The best tell is where no viable alternatives are being used. Give someone a hammer and all that.. Don't get me started on JS! ;)
Re: (Score:2, Insightful)
To me that same argument that high level programming languages are "smarter because you save time" means you should never code anything, you should just wait for someone else to code it.
The reality is, the best programs and computer functions are as fast as possible, given the relative speed needs. Fast is not just good, fast is a measure of how good something is. If you could write a program that beat all humans at chess all the time, only it took 1 year to make each move, it would be a terrible program.
Wh
Re: (Score:2)
Re: (Score:2)
> means you should never code anything
It does not. It simply means - Don't keep reinventing the wheel. Else, your final solution will be slower for it, even if you use a much faster tool.
It is perfectly possible for a C program developed in x hours to be slower than a Python program in the same x hours. High-level code makes it easier to identify low hanging optimizations since you are less wrapped up with unnecessary lower level concerns.
> you should just wait for someone else to code it.
No, but do t
Re: (Score:2)
I'd rather wait another 2 years for a program that is great than get it immediately and shave off a year of my life due to inefficiencies; gratuitous exaggeration intended.
... because you lose more money by not having the solution versus having a slow solution.
Unfortunately solutions are often needed ASAP
I once replaced a bunch of interacting Excel Sheets based on Access that partly were only front ends for Oracle with a Java Application. Before optimization the Java program (that was around 2003, Java 1.4
Re: (Score:2)
Re: (Score:2)
As a professional programmer, I am starting to see the fine edge of using a language with static typing versus dynamic typing.
Using Assembly vs C++ vs Python is less relevant as long as there is good tooling
Static typing does allow for much easier analysis and refactoring which is critical for building large complex software system.
Unit tests help but the tooling for them is inconsistent. (Code coverage metrics are important but not there yet...)
If Python does gain a form of static typing like TypeScript, t
Re: (Score:2)
It must be nice to work in an industry where performance is unimportant. For many of us in the business, run time is money.
In a cloud environment, more efficient code means smaller, fewer, and cheaper instances. On big data, more efficient code is the difference between a feasible solution and an infeasible solution. On mobile and embedded devices, more efficient code means better battery life and a more responsive user experience. And on anything to be deployed on machines you don't own, more efficient co
Re: (Score:2)
Sure. Whenever there is an explicit case with clear cost, by all means, you pick something more appropriate.
Python is rare on mobile devices. Go is perhaps popular for micro-services. Whatever works.
My point was about *pre-mature* optimization, not to avoid optimization in every case.
I would have preferred a language like Scala or Nim, if it wasn't so darn easy to find a library (often libraries that are high-performant) for everything in Python.
Purely as languages, either is even more high-level than Pytho
Re:now we know why its so darn slow to start... (Score:5, Informative)
It is called an Interpreted Language.
When I was younger I use to do everything in compiled languages, the lower level the better. But as my career continued. I rarely made software designed to be put in a Box and shipped to someone to install on their PC where we need the easy to click .EXE file to run the program, but software that is running on a server, with a complex set of configurations and security rules, in an environment where the software will always need to be changed, and tweaked, also many components will run fine for much longer then my time at the company, or source code retention and versioning policies stay in place. They were many times where I needed to reprogram a perfectly functioning program someone made 2 decades ago, just because they hard coded the File path locations, and the file locations have changed servers, because it was compiled into an executable. and the source code was long gone. Across multiple CTO who change policies from a lot of development to none then to when appropriate. Having the software as interpreted means the source code is always available, and makes those fixes on forgotten software possible. (Ok once I fixed an exe with a Hex editor, but I was lucky enough the change took the same amount of bytes.)
NASA used LISP for many of their satellites and space probes, because it allows the code to be changed while the program is still running.
Then you finally get to cross platform development, It is easier to port your interpreter to a different platform, then converting and recompiling all your programs.
In the 1980's Mainframes (Unix, VMS, PrimeOS... ) were the big thing, 1990's The rise of the Desktop Personal PC's (MS DOS/MS Windows), 2000's PC Based Servers (Windows Server/Linux), 2010's Cloud services (who really knows anymore, usually Linux) There is a lot of code that you don't need to rewrite, if you can get it ported to the new environment. That python program I made in 1999 to run on a Solaris Unix box will still run in 2019 on a Windows Server Box. The program of that level of complexity would have a 50/50 chance of recompiling in C, do to the differences in system architecture between Solaris for Ultra Sparc, and Windows for x86.
Re: So, perfect for you then? (Score:2)
And yet you use literally wrongly
Re: (Score:2)
No. Your just an arse. The above is grammatically correct you nincompoop.
Re: (Score:2)
His what?
Re: now we know why its so darn slow to start... (Score:1)
It is compiled to bytecode.
Re: (Score:2)
Then why do I have a Python.exe file for Windows. and if I download Python from source I need to use GCC to compile it?
Re: (Score:3)
a quick google search for python bytecode brings up:
https://opensource.com/article... [opensource.com]
Python is often described as an interpreted language—one in which your source code is translated into native CPU instructions as the program runs—but this is only partially correct. Python, like many interpreted languages, actually compiles source code to a set of instructions for a virtual machine, and the Python interpreter is an implementation of that virtual machine. This intermediate format is called "bytecode."
Python.exe will check the source .py file and see if it changed and if so, initiate a comple to a .pyc (or .pycache in python3). If you run the same program over and over the compile is only done once and subsequent invocations of the Python VM go straight to the compiled bytecode.
Many modern languages (java, etc.) now compile to bytecode and use a byte code interpreter. If you really need the bytecode compied to native
Re: (Score:3)
Then why do I have a Python.exe file for Windows. and if I download Python from source I need to use GCC to compile it?
For the same reason that you have a java.exe file for Windows, and for the same reason thar you don't need GCC to compile Java programs.
Re: (Score:2)
Re: (Score:2)
PYC = Pretty Young Child?
Re: now we know why its so darn slow to start... (Score:5, Funny)
Yes ... Yes it is. What does the .pyc extension mean?
That it's compiled to Russian?
(Disclaimer: familiarity with the Cyrillic alphabet is required)
Re: (Score:2)
Re: (Score:2)
Compiled, not compiled, whatever... real programmers burn every algorithm into an ASIC.
Wait! That's what I do.
Re: (Score:2)
Re: (Score:2)
Yes ... Yes it is. What does the .pyc extension mean? Oh that's right. Look who I'm asking! Some clueless AC.
By that logic, my bash code is "compiled" by me, using vim, into a .sh file!
(Quick, what's the difference between using a runtime to run .pyc files and a runtime to run .sh files?)
Behind Python is C (Score:2)
Re: Behind Python is C (Score:1)
Re:Behind Python is C (Score:5, Insightful)
Re: Behind Python is C (Score:4, Informative)
Re: (Score:2)
Behind Python is C C is found in the Linux kernel, most of userspace, high-performance network code, the Python implementation itself.
And behind C is C++ because that's what all the compilers are written in these days.
Re: (Score:3)
You clearly haven't examined a process try while GCC or clang is running. The assembler is still very much there, invoked automatically by the compiler wrapper (as are the preprocessor and the compiler proper).
Thanks? (Score:5, Insightful)
you can thank popular programming language Python
Why? Do we think if python hadn't been around or not up ti the job, netflix would have shrugged and said can't be done then?
Re: (Score:2)
More importantly (Score:4, Interesting)
It's also probably behind the actual content creation process.
Most of the VFX/Animation industry software uses python as the scripting language as a standard (Except for Adobe which has its own shitty outdated implementation of Javascript) and by extension the internally developed production pipelines also generally use Python.
Great (Score:1)
Asphyxiation (Score:5, Funny)
So one could say that Python is helping to strangle romance? [slashdot.org]
Nonsense, it's all C/C++ (Score:5, Informative)
What he describes is a few minor, light-weight tasks like the "recommendation algorithms" and "Open Connect, which ensures that content is streamed from network devices that are as close as possible." All the actual work is being done in C/C++, so while the trivial code to determine the closest network element might be written in Python, all the code that does the streaming, from the firmware on the network equipment to the software on the servers, will be written in C/C++. While the trivial recommendation algorithm might be written in Python, all the code involved in encoding and decoding the video will be written in C/C++.
To say Python is "behind every film you stream" is nonsense. C/C++ is behind everting and Python is only being used for minor tasks on the side.
Re:Nonsense, it's all C/C++ (Score:4, Interesting)
Re: (Score:3)
All the actual work is being done in C/C++
Actually, I believe a lot of it is in Java.
https://go.java/netflix.html [go.java]
https://netflix.github.io/ [github.io]
I spot checked a bunch of their OSS projects on the github.io page above and they were all written in Java (or had large Java based components).
Re:Nonsense, it's all C/C++ (Score:5, Informative)
While what you say isn't false, the amount of code that gets written in these kind of companies is quite vast, and most of it isn't very performance sensitive (ie. they'll just spin up an extra thousand boxes to cover the amount of requests, but the actual request doesn't have to be finished in a couple of ms). It makes sense from a productivity and security point of view to write these in python instead of C - especially as they include lots of network requests and string processing.
The point is, for any given user, you are probably touching vastly more Python code than C code, even if by your definition the C code is doing more important stuff.
About importance, from the Netflix point of view, the business logic and websites that are implemented in Python aren't quite as unimportant as you think they are though.
employee turnover (Score:2)
Re: (Score:1)
How can you tell somebody is a python programmer? (Score:5, Funny)
You don't have to - they'll tell you.
Re: (Score:2)
2 or 3?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
No self-respecting C or C++ programmer would ever call themselves a "C/C++" programmer.
Re: (Score:2)
they'll tell you
Haltingly. With lots of empty space between sentences.
Re: (Score:2)
You don't have to - they'll tell you.
How can you tell somebody is not a Python programmer...
Can we see where this is going?
Re: (Score:2)
Quick! (Score:1)
Quick! If Netflix are doing it, so should we. Start porting all that Java to Python right away!
Wake me up when... (Score:1)
...somebody writes a major piece of software (operating system, file system, database, etc.) in Python and it isn't half as fast as the competition.
Buggy and Slow (Score:1)
I've been considering closing my netflix account because it's buggy and slow so this probably isn't a glowing recommendation for python.
explains a lot.. (Score:2)
Re: (Score:2)
Re: (Score:2)
OSS CDN Programming Languages (Score:2)
Re: (Score:1, Insightful)
Perhaps a similar reason as to why PHP became so popular. Ugly syntax, lots of nonsense, and yet people are prepared to defend it to the death. Why? Well, it's remarkably easy to seem to get small stuff done in it. A quick page that munches forms, that sort of thing. Big stuff gets annoying in a hurry. Fun fact: "A fractal of bad design" is written by someone who "looooves python", though isn't blind for its failings. Might ask them why.
PHP was, frankly, written by incompetent idiots, and appeals to that de
Re: So? (Score:2)
In Soviet Russia, PHP uses you!
Re: (Score:2, Funny)
"I use PHP. I've spent 20 years learning it..."
What a senseless waste of human life.
Re:So? (Score:4, Insightful)
Some people have 20 years of experience. Others have one year of experience, 20 times.