Why Jupyter is Data Scientists' Computational Notebook of Choice (nature.com) 58
Jeffrey M. Perkel, writing for Nature: Perched atop the Cerro Pachon ridge in the Chilean Andes is a building site that will eventually become the Large Synoptic Survey Telescope (LSST). When it comes online in 2022, the telescope will generate terabytes of data each night as it surveys the southern skies automatically. And to crunch those data, astronomers will use a familiar and increasingly popular tool: the Jupyter notebook. Jupyter is a free, open-source, interactive web tool known as a computational notebook, which researchers can use to combine software code, computational output, explanatory text and multimedia resources in a single document. Computational notebooks have been around for decades, but Jupyter in particular has exploded in popularity over the past couple of years. This rapid uptake has been aided by an enthusiastic community of user-developers and a redesigned architecture that allows the notebook to speak dozens of programming languages -- a fact reflected in its name, which was inspired, according to co-founder Fernando Perez, by the programming languages Julia (Ju), Python (Py) and R.
[...] For data scientists, Jupyter has emerged as a de facto standard, says Lorena Barba, a mechanical and aeronautical engineer at George Washington University in Washington DC. Mario Juric, an astronomer at the University of Washington in Seattle who coordinates the LSST's data-management team, says: "I've never seen any migration this fast. It's just amazing." Computational notebooks are essentially laboratory notebooks for scientific computing. Instead of pasting, say, DNA gels alongside lab protocols, researchers embed code, data and text to document their computational methods. The result, says Jupyter co-creator Brian Granger at California Polytechnic State University in San Luis Obispo, is a "computational narrative" -- a document that allows researchers to supplement their code and data with analysis, hypotheses and conjecture. For data scientists, that format can drive exploration.
[...] For data scientists, Jupyter has emerged as a de facto standard, says Lorena Barba, a mechanical and aeronautical engineer at George Washington University in Washington DC. Mario Juric, an astronomer at the University of Washington in Seattle who coordinates the LSST's data-management team, says: "I've never seen any migration this fast. It's just amazing." Computational notebooks are essentially laboratory notebooks for scientific computing. Instead of pasting, say, DNA gels alongside lab protocols, researchers embed code, data and text to document their computational methods. The result, says Jupyter co-creator Brian Granger at California Polytechnic State University in San Luis Obispo, is a "computational narrative" -- a document that allows researchers to supplement their code and data with analysis, hypotheses and conjecture. For data scientists, that format can drive exploration.
Wasn't this recently discussed? (Score:5, Interesting)
Economics Nobel Laureate Paul Romer Is a Python Programming Convert [slashdot.org]
They're oblivious to who owns their code (Score:1)
Especially when that code provides a competitive advantage in grant proposals.
It's Matlab in python clothing (Score:4, Interesting)
I love jupyternotebooks. But it's matlab. Well a broken inferior matlab. I do like python syntax better than matlab but that's just a sugar.
The upcoming Jupiterlab is a slavishly copy of the matlab ide.
It reminds me of how Linux desktop managers were always copying the last generation of windows.
I'm not complaining! I use mint and it owes a lot to windows too.
Mint however is actually superior to windows now.
But look at something like staroffice libre office. Ow... the pain. It's like a bad ms office 5, except you can only use it if you have thumbtacks in your shoes. They copied everything that was bad just so it was the same.
Jupiter is really nice and I use it in preference to matlab because it's so portable and I can use other python packages. But unless you used matlab you may not realize it's just a fast follower of ideas already tested out by matlab
Re: (Score:2)
Not quite.
- I'm not aware that you can typeset whole documents, including mathematics, in the Matlab UI. see nbviewer website [jupyter.org]
- I'm not aware that you can do slideshow presentations in Matlab, using something as simple as markdown.
- Plots and graphics are not embeddable in the Matlab command line.
On the other hand you can use Matlab as a GUI builder, which you cannot do as easily with Jupyter. see dashboards in jupyter [dominodatalab.com]
Huge Notebook fan. (Score:5, Interesting)
Anecdotal, but I do 90% of my python 'development' in Jupyter Notebooks.
For work I can make a nice notebook and have it generate a PDF for archiving. It'll output to LaTeX, html, .py and a number of other formats.
Now you can include multiple languages in the same notebook including R and Matlab, both popular in their own niches of use.
Re:Huge Notebook fan. (Score:5, Interesting)
For the past year, I've begun using Jupyter and although I like it, there are some features that really bother me, and worry me when it comes time to create reproducible science. 1) Jupyter doesn't integrate automatically with any kind of source control software, and in the circles I run in, it is largely ignored. Data scientists act like they've never heard of source control, and what makes it worse, my local university is pumping out student after student where they introduce them to data science with Jupyter, but never bring up the topic of coding standards and recoverability. 2) Jupyter allows you to execute cells out of order. While this definitely helps speed up development (when you make a mistake, and just want to fix the relevant line and continue, rather than re-loading your entire data set), it presents a unique risk when someone thinks they've discovered something amazing, only to be unable to reproduce it after a restart, or when sharing the notebook with someone else. This can happen when race conditions exist, or when code makes changes to the database, and your out of order execution causes spooky behavior. 3) Jupyter doesn't encourage enterprise deployment. Too often I see experimental data science done well, but due to the nature of rapid development, nothing is modular, nothing is object oriented, and so if the solution was a one off answer, everything is great, but if the solution is to be made into proper enterprise ready code, the entire notebook must be transcribed into truly disciplined code. (as an aside, this process is massively difficult because data scientists often don't understand the principles of object oriented programming, and the programmer doesn't understand the principles specific to the data science objective the code was written to solve.)
I expect to use Jupyter a lot more frequently in the coming years, but I fear it will feel like a huge step back in terms of the things that computer scientists have solved, that data scientists are ignoring.
Re: (Score:2)
Re: (Score:3)
A bit like programmers, then?
Re: (Score:2)
They don't have the excuse that they weren't shown a better way.
Re: (Score:3)
I think you misunderstood a few things. On my first point I said there is no INTEGRATION with source control. In Visual Studio, you can commit and check out directly in your IDE. In Eclipse, you can commit and check out directly in your IDE. In Pycharm you can commit and check out directly in your IDE. In Jupyter you can not (at least I'm not aware of how).
On your second comment, there are many ways to test that the code compiles and runs correctly, but that will not guarantee that it is right. I, lik
Re: (Score:3)
You can execute shell commands from within a notebook with !
!git commit -am "Commit message"
Re: (Score:2)
3) Jupyter doesn't encourage enterprise deployment. Too often I see experimental data science done well, but due to the nature of rapid development, nothing is modular, nothing is object oriented,
I'm not aware of anything like this that works generically for Jupyter notebooks, but I've been using OpenCPU to provide this functionality for R. RStudio is kind of like Jupyter, but designed more specifically for R, and it has templates for turning R scripts into packages. So you start with R Notebook, modularize into part Notebook / part R script with embedded functions, the Notebook part can be bootstrapped into a basic UI, then wrap up everything in an R package and deploy with OpenCPU. It's not quite
Re: (Score:3)
I wrote a book in latex, with lots of python code generating data and gnuplot and matplotlib generating pictures. This worked well (try doing a 426 page mathy book in word) and it was all text files, so source control and offsite backup via git worked well.
Would you recommend Jupyter for that kind of thing. Would the output always look like a paper - or could you make it work for technical book writing to eliminate some of the scripting and hand integration done with raw latex?
My experiments were not encour
Re: (Score:3)
It looks exactly like it would on paper, but if you plan on printing it I don't know if it handles the page break stuff. It does make a great PDF.
I know that you can integrate LaTeX templates.
The nice part about it is you don't have to deal with 99% of LaTeX and can just focus on writing the equations.
There's a free online way to try it: https://jupyter.org/try [jupyter.org]
Re: (Score:2)
LOL you almost got me, but I since I never believe anything anybody says I looked it up and found out you're wrong.
YOU'RE ON THE INTERNET, AND YOU'RE WRONG! (That's worse than genocide, BTW)
Literate Programming (Score:2, Informative)
Knuth tried to teach us to do this decades ago, but nobody listened.
Re: (Score:2)
Utterly unrelated with only the slightest of superficial similarity.
Furthermore, plenty of people "listened" and tools exist today not unlike those Knuth advocated.
At least you spelled Knuth right.
Re: (Score:2)
Knuth tried to teach us to do this decades ago, but nobody listened.
If that was true, we wouldn't be talking about this implementation!
It took 30 years after Maxwell published his Equations that people realized he was right, because it implied relativity, which would contradict Newton.
A few decades of pause in the face of a new idea is nothing to be concerned about.
That said, IMO it is only useful for casual programming, mostly because the words used in English to describe what literate programming is supposed to be don't really prepare people for the mess of mixed source a
Because Jupyter aligned with Mars? (Score:4, Funny)
Re: (Score:2)
Didn't we do that back in the 1960s? At least I think so; they say that if you remember the 60s, you weren't there.
A rare sort of development in the software world.. (Score:4, Interesting)
Jupyter is something that is relatively unique, useful in its field, and *not* crammed down the throats of people for whom it isn't really relevant.
I applaud the way that project is executed, adopted, and evangelized as being on point and solidly executed...
Re: (Score:1)
If you come from your mindset of replacing Excel, yes it is.
However, if you come from the mindset of using it as the prodcution code for the backend model for your startup, then no.
I can see how it works great for prototyping, experimenting, and one-off type work especially when all the users of the tool will be reasonably familar with it.
But I wouldn't use it to run production code behind a website. Maybe you'll build your prototype version of the model in there, and then port it to a production framework
Re:rare sort of development in the software world (Score:2)
"But no one says is the best tool for every job." I wish someone knowledgeable would explain which jobs Jupyter is good for, and which it isn't. I see claims here by one poster, and then a reply below that post saying they're all wrong... Doesn't generate trust, at least not for me.
For the record, I do a lot of Python programming (including object oriented, which one poster said you couldn't do in J, and the very next poster said you could; case in point). Also XML, LaTeX, and finite state transducers,
Re: (Score:1)
Having said all that... the order of execution isn't really fixed. Unlike Excel where making any change normally recomputes everything affected, making a change in a notebook doesn't recompute anything untill that 'cell' is executed.
This means the 'execution state' doesn't necessarially match what's in the notebook (it reflects the state of the cells based on the order they were executed in, which isn't guarenteed to be top-to-bottom but rather whatever order you triggered them in.)
So, if you're a programm
Terrabytes/day is not much for astronomy (Score:2, Interesting)
In early nineties, my experiments used to generate 5 terabytes of data a day (limited by ability to store and post process. the telescopes had capability of generating much larger data. In fact in late 70s and early 80s, it was common to generate terrabytes of data a day in radio astronomy VLBI experiments. These were stored on regular video cassettes (7 GB/tape). A single experiment will use anywhere from 5-25 recorders simultaneously 24 hrs a day.
Re: Couldn't be STUPYDR (Score:2)
Re: (Score:3)
I get it. You print out the suspect word with your 3D printer, and then you use your sterling hand to trace around the printed artifact for bumps, hollows, ridges, descenders, and cisterns—which you simultaneously compare and contrast to the irregular outlines of your Mario Kart Kamikazi Kukmumbr as braced by your other fleshy mitt.
Phrenology, with benefits.
Re: (Score:2)
Agreed, why qan't they replase 'c' with 's' or 'q'.
Literate programming (Score:4, Interesting)
For decades, we talked about Knuth's literate programming. Jupyter is finally an open source tool that made it usable for everyone.
There is no better way to explain the use of a library than making a Jupyter notebook available.
Most of my Python use lately is for one-off analytics with heavy libraries. Jupyter suits this workflow very well.
IPython already has decent hooks for IDEs (PyCharm, Spyder), but I hope this gets even better.
it is? (Score:2)
Sagemath has more libraries (and can deal with jupyter notebooks too)