The downsides of using Jupyter for Software Development (Part 1)
Big Fat Disclaimer: I absolutely love Jupyter Notebooks
Making sure there are no misunderstandings.
Jupyter Notebooks and the Jupyter ecosystem as a whole have revolutionized the world of scientific computing. I was an Academic Scientist when the iPython project released “iPython Notebooks'. When this happened, amongst the group of researchers I was part of, Python went from being just a weird language the more “Computer Science like” researchers like using for scripting to a serious contender to the competing Interactive Computing Environments (ICE) of the time. As a heavy Python user at the time, I was both elated and an early adopter.
Since the rename of iPython to Jupyter, the project has exploded and has become one of the biggest Free Open Source Software (FOSS) ICEs (maybe the biggest?), and not only FOSS. All major cloud vendors (AWS, GCP, Azure) and some other players (Databricks, Jetbrains, and others) have their own Jupyter-based systems, and is quickly becoming a standard in scientific computing. All of this makes me extremely happy and optimistic about the quality of scientific tooling, which is something I deeply care about.
I write all this because I don’t want to give the impression that I’m throwing shade at the Jupyter Project or the Jupyter Ecosystem as a whole. In fact, I maintain 2 small FOSS libraries that I use regularly in Jupyter and if anything, I’m just trying to make sure that Jupyter is used for its intended purpose and it’s not stereotyped as “that horrible thing Scientists use” by software developers. (The thing is, I used to be a Software Developer eons ago and I know how quickly this community delves into disdain. I can already feel it happening).
Why are you talking about not using notebooks then?
Please keep in mind there is quite some nuance in this post. I’m still advocating notebooks for analytics and prototyping code, what I’m saying is: Stop using notebooks for developing code and using .ipynb files as production artifacts.
Because of notebooks becoming so ubiquitous, we have seen less than optimal projects where notebooks played a central role where they shouldn’t. These are the main three categories in my mind:
Why are these bad? Lets start with the .ipynb format itself.
.ipynb is not an efficient for holding code
Consider this simple example:
The objective is to make sample_from_beta() available as part of a data pipeline. Forget about the tests for now, we will talk about them later. If you C&P this function in a .py file it will (not surprisingly) look like so:
Recommended by LinkedIn
import numpy as np
def sample_from_beta(a, b, samples=2000):
"""
Same as np.random.beta, but increasing the default sample size and
renaming 'size' for 'samples'
"""
return np.random.beta(a, b, size=samples)
However (even after removing the tests) this is now the insides of a .ipynb file look like:
"cells": [
{
"cell_type": "code",
"execution_count": 8,
"id": "9d2d2032-8b7b-4c4c-983b-b388bca79d33",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"\n",
"def sample_from_beta(a, b, samples=2000):\n",
" \"\"\"\n",
" Same as np.random.beta, but increasing the default sample size and \n",
" renaming 'size' for 'samples'\n",
" \"\"\"\n",
" return np.random.beta(a, b, size=samples)"
]
},
{
"cell_type": "markdown",
"id": "1d1b09e1-91a0-481f-abb6-a751d551bf06",
"metadata": {},
"source": [
"# Testing my function"
]
},
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}{
At this point, you might wonder what the problem is here exactly. After all, this is not the 90s anymore and modern computers can handle verbose text files with ease. Aside from making a simple script impossible to inspect outside of a Jupyter Environment (more of that in Part 2), here are the extra consequences:
Jupyter does not incentivize Unit Testing
In the notebook example above, I wrote some “tests” in order to validate that my function was working as intended. This is one of the reasons why ICEs are so nice! It’s really easy to write simple, fast tests and validate your code as you write it. However, this habit is counterproductive when having to write Unit Tests… And if you are developing serious software, like a Python library that needs to be uploaded to PyPI, or something that goes into a data pipeline that needs to be resilient, Unit Tests are not a nice-to-have, they are almost a necessity. I have found myself postponing writing unit tests because of how comfortable it is to sloppily “test” in the notebook, and I’ve seen countless examples of notebooks quickly becoming unmaintainable.
As a side note, do you know why Rust is becoming so popular amongst developers? It’s because the way the language is designed it’s very hard to get your code to run without it being good quality. ICEs like Jupyter make it as easy as possible to write disposable code that runs, which again, is great for analytics! But please consider the risk of using it (exclusively) for software development.
Notebooks don't incentivize a modular Software Architecture
Furthermore, when every code block is executable it feels wrong to make functions (so lets not even talk about OOP). When "writing notebooks" every cell feels like a function and writing something as basic as the sample_from_beta() function above when you can simply do this:
Which again, it’s great when quickly prototyping a complex piece of code! I personally often start prototyping projects in notebooks. However, I’ve also often found myself repackaging my endless notebook spaghetti code into functions with the lingering feeling that I should have done it earlier. This is because it’s comfy to prototype in Jupyter until it’s almost too late.
(In the 2nd part I'll talk about why IDEs are awesome too, idempotence, markdown vs code, and fat fingers)
Haha is this the hill you die on? Would be interesting to hear what the view at Netflix is four years later https://netflixtechblog.com/scheduling-notebooks-348e6c14cfd6