Aayushi Johari · Follow
Published in · 6 min read · Jun 22, 2019
--
The one-stop destination for every movie buff is, of course, Netflix. But what if you were watching your favorite movie and it keeps buffering every now and then? You would just shut down the application and choose another option. But, how does it manage the traffic of millions of users swiftly? Thanks, to PYTHON. In this article, let’s explore how Netflix uses Python.
Let’s begin by taking a quick look at the themes that fill this article:
- Introduction to Netflix
- How Netflix uses Python?
- Open Connect
- Demand Engineering Team
- Machine Learning Infrastructure
- Big Data
- Scientific Experimentation
- Video Encoding / Media Cloud Engineering
- Netflix Animation and NVFX
- IS (Information Security)
- Monitoring and Auto-Remediation
So let’s get started. :)
Netflix is an American company which renders Video on Demand (VOD) services. Headquartered in Los Gatos, California, Netflix has about 148 million subscribers throughout the world and the number, however, keeps growing each day. In a period of approximately two decades, Netflix has emerged as the ‘King of the clan’ for the biggest Tv Series and Movies throughout the world. Being the fastest growing brand of America and having a revenue of $20.5B in 2019, is enough for it to be an ‘eye-catcher’, thereby interesting all into its technological spheres.
Based on the same area of interest, Netflix has revealed how it makes use of the most trending language, Python, for its infrastructure.
So now let’s move on to see how actually Netflix uses Python?
We use Python through the full content life cycle, from deciding which content to fund all the way to operating the CDN that serves the final video to 148 million members” — Engineers at Netflix
Ranging from Administrative domains to Reliability and Data Science to Machine Learning etc, Netflix uses Python for nearly every edge of their business.
Now let’s take a deeper look at how Python is used in various domains at Netflix:
Open Connect:
The CDN (Content Delivery Network) that Netflix makes use of is, Open Connect. Open connect basically come into picture when you click on the ‘play’ button. All the content delivered to the end user is looked after by this CDN.
Open connect requires various other software systems to design, build and operate it which are in turn written in Python. Not just this, the network devices underlying this CDN are Python applications since Python is prominent in solving network issues.
Demand Engineering Team:
The Demand Engineering team is responsible for handling the Netflix cloud’s Regional Failovers, Traffic Administration, Capacity Operations Management (looking after the limit up to which the content can be made serviceable), and Fleet Efficiency. The elements of Python used by this team are:
NumPy and SciPy:
NumPy and SciPy are the libraries used for scientific computing. Netflix uses these Python libraries to perform numerical analysis thereby allowing management of Regional Failovers.
Boto3:
Boto3 is the Software Development Kit (SDK) of AWS (Amazon Web Services) for Python. This helps Python developers integrate Python into AWS thereby allowing development in the infrastructure.
RQ (Redis Queue):
This is a Python library that helps keep track of tasks that are present in the queue and allows their execution thereby allowing the management of asynchronous workloads.
Flask:
Finally, Netflix uses Flask (Python Web Development library) API’s to bind all of the previous segments together.
Netflix makes use of Jupyter Notebook which is an open-source web app, used for Python development along with nteract (extension for Jupyter) on a large scale. Jupyter is known to be popular for data analysis. It serves very well in operational data analysis and visualization which in turn help in detecting capacity regressions.
Machine Learning Infrastructure:
Machine Learning ranges from creating Personalization algorithms to figuring out the use cases. Personalization algorithms help to train the Machine Learning models as per the Netflix standards. It provides personalized recommendations, outlines on a day-to-day basis, label generations, etc.
The libraries required to learn Deep Neural Networks are TensorFlow, Keras, and Pytorch whereas XGBoost and LightGBM for Gradient Boosted Decision Trees. They have also developed quite a few higher-level libraries that help in combining with the work areas such as fact logging, feature extraction, publishing, etc. Apart from all this, Netflix also uses MetaFlow to create machine learning projects.
Metaflow pushes the limits of Python: We leverage well parallelized and optimized Python code to fetch data at 10Gbps, handle hundreds of millions of data points in memory, and orchestrate computation over tens of thousands of CPU cores” — Netflix
Big Data:
The Big Data team is responsible to execute ETL (extract, transform, load) and Adhoc pipelines. A major part of this orchestration is written in Python. This team uses a scheduler which runs on Jupyter Notebooks with papermill to produce job types with templates, for example, Spark, Presto, etc.
In addition to this, the team has also created an event-driven platform which is built completely on Python. They have created a number of events and combined it into a single one allowing Netflix to filter, react and route events. Pygenie is also a part of this infrastructure which interface with Genie (featured job execution service).
Scientific Experimentation:
This is a platform created by the scientific experimentation team to allow A/B testing along with some other experimentations. Here, scientists and engineers can present new innovations in data, statistics, and visualization.
The Python framework that is implemented here is Metrics Repo which is based on PyPika and allows writing of reusable parameterized queries. For the statistics sector, PyArrow and RPy2 are used so as to calculate statistics in either Python or R. Plotly helps in visualizations.
Video Encoding / Media Cloud Engineering:
This team is responsible for encoding and re-encoding tasks for the Netflix catalog. Python is used approximately for 50 projects such as VMAF ( Video Multi-Method Assessment Fusion) and MezzFS(Mezzanine File System), Computer Vision Solutions (deals with imagery) using Archer, etc.
Netflix Animation and NVFX:
Python forms the base for all Animations and Visual Effects (VFX) at Netflix. All of the Maya and Nuke unions are done on Python.
IS (Information Security):
Netflix uses Python powered IS systems for auto-remediation, security automation, risk classification, etc. The most active open source Python project of this team is Security Monkey. Netflix also uses BLESS (Bastion’s Lambda Ephemeral SSH Service) to protect SSH (Secure Shell) resources. RepoKid is used to grant IAM permissions and TLS certificates are allotted through Lemur. Both of these tasks rely mainly on Python.
Monitoring and Auto-Remediation:
This team is known as the Insight Engineering team. They build and execute tools for operational insight, diagnostics, auto-remediation, and altering. For most of its services, this team makes use of Python, for example, the Spectator Python client library. This library is used for recording dimensional time series. Along with these libraries, products like Winston and Bolt are also built on Python frameworks which are Flask, Gunicorn, and Flask-RestPlus.
Summing it all up, one can easily claim Python to be the driving force for Netflix. With this, we have reached the end of this article on “How Netflix uses Python?”. I hope you’re clear all that has been discussed.
If you wish to check out more articles on the market’s most trending technologies like Artificial Intelligence, DevOps, Ethical Hacking, then you can refer to Edureka’s official site.
Do look out for other articles in this series which will explain the various other aspects of Python and Data Science.
1. Python Tutorial
2. Python Programming Language
3. Python Functions
4. File Handling in Python
5. Python Numpy Tutorial
6. Scikit Learn Machine Learning
7. Python Pandas Tutorial
8. Matplotlib Tutorial
9. Tkinter Tutorial
10. Requests Tutorial
11. PyGame Tutorial
12. OpenCV Tutorial
13. Web Scraping With Python
14. PyCharm Tutorial
15. Machine Learning Tutorial
16. Linear Regression Algorithm from scratch in Python
17. Python for Data Science
18. Loops in Python
19. Python RegEx
20. Python Projects
21. Machine Learning Projects
22. Arrays in Python
23. Sets in Python
24. Multithreading in Python
25. Python Interview Questions
26. Java vs Python
27. How To Become A Python Developer?
28. Python Lambda Functions
29. What is Socket Programming in Python
30. Python Database Connection
31. Golang vs Python
32. Python Seaborn Tutorial
33. Python Career Opportunities
Originally published at https://www.edureka.co on June 22, 2019.
