Python Pants - A build system for large codebases with multiple dependencies - PyConSG 2016
Speaker: Angad Singh
Description
This talk is geared towards Infrastructure and system engineers who are interested in learning about structuring a large monorepo codebase, consisting of multiple micro services that share many dependencies. This talk will introduce Pants as a build system for such large monolithic codebase and how it ties with today’s container ecosystem principles.
Abstract
Pants was developed at Twitter as the need grew for developers to be able to build large code bases, really fast. A simple way to speed up code builds is to split the code into hundreds of small repositories (with a one-to-one mapping between each micro service and its repository). But that solution does not scale as you end up with hundreds of non-standard repositories with shared code and dependency management issues. A large unified codebase promotes better engineering collaboration. Pants was developed as a build system for such large codebases.
In web companies of all sorts and sizes, Python is a popular choice for writing infrastructure/operations scripts as well as large scale web projects. I have worked at two companies, Twitter and Viki, where Python is a core language for Infrastructure, used for automation, job scheduling, deployments, infrastructure provisioning and operational dashboards. Across both companies, we have been using Pants as a build system for our Python projects.
With the rise of the container ecosystems, the importance of strong build systems has increased even more. Golang with its statically linked binaries is a beautiful example of “drop your binary into a container and run”. Pants introduces a similar concept of packaging the dependencies along with the target for python projects. It can run your tests and build standalone python executables called PEX files. It gives benefits similar to a virtualenv and can even give you a REPL for your python project. All of this is helpful as we ship python executables in our Docker containers, which is used in both development and production environments.
There are other python build systems as well such as Bazel, http://bazel.io/docs/be/python.html and Buck https://buckbuild.com/ with all of them providing almost similar benefits. The goal is to encourage the usage of a build system to ensure standardisation and increased productivity of the development team.
The talk will include a quick overview of the Pants command line as well as how we can ship a simple Flask application, which has multiple library dependencies as a Python executable inside a Docker container.
Event Page: https://pycon.sg
Produced by Engineers.SG
Help us caption & translate this video!
