by Charles Brunet
At Optel, we use Python to connect our systems with our client's specific systems. I will show you how we used namespace package and a few setuptools hacks to split a big monolithic library into an ecosystem of small versioned packages, to easily adapt to each client needs.
'Bridge' is the name of the Python library we use to connect our systems to our clients' ERPs. Up until recently, it was a monolithic library with a lot of custom code for each client. However, it was difficult to maintain, as each single feature or bugfix for a given client required a new version of the whole bridge library. Furthermore, the deployment of the bridge for a given client required a special tool, the deployer, used to copy only the specific required parts of the bridge into the client system.
To solve those problems, we split the bridge into an ecosystem of small interrelated packages. Each package now has its own version number and dependencies along both the bridge itself and third-party Python libraries. This allows to upgrade a single client package without affecting the whole bridge product. It also eases the installation because the client's package knows what are the required packages to install.
On the technical side, first part of the solution was to use namespace package. This little-known feature of Python allows many related packages to be imported from the same base package name. There are three ways of declaring namespace packages:
setuptools.declare_namespace. While PEP 420 namespace package is the modern way to go, we used
pkgutilnamespace because we still need to support Python 2. Hopefully, the transition to PEP 420 should be easy once we will be able to drop Python 2. While not mandatory, the namespace package allows to:
The second part of the solution concerns
setuptools. Nothing special is required in
setuptools to use namespace package and declare dependencies between the packages. However, using a special
setup.py file at the root of the project allows to manage the set of micropackages as a single library for easier integration with CI tools. In the second part of my presentation I will show you how I hacked
setuptools to build and install all the micropackages in a single command.
Finally, I will conclude showing you how our new package structure eases the maintainability and the deployment of our solution.
About the Author