WebDev/Deployments

This covers deploying of production systems using pip and PyPI.

Goals

• to use pip and Python Package Index (PyPI) whenever possible
• to allow any contributor to easily install and contribute WebDev projects
• to have them deployed quickly and securely internally

Using requirements for a deployment

For a deployed product (such as a website) specify its packages in requirements files. We recommend splitting the packages down into separate file for separate roles, for example: requirements/dev.text, requirements/prod.txt (this is also used for staging), and requirements/test.txt (those needed only for CI builds).

All requirements should be pinned by version number, e.g.:

  foo==0.3

Use of >= or not pinning to a version is not recommended. This could result in deploying broken, untested versions of packages to production or during local development.

When running pip use the flag: --no-deps. This ensures that packages will not pull in more untested versions of packages. It also means that the requirements files are a definitive source of packages used. This allows security faster audits of who is using what package.

Note: Packages on PyPI can be removed then replaced so even pinning a version does not mean it can be trusted if you are not using the internal mirror.

Using requirements for a library

When building a library for deployed products to install, we recommend creating a single requirements.txt file in the root of the project. Unlike deployed products, do not pin requirements for libraries. Instead, keep them loose like:

   django
   django-aesfield

If you require certain versions, specify the minimum version required:

   django>=1.4

Using loose requirements for libraries helps to avoid version conflicts when a deployed product has multiple packages that both require django (or some common package).

The deployed product is installed with --no-deps, allowing the deployed product to choose all its dependencies, including those required for the library.

External package server

When uploading a package to PyPI, please add in a shared account that will be able to push new packages to that account. This will allow members of the team to push new fixes, including security fixes. In PyPI go to your package and click "admin".

Existing shared accounts:

• Username: marketplacedevs

Internal package server

When a package is to be used, it must be uploaded to the internal package server.

Existing servers:

• For add-ons and marketplace: pyrepo1.

The trusted users who can upload apps to the server are responsible for validating the packages before uploading them. It's up to the uploading user to validate that the package meets security needs.

Contributors and external users will continue to use PyPI as normal. Internal production deployments will use this mirror by using the --no-index and --find-links. For example in Marketplace:

   ./venv/bin/pip install --exists-action=w --no-deps --no-index 
   -f https://pyrepo1.addons.phx1.mozilla.com/ -r requirements/prod.txt

Future goals

If we can build RPMs prior to deploying, this will allow security to more easily audit the source of files. Currently services and socorro do this.