The python module uses this data tree to get python packages.

In general, these are just downloads from pypi. However, a few packages are special - see below.

These packages are at

http://repos/python/packages

and can also be sourced at

puppet:///python/packages

in file resources.

Adding files

In general, you do not need to manual add files to the Python data directories. The Distingished Puppet Master will automatically walk through the Puppet Manifests and download any missing Python packages defined in a requirements file.

If you are deploying a package that is not hosted on Pypi (for example, a scriptworker package) you must add it to the Python package directory manually, and you should mark it as "# puppet: nodownload" in the requirements file.

Debugging failed downloads

The pip download script will send mail to the releng-puppet-mail mailing list if any errors occur when downloading packages. Most commonly, this will happen if it tries to download a package or package version that doesn't exist on pypi (see the note above about marking internal packages as nodownload). If you need to do additional debugging, you can hop on the distinguished master and edit the "pip download" command in /etc/puppet/puppetmaster-pip-download.sh to get additional output on the next run (Puppet will revert your changes, so you may have to do it more than once depending how long it takes you to debug).

Mirror Access

If you would like to point your 'pip' at the python repository, you'll want a ~/.pip/pip.conf that looks something like

[global]
no-index = true
find-links = http://puppetagain.pub.build.mozilla.org/data/python/packages

Special Cases

virtualenv.py

An un-packaged copy of virtualenv.py is at python/packages/virtualenv.py. More recently, updated versions of virtualenv have been unpacked to "virtualenv-$version" directories in both python/packages and python/packages-3.x. These are all used to bootstrap new virtualenvs.

Twisted

Some virtualenvs need to be installed on systems without compilers. In particular, Talos systems don't have compilers or development libraries installed. For these cases, we install packages that can fall back to a pure-python implementation when the compile fails. The tarball of Twisted-10.2.0 is *not* the one from pypi; that tarball was unpacked, conditionalExtensions was set to [] in both setup.py and twisted/topfiles/setup.py, and a new tarball made with setup.py sdist, and the result is in the python-packages directory. Newer Twisteds may not need this treatment -- who knows.