Firefox/Python 3 Migration

This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

Why This is Important

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[1]

Python 2 will reach it's End Of Life on January 1st 2020 (here's a countdown!).

What Will Happen If We Don't Upgrade?

The PSF said it best:

"If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time."

How We Get There

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

1. Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
2. Make key support libraries run in both Python 2 and Python 3. (mach-core)
3. Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
4. Port Python 2-only components to Python 3 piecemeal.
5. Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

Getting Involved

Find the others

Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.

What you'll need

You will need a working Firefox build environment.

Where to start:

Remove more of the "excluded" paths from the py2 and py3 linters

See this Etherpad for instructions. (Etherpad is blank, needs to be fixed)

Roughly:

• Pick a directory from the list, and put your name beside it so we know it's ported (this etherpad is now blank, ask in #py3 for details)
• Edit py2.yml and/or py3.yml and remove that directory from the exclusion list (from tools/lint/py2.yml and tools/lint/py3.yml in https://hg.mozilla.org/mozilla-central/)
• Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
• To get a list of files that need fixing:

./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt

• To install `futurize`:

pip install --user future

• To run through futurize

cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w

• Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

1. Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
2. Use bug 1559975
3. Set one of callek, ahal, or catlee as the code reviewer

Enable Python 3 on more Python unittests

Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:

1. Find a test that is skipped with Python 3, you can use this query to find these tests.
2. Edit the manifest and remove the Python 3 skip-if annotation.
3. Run the test with `./mach python-test --python 3 path/to/test`.
4. Fix any errors and repeat the previous step.
5. Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in python.yml.
6. Push the task to try on all platforms it is configured for.

Port a Mach Command

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2.

The whitelist contained in the mach command is py2commands

py2commands="
   addtest
   analyze
   android
   android-emulator
   artifact
   awsy-test
   bootstrap..."

Pick the command that you'd like to convert and then:

1. Remove it from the whitelist
2. Run: ./mach <command>
3. See what breaks and fix it!
4. Repeat with a variety of different flags and arguments
5. Make sure any Python unittests running in CI have Python 3 enabled (see above)

How to Port the Code

Before starting

If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.

Porting modules in the Firefox source tree

• We've already standardized on Python 2.7 across the tree.
• Make sure you can test the code you're working on with python3!
  • If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
  • If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
• `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
• The `six` module is available in-tree, feel free to use it as a dependency.
• Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).

Porting examples

Prefer six when possible. Make use of six.PY2, six.PY3, six.moves, six.string_types, six.integer_types, etc. This make it easy to find the py2 code again so we can remove it.

import statements

Use the six.moves module:

Before:

import __builtin__

__builtin__.__import__ = myimporthook()

After:

from six.moves import builtins

builtins.__import__ = myimporthook()

if statements

Use six.PY2 and six.PY3:

import six

if six.PY2:
    # py2 stuff
else:
    # py3 stuff

If six isn't available or you are in the imports section of the module then this is fine too:

if sys.version_info < (3,):
   # py2 stuff
else:
   # py3 stuff

String data

Use six.string_types, six.text_type, six.binary_type, etc. Also see the six docs for "Binary and text data":

Before:

# using unicode
def write(self, buf):
    if isinstance(buf, unicode):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)


# using types.StringTypes
if isinstance(value, types.StringTypes):
    ...

After:

import six

# fixed unicode
def write(self, buf):
    if isinstance(buf, six.text_type):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)


# fixed types.StringTypes
if isinstance(value, six.string_types):
    ...

subprocess.check_output()

Python 2.7 code will often call subprocess.check_output() and then do string manipulation on the output, such as str.splitlines(). The code will fail under Python 3 because subprocess.check_output() now returns a binary data stream. However we can use the universal_newlines=True keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

output = subprocess.check_output([callargs])
lines = output.splitlines()  # Broken in Python 3

After:

output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()

See the subprocess documentation for details.

Porting Resources

Porting Guide on python.org

Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

References

Bug 1496527 tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

Python 3 tree migration notes

mach Python 3 migration roadmap