Packaging Android host utilities

Host utilities are executable files that run on a *host* machine. These utilities provide services to an Android *target* device, such as running a web server or a certificate authority. Host utilities must be periodically updated with a good build from mozilla-central or autoland due to addition of new tests targeted at Android and/or changing of component behavior.

This page will provide instruction on how to package new versions of the host utilities.

Generate new archive

On Treeherder, identify a target build, preferably from mozilla-central or autoland. Check the test output to ensure a reasonably good build (one without too many failures).

example: for the version 66.0 update, this build was selected. For more details, please see bug 1514075.

Linux

For Linux, the steps are fairly straightforward. Follow these steps to locate and download the necessary files.

From the selected treeherder build, identify Linux x64 opt.
Click on the green B icon, which will bring up a pane covering bottom third of the window.
On the bottom left pane is a header named Task. Click on the hash.
A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
Download target.common.tests.tar.gz.
Download target.tar.bz2.
Follow the contents of the script below:

tar xvf target.tar.bz2
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.linux-x86_64
tar cvf host-utils-66.0a1.en-US.linux-x86_64.tar host-utils-66.0a1.en-US.linux-x86_64
gzip host-utils-66.0a1.en-US.linux-x86_64.tar

Repeat for Linux 32bit, substituting x86 for x86_64 where necessary.

macOS

Preparing for macOS is not quite so simple. Due to requirements of macOS, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If binaries are unsigned, users will be prompted at every invocation to "allow network connections". For more information on signing macOS binaries, see here.

Navigate to Mozilla FTP.
Drill down to the appropriate year, month. Select a date that is not l10n.
Ensure files in the directory are prefixed with the version of Firefox that is desired.
Download firefox-66.0a1.en-US.mac.common.tests.tar.gz.
Download firefox-66.0a1.en-US.mac.dmg.
Follow the contents of the script below:

tar xvf firefox-66.0a1.en-US.mac.common.tests.tar.gz 'bin/*'
open firefox-66.0a1.en-US.mac.dmg
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/* bin
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/Resources/* bin
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -
mv bin host-utils-66.0a1.en-US.mac
tar cvf host-utils-66.0a1.en-US.mac.tar host-utils-66.0a1.en-US.mac/*
gzip host-utils-66.0a1.en-US.mac.tar

Windows (experimental)

Similar to Linux. Follow these steps to locate and download the necessary files

From the selected treeherder build, identify Windows 2012 debug (Windows 2012 opt isn't being built any longer).
Click on the green B icon, which will bring up a pane covering bottom third of the window.
On the bottom left pane is a header named Task. Click on the hash.
A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
Download target.common.tests.tar.gz.
Download target.zip
Follow the contents of the script below:

unzip target.zip
mkdir temp_common
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.win32
tar cvf host-utils-66.0a1.en-US.win32.tar host-utils-66.0a1.en-US.win32
gzip host-utils-66.0a1.en-US.win32.tar

Uploading to ToolTool

Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.
Ensure uploading user has a valid token at Mozilla Releng.
Add file to temporary manifest

python tooltool.py add --unpack --visibility public [file]

Upload the archive:

python tooltool.py upload --authentication-file=[token_location] --message [commit_message]

Update the manifest in testing/config/tooltool-manifests/macosx64/hostutils.manifest.

If updating host utilities for Linux, repeat using 32bit/x86 archives.

Do the same for Windows. testing/config/tooltool-manifests/win32/hostutils.manifest

Verification

ToolTool will return an updated manifest once the archives are uploaded and processed. This new manifest must first be tested, follow the steps below:

Linux

Changes to host utilities for Linux can be tested on the try server.

Update to current tip by hg pull and hg update -r tip -C
Copy and paste the updated manifest to the hostutils.manifest file for Linux 64 and Linux 32.
Commit change and run moz-phab.
Initiate a try run of all Android tests:

# all (use this by default)
./mach try fuzzy --no-artifact -q='test-android-'

# only hardware
# ./mach try fuzzy --no-artifact -q=test-android-hw-
# only emulator
# ./mach try fuzzy --no-artifact -q=test-android-em-

macOS

Changes to host utilities for macOS cannot be tested on try server, therefore the following steps are taken:

Update to current tip by hg pull and hg update -r tip -C
Copy and paste the updated manifest to the hostutils.manifest file for Mac.
Commit change and run moz-phab.
Ensure the currently selected MOZCONFIG designates an Android build target.
Sanitize and rebuild with the following:

./mach clobber
./mach build

Once build is complete, package the apk using ./mach package
Install on device with ./mach install
Run a short test with ./mach mochitest testing/mochitest/tests/Harness_sanity
Accept offer to update host utilities.

If tests complete without issues, the generated host utilities are deemed to be good.

Download existing archive

It is possible to download existing host utilities.

Locate the existing manifest file using [1].
Using ToolTool, download the host utilities:

python tooltool.py fetch -m old_hostutils_manifest.tt

Packaging Android host utilities

Contents