Mobile/Fennec/Android/UITest

UITest is a base test class for Robocop (UI-centric) tests. This and the related classes attempt to provide a framework to improve upon the issues discovered with the previous BaseTest implementation by providing simple test authorship and framework extension, consistency, and reliability.

UITest was originally implemented in bug 910859.

Architecture

UITest is divided into the base package, components/, and helpers/.

The base package

This package contains the UITest base class, its interface (UITestContext), the tests, and any helper classes also used by BaseTest (e.g. StringHelper).

components/

This package contains any helper classes that can be equated with a Firefox for Android component, a component being any small section of self-contained functionality (e.g. the about:home screen (AboutHomeComponent), the toolbar (ToolbarComponent)).

helpers/

This package contains static helper classes that are designed to utilize one or more components to make testing simpler (e.g. an assertions class (AssertionHelper), a navigation class (NavigationHelper) - go forward in history, go back in history). Note that these classes are not intended to be instantiated and should be used statically.

Writing UITests

Using tests should be very simple since the framework should be doing all of the heavy lifting! If you find yourself writing more than a line or two to take an action before an assertion, ensure the appropriate action is not a part of the framework! If it isn't, file a bug (and consider adding it yourself)!

For example, a simple test might look like testSessionHistory (on 11/27):

public void testSessionHistory() {
    GeckoHelper.blockForReady();

    NavigationHelper.enterAndLoadUrl(StringHelper.ROBOCOP_BLANK_PAGE_01_URL);
    mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_01_TITLE);

    NavigationHelper.enterAndLoadUrl(StringHelper.ROBOCOP_BLANK_PAGE_02_URL);
    mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_02_TITLE);

    NavigationHelper.enterAndLoadUrl(StringHelper.ROBOCOP_BLANK_PAGE_03_URL);
    mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_03_TITLE);

    NavigationHelper.goBack();
    mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_02_TITLE);

    NavigationHelper.goBack();
    mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_01_TITLE);
}

Samples

For some sample UITests, check out this mxr search.

Required code

Assuming the package names do not change, starting a UITest is as simple as:

package org.mozilla.gecko.tests;

import static org.mozilla.gecko.tests.helpers.AssertionHelper.*;

import org.mozilla.gecko.tests.helpers.*;

// TODO: Change this test name!
public class testSomething extends UITest {
    public void testSomething() {
        // Test code here...
    }
}

In particular, note the AssertionHelper import.

At the start of your test method, you'll likely also want to call:

GeckoHelper.blockForReady();

Just one test() function please!

When you write that testSomething() function, you don't need to register it anywhere -- the test framework will find your function and run it automatically as long as it is public and named test<something>. Unfortunately, if you have more than one test function, the framework will try to run that too -- and that can cause problems! (bug 1113751)

// DO NOT DO THIS
public class testSomething extends UITest {
    public void testAAA() {
        // Test code here...
    }
    public void testBBB() {
        // More test code here...
    }
}

// You can do this instead!
public class testSomething extends UITest {
    public void testAAA() {
        // Test code here...
        checkBBB()
    }
    private void checkBBB() {
        // More test code here...
    }
}

Assertions

Assertions are imported from helpers.AssertionHelper. They take the form:

fAssert*(...)
fFail*(...)

and typically correlate with JUnit's `assert*(...)` and `fail*(...)` methods. For a full list of methods, see AssertionHelper.java. To prevent accidental JUnit usage, if a JUnit method is called, an Exception will be thrown.

What to assert

Only verify what is visible on the screen. Consider it a visual API to the user: the internal state of the app may change as bugs are fixed and code is refactored, but what the user sees should remain the same.

If you're trying to assert what is not on the screen, considering using JUnit (see bug 903528)!

Note that if there is an action-triggered transition between UI states (e.g. tapping the toolbar to enter editing mode), all state that is asserted in a test suite should be waited on by the framework, otherwise race conditions may arise where either the assertion or the state change may occur first. If you're receiving intermittent failures while writing tests, you should look at the underlying methods in the framework to see which states are being waited on. For more information, see the section on waiting below.

Assertion messages

Assertion messages should print what state the UI would be in had the assertion passed. For example:

assertEquals("The HomePager is visible", View.VISIBLE, getHomePagerView().getVisibility());
assertNotNull("url is not null", ...);

Test length and specificity

In an ideal world, each test would be extremely short and only test one particular aspect of functionality, to better isolate potential regressions.

However, since the test suite is run in full on real (and sometimes slow/old) devices, we have to be concerned about test setUp and tearDown time. In particular, test setUp is also required to wait for Gecko to load (for specifics, see here).

Therefore, tests should fairly comprehensively test some aspect of the UI. When adding to the tests:

• See if you can add your assertions to an existing one.
• Use comments or helper functions to specify what might better be created as a new test.
• Be careful to avoid changing too much browser state if its unnecessary, and be very clear when this occurs. It may be better to save large state changes for the end of a method

Test on tablets

Pretty please! ^_^ Our automated testing framework, tbpl, runs on tablet-sized devices so make sure your changes work there too!

Cleanup

The profile directory is removed between tests, however, any files saved outside of this profile directory or the /mnt/sdcard/Downloads directory are not removed. Make sure you clean up these files! See bug 968200 (downloads) for a failure caused by not cleaning up.

Extending the framework

The framework should handle all of the edge cases making it easy for other developers to write tests, therefore be very aware of these tiny edge cases! If you find one, please add it below!

Be sure to also read the section above on writing UITests and the Robotium API docs (note that this link may not be valid to the version we are currently using!).

Adding a new Helper

The best way to learn would be to look at the existing code! Helpers are located in the helpers/ directory and should be named "*Helper.java". Be sure to:

• Set the default constructor to private access (since helpers are expected to be used statically, we don't want them being instantiated!)
• (Optional) Initialize any state in a static init(UITestContext) method and call this from UITest.initHelpers()

Adding a new Component

The best way to learn is to look at the existing code! Components are located in the components/ directory. Be sure to:

• Extend BaseComponent
• Create constructors which take in UITestContext and pass this to super class
• "return this;" from all assertion and actions methods to allow chaining of function calls
• Add your component as a member var to UITest and instantiate it in UITest.initComponents
• Add the enum name to the UITestContext.ComponentType enum
• Return your component from UITest.getComponent
• Order the methods in your files as such:
  • constructors
  • assertions (alphabetize)
  • getViews (alphabetize)
  • actions (generally put public methods towards the top, but use your judgement)

Keep in mind that if some bit of functionality would be useful to all Components, consider adding it to BaseComponent.

Waiting

Most public methods in the framework will require two phases: acting and waiting. For example, when you click on the toolbar, you must wait for the toolbar to fully load and open before returning control to the test suite. If you do not, the test suite may make an assertion about the loaded state (e.g. what is the current toolbar text?) that may not have been loaded, causing race conditions and other hard to diagnose bugs. Any state that is asserted in the test suite should be wait on in the framework!

For example, from components/ToolbarComponent.java (12/2):

public ToolbarComponent enterEditingMode() {
    assertIsNotEditing();

    mSolo.clickOnView(getUrlTitleText(), true);

    waitForEditing();
    WaitHelper.waitFor("UrlEditText to be input method target", new Condition() {
        @Override
        public boolean isSatisfied() {
            return getUrlEditText().isInputMethodTarget();
        }
    });

    return this;
}

Gecko events

Sometimes, waiting on a gecko event is not enough to making assertions about the UI state because these same gecko events are often used by the front-end code to change the UI state, resulting in a race condition over whether the UI state we're asserting changes first or that the assertion runs.

For example, there was an intermittent failure in testAboutHomeVisibility:

NavigationHelper.enterAndLoadUrl(StringHelper.ROBOCOP_BLANK_PAGE_01_URL);
mToolbar.assertTitle(StringHelper.ROBOCOP_BLANK_PAGE_01_TITLE); # Assertion failure: Title was ""

At the end of NavigationHelper.enterAndLoadUrl, the WaitHelper.waitForPageLoad method is called. waitForPageLoad was originally written to wait on the "DOMTitleChanged" event (among others), after which it would return control to the test suite.

However, the Tabs class in the main code base also waits for the "DOMTitleChanged" event (after propagating through js) before calling (in Tabs.handleMessage):

tab.updateTitle(message.getString("title"));

Sometimes mToolbar.assertTitle would run first, failing when it received the incorrect title. Other times tab.updateTitle would run first and the test would run successfully.

This was fixed by ensuring the text in the View associated with the title has actually changed before returning control to the test suite (see tests.helpers.WaitHelper.ToolbarTitleTextChangeVerifier).

View caching

Don't cache views unless its absolutely necessary for performance reasons! View references can become invalid as they are added and removed from the View hierarchy and hanging onto an invalid View reference can cause actions and assertions to take place on the wrong View object. This is so important that Google's UI testing framework, Espresso, does not allow you to cache views at all!

Avoid using View indices

The Robotium API allows you to retrieve views by index (see Solo.getView(Class<T>, int)). This behavior is very fragile because the ordering of the Views in the hierarchy may change without changing what is actually visible to the user (this can happen easily when code is refactored). This is so important that Google's Espresso permits only one View to ever be matched and throws an Exception if more than one is matched.

Assert all the things!

Keep the test framework sane! Make assertions about all possible state you are assuming when writing the framework! Examples:

• Assert function arguments are not null (when applicable)
• Before tapping on a View, make sure it's visible first (see AboutHomeComponent.swipe for an example)
• When taking actions on the toolbar, assert that it is (or is not) in editing mode (assert editing mode solely on visible View state, of course!)

Importing Fennec code

Any code imported from Fennec (e.g. "import org.mozilla.gecko.*") must have the "@RobocopTarget" annotation placed on the imported methods, classes, etc. so that ProGuard knows what it can or can't optimize out. The scope of the annotations should be kept minimal (e.g. don't annotate a class when you can annotate a single one of its methods) to allow ProGuard to optimize as much as possible.

Miscellaneous

• The profiles for Robocop tests are generated by automation and thus don't run the typical first-run Fennec startup code path. Beware of errors that can arise from this behavior such as empty or inconsistent databases!