QA/ContainersTestPilot

Summary & Purpose:

The following document describes the plan for testing the containers feature using the test pilot implementation which can be found here. The goal of this project is to allow users to separate their contexts while browsing the web on Firefox. Each context will have its own local state which is separated from the state of other contexts.

Currently, the "platform" implementation of containers which the test pilot is built on top off is only enabled under Nightly by default:

• FX Nightly - Enabled by default (privacy.userContext.enabled;true)
• FX BETA - Disabled by default (privacy.userContext.enabled;false)
• FX Release - Disabled by default (privacy.userContext.enabled;false)

Installing the test pilot under Release or BETA will enable the containers feature via privacy.userContext.enabled while installing the experiment.

Firefox Builds & Platforms

The following test plan should be tested against all three platforms if time permits:

• Windows 10
  • all three of the latest fx versions should be tested (m-c, m-b, m-r)
• Linux (Ubuntu)
  • all three of the latest fx versions should be tested (m-c, m-b, m-r)
• macOS
  • all three of the latest fx versions should be tested (m-c, m-b, m-r)

Using unsigned XPI's

Because Release/BETA doesn't allow users to install unsigned XPI's, you'll need to use unbranded versions of both Release/BETA. You can find the instructions here.

Once you've downloaded and installed the unbranded version of either Release/BETA, disable the following pref before attempting to install the XPI:

• xpinstall.signatures.required;false

Container Prefs

Note: Some of these prefs don't apply to the containers test pilot experiment but I've added them here for clarity.

• privacy.userContext.enabled
  • enables the containers feature
• privacy.userContext.longPressBehavior
  • enables the "long press" access point via the "+" new tab button (See Bug#1272256)
• privacy.userContext.ui.enabled
  • enables access to about:preferences#containers via about:preferences#privacy
  • Note: This is disabled in the test pilot and not used.. Users shouldn't see an access point via about:preferences#privacy (See Isse#114)
• privacy.usercontext.about_newtab_segregation.enabled
  • Using private containers for thumbnail loads (See Bug#1310112)

Checking Cookies [Important]

Because cookies are segregated per container, it's important to check and ensure that cookies are being placed into the correct containers after loading websites. This can be done using the following two methods:

First Method

This is the simplest method which requires you to simply open about:preferences#privacy and manually go through each cookie via the "Cookie Manager" and ensure that it's placed into the correct container/context. Quick Example:

Opening Facebook in the personal container should place all the cookies from FB into the "Personal" container is demonstrated by the screenshot below. Notice the "Container: Personal" at the bottom of the cookies properties.

Second Method

The second method requires opening the cookies.sqlite database that's in the users profile and ensuring that the correct userContextId is being used. Using the same example as above, the FB cookies will have a userContextId=1 which basically means the "Personal" container, example:

You'll notice that the cookies that were created when loading FB all have the correct userContextId associated with them.

Creating a manual Test Pilot XPI

For more more information, you can visit the Container github page and read through the README.

Prerequisites:

• node.js and npm installed.... Instructions can be found here.

If there's no available XPI's for the new version of the test pilot, you'll need to create an XPI from the latest master branch using the following steps:

• clone the latest master branch with git clone https://github.com/mozilla/testpilot-containers.git
• once cloned, move into the testpilot-containers directory via cd testpilot-containers/

If the version number of the test pilot experiment hasn't been bummed to the next version, you'll need to manually do it via the following:

• within the testpilot-containers directory, open package.json within a text editor
• you'll need to adjust the version number next to "version":
  • example: in this case, I would change "version": "2.3.0" into "version": "2.4.0"

Once you've changed the version number under package.json, you can continue with building the XPI using the following:

• Within the testpilot-containers directory, type in npm install
• Once npm install completes, create the XPI using jpm xpi

You should see similar output within the terminal/console:

 kjozwiak:testpilot-containers kjozwiak$ jpm xpi
 JPM [info] Starting jpm xpi on Containers Experiment
 JPM [info] Successfully created XPI at /Users/kjozwiak/projects/testpilot-containers/testpilot-containers.xpi (1171ms)

Creating a manual Shield Study XPI

For more more information, you can visit the Container github page and read through the README.

Prerequisites:

• node.js and npm installed.... Instructions can be found here.

If there's no available XPI's for the new version of the test pilot, you'll need to create an XPI from the latest master branch using the following steps:

• clone the latest master branch with git clone https://github.com/mozilla/testpilot-containers.git
• once cloned, move into the testpilot-containers directory via cd testpilot-containers/

If the version number of the shield study experiment hasn't been bummed to the next version, you'll need to manually do it via the following:

• within the testpilot-containers directory, open package.json within a text editor
• you'll need to adjust the version number next to "version":
  • example: in this case, I would change "version": "2.3.0" into "version": "2.4.0"

Once you've changed the version number under package.json, you can continue with building the XPI using the following:

• Within the testpilot-containers directory, type in npm install
• Once npm install completes, run npm install -g shield-study-cli
• create the XPI using npm run build-shield

You should get a similar output as shown here.

Test Cases

After initial Installation

• ensure that the "Container" button appears under the toolbar
• ensure that the containers experiment appears correctly under about:addons -> Extensions
• ensured the correct version appears in about:support under "Extensions"
• ensure that privacy.userContext.ui.enabled;false under about:config
• ensure that the "NEW" label appears over the Container button under the toolbar once a new version is installed/updated
• ensure that the "NEW" label disappears once a user clicks on the "Container" button under the toolbar
• ensure that the "NEW" label doesn't appear the second time around when users completely clear their history/cookies
• ensure that the user receives the onboarding panels when initially clicking on the "Container" button under the toolbar
  • "a better way to manage all the things you do online"
  • "Put containers to work for you"
  • "A place for everything, and everything in it's place."
  • "Always open sites in the containers you want"
  • (2.4.0+) "Long-press the New Tab button to create a new container tab."
• ensure that the following default containers are being listed:
  • Personal (Blue Thumbprint)
  • Work (Orange Briefcase)
  • Baking (Green Dollar Sign)
  • Shopping (Pink Shopping Cart)

Access Points

• ensure that you can view/use and access all your current containers via the following access points:
  • (<= 2.3.0) hovering over the "+" new tab button
  • (2.4.0+) long-pressing the "+" new tab button
  • clicking on "File -> New Container Tab"
  • accessing containers via the right click context menu -> Open Link in New Container Tab
  • clicking on the "Containers" button under the toolbar

Creating new Containers

• ensure that you can create new containers via the "+" button under the containers panel
• ensure that all the images and colours can be used when creating new containers
• ensure that the new containers are correctly appearing under all the access points including images/colours
• create 10-15 random containers and ensure that you can open all of them without any issues
  • ensure that the correct container is opened
  • ensure that there's no jank/unexpected results when scrolling through the list of containers

Opening Containers

• ensure that you can open new containers via all available access points
• ensure that a new tab is launched with the correct UI
  • the correct icon/image is being used
  • the correct underline colour is being used
• ensure that hovering over each container under the "Container" panel shows a "+" button

Editing Tabs

• ensure you can edit container names and icons via "Edit Containers" under the Containers panel button
• ensured that pressing on the "<" button correctly applies all the changes
• ensure that renaming/editing a container correctly propagates all the changes to all tabs that are currently opened and belong to that specific container
• ensure that the changes correctly propagate to all the access points
• ensure that there's no conflicts if the same name/images are used for multiple containers
  • example: creating several containers that are using the same name/image/colour

Deleting Containers

• ensure you can delete containers via "Edit Containers" under the Containers panel button
• ensure that deleting a container that has several tabs opened removes/closes those tabs while the container is being deleted
• ensure that users get a warning when clicking on the delete button
  • clicking on "Cancel" should cancel the action and not remove the container
  • pressing "ESC" should cancel the action and not remove the container
  • clicking on "OK" should confirm the action and remove the container
• ensure that all the cookies associated with the removed container are also deleted (check using methods outlined here)
• ensure that the container as been removed from all available access points

Private Browsing Mode

• ensure that users can't use the Containers feature in PB mode, this includes:
  • all access points should be disabled and not accessible
  • users shouldn't be allowed to assign specific websites via "Always Open in This Container" under PB mode
  • users should be able to access all websites under PB that have been assigned to specific containers via "Always Open in This Container"
    • example: assign google.com to the work container... Opening google.com under PB shouldn't be using the work container
  • users shouldn't be allowed to drag and drop container windows into a PB window
• ensure that cookies aren't being stored for websites being opened in PB
  • you can check in either the "Cookie Manager" under about:preferences#privacy or the cookies.sqlite database under the profile folder

Session Restore

• ensure container tabs are restored when using about:home -> Restore Previous Session
• ensure container tabs are restored when using about:preferences -> Show your window and tabs from last time
• ensure container tabs are restored when using History -> Restore Previous Session
• ensure hidden container tabs are restored and NOT removed when using about:preferences -> Show your window and tabs from last time

When restoring tabs, ensure:

• logged in sessions are not being lost
• the correct UI is still being used (names, images, colours and underlines)
• ensure you can restore multiple container tabs at the same time without any conflicts/issues
• ensure websites are restored in the correct containers
  • example: two tabs opened (FB inside Personal) (LinkedIn inside Work), restore and check the cookies to make sure they're still correctly being segregated
• ensure there's no issues restoring regular non-container tabs (try restoring a combination of both non-container and container tabs)

Assigning Websites to Specific Containers

• ensure that "Always Open in This Container" doesn't appear under the right click context menu for non-containers
• ensure that "Always Open in This Container" appears under the right click context menu for containers

• ensure that assigning websites using "Always Open in This Container" via the following access points displays the "Successfully added site to always open in this container" notification bar:
  • assigning via the right click context menu
  • assigning via the "Always open in [XXXX]" under the Container panel

• ensure that deselecting "Always Open in This Container" via the following access points displays the "Successfully removed site to always open in this container" notification bar:
  • deselecting via the right click context menu
  • deselecting via the "Always open in [XXXX]" under the Container panel
  • deselecting by removing the websites assignment via "Edit Containers"

• ensure that opening a website that's been assigned to a particular container in either a non-container or a different container warns the user with the following tab:

• ensure that the above warning tab isn't displayed once a user selects "Remember my decision for this site"
  • if the user selects "Open in [XXXX] Container", ensure that the website is always opened in that specific container
  • if the user selects "Open in Current Container", ensure that the website is always opened in that specific container
• ensure that you can remove website assignments via the following access points: (once unassigned, ensure it opens in the correct tab)
  • deselecting via the right click context menu
  • deselecting via the "Always open in [XXXX]" under the Container panel
  • deselecting by removing the websites assignment via "Edit Containers"

Sorting Tabs

• ensure that the "Sort Tabs" feature under the container panel correctly sorts all of the tabs that are currently opened in the specific window
• ensure that sorting doesn't cause any issues in terms of closing tabs, terminating sessions
• ensure that sorting doesn't cause significant jank/performance issues
• ensure that sorting only affects the current window and doesn't sort tabs in other windows
• ensure that sorting also applies to default non-container tabs

Showing/Hiding Containers

• ensure that hiding a particular container closes all tabs that are associated with that specific container and displays each tab under the Container panel
• ensure that creating a new container tab restores all associated hidden tabs with that specific container
  • example: having all personal containers hidden, creating a new personal tab would restore ALL of the personal tabs that are hidden within that window
• ensure you can restore hidden tabs using "Show this container" via the Container panel
• ensure that you can restore tabs into new windows using "Move tabs into a new window"
• ensure that hiding/restoring tabs doesn't create duplicate entries in the Container panel
• ensure that sessions are not being lost when hiding/restoring several websites that are logged into accounts

Updating from a previous version

Install the latest version of the test pilot found here. Once installed:

• create several new containers
• using several containers, login into several different websites
• assign several different websites to a specific container using "Always Open in This Container"

Note: if you don't have the latest XPI available, you can create your own XPI with the latest master branch using the instructions found here.

Update the test pilot to the newest version and ensure the following:

• doesn't remove users data and cookies
• websites are still associated to specific containers via the "Always Open in This Container"
• displays the "NEW" label on top of the "Containers" button on the toolbar
• ensure that nothing obvious has been broken in terms of UI regressions
• ensure that new features are working as expected after the update

Removing/Deleting the Containers Test Pilot

• ensure that all the container cookies have been removed from the cookies.sqlite database
  • note: best way of checking this is using method 2 here
• ensure that "privacy.userContext.enabled;false" under about:config
  • IMPORTANT NOTE: "privacy.userContext.enabled" should be "true" as the platform implementation of containers is enabled by default under nightly
• ensure that the container experiment has been removed from the following locations:
  • about:support
  • about:addons -> Extensions
• ensure that all the all access points are removed

Helpful Container Resources/Links

• Containers (Contextual Identify) Wiki
• Containers Test Pilot GitHub Project
• Containers Test Pilot GitHub Issues
• Previously used Test Cases