Test Pilot/UX Workflows
Contents
The Test Pilot program ended in January 2019.
To read more, please see: Adios, Amigos. The information below is being retained for posterity. Thanks to all who traveled along with us!
Reviewing and Maintaining Test Pilot
The TL;DR of this section is is this this:
You should be looking at Test Pilot constantly. If something looks wrong FILE AN ISSUE
Engineers rarely nail it 100% when it comes to implementing UI and UX in code on the first try. Also, sometimes you might design something that looks great in Sketch but winds up a bit meh when you see it implemented. As a designer it’s your job to make sure that:
- Your designs look right upon implementation in Test Pilot
- In general, Test Pilot looks and feels really, really good
So that’s the basic thing, but it’s important to understand how Test Pilot engineering works so you can make sure things look and feel right as they’re implemented.
The Test Pilot Engineering Cycle
There’s a Test Pilot Public Meetings Calendar where all of this stuff is outlined (you should be on that now), but here’s an in-depth overview of the engineering cycle:
- Test Pilot code is written and shipped in two week cycles called sprints.
- At the beginning of each sprint the team meets to plan the work in that sprint. During the sprint planning meeting, we look at a list of all open Github issues and move all of the issues we want to resolve into a Sprint Milestone. Some sprints are UX heavy while others tend to focus on bugs, code optimization or other specific tasks like L10N. Pretty much every sprint will include some engineering that modifies site UI in one way or another.
- As engineers close issues in a sprint, their new code is pushed to GitHub and they open something called a *pull request* which just means that their new code is under review by another engineer. Once the new code passes review, it gets merged into the master branch, and the pull request and underlying issue are marked as *closed* in GitHub. Also, the Test Pilot Dev Server is updated automatically to reflect all changes to the Test Pilot master branch.
- Sprints end on the second Thursday after they start. When the sprint ends we *tag* a commit in GitHub. This is simply an annotation that tells us that all commits to master made before this point constitute the work completed in the sprint and will eventually be deployed into production. All work done after this time will be included in the subsequent sprint.
- Once we tag a sprint as complete, we deploy the work from that sprint into the Test Pilot Staging Server. The staging server is used by QA to review and verify all changes made in a sprint. This verification process goes from Thursday to the following Tuesday. While the staging survey is under review, work begins on the next two-week sprint.
- Finally, once QA has verified everything on the staging server, that sprint is then deployed in Production and released into the world.
The Four Test Pilot Environments
So there are four places where you can see test Pilot:
- Production - This is the live site that the world sees
- Staging - This is the version of the site used by QA to verify changes in each sprint before a production deployment
- Dev - This is the version of the site that is constantly updated to match the latest commit to master
- Local - You can pull down Test Pilot and run it locally if you like. It’s actually pretty darn easy!
Running Test Pilot Locally and Switching Contexts
Running Test Pilot locally is a good idea, particularly if you want to mess around with how the site looks (through CSS, changing copy, changing or adding experiments, or doing things like graduating experiments to see how that looks). Getting everything up and running is relatively easy, and you should be able by doing the following:
- Install the Current version of Node: Download Node.js
- Follow the Development Quickstart guideQuickstart guide
- Build and Install Ground Contol for fast environment switching (so you can go between local, dev, stage and prod)
If you have trouble with these steps contact me and I’ll help you out!
The User Experience Job In Detail
Like I said at the beginning, it’s your job to look at Test Pilot and file Github issues if you see something wrong. In your issues you should be clear, and precise. Adding images showing the problem helps as does listing steps to reproduce the issue, or providing a comparison between the current bad thing and the thing you expect to be there. Engineers don’t read minds so the more precise you are here the better.
Taking this core job a step further, what you should really be doing is paying attention to ongoing work in the current sprint and ensuring that things look like they should as they’re being built. In general this will mean keeping an eye on the Test Pilot dev server (which shows the latest master branch), but you can also get involved by reviewing Pull Requests in Github. For example, don’t be afraid to ask engineers for screenshots of their work, so you can give them pointers about things they may have missed while things are still being worked on. If you see a problem with a pull request while it’s under review SAY SOMETHING IN THE PULL REQUEST If you see an issue with UI on the Dev server FILE AN ISSUE.
