QA/BrowserID/TestPlan

From MozillaWiki
< QA‎ | BrowserID
Jump to: navigation, search

Back To ==> Mozilla BT Services

Overview

This test plan covers the general weekly testing that will happen against a built, unit-tested, and deployed BrowserID product in the Stage Test environment. The goal is to ensure a defined/consistent amount of quality and usability in the server-side and RP/client side portions of the BrowserID product. Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product.

Strategy

Create a consistent and repeatable set of tests for qualifying and releasing the weekly builds and deployments for BrowserID. The focus needs to be on quality, but also on providing accurate feedback on successes and issues to Dev and PM so that the Production releases provide a useful product for the Mozilla and development communities.

  • Ad Hoc testing to define test paths on the RP/client and server side.
  • Create useful RP/client-side and server-side test cases that can be imported into Litmus for repeatable manual testing.
  • Create automated RP/client and server tests that can more accurately cover the full functionality of the UI, the API, and the main roles (PIA, RP, IP) and various flows.
  • Emphasis on security and privacy issues that might come up across OS, browsers, accounts, and emails on the various RPs/clients tested against the server.
  • Identifying and tracking issues in GitHub and Bugzilla.
  • Provide useful metrics to Dev and PM.

Scope of Testing

  • RP/Client-side testing will cover the following areas: basic functionality and UI, accounts and emails, interaction with the Server, security/privacy, and usability/compatibility across OS and browsers.
  • Server-side testing will cover the following areas: basic functionality, support for multiple RP/client sites, user security/privacy, communication with various RPs/clients, information handling and storage, security/privacy of information, information persistence across deployments, and logging.
  • Areas not covered
    • Server and RP/Client automation
    • Security: VEP, VES, VEC
    • Flows/End-To-End
    • Additional Stability Testing
    • FireFox Add-On
    • Internationalization/Localization
    • Certificate length and session timeout/length
    • Primary/secondary identifiers
  • You can find notes on these areas at the bottom of the Test Plan

General Test Information

Links and Documentation

Weekly Test Schedules

  • BrowserID testing runs on a weekly "train" or schedule:
    • Wednesday afternoon: deployment to Production, Stage (QA)
    • Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
    • Following week: Monday - Wednesday: QA testing and sign off of current deployment
  • NOTE: Dev deployments are now back in the hands of the Dev team, so they can control incremental builds and releases.
  • NOTE 2: OPs is working on a Test/CI environment that will be deployed nightly for Dev and QA testing purposes (automation, etc.).

Weekly Meetings

Email and IRC

  • Key players in IRC: #identity
    • lloyd = Lloyd Hilael developer
    • stomlinson = Shane Tomlinson developer
    • benadida = Ben Adida developer
    • thunder = Dan Mills PM
    • gene = Gene Wood OPs
    • jbonacci = James Bonacci QA
    • jrgm = John Morrison QA
    • ozten = Austin King developer

Bugs and Open Issues

  • Security, OPs, and Apps-related bugs in Bugzilla
    • Classification: Client Software
    • Component: Server: Identity
    • Product: Mozilla Services
  • Deployment Requests in Bugzilla
    • Classification: Client Software
    • Product: Mozilla Services
    • Component: Operations: Deployment Requests

Quick Architectural Overview of Stage

  • The Stage environment is a set of physical Linux systems that run the various BrowserID services/processes:
    • Webheads: These servers run the main browserid services and the verifier services.
    • Secure Webheads: These servers run the dbwriter services.
    • Keysigners: These servers run the keysigner services.
    • Databases: These servers do all the MySQL work.
    • Zeus: the front end of the Stage cluster, handles the load balancing of all the requests.

Specific RP/Client Features

  • TBD based on updated content and UI flows for http://123done.org, including session length for personal vs. shared computers.

RP/Client and Server Test Environments

Supported OS and Browsers

Stage Logging

  • Log file locations on all services except Zeus:
    • /var/log/browserid -> /var/browserid/log
  • Log file locations on Zeus:
    • /var/log/zeus
  • webheads: /var/log/browserid
    • browserid.log
    • router.log
    • verifier.log
    • verifier-metrics.json
    • verifier-compute.log
    • static.log
    • router-metrics.json
Note: router-metrics.json replaces browserid-metrics.json
  • secure webheads: /var/log/browserid/
    • dbwriter.log
  • keysigners have /var/log/browserid/
    • keysigner.log
    • keysigner-compute.log
  • dbs: various db logs
  • zeus: various zeus logs
  • Note: the browserid-related logs currently are not being rotated or archived in Dev or Stage, but they are in Production.

Major Test Areas

Code Verification

Verify fixes/code changes by direct inspection (as required)

  • Or, you can verify fixes by checking the commits listed in the Closed issues.
  • Note: This is only for cases where QA has no direct/easy way to verify the fix in using BrowserID.

Installation/Deployment Verification

Verify the deployment version, processes, and logging in Stage.

  • Stage Processes
    • webheads:
      • node bin/browserid
      • node bin/verifier
      • node bin/router
      • node bin/static
      • /usr/bin/node /opt/browserid/lib/verifier/verifier-compute.js
      • /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
    • secure webheads:
      • node bin/dbwriter
      • /usr/bin/node /opt/browserid/lib/bcrypt-compute.js
    • keysigners:
      • node bin/keysigner
      • /usr/bin/node /opt/browserid/lib/keysigner/keysigner-compute.js
    • databases:
      • /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql --pid-file=/var/run/mysqld/mysqld.pid
      • /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql --user=mysql --log-error=/data/mysql/db1.iddb.scl2.stage.svc.mozilla.com.err --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
      • perl /usr/bin/pt-heartbeat --create-table -D browserid -u root --update
    • Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify the processes.
  • Verify Stage logging is active
    • See Stage Logging section above.
    • Note: This assumes access to the main Stage Admin server. If you do not have SSH access, then get QA staff or Dev to verify that the logs are active.

Local Installs

  • Verify at least one local install to Mac or Linux
  • Verify at least one local install with locale support to Mac or Linux
  • Run the unit tests on one local install (see next section)
  • See the following wiki for more information on basic/local installs:

Unit Testing

  • Dev has created three sets of unit tests
    • Front-End Unit Tests - run in any browser
    • Back-End Unit Tests - run from the BrowserID install directory
    • "Headless" Front-End Unit Tests - run without a browser, from the BrowserID install directory
  • The QA goal is to run these unit tests in a local install and report results.
  • This testing, along with the Sanity/Acceptance/Smoke testing below, will be conditions for entering full Stage cycle testing.

Automation Testing

  • There are two main ways to run automation testing:
    • Directly through the Selenium/Jenkins/Sauce Labs automation environment
    • Locally using a stand-alone Selenium server and the built-in automation tests

Sanity/Acceptance/Smoke

  • Small, repeatable set of tests with known, good, expected results.
  • Manual and automated testing on client and server to pass a minimal level of acceptance without which, QA testing of BrowserID can not proceed.
  • See the Test Cases (listed in the "Links and Documentation" section above) for more information.
  • Client-side (http://myfavoritebeer.org or http://123done.org using one desktop OS with one browser)
    • Sign In using one RP/client to create a new account with one email
      • Verify correct email verification sent to correct account (email provider)
    • Logout/Sign In/Logout using the same account from a different RP/client
    • Sign In and add one BID email to the same account from a different RP/client
      • Verify correct email verification sent to correct account (email provider)
    • Sign In and add one primary email to the same account
    • Sign In and change the password on the same account
      • Verify correct email verification sent to correct account (email provider)
    • Sign In to the same account and "sign out" by using the This is not me... link
    • Delete/Remove the account from the server-side
  • Client-side (http://beta.myfavoritebeer.org or http://beta.123done.org using Android with stock browser)
    • Sign In using one RP/client to create a new account with one email
      • Verify correct email verification sent to correct account (email provider)
    • Logout/Sign In/Logout using the same account from a different RP/client
    • Sign In and add one BID email to the same account from a different RP/client
      • Verify correct email verification sent to correct account (email provider)
    • Sign In and add one primary email to the same account
    • Sign In and change the password on the same account
      • Verify correct email verification sent to correct account (email provider)
    • Sign In to the same account and "sign out" by using the This is not me... link
    • Delete/Remove the account from the server-side

Bug Verification

  • Manual testing of bugs/issues resolved for this weekly cycle of testing.
  • Dev will list all resolved/closed issues in the deployment ticket for the weekly cycle of testing.
  • Test cases generated during this testing can be moved to an automation tool for bug regression (see below).
    • Note in GitHub all the issues marked with the "needs testcase" label

Basic Functional

Manual and automated testing on the client and the server to verify basic functionality of BrowserID:

  • Accounts and Email Verification
    • Creating an account from the server site using Sign Up
    • Creating an account inline (at first use of an RP/client Sign In) AUTO
    • Email notifications for new accounts: verification email through email provider with proper email account listed, live verification link, etc. AUTO
      • Test with emails/accounts on various, popular email servers/services
      • See more details in the next bulleted list
    • Creating multiple accounts with one or more emails
    • Deleting one or more accounts (cancellation) from the server site
  • Email Notifications for server or RP/Client
    • On the same OS
      • Email notification and verification using one browser AUTO
      • Email notification on one browser, verification on an another
    • Across multiple OS
      • Email notification on one specific browser on one specific OS
      • Verification on the same browser on a different OS
    • Other cases
      • Email notification on one specific browser on one specific OS
      • Verification on a different browser on a different OS
    • Mail Servers
      • Check functionality when the user does not verify by email (skips, forgets)
      • Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
    • BrowserID Server site
      • Check functionality when BrowserID server is unavailable (down or user is off the net)
      • Check functionality when BrowserID server is available but user has slow connection (like a public wifi)
    • Stage server logs (if you have access)
  • Accounts and Emails
    • Adding additional emails to an account AUTO
    • Attempt to add an email (that may or may not be yours) from another account
    • Deleting one or more emails (without actually deleting the account)
    • Leaving/returning to sites (while signed in, after signing out)
    • Browser restart after creation of account or access of an account
    • Always logging out from sites vs. never logging out from sites (session timeouts)
    • Shared access to same computer or profiles or accounts with different users
    • Browser settings and preferences, esp. pop-ups, cookies, security, privacy
    • Cancelling accounts
    • Copy/Pasting emails (names) or passwords from other sources
    • Auto-completion of emails and passwords
    • Merging one or more accounts (that may or may not be yours)
  • Email and Password fields
    • Email strings/types/limits
    • Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
    • Password strings/types/limits
  • More on Emails and Passwords
    • Use of passwords (strict) vs. pass phrases
    • Verify minimum/maximum sizes of emails and passwords (length)
    • Password reset, password remember/restore
    • Unique/unusual/edge case emails and passwords
    • Email and Password character compatibility
    • Valid vs. invalid email formats
    • Valid vs. invalid password formats
    • Different accounts using same email/password combos
    • Try to setup a new account with a password and/or email already in use
    • Copy/Pasting passwords from other sources
    • Verify that passwords are never stored in LocalStore on the user machine
    • Verify whether or not passwords are stored client-side (Stage RP)
    • Verify whether or not passwords are stored on the server (Stage server)
    • Verify proper formatting with very long emails and/or passwords.
  • Cross site activity
    • Creating an account on one RP/client, verifying the account/email on another RP/client
    • Adding an email on one RP/client while logged into another RP/client
    • Deleting an email on one while logged into other RPs/clients
    • Cancel account/delete email from the server while signed into an RP/client
    • Deleting the whole account while logged into one or more RPs/clients
      • This must be done from the server
    • Verify how account information on the server reflects the changes for each of these tests
  • Other Areas
    • Verify that the user cannot log in with an email if he/she did not confirm the used email
    • Log in simultaneously in two different browsers with the same email, then log out from one of the two browsers
    • Log in with different emails for different clients in the same browser/different browser
    • Log in with the same email for different clients in the same browser, then log out from one of the browsers
    • Verify that the log in is kept when restoring a session after a browser crash (or after closing a browser with the session restore option selected (is this Firefox only?)

Support for Primaries

  • Verify basic Primary support through the use of one or more test Primary sites
  • Primary: https://eyedee.me/
  • Other sites: TBD
  • Links
  • Verify all links on the Primary, especially if they open a new page/tab or replace the current page/tab
  • Email and Password fields
    • Email strings/types/limits
    • Verify all legal combinations of characters for both "local name" and "domain name" parts of the email string
    • Password strings/types/limits
  • Direct Primary usage
    • Signing in from Primary
    • Sign-in redirection to Primary from an RP/client
    • Sign-out from Primary
    • Sign-out from Primary while signed into an RP/client
    • Sign-out from Primary while signed into the Server site
  • BrowserID tests (from Server or from RPs)
    • Creating an account with Primary emails
    • Creating an account with mixed emails (Primary/BID)
    • Adding/Deleting a Primary email from a Primary account
    • Adding/Deleting a BID email from a Primary account
    • Adding/Deleting a Primary email from a BID account
    • Adding/Deleting a BID email from a BID account
    • Deleting an account with Primary emails
    • Deleting an account with mixed emails (Primary/BID)
    • Account Manager password changes on accounts with both primary and secondary emails
  • Network interruptions: latency or Primary site being down
  • Primary site UI flow
    • General tests for navigating the site
    • Include failures, cancellations, backing out
  • Local Installs: run the check_primary_support script on eyedee.me, other Primaries, and at least one non-primary
  • Regression: attempt Primary use with and without third-party cookie support set in the browser

Specific RP/Client Functionality

  • http://beta.myfavoritebeer.org
    • Verify that this site allows for standard authentication, sign in, log out, password changes, and email additions to a new or current account.
    • Verify different sign-in/verification paths for Primary emails vs. BID emails.
    • Regression: attempt RP Sign In use with and without third-party cookie support set in the browser

UI

Manual and automated testing on the RP/client and the BID server to cover all aspects of the current UI.

  • Sign-in UI
  • Email field
  • Password field
  • "forgot your password?" link
  • "learn more" link
  • "This is not me" link
  • "Use a different email" link
  • "Terms of Service" and "Privacy" links off of RP sites
  • Account management: https://www.diresworb.org Account Manager page
  • Sign Up
  • Sign In
  • Edit button
  • Remove button
  • Password button
  • "cancel your account" link
  • UI for new user
    • Email field
    • Verify button
    • Verification email
    • Verification link
    • Password
    • Email selection - radio button vs. email string
  • Confirm your Email UI - verifying accurate "prove" link
    • Confirm email verification from client-side and server-side, as defined
  • Confirm Email verification UI
  • UI to Add new emails ("Use a different email") while still logged into RP/client
  • UI to Change a password ("Forgot your password?") on the RP/client
  • UI to select Terms of Services, Privacy, Learn More on the RP/client
  • UI to Edit the account on the Server from the Account Manager
  • UI to Remove an email on the Server from the Account Manager
  • UI to Change the Password on the Server from the Account Manager
    • Verify functionality with only BID emails
    • Verify functionality with only Primary emails
    • Verify functionality with a mix of email types
  • UI to Cancel an account (all emails) on the Server
  • General UI navigation to cover the following:
    • Error screens and dialogs
    • Email verification pop-ups and in browser (the fade/change to a server page)
    • Closing pop-ups manually rather than with a Cancel, OK, Continue, or other button
  • Various UI scenarios and navigation not covered above...
    • The affects of Stay logged in vs. always logging out
  • General ease of use
    • Mouse/pointer: left (click), right/ctrl click, hover over links, etc.
    • Keyboard: selection, tabbing, arrows, etc.
    • Mobile: touchscreen, keyboard, etc.

Tool Tips, Messages, and Errors

  • Verify tool tips for all text fields and buttons (if supported).
  • Verify all error messages for various normal activities with BrowserID
  • Verify all error messages for various browser-level activities (browser reset, auto updates, reset of prefs, cookie deletion, etc.)
  • Verify all error messages for various system-level activities (network issues primarily)
  • Verify all error messages/pop-ups for disabled main/third-party cookies

Localization

  • Verify the UI flow with specific focus on localization, all text-based content
    • Page content, layout, and strings
    • Pop-up content, layout, and strings
    • Email content, layout, and strings
    • Messages, tool tips, errors
      • Errors: will need to create/force error situations in order to verify localization of error strings
  • Setting locale in a browser
    • Firefox: Preferences > Content > Languages
    • Safari: Apple menu > System Preferences, and then click Language & Text.
      • Open Language & Text preferences
      • Drag your preferred language to the top of the Languages list.
      • If Safari is open, quit it, and then open it again.
      • NOTE: In the Language tab, you will need to click on "Edit List..." and add any special or test languages as a selectable language.
    • Chrome: Tools > Preferences > Under the Hood > Web Content > Languages and Spell-Checker Settings
    • Opera: Preferences > General > Language
      • Apparently can only pick through languages here: /Applications/Opera.app/Contents/Resources/
    • IE8: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
      • You can also create a User-Defined language by adding the language code directly
    • IE9: Tools > Internet Options > General > Appearance > Languages > Language Reference > Add...
      • You can also create a User-Defined language by adding the language code directly
    • iOS/Safari: TBD
      • Note: This will change the locale (at least temporarily) for all apps on iPhone and/or iPad
  • OS-specific changes, if any:
    • Ex: Rebooting to clear cache, data, cookies, etc.
    • Ex: text/font encodings
    • Custom language entries for browsers

Accessibility - Desktop only

  • Verify minimal accessibility in the UI - keyboard only
    • Creating a new BrowserID account
      • Email verification from mail application and from BrowserID
    • Adding a new email to a current BrowserID account
      • Email verification from mail application and from BrowserID
    • Changing/resetting a password - RP flow
      • Email verification from mail application and from BrowserID
    • Changing a password - Account Manager
    • Deleting an email from an account - Account Manager
    • Deleting an account - Account Manager
    • Other UI flows: This is not me, BrowserID links, etc.
  • Note: this testing may require changes at the OS-level (like Mac OS) or changes at the browser level to more fully support keyboard-only access to a site.

Bug Regression

  • Ideally, this regression would be in the form of a quick automated test/script to check key fixes/changes/updates.
  • QA to work with Dev and PM to come up with a list of bugs/issues candidates to from the test cases for manual and automated testing.
  • Tagging Regressions
    • If an old issue is found to exist in the code, then file a new issue and tag it with [regression] in the Title
    • Give that regression issue a 5-star label

OS/Browser integration

Manual testing/verification of the required OS and browser configurations:

  • Verify access and use across browsers on same OS, different OS
  • Verify access and use across OS platforms, same browser, different browser
  • This includes account creation/update/deletion and email addition/deletion
  • OS/Browser-specific local storage verification
  • Browser preferences, esp for privacy/security
  • Browser synchronization - same platform, across platforms
  • Verify RP and main site access
    • With and without third-party cookies enabled
    • With and without all cookies enabled
  • Verify cached headers, status, content across browsers
  • Verify browser cookie information/inspection (per browser)
    • Inspect the client and server browser cookie information for BrowserID specific information
    • Privacy settings in browsers: start with 3rd-party cookies due to issues with eyedee.me
  • Verify door hanger information (per browser, desktop and mobile)
    • Inspect/verify the doorhanger information and links (if any)
  • On the FireFox browser, various tools can be used to help test/verify BrowserID:
    • Tools > Web Developer > Web Console: can be used to track live GET and POST messages from the RP (client) site.
    • Tools > Web Developer > Error Console: to track any errors, warnings, or messages from the RP (client) site.
    • From the mobile version: TBD
  • On the Safari browser:
    • Preferences > Advanced > "Show Develop menu in menu bar"
    • Now, reference all tools through the Develop drop-down menu"
      • Error Console
    • From the mobile version: Settings > Safari > Advanced > Debug Console > On
  • On the Google Chrome browser:
    • Tool icon > Tools > Developer Tools
    • View > Developer > Developer Tools
  • On the Opera browser:
    • View > Developer Tools > Error Console
  • On the IE8/IE9 browsers:
    • F12 or Developer Tools
  • On the Android Stock browser: TBD

BrowserID Logs/Messages

  • Verify that the main BrowserID logs contain meaningful information for inactive and active messaging:
    • browserid.log
      • QA is most concerned about any warnings or errors that come up during testing, and any debug messages that do not belong (talk to Dev about list of current debug messages)
    • verifier.log
      • QA should not see any debug, warn, or error messages in this log (verify with Dev)
    • keysigner.log
      • QA should not see any debug, warn, or error messages in this log (verify with Dev)
    • dbwriter.log
      • QA should not see any debug, warn, or error messages in this log (verify with Dev)
    • All *-error.log files should be empty. If they are not, this could point to an issue requiring debug and Dev/OPs interaction.
    • NOTE: archived logs (in gzip format) may need to be reviewed if testing is generating a lot of log activity or if issues need to be debugged over two or more log rotations.

Mobile Testing

Mobile-specific tests should be added to verify compatibility/stability/functionality of BrowserID across Android and iOS tablets and phones

  • Complete the Sanity/Acceptance section for phone, for tablet
  • Complete the Basic Functional and UI sections on phone, on tablet
    • Panning, zooming, scrolling, tab access and flow between tabs
  • Complete the Specific RP/Client tests on phone, tablet
  • OS/Browser interaction using FireFox and stock browser on Android phones and tablets
  • OS/Browser interaction using Safari browser on iPhones and iPads
  • Compatibility - review actions and processes that are unique to mobile devices
  • Verify compatibility and functionality when working with Android/iOS Settings and with browser app-specific settings.
  • Most of the testing will be based on the final matrix located here:

Network/Internet

Verify the impact of various ways to access the internet: timing, communication, messaging, outages, errors, traffic, etc.

  • Ethernet - personal vs. office, with and without VPN
  • Public WiFi
  • Private WiFi and other home setups
  • 3g, 4g

Client-side LocalStorage

  • Most, if not all, of the BrowserID-specific information is now kept on the server.
  • But, LocalStorage should still be examined for any unexpected changes based on BrowserID use.
  • Work with Dev to come up with meaningful tests to verify BrowserID's use of, and impact on, LocalStorage.
  • Example issue: 688: IE8 blowing up trying to do "delete" on localStorage members.
  • All session data is stored in the cookie (per Ben Adida).

Server-side Testing

  • Verify some UI elements and actions only found on the server
    • See UI section above.
  • Log inspection to verify accurate information
    • See the Logging section above for more details
  • Load Testing and Monitoring
    • See OPs section below for more information
  • API automation (if applicable/useful)
  • Unit Tests (work with Dev)
  • Headless frontend testing (work with Dev)
  • QA should look into headless (no browser/client) testing of server
  • Server/service scenarios - failover/recover, temp outage, traffic flow, etc.
    • Zeus/load balancer issues
    • webhead for browserid, verifier, router, and static
    • swebhead for dbwriter
    • keysigner for keysigner
    • mysql/db

Load Testing

  • Load Testing in Stage is handled by QA using BrowserID code provided by Dev.
  • QA should monitor all load tests
  • QA should also schedule specific load tests to cover various requirements for server-side testing.
    • Work with OPs to queue up specific load tests to check Server stability, various metrics
  • Monitor load tests with OPs dashboards - e.g., http://pencil.scl2.stage.svc.mozilla.com/dash/stage/bid_webhead (requires access to the MPT VPN)
  • Analyze text and visual data for functionality and peformance
An example command line to run the load test:

    nohup bin/load_gen -s https://verifier.login.anosrep.org -o -m 250000 -u 1/250

-s  -- the URL to the browserid server
-o  -- don't request static resources
-m  -- maximum active users to simulate (ADU)
-u  -- use this range of #@loadtest.domain users (these users already exist in the 
       stage database. If running your own instance, do `CREATE_TEST_USERS=1000 
       BROWSERID_FAKE_VERIFICATION=1 NODE_ENV=test_mysql npm start`).

I typically run the above command piped into this:  

    | /usr/local/bin/tai64n | /usr/local/bin/tai64nlocal \
      >> /path/to/load_gen.log-`date +%Y%m%d-%H%M` 2>&1 &

which will timestamp the output and save it in a logfile, etc.

QA Sign-Off for Stage

  • Requirements and procedures:
    • Complete all required testing for the current weekly train: resolved/closed issues, suggested areas for QA focus, specific features and areas of test coverage, automation, etc.
    • Update the deployment ticket with a summary of the completed testing, plus the following: issues verified, issues opened, issues reopened, any testing marked incomplete, delayed, or removed from this release, etc.
    • Add a sign-off to the deployment ticket if this weekly release passes QA
    • Add additional comments (without a sign-off) if this weekly release fails QA

QA Testing for Production Pushes

This is a work in progress!...

  • Verify push to PHX1 (/etc/hosts hack)
    • Standard sanity testing across platforms, OS, and browsers
  • Verify push to SCL2 (/etc/hosts hack)
    • Standard sanity testing across platforms, OS, and browsers
  • Locale verification
    • Based on final pull request for config/l10n-prod.json
    • Quick sanity test of new/changed/removed locales
  • Check the verifier service before OPs undrains a datacenter during a push (for PHX1 and SCL2)
    • REF: Bug 753728 - browserid.org/verify is on occasion throwing 500 errors (being caught on the load balancer)
    • REF: Bug 753828 - monitor the state of zeus pools

QA Sign-Off for Production

  • TBD: Requirements and procedures:
  • Similar to sign-off for Stage, but applies to both data centers
    • Add the Services Maintenance Window information to the deployment ticket.
    • Complete all required testing for the push to Production.
    • Update the deployment ticket with a summary of the completed testing.
    • Add a sign-off to the deployment ticket if this Production push passes QA.
    • Add additional comments (without a sign-off) if this Production push fails QA.


Back To ==> Mozilla BT Services

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/BrowserID/TestPlan&oldid=672405"