QA/BrowserID/BrowserID Weekly Trains Beta

From MozillaWiki
< QA‎ | BrowserID
Jump to: navigation, search

Back To ==> Mozilla BT Services


  • Legend:
    • RP = Relying Party (think test client)
    • PIA/IP = Primary Identity Authority or Implementation Provider (think test server)
    • Seconary Identity Authority = none yet

Overview and Purpose

  • This wiki is set up to help QA with the background and information necessary to perform weekly testing of BrowserID.
  • Use this wiki alongside the BrowserID Test Plan, automation tools, and other materials for the completion of testing and reporting of results to the BrowserID team.

Main Links

NOTE: This section needs to get updated

QA Test Documents

Schedules

  • BrowserID testing runs on a bi-monthly "train" or schedule:
    • (Day 1 / Day 15) Wednesday afternoon: deployment of current train to Production
    • (Day 2-3) Thursday/Friday: open testing and experimentation on Dev environment
    • (Day 3) Friday afternoon: deployment of next train to Stage
    • (Day 6-10) Following week: Monday - Friday: QA testing
    • (Day 13-14) Third week: Monday - Tuesday: Final QA testing and sign off of current deployment
    • (Day 14) Tuesday : Rebuild next train with new localizations and deploy to Stage one day ahead of the Day 15 production deploy
  • NOTE: Dev deployments are now back in the hands of the Dev team, so they can control incremental builds and releases. Tuesday is also the day to rebuild the rpm with the updated translations for L10N, and a re-deploy of that to stage.
  • NOTE 2: Continuous integration is done with travis-ci.org; commits and pull requests exected the back, back_mysql and front tests, with work underway to execute selenium tests as well.

Weekly Meetings

GitHub Access

Email/Group Access

  • Services QA
    • Sign up for services-qa@mozilla.com (talk to Tony Chung to get added)
    • Use this email list for test and QA-related questions and information.
  • BrowserID/Identity email list
    • Sign up for access to dev-identity: https://lists.mozilla.org/listinfo/dev-identity
    • Use this email list for general questions and issues specific to BrowserID.
    • Email will be sent out to all members of this list including QA, Dev, Security, UI, and PM.

Bugs and Issues

Bugs and Open Issues

  • Security, OPs, and Apps-related bugs in Bugzilla
    • Classification: Client Software
    • Component: Server: Identity
    • Product: Mozilla Services
  • Deployment Requests in Bugzilla
    • Classification: Client Software
    • Product: Mozilla Services
    • Component: Operations: Deployment Requests

Filing Issues Against BrowserID

Since GitHub is being used for recording all bugs/issues, please use the following template to better match the fields we would normally use in Bugzilla:

  • Bug Number: Issue # is generated by GitHub
  • Title: similar as with Bugzilla
  • CC: GitHub will list the creator of the issue and pass it along to everyone associated with BrowserID, so there is no need to CC anybody or email anybody with the issue information.
  • QA Contact: Assume that you, or the QA team as a whole, will be the contact for this issue.
  • Platform information
    • Platform or mobile device used for testing (examples: Mac, PC, phone, tablet)
    • OS version (examples Windows, Mac, Android, iOS)
    • Browser and version, plus any specific Add-Ons you installed
      • (examples: FireFox, Safari, IE, Google Chrome, Opera)
  • Test Environment: Stage, Dev, and/or Production
  • Description
    • Include background on issue and include steps to reproduce.
    • Include information on whether or not this was checked against Dev or Prod environments.
    • Include information on whether or not this was checked against other platforms, OS version, browsers, etc.
  • Additions:
    • For regressions, please put "[regression]" at the beginning of the Title
    • For locale bugs, please put the Locale and country code at the beginning of the Title
      • Example: Italian [it]: I saw this bug...
    • For Dev environment bugs, please add "Dev:" at the beginning of the Title
    • For local install bugs, please add "Local Install:" at the beginning of the Title
    • For load test bugs, please add "load_gen:" at the beginning of the Title

Reporting Results and Status

  • Use MoPad to report both train test results and weekly status for BrowserID:
    • mozbtech-<DATE>
    • (where <DATE> is the current week as set by Browser Technologies QA team)
  • Questions and comments should go to the email lists mentioned above...

QA Test/Automation Environment

NOTE: This section needs to get updated.

  • qa3.mtv1.dev.svc.mozilla.com (IP=10.250.64.149)
    • Note, this VM has not been set up yet, but will include an auto-process for deploying BrowserID builds (all-in-ones) so that it can be used as RP/client and server.
  • Installing a BrowserID Server
    • Used for QA testing, tools development, and automation.
    • This is planned for a QA server HW that has yet to be ordered.
  • Installing a BrowserID Client
    • Used for QA testing, tools development, and automation.
    • This is planned, but not currently scheduled.

BrowserID Test Cases in MozTrap

BrowserID Automation

Dev, Stage, and Production Environments

  • See the BrowserID Test Plan for more information

Browser and Mobile Support

  • See the BrowserID Test Plan and the Support Matrix for more information about mobile devices and required testing.
  • Clearing BrowserID settings
  • Safari:
    • History > Clear History
    • Empty Cache
    • Preferences > Privacy > Remove All Website Data
  • FireFox: Prefs > Privacy > Change History to "Use custom settings for history" > Clear history when Firefox closes > Settings > select all settings
  • Google Chrome: Tool icon > History > Edit items > Clear all browsing data
  • Opera: Preferences > Advanced >
    • History > Addresses > Clear
    • History > Disk Cache > Empty Now
    • Cookies > Delete new cookies when exiting Opera
  • NOTE: Still need to add the steps for mobile browsers on iOS and Android.

Profiles and Accounts: BrowserID

  • Profiles
    • It is highly recommended to create accounts/profiles per OS/platform and per browser to really expand on various testing scenarios.
    • FireFox: use the -ProfileManager option to create specific test profiles
    • Other browsers: TBD
    • Mobile: may not be applicable or necessary
  • Accounts
    • Create unique accounts across OS/platform and browser for specific testing scenarios
    • BrowserID accounts can be created uniquely for each test or can be grouped by OS, by platform, by browser, etc.
  • Emails
    • Use the free and the more popular email services/servers to generate test email accounts
      • Examples: ISPs, Yahoo, Gmail, HotMail, Mailinator, etc.
    • Broaden the use of each email account by creating multiple emails using the +TEXT string
      • Examples: mymail+test1@yahoo.com, mymail+test2@yahoo.com, mymail+test3@gmail.com
  • Passwords
    • BrowserID should have a requirement now for password strings between 8 and 80 characters, inclusive.
    • Unique passwords of both weak and strong categorization should be created and used for testing.
    • Passphrases are also acceptable for testing: groups of characters rather than a seemingly random selection, as is required by most authentication systems.

Testing Firefox Channels

Along with the Platform/OS/Browser test matrix, all Firefox live versions (channels) should be used for testing:

FireFox Release (example 13.0)

FireFox Beta (example 14.0)

FireFox Aurora (example 15.0)

FireFox Nightly (example 16.0)

Using Browser Tools

  • On the FireFox browser, various tools can be used to help test/verify BrowserID:
    • Tools > Web Developer > Web Console: can be used to track live GET and POST messages from the RP (client) site.
    • Tools > Web Developer > Error Console: to track any errors, warnings, or messages from the RP (client) site.

Deep-Dive Documentation

NOTE: This section needs to get updated

Back To ==> Mozilla BT Services

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/BrowserID/BrowserID_Weekly_Trains_Beta&oldid=492535"