MDN/QA/testplan

From MozillaWiki
< MDN
Jump to: navigation, search

We are the MDN (Mozilla Developer Network) durable team. MDN's mission is to provide complete, accurate, and helpful documentation for everything about the open Web, whether it's supported by Mozilla-built software or not. Read more about our mission.

For additional information about the project team visit here.

See what the team is currently working on

2016 QA Priorities

  • Overview and article pages behave as expected.
  • Editing pages is a consistent and user friendly experience.
  • Feedback mechanisms on MDN exist and work.
  • Localized experiences persist when a user arrives, uses, and leaves the site.
  • The search experience is consistent.

Open Questions

  • Lack of test data - Stage currently isn't sync'd with production data, the database schema is. How can we solve this?
  • Is there telemetry in place for Kuma script rendering errors
  • Data integrity
    • How is this maintained, persisted, prevented from regressing?
    • How is user content quality maintained & updated for relevancy over time?
  • Reliability discussion - https://etherpad.mozilla.org/MDN-reliability-plan
  • l10n testing
  • Page load times

Outline of ownership of quality components

Shared responsibilities

**project management, developers, qa, UX, l10n**

  • Deep knowledge of the application and the ability to identify areas of risk
  • Provide feedback on feature definition to the project manager and team
  • Identifying, creating, and maintaining the functional and end-to-end test automation
  • Developing the appropriate mitigation strategies to lower risk of regressions landing in production
  • Staying current on best practices and pushing the team to explore the applied relevancy
  • Continually discussing and fine-tuning test coverage
  • Opening avenues for community collaboration on the project

What are QA’s responsibilities

  • Acting as a user advocate.
  • Leading end of sprint post-mortem discussions that identify and realign the risk areas that need to be mitigated.
  • Moving user stories (acceptance criteria) to TestRail
  • With a focus on community and developer inclusion; collaborative creation and maintenance of end-to-end test automation.

What are the dev team’s responsibilities

  • Developing new features for the project and implementing fixes for issues
  • Developing unit tests
  • Maintaining code quality via code review and documentation of standards

Complete a risk assessment

  • Stage - schema is up-to-date, data is 9 months out-of-date
  • Dev environments are significantly different than stage and production
  • Removal of low value features - tracking of feature usage and sunsetting of features that don't see uptake.

User stories and features are grouped into three categories

Cannot regress in production

  • Features that users should never see broken in production (in order of throughput/traffic)
    • Edit a doc article
    • Translate a doc article
    • View the home page
    • Log in
    • Search docs
      • Topic pages and articles render
    • Localization persists

Can regress in production for a finite amount of time

  • Features that can go to production broken but need to be discovered and fixed within a specific time period

Can regress in production - fix when we notice it's broken

  • Features that can regress on production and will be fixed once a user identifies the problem

Manual Testing

Feature testing

Not a dogmatic rule, in general new features or defect fixes should be manually tested and verified. Exploratory testing is the rule. Depending on the risk assigned to a feature, projects have the option of using waffle flags. The use of feature flags allows us to hide features from the general public while allowing a team or a group of beta testers to test on production with real data. An important note about feature verification

If a feature is deemed as low to medium risk, anyone on the team is empowered to test and verify it. If it is a feature that requires deeper investigation and poses high risk, the team can reach out to QA and request our input. If a feature set is big or user workflow is changed it is important to engage the test team to flush out defects and usability concerns.

  • Features sent to QA for manual verification must include Bug numbers.
  • Important workflows that need to be tested will live in TestRail TODO: update url to prod testrail once it's launched.
    • Workflows that live in TestRail are candidates for automation.

Test Automation

Currently, writing and maintenance of the tests written using Intern are owned by the developer team.

  • This test runner and the test suites will be phased out in Q3 of 2016.
  • Test will be rewritten in Python and use pytest and pytest-selenium
  • Test ownership and maintenance will be expanded to include QA.

Overview

Feature work that falls into the category of “better never break” should see a heavy amount of exploratory testing before it is exposed to the public.

  • Each pull request that includes a new feature that is exposed in the front-end UI should include theIntern.js test case(s).
  • API end points should include end-to-end tests that exercise these routes.

A component of the automation strategy includes reviewing the amount of test coverage that follows a developers pull request. Often times the team is able to influence and enhance the depth of coverage by simply asking for more tests earlier in the workflow [Unit + Integration].

Coverage Areas

  • Topic and article pages behave as expected.
  • Editing pages is a consistent and user friendly experience.
  • Feedback mechanisms on MDN exist and work.
  • Localized experiences persist when a user arrives, uses, and leaves the site.
  • The search experience is consistent.

TODO Environment/test matrix

  • Linux?
  • Mobile?
  • Windows 8
    • IE
  • Windows 10+
    • IE
    • Firefox latest
    • Chrome
    • Opera?
    • Brave?
  • OSX
    • Firefox latest
    • Safari
    • Chrome
    • Opera?
    • Brave?

Unit tests

Blocks Build
  • Run by TravisCI
    • Unit tests
    • Flake8
    • Sphinx Docs build
    • Dennis
Non-Blocking

Functional UI tests

End-to-end tests

  • Integration tests - work will begin late Q2 and through Q3
  • API tests - none planned

Stress and load testing

  • None planned for Q2

Monitoring tools

Style Guides

One-time Events

Test Events

Test days are a great way to engage both longstanding contributors as well as onboarding new people on projects. As project leads and core community it is our role to create an engaging, fun, yet focused experience when people work on our projects. When drafting a test plan for it is important to know who your audience is as well as tightly define the parameters of the test day.

Each test day is an opportunity to tell the story behind why a project exists and why that application is important to the matrix of projects that Mozilla supports. Suggested components when crafting a test plan:

  • Give a brief introduction of what the project is and why it’s important. For example; Mozillians.org is a community phonebook that is integral to helping people connect with one another as well as allows access to members only areas of Mozilla.
  • Use social media to get the word out.
  • Briefly discuss what precipitated the need for a test day - a new feature or set of features. Explain why there is a need for helping reduce project risk. This helps set the tone for the event and allows people to determine if they’re interested in helping on this project.
  • Describe what expertise and environment are needed to participate. Provide information that points people to the information they need to get setup. A small suggestion, try to create a reusable body of documentation you can point people to.
  • Teach people how and where to file bugs: what components make for a good bug, what to leave out of a bug. Include a Bugzilla URL with the correct GET parameters that point to the correct product, component, flags, etc.
  • To track bugs filed during a test event, include a whiteboard flag that is unique and easily queryable.
  • Include online and offline ways for people to get in touch with you. Emphasize pathways that lead people to your community: irc and mailing lists. Also include personal contact information.
  • Lastly, warmly thank people for taking a moment to read your post. Test event’s don’t need to be real-time, but you should think of them as vehicles for creating lasting relationships that help coach people.

Using Social Media

Social media should be viewed as any medium that your team currently uses when creating one way conversations with the world. They are a great way to blast out a small amount of info, a breadcrumb trail, that leads people to an event.

  • Use twitter to notify the world of an upcoming event, make it sound intriguing, and link to a blog post/test plan that contains all of the information.
  • Use blog post to expand upon why the event is important, what people will learn, and how they can reach out for help.

Templates

Sample Testday

The latest feature set [include Bugzilla query] of the <name of project> has landed [include URL to where testing will take place] and is ready for testing. The <project name> a project that does X and is important because of Y.

What’s new in this release/what is being tested:

  • This new feature(s) will allow users to do ..
  • Combined with this old feature users can do ..
  • .. etc ..

Test plan:

These are the areas to focus on .. This can be a list of bugs that need verification, a unified bugs that that represent a new feature set that you’d like people to test wholistically (exploratory testing, etc). Be very specific about what is being tested and what to ignore. Help set a cadence for what is relevant for this event.

Setup for manual or automated testing:

Create a list of things that will be needed to participate in this event. Include appropriate links/urls/screenshots/etc that help paint a colorful picture of what’s being worked on. Consider that some people learn through reading words, whereas others prefer a combination of images and text.

Filing bugs:

Give direction on how to file a good bug as well as

  • Write good bugs that provide clear steps to reproduce the problem. Read this document for tips.
  • Use this form to file new bugs. [Buzilla form that is pre-filled using GET params]
  • Bugzilla etiquette - be polite and treat people with respect, we are a friendly community.
  • IRC etiquette - same as Bugzilla; relax and have fun.

We really appreciate your enthusiasm and help with making the <project name> better. This is fully a community initiative and wouldn't exist without you.

Cheers,

<your name>

<url to your mozillians.org public profile>

oneanddone.mozilla.org

  • For less time sensitive testing tasks we can use oneanddone.org as a task manager to aid in funneling community members into our onboarding flows.

Other

Retrieved from "https://wiki.mozilla.org/index.php?title=MDN/QA/testplan&oldid=1137559"