B2G/QA/Device Test Plan/NewDevice

From MozillaWiki
< B2G‎ | QA‎ | Device Test Plan
Jump to: navigation, search

Test Plan

Introduction

This test plan is a test plan for testing new devices and some of the things to look out for. This is a guideline; there may be more to test depending on the feature of the phone.

Design Documentation/Background Research

Listed are the related documentation for testing:

Module Owners

Requirements:

Bugs tracking features:

Specs:

Tips and Tricks for testing:

OS Diagrams

Schedule Milestones

Hardware Requirements

  • A desktop machine capable of running ADB is necessary for testing for getting debug information and flash the device.
  • Mobile devices listed
  • Bluetooth headset
  • headset with microphone
  • headset without microphone
  • another device/machine with bluetooth transfer capabilities
  • Optional: Build machine to build your own gaia/gecko
  • Optional: Build OS Storage machine to make it easier to find regressions

Mobile Device

  • New Device

Software Requirements

  • ADB
  • fastboot
  • New Device Flashing Tool and USB Driver
  • System for Flashing Tool (Typically a Windows Machine)

System Testing

Mobile OS

  • Depends on device

Accounts

  • See Test Accounts
  • Facebook Contact Import accounts also listed in the above link

Email Servers

  • Hotmail.com
  • Gmail.com
  • Mozilla.com

Strategy

  • smoke test testing
  • target testing for interface between hardware and software
  • Exploratory testing

Currently not tested:

  • Performance against different mobile data types

Test Deliverables

  • Test Results

Dependencies/Risk

The drivers on most devices are given from the OEM vendors. This poses a risk if we do not get an update to a potentially critical bug.

Signoff/Exit Criteria

  • no critical bug is found during testing

Test Outline

Note : This test out line is a brain storm of all possible conditions to consider for testing. The scope of the testing will be narrowed from this list based on design information, risks of breakage, etc.

Positive Smoke/Alpha Tests

Functional

Smoke Test

Email
  • Test email on wifi and mobile data
Gallery
  • test various images and various image size (supported/unsupported)
Camera
  • test orientation
  • test video capture
  • test camera/camera preview
  • flash
  • front camera/back camera
FM Radio
  • test audio for stereo output
Music
  • music (supported/unsupported encoding)
Video
  • video (supported/unsupported encoding)
  • streaming video (youtube)
Usage
  • Usage Graph
Geolocation
  • GPS test app or maps.google.com
SIM
Dialer
  • make a call in
  • make a call out
  • conference call
  • call waiting
  • dual sim support
    • 0 sim, 1 sim, 2 sim
  • sim lock
  • airplane mode
  • sms
tethering
  • tethering
Wifi Hotspot
  • wifi hotspot
touch screen
  • sensitivity
bluetooth
  • file transfer
  • music control
  • dialer
NFC
Buttons & Misc
  • power button
  • volume buttons
  • Vibration
  • headset
  • microphone
  • orientation/angle
OTA
  • check wifi
  • check bluetooth
  • check NFC
  • check camera
    • front and back
    • check flash
    • check autofocus
  • check music
  • check geolocation
  • check dialer
  • check SMS
  • check USB storage
  • check FM Radio
  • check email
  • check logshake
  • check light sensor ( ie turn on auto brightness and place finger over light sensor )

Beta Conditional / Exploratory Items

  • OTA
    • Shallow flashing
      • testing full
    • changing the channel to nightly/aurora, updating
      • testing full
  • Physical Conditions
    • Power Consumption
    • Performance
      • Speed of functions
      • Processor utilization
      • Memory utilization
        • Memory leaks?
        • Memory efficiency
    • Storage
      • Disk
      • Sim
  • Race Conditions
    • Lock conditions?
    • tooltips or other dialogs?
    • Network Conditions (Should have no impact)
      • WiFi (various supported ranges)
      • Bluetooth (Devices)
      • Mobile Data type :
        • 3G/4G?
        • signal frequency
      • SMS
    • Signal Conflict
    • Battery
        • Battery Low
        • No Battery
        • While Charged
        • While not Charged
      • Other Apps
        • Calendar notification
        • Alarm Clock notification
        • Sleep notification
        • Screensaver / Lock (system)
        • EnergySaver (Fade) (system)
        • Other?
        • Music playing (should not affect)
        • plugins (youtube and other websites with different media types)
        • Extensions ( addons to fennec)
    • Buttons
      • Hard reset
      • Soft reset
      • Turn off phone
      • force quit
      • Other buttons being pressed on the phone (zoom, picture, power )
    • Endurance Testing

Usability

  • Usability
  • Accessibility (508 compliance)