Mobile/e10s/Testing

From MozillaWiki
< Mobile‎ | e10s
Jump to: navigation, search

Contents

Fennec Mochitest and Reftest

Build Fennec with enable-tests.

  • required patches:
    • bug 648104, 649776.

Running Tests

Make sure you have --enable-tests in your mozconfig as you build. You can then use normal Firefox desktop commands to run tests, like

TEST_PATH=modules/libpr0n/test/reftest/reftest.list make reftest

(do this in the OBJDIR). You will need to make a symlink called fennec-bin (pointing to fennec). (ln -s /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec-bin)

Another way to build tests is (for example, for libpr0n)

make package-tests
cd dist/test-package-stage/reftest
python runreftest.py --app=../../bin/fennec tests/modules/libpr0n/test/reftest/pngsuite-chunkorder/reftest.list

This however requires make package-tests, and you will not have your tests symlinked (so changes to them will require another make package-tests each time).

If you run a debug build, you will hit lots of "assertion count X is more than expected Y assertions". For now you can ignore those (but we should fix them eventually!)

Also, you can more easily compare errors using the tool checked in at: file:///builds/mozilla-central/layout/tools/reftest/reftest-analyzer.xhtml

Reftests

abs-pos

successful

backgrounds

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/mozilla-central/layout/reftests/backgrounds/background-size-zoom-no-repeat.html


Filed bug 650331

bidi

Unexpected: 7 (less when pango is enabled)

Existing linux bug for bidi-004.html and bidi-004j.html bug 386567

border-image

successful

border-radius

2 failures ====file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-1.html file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-2.html


Filed bug 650337

box-ordinal

successful

box-properties

one failure: file:///builds/mozilla-central/layout/reftests/box-properties/CSS21-t100301.xhtml

box-shadow

Unexpected: 4 boxshadow-blur-2.html | image comparison (!=) (looks clipped)

boxshadow-button.html | image comparison (==) -

boxshadow-fileupload.html | image comparison (==)

box-shadow/611574-1.html | image comparison (==) - color problems

box-shadow/611574-2.html | image comparison (==) - color problems

bugs

Hangs, eventually killed with |load failed: timed out waiting for pending paint count to reach zero (waiting for MozAfterPaint)| on

  • 621253-1-externalFilter.html
  • 621253-1-internalFilter.html

Filed bug 649859

canvas

file:///builds/mozilla-central/layout/reftests/canvas/text-font-lang.html

columns

successful

counters

successful

css-calc

successful

css-charset

successful

css-default

successful

css-disabled

8 failures.


file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-3.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-2.html

file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-3.html

css-enabled

8 failures

button-fieldset-2.html | image comparison (==) -- looks like we are overriding a style, although I'm not currently sure where that style is set (wesj)

button-fieldset-3.html | image comparison (==)

button-fieldset-legend-2.html | image comparison (==)

button-fieldset-legend-3.html | image comparison (==)

select-fieldset-2.html | image comparison (==)

select-fieldset-3.html | image comparison (==)

select-fieldset-legend-2.html | image comparison (==)

select-fieldset-legend-3.html | image comparison (==)

css-gradients

successful

css-import

successful

css-invalid

6 failures.

css-mediaqueries

successful

css-namespace

successful

cssom

successful

css-optional

successful

css-parsing

successful

css-placeholder

11 unexpected

css-required

successful

css-submit-invalid

successful

css-transitions

successful

css-ui-invalid

7 failures.

css-ui-valid

6 failures.

css-valid

6 unexpected

css-valuesandunits

successful

css-visited

datalist

successful

dom

successful

editor

17 failures.

first-letter

2 failures

first-line

successful

floats

successful

font-face

HANGS

font-features

4 unexpected

font-matching

11 failures. maybe because we don't have pango?

fonts

forms

23 failures

generated-content

ib-split

image

  • background-image-zoom-1.html and image-zoom-1.html - looks like we zoom differently than Firefox? In Fennec the image begins so tiny it can't be seen. Doing some zooming in will show it properly in both tests.

image-element

3 unexpected

image-rect

successful

image-region

successful

inline-borderpadding

successful

invalidation

successful

layers

successful

line-breaking

successful

list-item

successful

margin-collapsing

successful

marquee

successful

mathml

1 failure: file:///builds/mozilla-central/layout/reftests/mathml/scale-stretchy-5.xhtml

native-theme

3 failures

object

successful

ogg-video

7 failures.

pagination

successful

percent-overflow-sizing

successful

pixel-rounding

successful

printing

successful

reftest-sanity

successful

scrolling

failure: file:///builds/mozilla-central/layout/reftests/scrolling/fixed-text-1.html

selection

2 unexpected

svg

8 unexpected

table-anonymous-boxes

successful

table-background

successful

table-bordercollapse

successful

table-dom

successful

table-width

successful

tab-size

successful

text

8 failures

text-decoration

1 unexpected

text-indent

successful

text-shadow

successful

text-transform

successful

transform

successful

unicode

successful

view-source

xul

successful

xul-document-load

successful

z

libpr0n

Most of these tests fail due to fennec disabling color correction (due to performance concerns). And some other will fail if using a 16bbp display. For example, see bug 648148. Also note that reftests that require drawWindow support will fail in Fennec. (For that you will want the patch in bug 648104, as mentioned before.)

cjones was able to make these mostly pass back in Feb, but since have decayed.

layout/font*

All pass except for bug 649809.

layout/reftests/image*

Some failures due to color correction or scaling/smoothing:

  • canvas-outside-document.html
  • element-paint-native-widget.html
  • element-paint-sharpness-01b.html
  • element-paint-sharpness-02b.html

MochiTest

layout/base

Most of these failures are related to using the old netscape universal permissions api. Instead, we should be using SpecialPowers -- the new elevated permission api.


Many of the mochitests use some sort of elevated permission. For example, sending an keypress or composition event requires elevated permissions. I converted some of the EventUtils over to SpecialPowers in bug 649360.

To run, you can:

make package-tests
cd dist/test-package-stage/mochitest
python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/base


layout/style is another test suite of interest. Currently this has over 90,000 tests and we fail <30 on fennec. Most of those are simple fixes for using SpecialPowers for preferences.

python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/style