Mobile/e10s/Testing
Contents
- 1 Fennec Mochitest and Reftest
- 1.1 Running Tests
- 1.2 Reftests
- 1.2.1 abs-pos
- 1.2.2 backgrounds
- 1.2.3 bidi
- 1.2.4 border-image
- 1.2.5 border-radius
- 1.2.6 box-ordinal
- 1.2.7 box-properties
- 1.2.8 box-shadow
- 1.2.9 bugs
- 1.2.10 canvas
- 1.2.11 columns
- 1.2.12 counters
- 1.2.13 css-calc
- 1.2.14 css-charset
- 1.2.15 css-default
- 1.2.16 css-disabled
- 1.2.17 css-enabled
- 1.2.18 css-gradients
- 1.2.19 css-import
- 1.2.20 css-invalid
- 1.2.21 css-mediaqueries
- 1.2.22 css-namespace
- 1.2.23 cssom
- 1.2.24 css-optional
- 1.2.25 css-parsing
- 1.2.26 css-placeholder
- 1.2.27 css-required
- 1.2.28 css-submit-invalid
- 1.2.29 css-transitions
- 1.2.30 css-ui-invalid
- 1.2.31 css-ui-valid
- 1.2.32 css-valid
- 1.2.33 css-valuesandunits
- 1.2.34 css-visited
- 1.2.35 datalist
- 1.2.36 dom
- 1.2.37 editor
- 1.2.38 first-letter
- 1.2.39 first-line
- 1.2.40 floats
- 1.2.41 font-face
- 1.2.42 font-features
- 1.2.43 font-matching
- 1.2.44 fonts
- 1.2.45 forms
- 1.2.46 generated-content
- 1.2.47 ib-split
- 1.2.48 image
- 1.2.49 image-element
- 1.2.50 image-rect
- 1.2.51 image-region
- 1.2.52 inline-borderpadding
- 1.2.53 invalidation
- 1.2.54 layers
- 1.2.55 line-breaking
- 1.2.56 list-item
- 1.2.57 margin-collapsing
- 1.2.58 marquee
- 1.2.59 mathml
- 1.2.60 native-theme
- 1.2.61 object
- 1.2.62 ogg-video
- 1.2.63 pagination
- 1.2.64 percent-overflow-sizing
- 1.2.65 pixel-rounding
- 1.2.66 printing
- 1.2.67 reftest-sanity
- 1.2.68 scrolling
- 1.2.69 selection
- 1.2.70 svg
- 1.2.71 table-anonymous-boxes
- 1.2.72 table-background
- 1.2.73 table-bordercollapse
- 1.2.74 table-dom
- 1.2.75 table-width
- 1.2.76 tab-size
- 1.2.77 text
- 1.2.78 text-decoration
- 1.2.79 text-indent
- 1.2.80 text-shadow
- 1.2.81 text-transform
- 1.2.82 transform
- 1.2.83 unicode
- 1.2.84 view-source
- 1.2.85 xul
- 1.2.86 xul-document-load
- 1.2.87 z
- 1.2.88 libpr0n
- 1.2.89 layout/font*
- 1.2.90 layout/reftests/image*
- 1.3 MochiTest
Fennec Mochitest and Reftest
Build Fennec with enable-tests.
- required patches:
- bug 648104, 649776.
Running Tests
Make sure you have --enable-tests in your mozconfig as you build. You can then use normal Firefox desktop commands to run tests, like
TEST_PATH=modules/libpr0n/test/reftest/reftest.list make reftest
(do this in the OBJDIR). You will need to make a symlink called fennec-bin (pointing to fennec). (ln -s /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec-bin)
Another way to build tests is (for example, for libpr0n)
make package-tests cd dist/test-package-stage/reftest python runreftest.py --app=../../bin/fennec tests/modules/libpr0n/test/reftest/pngsuite-chunkorder/reftest.list
This however requires make package-tests, and you will not have your tests symlinked (so changes to them will require another make package-tests each time).
If you run a debug build, you will hit lots of "assertion count X is more than expected Y assertions". For now you can ignore those (but we should fix them eventually!)
Also, you can more easily compare errors using the tool checked in at: file:///builds/mozilla-central/layout/tools/reftest/reftest-analyzer.xhtml
Reftests
abs-pos
successful
backgrounds
REFTEST TEST-UNEXPECTED-FAIL | file:///builds/mozilla-central/layout/reftests/backgrounds/background-size-zoom-no-repeat.html
Filed bug 650331
bidi
Unexpected: 7 (less when pango is enabled)
Existing linux bug for bidi-004.html and bidi-004j.html bug 386567
border-image
successful
border-radius
2 failures ====file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-1.html file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-2.html
Filed bug 650337
box-ordinal
successful
box-properties
one failure: file:///builds/mozilla-central/layout/reftests/box-properties/CSS21-t100301.xhtml
box-shadow
Unexpected: 4 boxshadow-blur-2.html | image comparison (!=) (looks clipped)
boxshadow-button.html | image comparison (==) -
boxshadow-fileupload.html | image comparison (==)
box-shadow/611574-1.html | image comparison (==) - color problems
box-shadow/611574-2.html | image comparison (==) - color problems
bugs
Hangs, eventually killed with |load failed: timed out waiting for pending paint count to reach zero (waiting for MozAfterPaint)| on
- 621253-1-externalFilter.html
- 621253-1-internalFilter.html
Filed bug 649859
canvas
file:///builds/mozilla-central/layout/reftests/canvas/text-font-lang.html
columns
successful
counters
successful
css-calc
successful
css-charset
successful
css-default
successful
css-disabled
8 failures.
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/button/button-fieldset-legend-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-3.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-2.html
file:///builds/mozilla-central/layout/reftests/css-disabled/select/select-fieldset-legend-3.html
css-enabled
8 failures
button-fieldset-2.html | image comparison (==) -- looks like we are overriding a style, although I'm not currently sure where that style is set (wesj)
button-fieldset-3.html | image comparison (==)
button-fieldset-legend-2.html | image comparison (==)
button-fieldset-legend-3.html | image comparison (==)
select-fieldset-2.html | image comparison (==)
select-fieldset-3.html | image comparison (==)
select-fieldset-legend-2.html | image comparison (==)
select-fieldset-legend-3.html | image comparison (==)
css-gradients
successful
css-import
successful
css-invalid
6 failures.
css-mediaqueries
successful
css-namespace
successful
cssom
successful
css-optional
successful
css-parsing
successful
css-placeholder
11 unexpected
css-required
successful
css-submit-invalid
successful
css-transitions
successful
css-ui-invalid
7 failures.
css-ui-valid
6 failures.
css-valid
6 unexpected
css-valuesandunits
successful
css-visited
datalist
successful
dom
successful
editor
17 failures.
first-letter
2 failures
first-line
successful
floats
successful
font-face
HANGS
font-features
4 unexpected
font-matching
11 failures. maybe because we don't have pango?
fonts
forms
23 failures
generated-content
ib-split
image
- background-image-zoom-1.html and image-zoom-1.html - looks like we zoom differently than Firefox? In Fennec the image begins so tiny it can't be seen. Doing some zooming in will show it properly in both tests.
image-element
3 unexpected
image-rect
successful
image-region
successful
inline-borderpadding
successful
invalidation
successful
layers
successful
line-breaking
successful
list-item
successful
margin-collapsing
successful
marquee
successful
mathml
1 failure: file:///builds/mozilla-central/layout/reftests/mathml/scale-stretchy-5.xhtml
native-theme
3 failures
object
successful
ogg-video
7 failures.
pagination
successful
percent-overflow-sizing
successful
pixel-rounding
successful
printing
successful
reftest-sanity
successful
scrolling
failure: file:///builds/mozilla-central/layout/reftests/scrolling/fixed-text-1.html
selection
2 unexpected
svg
8 unexpected
table-anonymous-boxes
successful
table-background
successful
table-bordercollapse
successful
table-dom
successful
table-width
successful
tab-size
successful
text
8 failures
text-decoration
1 unexpected
text-indent
successful
text-shadow
successful
text-transform
successful
transform
successful
unicode
successful
view-source
xul
successful
xul-document-load
successful
z
libpr0n
Most of these tests fail due to fennec disabling color correction (due to performance concerns). And some other will fail if using a 16bbp display. For example, see bug 648148. Also note that reftests that require drawWindow support will fail in Fennec. (For that you will want the patch in bug 648104, as mentioned before.)
cjones was able to make these mostly pass back in Feb, but since have decayed.
layout/font*
All pass except for bug 649809.
layout/reftests/image*
Some failures due to color correction or scaling/smoothing:
- canvas-outside-document.html
- element-paint-native-widget.html
- element-paint-sharpness-01b.html
- element-paint-sharpness-02b.html
MochiTest
layout/base
Most of these failures are related to using the old netscape universal permissions api. Instead, we should be using SpecialPowers -- the new elevated permission api.
Many of the mochitests use some sort of elevated permission. For example, sending an keypress or composition event requires elevated permissions. I converted some of the EventUtils over to SpecialPowers in bug 649360.
To run, you can:
make package-tests cd dist/test-package-stage/mochitest python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/base
layout/style is another test suite of interest. Currently this has over 90,000 tests and we fail <30 on fennec. Most of those are simple fixes for using SpecialPowers for preferences.
python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/style