Releases/Firefox 4.0b3/BuildNotes
From MozillaWiki
< Releases | Firefox 4.0b3
note re: bug 582700. If Windows 2000 is still busted in 4.0b3 we should set updateType=major in windows snippets. Per Rob Strong's comment it should NOT prompt despite being a MU.. Windows 2000 support was restored for 4.0b3, nothing to worry about now.
Contents
- 1 Build Engineers
- 2 Signed-off Revision(s)
- 3 Tags
- 4 Build data
- 5 Bugs that we have hit on this release
- 6 Notes
- 6.1 Build 1
- 6.2 Build 2
- 6.3 Build 3
- 6.3.1 Tag
- 6.3.2 Source
- 6.3.3 Build/Repack
- 6.3.4 Unit tests
- 6.3.5 Sign
- 6.3.6 L10N verification
- 6.3.7 Updates
- 6.3.8 Update verify
- 6.3.9 Final checks before push
- 6.3.10 Virus check
- 6.3.11 Update Bouncer
- 6.3.12 Push to mirrors
- 6.3.13 Final verify
- 6.3.14 Push snippets to beta channel
- 6.3.15 Remove index.html files
- 6.3.16 Release
Build Engineers
armenzg, aki - tracking bug
Signed-off Revision(s)
- Build1: 2f187db8f5f6
- Build2: 69f2d0457750
- Build3: 900fdd7fb8b2
Tags
On mozilla-central:
| Build # | Tag | Changeset |
| 1 | GECKO20b3_20100804_RELBRANCH | 2f187db8f5f6 |
| FIREFOX_4_0b3_BUILD1, FIREFOX_4_0b3_RELEASE | 1fdcff2dbfdf | |
| 2 | FIREFOX_4_0b3_BUILD2, FIREFOX_4_0b3_RELEASE | 69f2d0457750 |
| 3 | FIREFOX_4_0b3_BUILD3, FIREFOX_4_0b3_RELEASE | 900fdd7fb8b2 |
Build data
| Build # | Type | Build ID | Build machine |
| 1 | [Linux compressed] | 20100804111201 | mv-moz2-linux-ix-slave22 |
| 2 | [Windows installer/zip] | 20100804193205 | mw32-ix-slave05 |
| [Mac dmg] | 20100804182810 | bm-xserve15 | |
| [Mac64 dmg] | 20100804183435 | moz2-darwin10-slave07 | |
| [Linux compressed] | 20100804182707 | mv-moz2-linux-ix-slave22 | |
| [Linux64 compressed] | 20100804183236 | moz2-linux64-slave03 | |
| 3 | [Windows installer/zip] | 20100805192522 | mw32-ix-slave05 |
| [Mac dmg] | 20100805181943 | bm-xserve15 | |
| [Mac64 dmg] | 20100805182338 | moz2-darwin10-slave07 | |
| [Linux compressed] | 20100805181633 | mv-moz2-linux-ix-slave22 | |
| [Linux64 compressed] | 20100805182321 | moz2-linux64-slave03 |
Bugs that we have hit on this release
- bug 565640 discovering which URL to pull l10n-changesets is not discoverable
-
bug 584574 - revision not found in buildbot-configs on hg.m.o - bug 584818 - release clobber branch broken
- select a master name and select "Any branch"
-
bug 584620 - Clobberer isn't working for 64bit release builders -
No bug - Fix ParseLocalesFile to handle multiple lines with the same locale. -
bug 584747 - Send sendchanges for releases to pm01:9009 after switchover to schedulerDB -
bug 570816 - Fix branch name for unittest release sendchanges -
bug 585082 - graphs does not recognise the "Firefox-Release" branch -
bug 585098 - Port branch name refactoring for talos sendchanges to 0.7.10/default masters (releases and tigers) -
bug 585108 - We don't need unit test builders for "mozilla-central" and "mozilla-2.0" releases - bug 586230 - No contrib directory for Firefox 4.0b3
Notes
Done on pm. Had to be restarted, because the master had 2 pending tag jobs.
Build 1
- Marked all builders for all releases on pm for a clobber (done by nthomas)
- Moved the following slaves to pm:
- from pm03:
- mv-moz2-linux-ix-slave22
- mv-moz2-linux-ix-slave23
- mw32-ix-slave05
- mw32-ix-slave13
- from pm01:
- mv-moz2-linux-ix-slave20
- mv-moz2-linux-ix-slave11
- mw32-ix-slave02
- mw32-ix-slave19
- bm-xserve06
- bm-xserve11
- bm-xserve12
- bm-xserve15
- from pm03:
- Confirmed beta 3 l10n changesets
- Downloaded l10n-changesets from here
- I only found the URL because Rail had mentioned previously. Axel filed bug 565640 to make it more discoverable
- Landed changes
- Reconfigured pm
- sendchange to get things running:
buildbot sendchange --username=armenzg --master=localhost:9010 --branch=mozilla-central -m "Firefox 4.0b3 build1" Go
Tag
mv-moz2-linux-ix-slave11: No problems
- Landed version bump
- Full stop at 12:11pm PDT
- only Linux had finished and had to stop the repackages for it
Build 2
- Removed *pyc and waiting
- Clobbered the release builders (NOTE: do not be connected to Build-VPN)
- Got the l10n-changesets from here
- Landed configs and l10n-changesets
- Stopped pm after checkconfig took too long
- Restarted pm after a successful checkconfig
- Tagging of our repos should have happened before the following sendchange
- This is what has to be done: Tagged tip of default branch of buildbot-configs, buildbotcustom, tools with FIREFOX_4_0b3_BUILD2 and FIREFOX_4_0b3_RELEASE
- Sendchange:
buildbot sendchange --username=aki --master=localhost:9010 --branch=mozilla-central -m "Firefox 4.0b3 build2" go
Tag
mv-moz2-linux-ix-slave11
- Closed m-c.
- Kept hitting "abort: destination 'be' already exists" in clone_be. The step didn't actually clobber the tag directory; I had to go back and clobber all release builders on pm in clobberer before it would work (nthomas says it was requesting branch=nothing)
- filed bug 584818
- This meant I did 3 sendchanges to get over the 2 failed tag steps. Luckily it went smoothly after that.
- Re-opened m-c.
- I tagged buildbot-configs, but ran into bug 584574 and decided to hold off on tagging buildbotcustom.
- Once bug 584574 was fixed, tagged buildbotcustom.
Source
mv-moz2-linux-ix-slave11 - no problems
Build/Repack
- Some L10N repacks failed due to bug 584620 (linux64 and macosx64).
- Clobbered and forced l10n repacks for the failed locales/platforms (see instructions:
# cltbld@pm PYTHONPATH=/tools/buildbotcustom:$PYTHONPATH python force_release_l10n.py -m http://localhost:8010 -t FIREFOX_4_0b3_RELEASE -v -s locales-file -n rail@mozilla.com
locales-file content:
it linux64 es-ES linux64 ru linux64 is linux64 de linux64 pt-PT linux64 nn-NO linux64 uk macosx64 pl macosx64 sv-SE macosx64 nn-NO macosx64 zh-TW macosx64
- nn-NO on linux64 never got triggered because the "nn-NO macosx64" line overwrote it. bhearsum landed a fix so this can't happen again
- I forced nn-NO for linux64
Unit tests
- There has been a switch over to scheduler db last night and the sendchanges are probably being sent to the wrong place. I am going to trigger them.
-
bug 584747 send sendchanges to the schedulerDB -
bug 570816 fix branch name for unit test releases sendchanges -
bug 585098 fix branch name for talos release sendchanges
NOTE: I send the sendchanges to production-master01.build.mozilla.org:9009 and I changed the branch name for each sendchange to "release-$branch-$platform-opt-unittest' instead of '$branch-release-$platform-opt-unittest' NOTE2: Note the white spaces on Mac's filenames and that I had to add an apostrophe. You will see mac unit tests red because for missing them.
buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-linux-opt-unittest --revision 69f2d0457750 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/linux-i686/en-US/firefox-4.0b3.tar.bz2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/linux-i686/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-linux64-opt-unittest --revision 69f2d0457750 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/linux-x86_64/en-US/firefox-4.0b3.tar.bz2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/linux-x86_64/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-win32-opt-unittest --revision 69f2d0457750 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/unsigned/win32/en-US/firefox-4.0b3.zip http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/unsigned/win32/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-macosx-opt-unittest --revision 69f2d0457750 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/mac/en-US/Firefox 4.0 Beta 3.dmg' 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/mac/en-US/Firefox 4.0 Beta 3.tests.zip' buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-macosx64-opt-unittest --revision 69f2d0457750 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/mac64/en-US/Firefox 4.0 Beta 3.dmg' 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2/mac64/en-US/Firefox 4.0 Beta 3.tests.zip'
- No need to trigger the talos sendchanges because we got a full stop for build2
Sign
Using instructions.
NOTE: Watch for bug 586230 in the next release.
en-US.xpi removed from stage:
#ffxbld@stage cd /pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2 rm en-US.xpi
Run usual signing script:
#!/bin/bash
set -x
set -e
PRODUCT=firefox
VERSION=4.0b3
BUILD=2
TAG=FIREFOX_4_0b3_RELEASE
REPO="mozilla-central"
EMAIL=armenzg@mozilla.com
KEYDIR=d:/2009-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make PRODUCT=${PRODUCT} VERSION=${VERSION} \
BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
KEYDIR=${KEYDIR} USE_NEW=1
Inspected stdout, signing log. They looked good. Ran the following:
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
L10N verification
No problems
Updates
No problems
Update verify
No problems
Build 3
- Full stop of build 2 because of bug 584057
- Taken this fix
- Clobbered all release builders
- Took no l10n-changesets at Axel's request.
- Landed config changes
- reconfiged pm
- sendchange:
buildbot sendchange --username=aki --master=localhost:9010 --branch=mozilla-central -m "Firefox 4.0b3 build3" doit
- tagged buildbot-configs and buildbotcustom
Tag
mv-moz2-linux-ix-slave11 - No problems.
Source
mv-moz2-linux-ix-slave11 - No problems.
Build/Repack
No problems.
Unit tests
- bug 585108 - We don't need unit test builders for "mozilla-central" and "mozilla-2.0" releases
- bug 585098 fix branch name for talos release sendchanges
- We currently can't run talos jobs for releases (even though they get triggered). See bug: talos for releases is currently broken
- We fixed prior build3 the name of the unit test branch name for its sendchanges with this branch name fix but we still need to land the where the sendchanges are sent
- NOTE I don't know why some talos sendchanges triggered 5 pending talos jobs instead of 1. For next release and after the aforementioned bugs are fixed let's see if it happens again.
List of sendchanges run manually:
buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange --branch release-mozilla-central-linux-talos --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-i686/en-US/firefox-4.0b3.tar.bz2 buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange --branch release-mozilla-central-linux64-talos --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-x86_64/en-US/firefox-4.0b3.tar.bz2 buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange --branch release-mozilla-central-win32-talos --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/unsigned/win32/en-US/firefox-4.0b3.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange --branch release-mozilla-central-macosx-talos --revision 900fdd7fb8b2 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac/en-US/Firefox 4.0 Beta 3.dmg' buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange --branch release-mozilla-central-macosx64-talos --revision 900fdd7fb8b2 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac64/en-US/Firefox 4.0 Beta 3.dmg'
buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-linux-opt-unittest --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-i686/en-US/firefox-4.0b3.tar.bz2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-i686/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-linux64-opt-unittest --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-x86_64/en-US/firefox-4.0b3.tar.bz2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/linux-x86_64/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-win32-opt-unittest --revision 900fdd7fb8b2 http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/unsigned/win32/en-US/firefox-4.0b3.zip http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/unsigned/win32/en-US/firefox-4.0b3.tests.zip buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-macosx-opt-unittest --revision 900fdd7fb8b2 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac/en-US/Firefox 4.0 Beta 3.dmg' 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac/en-US/Firefox 4.0 Beta 3.tests.zip' buildbot sendchange --master production-master01.build.mozilla.org:9009 --username sendchange-unittest --branch release-mozilla-central-macosx64-opt-unittest --revision 900fdd7fb8b2 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac64/en-US/Firefox 4.0 Beta 3.dmg' 'http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/mac64/en-US/Firefox 4.0 Beta 3.tests.zip'
Results
These were the only oranges we had. The oranges are known and the last one is permanent:
- [intermmitent] Linux64 mochitest-1/5 - bug 567954
- [intermmitent] 10.6 reftest - bug 565842
- [permanent][hidden on m-c] 10.6 xpcshell - bug 561350
Sign
Using instructions.
- tagged tools
en-US.xpi removed from stage:
#ffxbld@stage cd /pub/mozilla.org/firefox/nightly/4.0b3-candidates/build2 rm en-US.xpi
Run the following:
PRODUCT=firefox
VERSION=4.0b3
BUILD=3
TAG=FIREFOX_4_0b3_RELEASE
REPO="mozilla-central"
EMAIL=armenzg@mozilla.com
KEYDIR=d:/2009-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make PRODUCT=${PRODUCT} VERSION=${VERSION} \
BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
KEYDIR=${KEYDIR} USE_NEW=1
Inspected stdout, signing log. They looked good. Ran the following:
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
L10N verification
No problems
Updates
No problems
Update verify
No problems
Final checks before push
# ffxbld@stage cd /pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/ find . ! -user ffxbld find . ! -group firefox find . -type f ! -perm 644 find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' ^^ Last line is WRONG, should be |find . -maxdepth 1 -type d -perm 2775 ! -name 'contrib*'|
Virus check
# ffxbld@stage clamdscan -m /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/4.0b3-candidates/build3/ /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/4.0b3-candidates/build3: OK ----------- SCAN SUMMARY ----------- Infected files: 0 Time: 1626.579 sec (27 m 6 s)
Update Bouncer
- bug 571869 has been verified to be fixed
- Moved mv-moz2-linux-ix-slave16 from pm02 to pm
- verified release_config.py still said 4.0b3 build3
- Forced bouncer_submitter build
- This went green!
- TODO verify the step went smoothly: Nick says it shouldn't have submitted the various languages. I think Nick was making reference to:
Locales: be, ca, cs, da, de, el, en-US, es-AR, es-ES, et, fi, fr, ga-IE, hu, is, it, ko, ku, lt, nb-NO, nl, nn-NO, pa-IN, pl, pt-BR, pt-PT, ro, ru, sk, sv-SE, tr, uk, zh-CN, zh-TW Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="824" name="Firefox-4.0b3"><language locale="be"/><language locale="ca"/><language locale="cs"/><language locale="da"/><language locale="de"/><language locale="el"/><language locale="en-US"/><language locale="es-AR"/><language locale="es-ES"/><language locale="et"/><language locale="fi"/><language locale="fr"/><language locale="ga-IE"/><language locale="hu"/><language locale="is"/><language locale="it"/><language locale="ko"/><language locale="ku"/><language locale="lt"/><language locale="nb-NO"/><language locale="nl"/><language locale="nn-NO"/><language locale="pa-IN"/><language locale="pl"/><language locale="pt-BR"/><language locale="pt-PT"/><language locale="ro"/><language locale="ru"/><language locale="sk"/><language locale="sv-SE"/><language locale="tr"/><language locale="uk"/><language locale="zh-CN"/><language locale="zh-TW"/></product></products> Adding product: Firefox-4.0b3-Complete Locales: be, ca, cs, da, de, el, en-US, es-AR, es-ES, et, fi, fr, ga-IE, hu, is, it, ko, ku, lt, nb-NO, nl, nn-NO, pa-IN, pl, pt-BR, pt-PT, ro, ru, sk, sv-SE, tr, uk, zh-CN, zh-TW Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="825" name="Firefox-4.0b3-Complete"><language locale="be"/><language locale="ca"/><language locale="cs"/><language locale="da"/><language locale="de"/><language locale="el"/><language locale="en-US"/><language locale="es-AR"/><language locale="es-ES"/><language locale="et"/><language locale="fi"/><language locale="fr"/><language locale="ga-IE"/><language locale="hu"/><language locale="is"/><language locale="it"/><language locale="ko"/><language locale="ku"/><language locale="lt"/><language locale="nb-NO"/><language locale="nl"/><language locale="nn-NO"/><language locale="pa-IN"/><language locale="pl"/><language locale="pt-BR"/><language locale="pt-PT"/><language locale="ro"/><language locale="ru"/><language locale="sk"/><language locale="sv-SE"/><language locale="tr"/><language locale="uk"/><language locale="zh-CN"/><language locale="zh-TW"/></product></products> Adding product: Firefox-4.0b3-Partial-4.0b2 Locales: be, ca, cs, da, de, el, en-US, es-AR, es-ES, et, fi, fr, ga-IE, hu, is, it, ko, ku, lt, nb-NO, nl, nn-NO, pa-IN, pl, pt-BR, pt-PT, ro, ru, sk, sv-SE, tr, uk, zh-CN, zh-TW Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="826" name="Firefox-4.0b3-Partial-4.0b2"><language locale="be"/><language locale="ca"/><language locale="cs"/><language locale="da"/><language locale="de"/><language locale="el"/><language locale="en-US"/><language locale="es-AR"/><language locale="es-ES"/><language locale="et"/><language locale="fi"/><language locale="fr"/><language locale="ga-IE"/><language locale="hu"/><language locale="is"/><language locale="it"/><language locale="ko"/><language locale="ku"/><language locale="lt"/><language locale="nb-NO"/><language locale="nl"/><language locale="nn-NO"/><language locale="pa-IN"/><language locale="pl"/><language locale="pt-BR"/><language locale="pt-PT"/><language locale="ro"/><language locale="ru"/><language locale="sk"/><language locale="sv-SE"/><language locale="tr"/><language locale="uk"/><language locale="zh-CN"/><language locale="zh-TW"/></product></products>
Push to mirrors
# ffxbld@stage cd /home/ftp/pub/firefox/releases mkdir 4.0b3 cd 4.0b3 rsync -av --exclude=*tests* --exclude=*crashreporter* --exclude=*.log --exclude=*.txt --exclude=*unsigned* --exclude=*update-backup* /pub/mozilla.org/firefox/nightly/4.0b3-candidates/build3/. .
index.html:
curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=456169 > index.html vi index.html # edit to change Beta 1 to Beta 3 for dir in `find . -mindepth 1 -type d`; do cp -pv index.html $dir/; done
Final verify
- went green after 2nd force build (after a wait). Moved mv-moz2-linux-ix-slave16 back to pm02.
- sent email when mirrors were absorbed for testing (11:58pm PDT)
- sent email when mirrors were absorbed for release ( 7:51pm PDT)
Push snippets to beta channel
- Upon GO from release-drivers
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20100805-Firefox-4.0b3 ~/bin/pushsnip 20100805-Firefox-4.0b3
Remove index.html files
Once we are fully announced (by the driver) we can remove the index.html files:
# ffxbld@stage cd /pub/mozilla.org/firefox/releases/4.0b3/ find . -name index.html find . -name index.html | xargs rm -v
Release
- Emailed the following to metrics@m.c
Firefox,firefox,firefox,4.0b3,4.0 Firefox,firefox,firefox,4.0b4pre,4.0
