User:Armenzg:L10n build roadmap
NOTE: Just look at sections "Goals" and "Patches" to have the latest info and to don't be overwhelmed
Contents
Goals
-
Run L10n repackages in buildbot in parallel which requires:-
getting logic from tinderbox client code into make targets and buildbot-
get en-US binary with make target -
upload packages with correct naming -
report to tinderbox pages
-
-
- Make full switch from tinderbox to buildbot
- Do l10n repackages in parallel:
- in 1.9 (getting there)
- in mozilla-central
- Have L10n nightly updates which requires:
- Script that will generate installer without missing entities (en-US entities added that are not yet added in a locale) - Gandalf might something in his Silme library
- Integrate compare-locales.py as part of the steps instead of the old compare-locales.pl
- requires checking out same source code cut off as the one used with en-US nightly
Patches/Status
This is to keep track of which bugs are open and what is the status of the patches. This might be the section out of this document that might me more up to date
description | CVS | hg | status |
---|---|---|---|
456386 - tracking bug - repackages in 1.9 in *paralell* | N/A | delayed after 3.0.3 release | |
446038 - add more slaves to 1.9 automation | N/A | ||
398954 - use same source cut-off from en-US | N/A | no patches yet | |
451225 - master.cfg steps release | N/A | blocked in other bugs | |
456321 - "mozilla/other-licenses/7zstub/firefox" | no patches yet | ||
456351 - fix usage of variables | no patches yet | ||
|
commited | N/A | TODO: File upstream bug |
Patches done
description | CVS | hg | status |
---|---|---|---|
|
commited | N/A | |
|
commited | N/A | |
|
commited | commited | |
|
commited | commited |
Roadmap + Status
All other milestones depend on M1. All others milestones do not block each other
Main fixes - porting to buildbot and make targets(M1)
-
Multiple builders pulling from same queue for locales - Do L10n nightly builds on staging 1.9
- changes in master.cfg adding multiple build objects per locale'
- buildbotcustom classes - NightlyL10n, L10nHelper and Build (it could be called BuildL10n or BuildQueueL10n)
- be able to wget the right en-US binary (Bug 445254)
- prepare repackages in the folder structure expected in ftp
- The normal TinderboxMailNotifier just needs to allow WithProperties for the "tree" parameter
- We will report to Firefox3.0-l10n and Mozilla-1.9.0-%locale%
- with two TinderboxMailNotifier instances is enough since we can now use WithProperties with the "tree" parameter as mentioned before
Status
Best scenarios
In the next 4 scenarios I try to cover what is required so everything is working with buildbot and parallel builds
1.9 nightlies
The perfect world would be to have L10n repackages happening just after an en-US build, which we currently can't since en-US is being generated in dedicated machines with tinderbox code. It also requires to separate dependent/clobber builds from nightly builds
L10n nightlies in release automation (M2.a)
- Use similar configuration as in staging-1.9
- Turn off dedicated L10n slaves AND do L10n nightly builds in production-1.9
en-US nightly builds in release automation instead of dedicated machines (M2.b)
- Run en-US nightly builds in production-1.9 automation instead of dedicated slaves using bootstrap and tinderbox code
Nightly & Periodic schedulers for en-US (M3 depends on M2.b)
- Turn off $OfficialBuildMachinery to avoid generating nightly when generating dependent/clobber
- Have a Nightly scheduler for en-US nightly builds and a Dependent scheduler for dependent/clobber builds
Trigger L10n repackages after en-US nightly (M4 depends on M3)
- Use DependentL10n scheduler instead of NightlyL10n to trigger L10n nightly builds as soon as the Nightly scheduler finishes
1.9 releases
fake release in staging (M2)
- Do a fake L10n release on staging 1.9
- Make sure that ftp structure is proper and everything is uploaded
- Download one of the fake releases and try to see if it would get an update
- use shiped-locales instead of all-locales to determine which locales to repackages
- what else to test for a fake release to be good?
production release (depends on M2)
- Run production 1.9 releases
moz2 nightlies
To be added more
moz2 releases milestones
To be added more
Pollers milestones
To be added more
Open discussions
When is the right time to remove things
- Once every 24 hours? How?
Reduce time before making installers
- We have to check out the following code:
- TBA
- We have to configure
Using the configure file that comes by default checks for many more things than what we need when doing a repackage
- Could we use another configure file?
- Could we do "make -f client.mk configure-l10n" and check for less things that regular configure?
- Could we move "make installers-foo" to a python script?
Checkout same source code
- en-US builds should check out the 3AM source code - this way mac, linux and windows are exactly the same
- l10n builds check out the en-US 3AM source code and the locale's tip (maybe the 3AM source code as well?? I think it makes more sense)
Compare-locales to be run from source in binary
- We added the timestamp in application.ini to be able to checkout the same en-US source code that was used for a nightly - couldn't we compare with the entities that are in the installers?
Issues
- "Force build" does not work with these changes - I think it is not a blocker, I know where it breaks but it might require changes in the Builder class that comes with buildbot which I do not want since we would be hacking buildbot
Bugs involved
- Bug 434878 – change how automation does l10n repacks
- Bug 434289 – create l10n builds for mozilla-central
- Bug 438436 – l10n - make files should reduce some steps to repackage a locale
- Bug 398954 – l10n build automation doesn't pick the right en-US source for the build
- Bug 399014 – we need an l10n-merge tool
- Bug 407319 – application.ini should supply branch/timestamp information
- Bug 446038 – add more slaves to 1.9 automation
- Bug 421411 – migrate firefox 1.9 nightlies to release automation
- Bug 445254 – migrate firefox 1.9 L10n nightlies to release automation
- [https://bugzilla.mozilla.org/show_bug.cgi?id=451056 Bug 451056 – add code to parallelize in buildbot the generation of locales (blocks 445254)
- Bug 439778 – use a Nightly scheduler to generate firefox 1.9 nightlies
- Bug 444957 – Separate dependent/clobber build from nightly build in tinderbox code to have a Dependent and a Nightly schedulers in automation 1.9
- Bug 439050 – de-xpfe and revive make-jars.pl to python