User:AxelHecht/Building l10n

I've been hacking on getting the l10n build infrastructure over to buildbot, but I've been lacking some verbal communication of the ups and downs of that.

I'll try to give an overview of what's easy, what's hard, what I tried, which of that succeeded and what failed.

Why l10n is easy

I should say relatively easy, creating a localized build is way easier than a real compile, but it's still harder than just editing a text file. Building l10n works like this:

• Get the source (*)
• Create an object dir
• configure --disable-compile-environment --enable-application=foo
• cd foo/locales and make target-ab-CD, where target can be langpack, installers, or repackage-zip.

This works across different applications (browser, mail, calendar) and different branches. It's pretty quick, too, below a minute per binary (windows installers target does two, takes about twice as long).

Done, off to lunch.

Why l10n is not so easy

Firstly, browser is on 50 languages, mail on almost 40, so the "just 2 minutes" on Windows blow up to a cycle time of 3 hours plus. Smells like we should only build what we need, when we need, aka, buildbot food, huh? More on that below.

A bigger problem still are the special ways in which a localization can break the build. Like any code, l10n can introduce errors into a build, so let's make a list of known failure modes.

l10n failure modes

There are some failures which straight out break the building process, and there are some failures that cause run time errors. To put the discussion into perspective, the goal here is to

• detect errors as early as possible (rather at buildtime vs at runtime), and to
• recover sensibly where possible.

Build time failures

include

missing files 

make-jars.pl doesn't like missing files from it's jar.mn

search engines missing 

the search-engine packaging doesn't like missing files

installer problems 

NSIS currently only supports UI strings in the right windows code page. So while we have the installer files encoded in utf-8 on disc, the build process converts those to the encoding specified in toolkit/installer/windows/charset.mk. In addition, there is build logic to test for a change in installer variables.

Runtime failures

missing localized strings 

There's no provision in our runtime l10n architecture to deal with missing localizable strings. Doesn't really matter if it's just a single string, or a complete file. So a missing entity in a DTD file causes a YSOD. That can be as bad as the browser not starting up.

encoding issues 

All our files are treated as utf-8 at runtime (- windows installer and updater), and if they're not, bad things happen. I recall crashing once.

grammar errors 

Real bugs, like <!ENTITY foo "< bad"> do trigger parsing errors, too.

bad variables 

Some of our localizable strings contain variables to be replaced, either entity references in DTD files, or %S stuff in .properties. If those don't match the callig code, bad things happen, including crashes.

Now this pretty much wraps up the failures, before we go on with addressing those, there's more

l10n status

Right now, we have 121 l10n tinderbox pages listed on showbuilds, each of which show three platforms, and something between 1 and 4 products. Just for trunk, it's 63, and I'm not counting the 1.8.0 boxens at all, those don't seem to have any builds these days.

Going into Firefox 3, we lacked any understanding of how those trees were looking, and we're still facing the same problem for Thunderbird 3. The lack of information here probably hid problems that are now delaying some of our localizations.

Nowadays, creating shipped-locales is already a tools-assisted process.

There are different customers of l10n status, too. There's the release drivers, that need to understand which locales they want to ship, and if they can. There's localizers, too, for which it's essential to find out what to spend their volunteering time on.

Fixing stuff

The first phase of fixing stuff is trying to make as many runtime errors actually build time errors.

The primary tool for this these days is mdc:Compare-locales. There are different versions of that flying around, with varying powers.

compare-locales open items

• check variables

compare-locales isn't really run as part of the build, but it's called by the build automation framework. Tinderbox is currently calling into the old perl implementation, l10n.m.o is calling into the python version above, and does so actually before the build, so things break when they're broken.

The next step in this direction is to actually do something reasonable for common failures, i.e., missing strings. This piece is nick-named l10n-merge (bug 399014). l10n-merge enables us to create builds of incomplete localizations, fixing grammar errors is beyond its scope, though. l10n-merge is now an option to compare-locales.

l10n-merge amends existing files with missing strings, and doesn't touch missing or good files. JarMaker.py picks up the merged files on demand, then the l10n files and as last fallback, again on demand, en-US files. The Makefile targets in browser/locales react to the LOCALE_MERGEDIR variable to switch l10n-merge on or off.

Buildbot

Switching over to buildbot. So the basic idea goes like this

• whenever you have to:
  • get the en-US source
  • get the l10n source
  • for each affected locale and product:
    • compare-locales
      • possibly l10n-merge
    • if those are cool:
      • build
      • (runtime test)

There are a few fuzzy terms here, I'll go into detail in some not-so random order.

build means repackaging an existing binary, and we should run our source checks against the source of that build. There is a 'ident' target in locales/browser/Makefile.in to get that, at least for hg builds.

whenever you have to is another interesting question, I don't have a full answer to that. The goal would be "when there's a good reason to build this app for this locale". For the l10n side of things, this is involved, but with a clear answer. When a localizer checks in, build all products that that check-in affected. More on that below.

For en-US, things are more confuzzled in my head. Let's take a check-in to mozilla-central. That may or may not affect localization, if it does, it would impact all locales for all apps that include that localizable string. Butt. At the time of that check-in, there's no build with that change yet. I think there's some value in running compare-locales at the point. I guess for en-US, we really need to figure out which builds we want to repackage, and what their source is. Right now, firefox nightlies would be good again, for an actively evolving line of development, being more agile is an option. Like, if localizers only have 24 hours to fix a late-l10n issue, we shouldn't wait for the next nightly to tell them. Not sure how much horse power a full repack run on each nightly would take, and for how many products we want to cough that up. Right now, linux cycles around 1.5 hours on l10n, dep en-US builds are around 4 per hour. Just doing a full compare-locales run is a lot quicker, though. Maybe compare-locales tip against tip, and l10n-merge against the time-stamped source of the build. (Editors note, the last two sentences are old, in particular the times.)

Lessons learned

Here is a list of things that I tried and that didn't work out very well

builder per build

Creating one builder per locale and app and platform doesn't scale. It's not really helpful in watching the waterfall, and, as there are naturally tons of builders per slave, it raises a bunch of interesting load balancing problems.

resolved 

I use build properties to denote application, locale, and tree (branch combo, set of locales)

Scheduler.fileIsImportant

fileIsImportant sounds like a cool idea to only build l10n-impact changes, but it leaves all non-important files in the changeset, which, once you're master is pulling /cvsroot and /l10n on separate branches, is just evil.

Current arch on l10n.m.o

Right now, I'm running a bonsai poller per branch and rep, so, as I'm only running trunk for l10n right now, two bonsai pollers. For Mercurial, there is a poller for each hg repo (mozilla and comm-central at this point). There is a single scheduler, which is using helper objects (Dispatcher) to map those changes to tree/application/locale tuples to be built.

Then the scheduler submits non-mergeable build requests, which the respective builders pick up. The scheduler sets build properties to communicate the locale and application/branch combination to the respective builders.

Outlook

In this scheme, application and locale and such are only available as build properties, that is, finding the latest 15 German builds is hilariously costly in standard buildbot. Thus I added a status plugin that stores state in a real database. I intend to extend that to match buildbot better, and then use that database to actually persist the status of pending builds, so the scheduler can go dumb again, and things would be restart and reconfig friendly.