makefilegen - Container makefiles

bug 623617 Decursify the build system

Problem

Traditional recursive make builds can introduce a non-trivial amount of overhead for builds.

• Directories must be traversed to perpetuate recursion, even within parent directories where no processing is required.
  • Traversal can be expensive for platforms like windows where each shell/process spawned will impose non-trivial overhead.
• Common optimizations within this hierarchy can include pruning or explicitly listing directories for traversal as a way to reduce the workload. Traversal of doc/ and pref/ directories would be skipped as an example. One inherent problem with the setup is maintenance of hardcoded lists and problems that surface when usage changes over time.

Proposed Solution

As an alternative to recursive traversal(s), container makefiles can be generated that able to traverse the sandbox a single time.

• Traverse the filesystem once and build a list of all interesting files.
• The list will be grouped by name, extension and/or content.
• Post traversal iterate over each file group and use templates to autogenerate container makefiles.
• A container makefile is a standalone makefile that has been configured to iterate over all files within a given group and process/invoke each in turn.

Benefits

• Removal of recursive build overhead. make will only be invoked within directories and for specific targets where and when they are needed.
• Threading is implicit within the setup
  • At a minimum, processing individual elements within the container will be threaded by default.
  • Support multiple thread levels
    • container - $(MAKE) -j[n] -f $container can be used to limit the number of directories traversed concurrently. This should help reduce process starvation and resource contention for the entire build.
    • elements - $(MAKE) -j is later used to process elements within the container makefile. This will allow unlimited job processing within a finite number of directories processed concurrently.
    • Templates can be configured to handle special cases and limit their impact on the overall build. Alternate file/directory target/processing rules can be specified which use $(MAKE) --jobs=1 --load=[n] to isolate resource hogs so they cannot overwhelm the entire build or bury a machine deep in swap.
• Scaling - processing new targets is almost a free lunch. Files have already been gathered, simply iterate over the new target/group and generate a container makefile to process the target.
• Setup will help isolate elements within the build
  • Support a more modular build with simplified elements.
    • Elements are built using only relevant elements from rules.mk
    • building elements in isolation can often uncover unwanted/un-necessary dependencies.
• Structure is able to support processing of arbitrary targets.
  • gather makefiles - process targets non-recursively
  • gather Makefile.am - autogenerate all Makefiles beneath $(MOZ_OBJDIR)
  • process makefile targets: export, deps, libs, tools, install, etc
  • gather *.py.test - launch standalone unit tests, python tests in this instance
  • gather *.latex, *.doxygen, *.jdoc - gather documentation files and generate derived content from them.
  • gather configure* - invoke configure, dependencies can be added to reduce the frequency configure needs to be run.
• Tier targets - existing functionality
  • Processing can be supported by auto-generating additional elements within the container makefile and processing them in a fixed order.
    • tier1=$file_list, tier2=$file_list.
    • all: tier1 tier2 moz_libs
  • Dependencies can also be added/generated to ensure synchronization between tiers allowing make to thread as needed.

Pending

• Identifying sandbox dependencies that interfere with threads and parallel builds.
  • Add conditional processing for .../unix/.../Makefile, Makefile.win and any other platform variants.
  • Optimal answer would be removal of special case logic completely by moving platform specific logic into sources or the makefile making it easier to add functionality on the "other" platform when the need arises.
• Add special handling for files that exist beneath a m!/tests?/! subdirectory
  • Separate /test/ files from the main list of files.
  • Write the main list of files into the makefile as usual
  • When appending /test/ files wrapper the list with an ifdef ENABLE_TESTS conditional
  • bugs: 6870246, 687511, 687515 & 515374

Process Overview

• makefilegen will be initiated by client.mk/rules.mk.
  • Invoked by config/makefiles/makefilegen.mk (dependency driven)
  • Script config/makefiles/makefilegen will be launched.
    • When config/makefilegen/* is modified.
    • When sandbox makefiles (both autoconf and generated) are created or modified.
    • Initial safety net: If generated makefiles are >[n] days old (default=1 day).
• From the sandbox root gather a list of all interesting files.
• Based on filename, extension or content identify supported functionality.
  • makefile targets: export, libs, tools, deps, install
  • scripts: configure.sh
  • extension: unit tests, actions: compile, install,
• For a given list of targets iterate over the file list
  • Generate a raw list of files that are relevant to the target.
  • Generate a container makefile able to process that target
    • Select a makefile template
      • Template will be used to generate a container makefile.
      • Generated makefile will dictate how the list of files/commands should be processed.
    • Dynamically populate template makefile macros
      • Define a list of files containing the command target of interest.
      • Define a dependency list: target list + autconf, autogen, ... source files that make can use to determine when container makefiles need to be re-generated.
    • All dependency lists and makefiles generated will be written into $(MOZ_OBJDIR)/generated.
    • client.mk or rules.mk will process each of the collection makefiles in turn.
      • cd $(TOPSRCDIR) && make -f $(MOZ_OBJDIR)/generated/makefile.gen.tools

Deployment Options

• Problems that will need a workaround
  • explicit use of $(MAKE) -j1 within the sandbox to prevent an odd deadlock condition on windows.
  • source dependencies / circular dependencies within the source tree.
  • serial -vs- parallel target/synchronization issues. Build should be fully threaded with make dependencies maintaining synchronization.
  • existing problems - modifying makefiles, esp the included/library flavor have a nasty habit of exposing all of the creepie crawlies and latent problems lingering in the sandbox. A lot of bugs will need to be opened and problems resolved.
  • project directories which assume $top is a subdir of the sandbox root.
    • js/src
    • nsprpub/

• Option #1 - maintain exclusion lists
  • gather all existing targets by name and generate containers for them.
  • maintain exclusion lists for problem areas.
  • may be problematic as new makefiles would not be excluded by default.

• Option #2 - new target naming convention moz_%
  • Selective processing, a makefile will only be considered when the target is present.
  • self documenting - targets would explicitly list what a makefile can be expected to do { sometimes hidden through macro and library makefile use }.
    • moz_%: % - can be added in rules.mk so moz_* targets will act as a simple wrapper around real targets.
    • Will allow a gradual transition into building targets non-recursively rather than all at once.

• Option #3 ?

Elements

• $(TOPSRCDIR)/client.mk - generator
  • config/makefiles/makefilegen.mk
  • config/makefiles/makefilegen.pl
  • config/makefiles/makefilegen.pm
  • config/makefiles/test/*

• generated files
  • $(MOZ_OBJDIR)/generated/deplist.makefiles
  • $(MOZ_OBJDIR)/generated/deplist.configure
  • $(MOZ_OBJDIR)/generated/makefile.gen.export
  • $(MOZ_OBJDIR)/generated/makefile.gen.libs
  • $(MOZ_OBJDIR)/generated/makefile.gen.tools
  • $(MOZ_OBJDIR)/generated/...

• client.mk targets
  • makefilegen - main workhorse target/receipe, dependency of configure and will be invoked after makefiles have been generated.

• config/makefiles/container.mk
  • makefile containing a target rule and macros for launching make on a container makefile.
  • CONTAINER_ARGS - command line macro used to pass -j & -l when make is invoked (default=--jobs=3)

• build/autoconf/make-makefile, make*
  • Generate makefiles from Makefile.in templates.

• building
  • gmake -k -f client.mk
  • gmake -f obj-ff-dbg/generated/makefile.gen.export
  • gmake -f obj-ff-dbg/generated/makefile.gen.libs
  • gmake -f obj-ff-dbg/generated/makefile.gen.tools

General Problems

• make clean - will launch the configure script, should be a nop.
• Why is TOOL_DIRS= involved with testing, convenience ?
  • toolbox/crashreporter/Makefile
  • ifdef ENABLE_TESTS TOOL_DIRS = test endif
    • conditional to only generate libs:: while running in test mode ?
• export target building [[1]] beneath mozilla-central rather than $(MOZ_OBJDIR).

js/src/editline/Makfile.ref

• Referenced by testing/sisyphus/bin/build.sh
  • use the old-style Makefile.ref build environment for spidermonkey
  • logic has been moved into a separate repository. Deletion submission to mozilla-incoming is pending.

Utilities

makefile templates - form letter generation

• config/makefiles/makefilegen.pl
  • Given a list of directories, templates to use and output directory.
  • Search a given set of directories for files of interest
  • Iterate over groups/classes of files to use as input
  • Load a template makefile, perform token and list substitutions
  • Post-process generated makefile if needed (for ex: add ifdef TEST_ENABLED/endif)
  • Write generated makefile into the output directory $(MOZ_OBJDIR)/generated/

makefile templates

• SRC: config/makefiles/templates: makefile.tmpl, makefile.configure.tmpl, makefile.idl2h.tmpl
• <|${token}|> strings are used to perform text substitution
  • <|time|>, <|makefile_template|>
  • <|var=token^makefile_var += $token|>
  • <|auto_alldeps|> - A list of all file gathered for the current file class type.
  • <|auto_target_files|> - a list of all makefiles who contain a given target.
  • <|$token|> - Any variable value $token declared by makefilegen

addtgts

#
# For a given list of sandbox directories
# add moz_{tgt} container makefile targets in bulk for processing
#
cd mozilla-central
make -f config/makfiles/makefilegen.mk addtgts dirlist="db editor"

# cleanup: delete all local changes beneath db/ and editor/
hg revert --no-backup db editor

container-only processing

# 1) Add container makefile targets for the db and editor directory
# 2) invoke configure
# 3) generate container makefiles
# 4) process container makefiles
# 5) exit
make -f config/makfiles/makefilegen.mk addtgts dirlist="db editor" MOZ_OBJDIR=$path
gmake -f client.mk distclean
/bin/rm -fr $(MOZ_OBJDIR)

make -f config/makfiles/makefilegen.mk addtgts dirlist="db jpeg"
gmake -f configure
gmake -f client.mk USE_CONTAINER=1

full build

configure/rules.mk, search for "$(container_make)"

setup

cd mozilla-central

# import the makefilegen patch
hg qimport patch.gml-proto

# modify makefile templates to taste
( cd config/makefiles/templates; ls )

# modify sandbox makefiles to include special 'moz_' container targets to be processed
gmake -f config/makfiles/makefilegen.mk addtgts MOZ_OBJDIR=$path dirlist="db jpeg"
** config/makefiles/addtgts.pl

# Off to the races
gmake -f client.mk configure
 -or-
gmake -f client.mk 2>&1 | tee log.build

examples

https://wiki.mozilla.org/User:JoeyArmstrong/makefiles/ContainerMakefileSample

Build Problems

• Platform specific makefiles: Makefile.win, .../solaris/Makefile.in, .../windows/Makefile.in
• [tier] dependencies
• testing/sisyphus
• bug 680246 - tickled a problem with packager.mk on win32

todo

• report build profiling
• publish make.c patch
• report baseline for nop builds & traversal
• elapsed time for processing *.idl files, 1700 shells x 3 (nsinstall)