User:JoeyArmstrong/makefiles/threadsafe mkdir

Intro

Directory creation within makefiles can now be handled in a consistent manner that is both thread safe and dependency driven. Directories will be created on demand and the mkdir command invoked within a shell only when needed. bug 680246 has added utility makefile logic to support the functionality.

makefiles


MKDIR ?= mkdir -p
TOUCH ?= touch

# Dependency generator function
mkdir_deps =$(foreach dir,$(getargv),$(dir)/.mkdir.done)

# Target rule to create the generated dependency.
%/.mkdir.done: # target rule
    @$(MKDIR) -p $(dir $@)
    @$(TOUCH) $@

Problems

inlined "mkdir -p" calls have built in overhead. A shell must always be spawned even when directories exist.

foo/bar.out:
    mkdir -p $(dir $@)
    touch $@

Placing pre-requisite dependencies on directories can be problematic as they contribute to generation of dependencies that are randomly stale (example: [dirdep.mk]).

foo/bar.out: foo
    touch $@
foo:
    mkdir -p $@

Race conditions for make -j creating directories and targets.

why the extra dep? (.mkdir.done)

Placing dependencies on directories is a very old makefile problem. They contribute to generated targets being artificially stale even after make has been run. Make will query the inode table when making timestamp comparisons. Unfortunately whenever directories/files/links are created/deleted/modified/renamed within a directory the inode table alteration will be detected by make as a stale dependency/target to regenerate. Generated targets that have the directory as a pre-requisite are re-created whenever the directory timestamp is newer than the target. This can occur immediately after make has been run.

mkdir_deps

A simple workaround for these problems is to use real makefile dependencies to control if/when a directory should be created.

The user function 'mkdir_deps' can be called, passing in a list of directory paths. The function will return a list of arguments suffixed by the filename '.mkdir.done'.

include $(topsrcdir)/config/rules.mk
deps = $(call mkdir_deps,foo,bar,tans)
$(info deps=$(deps))

deps=foo/.mkdir.done bar/.mkdir.done tans/.mkdir/done

mkdir_deps target rule

The second component if the library is a wildcard target rule that will match (~directory) dependencies suffixed by '.mkdir.done'. The rule will use the given target name to create a directory and dependency file contained within that directory.

%/.mkdir.done:
    $(MKDIR) -p $(dir $@)
    $(TOUCH) $@

operation

Using dependencies for directories have a few benefits. They convey filesystem state information to gmake so the command can decide when a directory should be created. The dep can be mentioned as often as needed as a pre-requisite to trigger directory creation. Overhead for each reference will be a file stat or far more likely checking cached information.

With dependencies in use, existence of a .mkdir.done file will short-circuit all directory creation attempts because the target is up-to-date. To resolve race conditions and critical section problems "mkdir -p" was used. When multiple threads attempt to create the same directory one thread will win and create it. All other threads will attempt creation, fail and ignore status because the directory already exists.

USAGE

Multiple makefile targets

By far the simplest syntax to use for directory creation is when multiple targets in a makefile require a common directory. Assign a list of directories to GENERATED_DIRS then include rules.mk. Dependencies will be generated to automatically create and remove (clean target) directory deps as needed.

dirs = foo bar tans
GENERATED_DIRS = $(dirs)
include $(topsrcdir)/config/rules.mk

libs::
    find $(dirs) -ls

Pros:
- simple syntax
- automated creation and cleanup.
Cons:
- Shotgun approach. Dependencies may be created that are irrelevant to a target.

Individual makefile targets

GENERATED_DIRS may be a fast and easy answer but it is not always correct. Sometimes a more refined answer may be needed . Logic requiring directory creation by common logic residing within an included makefile or may be target specific can directly utilize the mkdir_deps function to create pre-requisites. An example of unwanted dependencies would be "gmake check" creating a directory only used by the install target.

Creating individual targets

Declare a local make var for the directory dependency or use ${target}-preqs if a target is using several.
$(call mkdir_deps,dir[,dirN]) to generate dependencies
Assign deps to targets as needed
Declare 'GARBAGE_DIRS =' if 'gmake clean' targets are allowed to remove the directories.

dirs = utils
GARBAGE_DIRS += $(dirs) # gmake clean

include $(topsrcdir)/config/rules.mk

# allocate dep once for re-use
dir-utils = $(call mkdir_deps,$(dirs))

# inline dep allocation for pre-requisites
install-preqs =\
  $(call mkdir_deps,install_here) \
  $(dir-utils) \
  $(NULL)

install: $(install-preqs)
  do_something_constructive.sh

export: $(dir-utils)

Pros
- surgical approach. Directory creation can be target specific.
Cons
- manual clean target removal required.
- care must be taken when using them as pre-requisites. If a dep is the only pre-req for a generated target unexpected results may occur.

caveats

mkdir_deps should be called after the function has been defined by "include rules.mk". Calling prior to the include will return $(NULL) rather than a dependency -- handled by make as a NOP instead of a directory creation request.
Be aware of how dependencies are being used as a pre-requisite for targets.
- Common targets [export, libs, tools] are flagged as phony and will function correctly.
- Targets explicitly marked as .PHONY: will function correctly.
- Targets that contain sources as a pre-requisite will function properly.
Targets that have no pre-requisites and are not .PHONY will not function properly w/o help:

dir = $(call mkdir_deps,mydir)
target: $(dir)
   do_something_productive.sh > $@

After the directory is created on the initial call subsequent checks will determine {erroneously} that the .mkdir.done dep exists and is older than the target so no work is required. At a minimum the target should be dependent on *.sh and any other source changes that should cause a change for the target.

client.mk example

An example from client.mk. Conditional directory creation is also supported with use of the $(if ) makefile directive.


configure:: configure-files
ifdef MOZ_BUILD_PROJECTS
        @if test ! -d $(MOZ_OBJDIR); then $(MKDIR) $(MOZ_OBJDIR); else true; fi
endif
        @if test ! -d $(OBJDIR); then $(MKDIR) $(OBJDIR); else true; fi
        @echo cd $(OBJDIR);
        @echo $(CONFIGURE) $(CONFIGURE_ARGS)
        @cd $(OBJDIR) && $(BUILD_PROJECT_ARG) $(CONFIGURE_ENV_ARGS) $(CONFIGURE) $(CONFIGURE_ARGS) \
          || ( echo "*** Fix above errors and then restart with\
               \"$(MAKE) -f client.mk build\"" && exit 1 )
        @touch $(OBJDIR)/Makefile

Using mkdir_deps, the creation calls can be written in terms of make dependencies and listed as pre-requisites for the target in place of an in-lined shell script reducing the number of shells that will need to be spawned.


configure-preqs = \
  configure-files \
  $(call mkdir_deps,$(OBJDIR)) \
  $(if $(MOZ_BUILD_PROJECTS),$(call mkdir_deps,$(MOZ_OBJDIR))) \
  $(NULL)

configure:: $(configure-preqs)
        @echo cd $(OBJDIR);
        @echo $(CONFIGURE) $(CONFIGURE_ARGS)
        @cd $(OBJDIR) && $(BUILD_PROJECT_ARG) $(CONFIGURE_ENV_ARGS) $(CONFIGURE) $(CONFIGURE_ARGS) \
          || ( echo "*** Fix above errors and then restart with\
               \"$(MAKE) -f client.mk build\"" && exit 1 )
        @touch $(OBJDIR)/Makefile

MAKEFILE: dirdep.mk

Directory dependency makefile example


# -*- makefile -*-

all: work/dep-on-dir work/dep-on-file

work/dep-on-dir: work
	date > $@

work/dep-on-file:
	sleep 3; touch $@

dep-3: work/dep-on-dir
	touch > $@

work:
	@mkdir -v -p $@

clean:
	$(RM) -r work

show:
	@echo
	/bin/ls -dl work work/*

First attempt: clean state

No targets exist, create directory 'work' and files 'dep-on-dir' and 'dep-on-file'.
Sleep to make dep-on-file newer than the containing directory 'work'.


% gmake -f dirdep.mk clean all
rm -f -r work
mkdir: created directory `work'
date > work/dep-on-dir
sleep 3; touch work/dep-on-file

Directory 'work' and both files exist so dependencies are expected to all be up-to-date.

% gmake -f dirdep.mk show
/bin/ls -dl work work/*
drwxrwxr-x 2 joey joey 4096 2012-04-12 12:12 work
-rw-rw-r-- 1 joey joey   29 2012-04-12 12:12 work/dep-on-dir
-rw-rw-r-- 1 joey joey    0 2012-04-12 12:12 work/dep-on-file

Second attempt: verify dependencies are all up-to-date

Re-run gmake a 2nd time to verify the assumption, "Nothing to be done for 'all"" should be reported.


% gmake -f dirdep.mk all
date > work/dep-on-dir

Alas no, since dep-on-file was modified 3 seconds after the containing directory 'work' was created dependencies are stale yet again and make will need to re-create dep-on-dir when called.

Third attempt: deps are hosed, tasks are being performed needlessly

Run make a 3rd time. Work is now newer than 'dep-on-file' so we are finally up-to-date.


% gmake -f dirdep.mk all
gmake: Nothing to be done for `all'.

The plot thickens

At least for this example the dependencies are now up-to-date after running gmake adnauseum (not a good practice...). The problem worsens as dependency chains expand and intra-target chains form circular dependencies that invalidate one-another or are forced to always be stale triggering rebuilds. As more dependencies are placed on work/dep-on-dir (target: dep-3) the chain will contribute to even more stale dependencies that require additional time and effort to try and resolve.


dep-3: work/dep-on-dir
    touch $@

% gmake -f makefile clean all
  [snip]
date > work/dep-on-dir
touch dep-3

## dep-on-dir, dep-3 and any targets dependent on them will be considered stale.
% gmake -f makefile all
date > work/dep-on-dir
touch dep-3

To uncover dependency problems like these, always run make twice in succession after modifying your makefiles.


% gmake clean all; time gmake all
gmake: Nothing to be done for `all'

If "Nothing to be done' is reported dependencies are functioning properly. If make reports any other progress status it is time to stop and fix the makefile. This is a clear sign that dependencies are not correct which in turn will forge gmake to spending extra effort needlessly regenerating targets, dependencies and doing random work.

User:JoeyArmstrong/makefiles/threadsafe mkdir

Contents

Intro

makefiles

Problems

why the extra dep? (.mkdir.done)

mkdir_deps

mkdir_deps target rule

operation

USAGE

Multiple makefile targets

Individual makefile targets

caveats

client.mk example

MAKEFILE: dirdep.mk

First attempt: clean state

Second attempt: verify dependencies are all up-to-date

Third attempt: deps are hosed, tasks are being performed needlessly

The plot thickens

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools