Gaia/Build/NewBuildSystem

Overview

After surveying, we decide to split build system into 2 steps: configure step and build step [1]. For detailed information, please visit change proposal in dev-gaia Update on Gaia build changes: A configure step, and Make v.s. Gulp.

New architecture plan is going to construct a powerful configure to generate an appropriate Makefile for build back-end as know as build step to consume. Thus, GNU Make will be adopted as our firstly choose build back-end tool. We're going to reference the architecture of CMake, they also have adopted the concept of splitting configure step and build step early and it's a fully fledged tool already have good solution to build up a powerful configure such as detecting the changing of ENV variable and computing those complicated file dependency to decide whether to re-generate Makefile or not.

Learning from CMake could be a huge help for us mitigate circumstances while we're fighting in new architecture transition. A study of CMake is ongoing within our implementation phase, and moreover we are trying to dig into CMake from it's source https://github.com/Kitware/CMake and play with those popular CMake projects such as opencv or libwebsockets

Our goal is to fasten incremental build and easier to customize and manage new/current building step. However, new architecture would be slow down clobber build which means first time build would take more time than old build system. We still believe that the only thing developers are concerned is the time of incremental build . As a result, the benefit of incremental build improvement is more importantly than clobber build. All of above glossary can be referred to Gecko build system document Glossary.

New build system architecture:

Configure

• This plan focuses on constructing a powerful configure as know as configure.js.
• There is a root configure.js which requires and executes every sub-configure.js (ex: svoperapp.js, apps-configure.js, bluetooth/configure.js...etc) to produce all Makefile scripts.

Build config manager

• A manager to provide an elegant way to tackle with build config, the ideal manager should be able to:
  • Load a default build config file
  • Combine default build config with user specified config to produce final config object
  • Store final config into cache
  • Provide well-defined API

Configure cache

• A cache mechanism to keep the status of last time configuring and speed up re-configure process. The notion of configure cache comes from CMakeCache.txt in CMake, so we're going to introduces a configure-decider to take responsibility of keeping these status.
• Configure status is useful to decide whether to re-run/skip a sub-configure step, status depends:
  • build config
  • file dependency for each sub-configure module

It means any changes of above things is likely to produce a different Makefile structure, so we should re-run configure to generate appropriate Makefile.

Well-defined common API

• Provide an interface to help you create your custom configure script or app-configure script.
• Built-in APIs are able to keep tracking file dependency when build script is trying to do anything related to a file. It strongly connects to configure cache and store these file dependency in cache for further use.

Robust incremental build

• Incremental build should generate same result every time no matter how build script be invoked, allowing user to launch 'configure' or 'make' command from everywhere.
  • We prefer to make default Makefile target redirect to configure and everything will work fine under this protective umbrella.
• Executing a Makefile in sub-directory will generate the same subset result as expect too.
  • Default behavior of executing a Makefile in sub-directory would take build config from cache as default argument, of course, user still could overwrite it by specifying build config.

Cleaning up

• No logic in $GAIA/Makefile
• Eventually run JavaScript with node from the command line with simple command (|node myscript.js|)

Plan

• Developer:
  • Ricky (MoCoTPE) (irc #gaia: rickychien)
  • George (MoCoTPE) (irc #gaia: gduan)

• Total expected schedule: 10 - 12 weeks

Milestone 1

Initial landing step. Since our new build system plan is around splitting a build process into two steps including configuring and building. This stage will focus on building up a configuration modules - configure.js and using GNU make as our build back-end to produce final target. It means configure.js part will run on XPCShell and require existed build modules at this stage in order to keep build system working properly. Finally, the new build system is going to be landed in code-base and replace current build system.

• Deliverable: A new Gaia Makefile target to trigger configure.js modules should be landed in Gaia code base. The configure.js can generate an all.mk (a temporary new Makefile) and produce a same PROFILE_DIR as usual.
• Expected schedule: 4 weeks
• Breakdowns:
  • New config step: adding configure.js to generate strong dependency all.mk and execute it.
  • New re-run step: in 2nd make, new build will check the change files and ENV to decide whether to regenerate all.mk or not, and then execute it.
  • Refactoring webapp-shared.js and copyToStage: based on new build system architecture, we'll refactor webapp-shared module and copyToStage into config and execute task in order to save time when re-run make and check if any possible risk before Milestone 2 (refactoring all build modules and app/build.js).

Full Query

ID	Summary	Assigned to	Status	Resolution
1154582	Build system configure - adding configure.js to generate strong dependency all.mk and execute it	(Inactive after June) George Duan [:gduan] [:喬智]	RESOLVED	FIXED
1154583	Configure re-run step - configure.js check the change files and ENV to decide whether to regenerate all.mk or not	(Inactive after June) George Duan [:gduan] [:喬智]	RESOLVED	WONTFIX
1154585	Refactoring webapp-shared.js for new build system	Ricky Chien [:rickychien] (inactive)	RESOLVED	FIXED
1168287	Merge new build script to master		RESOLVED	WONTFIX

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);

Milestone 2

After initial landing step being done, we should abandon GNU make's PHONY targets as many as possible, so we could have a faster build by taking the advantage of GNU make's incremental build. The configure.js should be able to connect with different build back-end. In order to achieve this aim, we eventually come out a new phase - config.status [2] for preparing the next step (replacing build back-end). The idea of config.status is charged with the task of tell back-end tool how to build the tree which includes scanning the build system definition and constructing a DAG data structure for representing task dependency and supporting parallel build natively. It means config.status will be generated by configure.js and output a build-file for back-end consumption (e.g. a `make` back-end will write a `Makefile`).

• Deliverable: Abandon GNU make's PHONY targets as many as possible to speed up incremental build and implement config.status phase.
• Expected schedule: 6 - 8 weeks
• Breakdowns:
  • Refactoring all modules and app/build.js into small tasks and split each module into configuring + building steps.
  • Implement config.status which will introduce DAG structure to configure step.

Full Query

ID	Summary	Assigned to	Status	Resolution
1154586	[meta] Refactoring all modules and app/build.js into small tasks and split each module into configuring + building steps		RESOLVED	INVALID
1154587	Implement config.status which will introduce DAG structure to configure step		RESOLVED	WONTFIX

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Milestone 3

We're going to measure incremental build performance from GNU make back-end. If the result is as good as our expectation, the decision of choosing build back-end would be GNU make. Otherwise, the new milestone would be targeting on constructing the new build back-end tool. After surveying in milestone 3 of previous plan RefactoringToNode, we believe that the Gulp or streaming base build back-end could offer a better performance especially in clobber build. Therefore, in this step, a streaming base build system which will be implemented.

• Deliverable: A blazing fast JavaScript build system
• Expected schedule: ? weeks
• Breakdowns:
  • Implement a JavaScript build system as our build back-end tool instead of GNU make

Q & A

Why new build system is quicker than current one?

• Clobber build
  • Current build: It manages process runner by app, which means the there would be idle processes if the last app takes longer time to complete.
  • New build: It manages process runner by file. The idle processes won't remain too long until all tasks complete.
• Incremental build
  • Current build: If one file has changed, it will rebuild all the app (see below).
  • 
  • New build: If one file has changed, it will only trigger all the related tasks (see below).
  •