Gaia/Build/RefactoringToNodejs
Contents
- 1 Overview
- 2 Profit from Node.js
- 3 Plan
- 4 Q & A
- 4.1 Which Gaia build task will continue to rely on b2g_sdk after Node.js transition?
- 4.2 What are the requirements removed/added of building Gaia before and after the transition?
- 4.3 Why are we giving up making Firefox as the only Gaia development dependency? Isn't that better and more inviting?
- 4.4 How do we ensure NPM package access to all parties, e.g. partner in China or Try servers
- 4.5 Buildbot should pre-install proper command tools (node.js, gulp) if we decide to use gulp, right?
- 4.6 Would the new build system slower than we have right now?
- 4.7 Should we choose io.js instead of node.js for developing new build system?
- 4.8 File dependency resolution to improve incremental build
- 4.9 Task dependency and parallel build
- 4.10 If we decide to build up our build system by GNU Make tool chain, considering to split up every task into small tasks would be very hard to implement. Furthermore, splitting into many small tasks mean that you have to deal with more I/O operations because you need to write more temporary file for communicating within each task, such thing will slow down performance
Overview
Our current build system comes with a Makefile, with each target took up to JavaScript scripts running in the Jetpack environment (XPCShell + CommonJS module loader). The JS runtime environment is quite unique and therefore comes with a few downsides:
- A lot of OS-specific detail leaks through, making modification hard to debug
- While providing a CommonJS interface, the environment does not come with common Node.js modules (e.g. fs, path, etc.), making reuse of code hard and creating new features harder.
In contrast, Node.js and the NPM packages runs on it can do things entirely simply in JavaScript. By substitute runtime dependency from XPCShell to Node.js, we would get a stable and common runtime environment, and gain access to NPM packages.
It's worthy to pointing out not building Gaia in XPCShell means we no longer considering making Firefox the sole dependency to build Gaia an eventual goal (i.e. develop Gaia with WebIDE & a "build Gaia" add-on and no extra dependency). Yet, considering the reward we decided using Node.js is the better way moving forward.
Profit from Node.js
- Easier to catch up: Web developers are familiar with JavaScript, also Node.js is the most popular and acceptable for those who are familiar with client-side JavaScript. It could attract more contributors to involve and improve Firefox OS.
- Abundant tool: NPM is a well-known build-in package manager of node, such tool is commonly used for web developers for years. It will make your life easier that developers can install third-party libraries and prevent us to reinvent the wheel if there is no such library provided by XULRunner.
- Pure JS: It would be great if we build up a pure JS system. To re-implement Makefile into JS is a hard work without third-party library, NPM provides a bunch of packages that let us build our pure JS system more quickly, flexible and maintainable.
Plan
Node.js refactoring plan will be split into 4 milestones .
- Developer:
- Ricky Chien (MoCoTPE)
- Scott Wu (MoCoTPE)
- Total expected schedule: 18 - 22 weeks (5 months)
Milestone 0
To ensure our work works on all environments and operating systems, we would like to create test suites on Gaia-Try/Try/b2g-inbound, namely (Gbn/Gbun) tests. They will be keep hidden and red until M1 is complete, as a proof for achieving M1 (explaining below).
- Deliverable: Enable test suites for node build script
- Expected schedule: 2 week
- Breakdowns:
| ID | Summary | Priority | Assigned to | Status | Resolution |
|---|---|---|---|---|---|
| 1131469 | Add RUN_ON_NODE switcher in Makefile | -- | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1207073 | RUN_ON_NODE=1 make is broken | -- | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Milestone 1
Tackle each of the build system core functions (parallel build, webapp-manifests, preferences, webapp-optimize, webapp-zip... etc). All of them exist certain specific code only use for XPCShell. We would also like to finish the previous attempt to wrap Node.js APIs into high-level Utils in (build/node-utils.js) (See also: bug 955988). During this milestone, we will ensure the build system continue to work on both XPCShell setup and Node.js. Once we complete all the work for the milestone, Gbn and Gbun should be green and set visible.
- Deliverable: Complete the implementation of node build scripts that coexist with XPCShell and passes Gbn / Gbun tests
- Expected schedule: 4 - 6 weeks
- Breakdowns:
| ID | Summary | Priority | Assigned to | Status | Resolution |
|---|---|---|---|---|---|
| 955988 | [meta] Running build scripts on node.js | -- | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 955989 | Running additional-extensions.js and download-manager.js on node.js | P2 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 955994 | Running preferences.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 955996 | Running svoperapps.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 955997 | Running webapp-manifests.js on node.js | P1 | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 955998 | Running webapp-optimize.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 955999 | Running webapp-zip.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1130934 | Refactoring uitls.js, utils-node.js, utils-xpc.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131468 | Running app.js and build-app.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131471 | Enable Gbn & Gbun for Node.js build script | P2 | Ricky Chien [:rickychien] (inactive) | RESOLVED | WONTFIX |
| 1131496 | Running build-app.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | DUPLICATE |
| 1131497 | Running build-test.js on node.js | P2 | Scott Wu [:scottwu] | RESOLVED | FIXED |
| 1131499 | Running clean-stage-app.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131500 | Running contacts-import-services.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131501 | Running copy-build-stage-data.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131502 | Running copy-common-files.js on node.js | P1 | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 1131503 | Running csslint.js on node.js | P2 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131504 | Running download-manager.js on node.js | P2 | Ricky Chien [:rickychien] (inactive) | RESOLVED | DUPLICATE |
| 1131505 | Running homescreen-manager.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131506 | Running import-config.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131510 | Running settings.js on node.js | P1 | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 1131511 | Running jsonlint.js on node.js | P2 | Scott Wu [:scottwu] | RESOLVED | FIXED |
| 1131512 | Running keyboard-layouts.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131513 | Running media-resolution.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131514 | Remove deprecated watcher.js and monitor.js | P3 | Scott Wu [:scottwu] | RESOLVED | FIXED |
| 1131515 | Running multilocale.js on node.js | P1 | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 1131516 | Running post-app.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131517 | Running post-manifest.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131518 | Running pre-app.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131519 | Running push-to-device.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131521 | Running r-wrapper.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131522 | Running rebuild.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131524 | Running shared-utils.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1131526 | Running webapp-shared.js on node.js | P1 | (Inactive after June) George Duan [:gduan] [:喬智] | RESOLVED | FIXED |
| 1131527 | Set tryserver Gbn / Gbun visible | P2 | Ricky Chien [:rickychien] (inactive) | RESOLVED | WONTFIX |
| 1138773 | Running search-provider.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1240678 | Running email build.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1240740 | Running all app's build.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1242326 | Running keyboard build.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1242327 | Running operatorvariant build.js on node.js | P1 | Ricky Chien [:rickychien] (inactive) | RESOLVED | FIXED |
| 1243351 | Running scan-appdir on node.js | P2 | Scott Wu [:scottwu] | RESOLVED | FIXED |
41 Total; 0 Open (0%); 41 Resolved (100%); 0 Verified (0%);
Q & A
Which Gaia build task will continue to rely on b2g_sdk after Node.js transition?
b2g_sdk run on ...
- Building Gaia (including sub-commands)
- Running build integration test
- Running push to device
- Running marionette integration test (must)
All of the above can be replaced with node.js smoothly except marionette test. Since the B2G Desktop is the test target for these tests, it's impossible to remove from our dependency. However, it's not necessary to download a b2g_sdk package if someone do not execute marionette test.
What are the requirements removed/added of building Gaia before and after the transition?
- Removed:
- Build Gaia without downloading b2g (but we still need to download b2g while launching marionette tests)
- Added:
- Requiring Node.js as a build prerequisite -- User must make available
nodebefore building Gaia. - We will need to download NPM packages and other NPM command-line tools
- Requiring Node.js as a build prerequisite -- User must make available
Why are we giving up making Firefox as the only Gaia development dependency? Isn't that better and more inviting?
Weighting the benefits, we believe requiring Node.js won't frustrate many users to get started with hacking Gaia, since Node.js comes with installers for all platforms. Again, using Node.js means we can build up new features quickly by NPM and also speed up rewriting Makefile to pure JS.
On the other hand, try to rewrite Makefile from scratch in XULRunner is proven to be hard. In sum, we believe that NPM ecosystem will bring lots of benefits more than XULRunner.
How do we ensure NPM package access to all parties, e.g. partner in China or Try servers
It should continue to work if we keep hosting our gaia-node-modules mirror to solve such situation. Replacing the make-shift gaia-node-modules mirror is not the scope of this work, but we are looking forward to find a way to do that.
Buildbot should pre-install proper command tools (node.js, gulp) if we decide to use gulp, right?
Buildbot has already installed node environment except gulp. For gulp, we can pre-install gulp -g in buildbot, or download gulp while first time build then launch it by relative path.
Would the new build system slower than we have right now?
Although we believe that M1 & M2 won't effect obviously on performance, it probably happen on M3. So we should ensure that we take full advantage of multi-core systems to speed up our workflow in Node.js environment (with gulp).
Should we choose io.js instead of node.js for developing new build system?
In my experience, build scripts doesn't take too many ES6 features, may be able to pass "node --harmony" in node.js environment. So there are no such immediate reasons to choose io.js in current phase. However, io.js provides more complete ES6 features and we'll consider to transfer to io.js in the future. We would have to figure out if buildbots can work with multiple Node.js versions (maybe with nvm) before using io.js.
In short, targeting io.js is not the scope of the current plan.
File dependency resolution to improve incremental build
It would work by using existed gulp plugins (ex: gulp-resolve-dependencies) or adding new feature into plugins.
Task dependency and parallel build
Gulp is not as much convenience as Makefile and doesn't support parallel building natively. Therefore, although plugins could help us build up a same parallel architecture of our current build system, it’s hard to split every task into dependency tree to take full advantage of multi-core CPU (like "make -j8").
If we decide to build up our build system by GNU Make tool chain, considering to split up every task into small tasks would be very hard to implement. Furthermore, splitting into many small tasks mean that you have to deal with more I/O operations because you need to write more temporary file for communicating within each task, such thing will slow down performance
It would work by condition plugins for instance gulp-if, gulp-filter, gulp-ignore…, but introducing these plugins mean that lots of condition logic will pollute inside gulpfile.js. (Can we solve this problem in GNU make tool chain?) Repack zip could be achieved by adding a new feature in gulp-zip (Issue is filed gulp-zip/issues/45) and customize hasChanged comparator callback.
