SummerOfCode/SampleApplications/1

This is one of three real applications from Summer of Code 2009 which we thought were good enough to put up as examples. Having said that, none of them are perfect, so please treat them as inspirational material, not as templates. :-)

nsIProcess2: A new interface to support piping in Mozilla

Abstract

The current nsIProcess interface in Mozilla's XPCOM module supports starting processes and passing arguments but it can't capture the output of those processes. A new interface, called nsIProcess2, will make this possible and will do it asynchronously using callbacks. Additionally, it will support Unicode on all platforms. The primary API will be a JavaScript API, but may be implemented partly using XPCOM code underneath. This functionality will be available through a JS module approach.

Content

James Boston
hiddenforprivacy@gmail.com
1-647-555-1234

Applications built on Mozilla code lack the ability to pipe I/O to or from other processes. Mozilla core contributor Dave Townsend says "It is one of the big requests I see all the time". The current nsIProcess interface in Mozilla's XPCOM module supports starting processes and passing arguments but it can't capture the output of those processes. I have consulted with Benjamin Smedberg, the maintainer of the XPCOM module, and he recommends that I implement a new interface, called nsIProcess2, that makes this possible. It will be possible to do this asynchronously using callbacks. Additionally, it will support Unicode on all platforms. The primary API will be a JavaScript API, but may be implemented partly using XPCOM code underneath. This functionality will be available through a JS module approach.

The existing nsIProcess interface has weaknesses other than the lack of support for piping. It doesn't support Unicode consistently across all platforms. Also, it is embedded in the XPCOM module, but the long term design goals for XPCOM make it a poor fit. In Mozilla 2, XPCOM may no longer offer a stable ABI and may not expose existing functionality through a public API. The long-term goal is to expose that functionality in a JavaScript API. Moreover, in XPCOM code it is not easy to pass array data around. Programmers need to specify an array and array length separately. Jason Orendorff, a JS API developer, has said that this a problem he would like solved. The existing code is also entangled with the Netscape Portable Runtime (NSPR) in a way that hampers innovations. For instance, Linux lacks Unicode support because nsIProcess relies entirely on NSPR for starting processes. Hacking NSPR is a very involved process that requires a lot of lead time because it is used in many projects independently of Mozilla applications. Benjamin Smedberg suggests that writing a new abstraction layer is far more practical.

The design of the interface will be similar to the subprocess module found in Python. Here is how it may appear as a JS API:

Components.utils.import("resource://gre/Process.jsm");

// Note: this only create the process object: the new process isn't actually
// launched until .run() is called
var p = new Process("c:\\myapp.exe");

// Set up stdin of the process
p.setstdin(null);  // the default value: inherit stdin from the current process

// open the file and use that as stdin of the new process
p.setstdin(new Process.InputFile("c:\\myinput.txt"));

// set up a writable pipe to send data to the new process
// later, you can write to the pipe using stdin.write(data)
// the writing is asynchronous...
// if buffers are full, the write() call may throw an exception
let stdin = new Process.WritePipe();
p.setstdin(stdin);

// Set up stdout/stderr of the process
p.setstdout(null); // the default value: inherit stdin from the current process

// open the file and use that as stdout of the new process
p.setstdout(new Process.OutputFile("c:\\myoutput.txt"));

// capture output and processes it via a callback
let stdout = new Process.ReadPipe();
p.setstdout(stdout);
stdout.addEventListener("dataReceived", function(d) {
  alert("Got data from child process: " + d);
});

// Add a callback to get the result code of running the process
p.addCallback("processCompleted", function(r) {
  alert("Child process finished with result: " + r);
});

// Launch the process
p.run(["argument1", "argument2"]);

This will not be implemented directly in XPCOM, but rather the Process object would wrap an internal nsIProcess2 object and hide the implementation details.

Potential new interfaces which implement the underlying functionality might be:

interface nsIProcess2 : public nsISupports
{
  void init(in nsIFile program);

  void inheritStdin();
  void pipeStdin(in nsIPOSIXOutputStream);

  void inheritStdout();
  void fileStdout(in nsIFile);
  void pipeStdout(in nsIPOSIXInputStream);

  void inheritStderr();
  void fileStderr(in nsIFile);
  void pipeStderr(in nsIPOSIXInputStream);

  readonly attribute unsigned long result;
  readonly attribute unsigned long pid;
  attribute nsIObserver listener;

  void run([array, size_is(argv)] in wstring argv, in unsigned long argc);
};

interface nsIPOSIXOutputStream : public nsIOutputStream
{
  /* the native POSIX file descriptor for this stream */
  readonly attribute unsigned long fd;
};

interface nsIPOSIXInputStream : public nsIInputStream
{
  /* the native POSIX file descriptor for this stream */
  readonly attribute unsigned long fd;
};

interface nsIPipeManager : public nsISupports
{
  nsIPOSIXInputStream createIncomingPipe();
  nsIPOSIXOutputStream createOutgoingPipe();

  nsIPOSIXInputStream readFile(in nsILocalFile file);
  nsIPOSIXOutputStream writeFile(in nsILocalFile file);
};

I think this project will be a great boon to extension developers and projects that are built on top of Mozilla. I have developed this proposal in consultation with my mentors. I have had discussions with Mark Finkle, Benjamin Smedberg, Jason Orendorff, Ted Mielczarek, Gijs Kruitbosch and other Mozilla developers . I have discussed this proposal with developers in the wider community who use Mozilla as the base for their projects. I know from comments and enquiries generated through my blogging on the topic that this is something that is wanted.

It is possible to initially develop this code as a binary component that can be packaged in an extension. This would allow the community to begin immediately using and testing it before it is ready to be merged into TRUNK. I like to blog about my coding as I do it. (See http://jamesboston.ca for my blog.) It helps generate interest and feedback if there is something people can download and use. For instance, I will start by releasing an extension that can capture I/O synchronously on one platform. This can be used by others to build extensions that need this functionality. Next I will make this cross platform. After this I will add asynchronous support. Unicode is an issue that will need to be addressed early in the development. When I feel it is feature complete I will begin the task of integrating it into Mozilla.

I have the programming experience necessary to complete this project successfully. I am already a contributor to the Mozilla community. A patch I wrote to implement the existing nsIProcess API is now part of the Mozilla codebase. Moreover, it was considered important enough to be back-ported from TRUNK for the Mozilla 1.9.1 release branch.

I am also familiar with open source development methods. I know how the Mozilla community uses bug tracking, mailing lists, wikis, IRC and blogging to collaborate and I am very much a participant in that. Also, I have taken a hands on course in open source development at Seneca College and worked for the Centre for Development of Open Technology at Seneca College.

I personally want to do this project because I enjoy the technical challenge of writing cross platform code. The way Windows, Mac OS X and Linux handle process creation, security, and Unicode can be very different, but it possible to present a single API with consistent behaviour across those platforms. I enjoy working on a project with a large codebase and a large community of contributors. I believe that no single person understands all of Mozilla. This makes it interesting to explore, much like a large city.

I also believe in the Mozilla Foundation's core mission to promote open standards on the web.