Subprocess Module

The Subprocess module allows a caller to spawn a native host executable, and communicate with it asynchronously over its standard input and output pipes.

Processes are launched asynchronously Subprocess.call method, based on the properties of a single options object. The method returns a promise which resolves, once the process has successfully launched, to a Process object, which can be used to communicate with and control the process.

A simple Hello World invocation, which writes a message to a process, reads it back, logs it, and waits for the process to exit looks something like:

let proc = await Subprocess.call({
  command: "/bin/cat",
});

proc.stdin.write("Hello World!");

let result = await proc.stdout.readString();
console.log(result);

proc.stdin.close();
let {exitCode} = await proc.wait();

Input and Output Redirection

Communication with the child process happens entirely via one-way pipes tied to its standard input, standard output, and standard error file descriptors. While standard input and output are always redirected to pipes, standard error is inherited from the parent process by default. Standard error can, however, optionally be either redirected to its own pipe or merged into the standard output pipe.

The module is designed primarily for use with processes following a strict IO protocol, with predictable message sizes. Its read operations, therefore, either complete after reading the exact amount of data specified, or do not complete at all. For cases where this is not desirable, read() and readString may be called without any length argument, and will return a chunk of data of an arbitrary size.

Process and Pipe Lifecycles

Once the process exits, any buffered data from its output pipes may still be read until the pipe is explicitly closed. Unless the pipe is explicitly closed, however, any pending buffered data must be read from the pipe, or the resources associated with the pipe will not be freed.

Beyond this, no explicit cleanup is required for either processes or their pipes. So long as the caller ensures that the process exits, and there is no pending input to be read on its stdout or stderr pipes, all resources will be freed automatically.

The preferred way to ensure that a process exits is to close its input pipe and wait for it to exit gracefully. Processes which haven’t exited gracefully by shutdown time, however, must be forcibly terminated:

let proc = await Subprocess.call({
  command: "/usr/bin/subprocess.py",
});

// Kill the process if it hasn't gracefully exited by shutdown time.
let blocker = () => proc.kill();

AsyncShutdown.profileBeforeChange.addBlocker(
  "Subprocess: Killing hung process",
  blocker);

proc.wait().then(() => {
  // Remove the shutdown blocker once we've exited.
  AsyncShutdown.profileBeforeChange.removeBlocker(blocker);

  // Close standard output, in case there's any buffered data we haven't read.
  proc.stdout.close();
});

// Send a message to the process, and close stdin, so the process knows to
// exit.
proc.stdin.write(message);
proc.stdin.close();

In the simpler case of a short-running process which takes no input, and exits immediately after producing output, it’s generally enough to simply read its output stream until EOF:

let proc = await Subprocess.call({
  command: await Subprocess.pathSearch("ifconfig"),
});

// Read all of the process output.
let result = "";
let string;
while ((string = await proc.stdout.readString())) {
  result += string;
}
console.log(result);

// The output pipe is closed and no buffered data remains to be read.
// This means the process has exited, and no further cleanup is necessary.

Bidirectional IO

When performing bidirectional IO, special care needs to be taken to avoid deadlocks. While all IO operations in the Subprocess API are asynchronous, careless ordering of operations can still lead to a state where both processes are blocked on a read or write operation at the same time. For example,

let proc = await Subprocess.call({
  command: "/bin/cat",
});

let size = 1024 * 1024;
await proc.stdin.write(new ArrayBuffer(size));

let result = await proc.stdout.read(size);

The code attempts to write 1MB of data to an input pipe, and then read it back from the output pipe. Because the data is big enough to fill both the input and output pipe buffers, though, and because the code waits for the write operation to complete before attempting any reads, the cat process will block trying to write to its output indefinitely, and never finish reading the data from its standard input.

In order to avoid the deadlock, we need to avoid blocking on the write operation:

let size = 1024 * 1024;
proc.stdin.write(new ArrayBuffer(size));

let result = await proc.stdout.read(size);

There is no silver bullet to avoiding deadlocks in this type of situation, though. Any input operations that depend on output operations, or vice versa, have the possibility of triggering deadlocks, and need to be thought out carefully.

Arguments

Arguments may be passed to the process in the form an array of strings. Arguments are never split, or subjected to any sort of shell expansion, so the target process will receive the exact arguments array as passed to Subprocess.call. Argument 0 will always be the full path to the executable, as passed via the command argument:

let proc = await Subprocess.call({
  command: "/bin/sh",
  arguments: ["-c", "echo -n $0"],
});

let output = await proc.stdout.readString();
assert(output === "/bin/sh");

Process Environment

By default, the process is launched with the same environment variables and working directory as the parent process, but either can be changed if necessary. The working directory may be changed simply by passing a workdir option:

let proc = await Subprocess.call({
  command: "/bin/pwd",
  workdir: "/tmp",
});

let output = await proc.stdout.readString();
assert(output === "/tmp\n");

The process’s environment variables can be changed using the environment and environmentAppend options. By default, passing an environment object replaces the process’s entire environment with the properties in that object:

let proc = await Subprocess.call({
  command: "/bin/pwd",
  environment: {FOO: "BAR"},
});

let output = await proc.stdout.readString();
assert(output === "FOO=BAR\n");

In order to add variables to, or change variables from, the current set of environment variables, the environmentAppend object must be passed in addition:

let proc = await Subprocess.call({
  command: "/bin/pwd",
  environment: {FOO: "BAR"},
  environmentAppend: true,
});

let output = "";
while ((string = await proc.stdout.readString())) {
  output += string;
}

assert(output.includes("FOO=BAR\n"));