IOUtils Migration Guide

Improving performance through a new file API

What is IOUtils?

IOUtils is a privileged JavaScript API for performing file I/O in the Firefox frontend. It was developed as a replacement for OS.File, addressing bug 1231711. It is not to be confused with the unprivileged DOM File API.

IOUtils provides a minimal API surface to perform common I/O tasks via a collection of static methods inspired from OS.File. It is implemented in C++, and exposed to JavaScript via WebIDL bindings.

The most up-to-date API can always be found in IOUtils.webidl.

Differences from `OS.File`

IOUtils has a similar API to OS.File, but one should keep in mind some key differences.

No `File` instances (except `SyncReadFile` in workers)

Most of the IOUtils methods only operate on absolute path strings, and don’t expose a file handle to the caller. The exception to this rule is the openFileForSyncReading API, which is only available in workers.

Furthermore, OS.File was exposing platform-specific file descriptors through the fd attribute. IOUtils does not expose file descriptors.

WebIDL has no `Date` type

IOUtils is written in C++ and exposed to JavaScript through WebIDL. Many uses of OS.File concern themselves with obtaining or manipulating file metadata, like the last modified time, however the Date type does not exist in WebIDL. Using IOUtils, these values are returned to the caller as the number of milliseconds since 1970-01-01T00:00:00Z. Dates can be safely constructed from these values if needed.

For example, to obtain the last modification time of a file and update it to the current time:

let { lastModified } = await IOUtils.stat(path);

let lastModifiedDate = new Date(lastModified);

let now = new Date();

await IOUtils.touch(path, now.valueOf());

Some methods are not implemented

For various reasons (complexity, safety, availability of underlying system calls, usefulness, etc.) the following OS.File methods have no analogue in IOUtils. They also will not be implemented.

void unixSymlink(in string targetPath, in string createPath)
string getCurrentDirectory(void)
void setCurrentDirectory(in string path)
object open(in string path)
object openUnique(in string path)

Errors are reported as `DOMException`s

When an OS.File method runs into an error, it will throw/reject with a custom OS.File.Error. These objects have custom attributes that can be checked for common error cases.

IOUtils has similar behaviour, however its methods consistently reject with a DOMException whose name depends on the failure:

Exception Name	Reason for exception
`NotFoundError`	A file at the specified path could not be found on disk.
`NotAllowedError`	Access to a file at the specified path was denied by the operating system.
`NotReadableError`	A file at the specified path could not be read for some reason. It may have been too big to read, or it was corrupt, or some other reason. The exception message should have more details.
`ReadOnlyError`	A file at the specified path is read only and could not be modified.
`NoModificationAllowedError`	A file already exists at the specified path and could not be overwritten according to the specified options. The exception message should have more details.
`OperationError`	Something went wrong during the I/O operation. E.g. failed to allocate a buffer. The exception message should have more details.
`UnknownError`	An unknown error occurred in the implementation. An nsresult error code should be included in the exception message to assist with debugging and improving `IOUtils` internal error handling.

`IOUtils` is mostly async-only

OS.File provided an asynchronous front-end for main-thread consumers, and a synchronous front-end for workers. IOUtils only provides an asynchronous API for the vast majority of its API surface. These asynchronous methods can be called from both the main thread and from chrome-privileged worker threads.

The one exception to this rule is openFileForSyncReading, which allows synchronous file reading in workers.

`OS.File` vs `IOUtils`

Some methods and options of OS.File keep the same name and underlying behaviour in IOUtils, but others have been renamed. The following is a detailed comparison with examples of the methods and options in each API.

Reading a file

IOUtils provides the following methods to read data from a file. Like OS.File, they accept an options dictionary.

Note: The maximum file size that can be read is UINT32_MAX bytes. Attempting to read a file larger will result in a NotReadableError.

Promise<Uint8Array> read(DOMString path, ...);

Promise<DOMString> readUTF8(DOMString path, ...);

Promise<any> readJSON(DOMString path, ...);

// Workers only:
SyncReadFile openFileForSyncReading(DOMString path);

Options

`OS.File` option	`IOUtils` option	Description
bytes: number?	maxBytes: number?	If specified, read only up to this number of bytes. Otherwise, read the entire file. Default is null.
compression: 'lz4'	decompress: boolean	If true, read the file and return the decompressed LZ4 stream. Otherwise, just read the file byte-for-byte. Default is false.
encoding: 'utf-8'	N/A; use `readUTF8` instead.	Interprets the file as UTF-8 encoded text, and returns a string to the caller.

Examples

Read raw (unsigned) byte values

OS.File

let bytes = await OS.File.read(path); // Uint8Array

IOUtils

let bytes = await IOUtils.read(path); // Uint8Array

Read UTF-8 encoded text

OS.File

let utf8 = await OS.File.read(path, { encoding: 'utf-8' }); // string

IOUtils

let utf8 = await IOUtils.readUTF8(path); // string

Read JSON file

IOUtils

let obj = await IOUtils.readJSON(path); // object

Read LZ4 compressed file contents

OS.File

// Uint8Array
let bytes = await OS.File.read(path, { compression: 'lz4' });
// string
let utf8 = await OS.File.read(path, {
    encoding: 'utf-8',
    compression: 'lz4',
});

IOUtils

let bytes = await IOUtils.read(path, { decompress: true }); // Uint8Array
let utf8 = await IOUtils.readUTF8(path, { decompress: true }); // string

Synchronously read a fragment of a file into a buffer, from a worker

OS.File

// Read 64 bytes at offset 128, workers only:
let file = OS.File.open(path, { read: true });
file.setPosition(128);
let bytes = file.read({ bytes: 64 }); // Uint8Array
file.close();

IOUtils

// Read 64 bytes at offset 128, workers only:
let file = IOUtils.openFileForSyncReading(path);
let bytes = new Uint8Array(64);
file.readBytesInto(bytes, 128);
file.close();

Writing to a file

IOUtils provides the following methods to write data to a file. Like OS.File, they accept an options dictionary.

Promise<unsigned long long> write(DOMString path, Uint8Array data, ...);

Promise<unsigned long long> writeUTF8(DOMString path, DOMString string, ...);

Promise<unsigned long long> writeJSON(DOMString path, any value, ...);

Options

`OS.File` option	`IOUtils` option	Description
backupTo: string?	backupFile: string?	Identifies the path to backup the target file to before performing the write operation. If unspecified, no backup will be performed. Default is null.
tmpPath: string?	tmpPath: string?	Identifies a path to write to first, before performing a move to overwrite the target file. If unspecified, the target file will be written to directly. Default is null.
noOverwrite: boolean	mode: 'overwrite' or 'create'	'create' mode will refuse to overwrite an existing file. Default is 'overwrite'.
flush: boolean	flush: boolean	If true, force the OS to flush its internal buffers to disk. Default is false.
encoding: 'utf-8'	N/A; use `writeUTF8` instead.	Allows the caller to supply a string to be encoded as utf-8 text on disk.

Examples

Write raw (unsigned) byte values

OS.File

let bytes = new Uint8Array();
await OS.File.writeAtomic(path, bytes);

IOUtils

let bytes = new Uint8Array();
await IOUtils.write(path, bytes);

Write UTF-8 encoded text

OS.File

let str = "";
await OS.File.writeAtomic(path, str, { encoding: 'utf-8' });

IOUtils

let str = "";
await IOUtils.writeUTF8(path, str);

Write A JSON object

IOUtils

let obj = {};
await IOUtils.writeJSON(path, obj);

Write with LZ4 compression

OS.File

let bytes = new Uint8Array();
await OS.File.writeAtomic(path, bytes, { compression: 'lz4' });
let str = "";
await OS.File.writeAtomic(path, str, {
    compression: 'lz4',
});

IOUtils

let bytes = new Uint8Array();
await IOUtils.write(path, bytes, { compress: true });
let str = "";
await IOUtils.writeUTF8(path, str, { compress: true });

Move a file

IOUtils provides the following method to move files on disk. Like OS.File, it accepts an options dictionary.

Promise<void> move(DOMString sourcePath, DOMString destPath, ...);

Options

`OS.File` option	`IOUtils` option	Description
noOverwrite: boolean	noOverwrite: boolean	If true, fail if the destination already exists. Default is false.
noCopy: boolean	N/A; will not be implemented	This option is not implemented in `IOUtils`, and will be ignored if provided

Example

OS.File

await OS.File.move(srcPath, destPath);

IOUtils

await IOUtils.move(srcPath, destPath);

Remove a file

IOUtils provides one method to remove files from disk. OS.File provides several methods.

Promise<void> remove(DOMString path, ...);

Options

`OS.File` option	`IOUtils` option	Description
ignoreAbsent: boolean	ignoreAbsent: boolean	If true, and the destination does not exist, then do not raise an error. Default is true.
N/A; `OS.File` has dedicated methods for directory removal	recursive: boolean	If true, and the target is a directory, recursively remove the directory and all its children. Default is false.

Examples

Remove a file

OS.File

await OS.File.remove(path, { ignoreAbsent: true });

IOUtils

await IOUtils.remove(path);

Remove a directory and all its contents

OS.File

await OS.File.removeDir(path, { ignoreAbsent: true });

IOUtils

await IOUtils.remove(path, { recursive: true });

Remove an empty directory

OS.File

await OS.File.removeEmptyDir(path); // Will throw an exception if `path` is not empty.

IOUtils

await IOUtils.remove(path); // Will throw an exception if `path` is not empty.

Make a directory

IOUtils provides the following method to create directories on disk. Like OS.File, it accepts an options dictionary.

Promise<void> makeDirectory(DOMString path, ...);

Options

`OS.File` option	`IOUtils` option	Description
ignoreExisting: boolean	ignoreExisting: boolean	If true, succeed even if the target directory already exists. Default is true.
from: string	createAncestors: boolean	If true, `IOUtils` will create all missing ancestors in a path. Default is true. This option differs from `OS.File`, which requires the caller to specify a root path from which to create missing directories.
unixMode: number	permissions: unsigned long	The file mode to create the directory with. Ignored on Windows. Default is 0755.
winSecurity	N/A	`IOUtils` does not support setting custom directory security settings on Windows.

Example

OS.File

await OS.File.makeDir(srcPath, destPath);

IOUtils

await IOUtils.makeDirectory(srcPath, destPath);

Update a file’s modification time

IOUtils provides the following method to update a file’s modification time.

Promise<void> setModificationTime(DOMString path, optional long long modification);

Example

OS.File

await OS.File.setDates(path, new Date(), new Date());

IOUtils

await IOUtils.setModificationTime(path, new Date().valueOf());

Get file metadata

IOUtils provides the following method to query file metadata.

Promise<void> stat(DOMString path);

Example

OS.File

let fileInfo = await OS.File.stat(path);

IOUtils

let fileInfo = await IOUtils.stat(path);

Copy a file

IOUtils provides the following method to copy a file on disk. Like OS.File, it accepts an options dictionary.

Promise<void> copy(DOMString path, ...);

Options

`OS.File` option	`IOUtils` option	Description
noOverwrite: boolean	noOverwrite: boolean	If true, fail if the destination already exists. Default is false.
N/A; `OS.File` does not appear to support recursively copying files	recursive: boolean	If true, copy the source recursively.

Examples

Copy a file

OS.File

await OS.File.copy(srcPath, destPath);

IOUtils

await IOUtils.copy(srcPath, destPath);

Copy a directory recursively

OS.File

// Not easy to do.

IOUtils

await IOUtils.copy(srcPath, destPath, { recursive: true });

Iterate a directory

At the moment, IOUtils does not have a way to expose an iterator for directories. This is blocked by bug 1577383. As a stop-gap for this functionality, one can get all the children of a directory and iterate through the returned path array using the following method.

Promise<sequence<DOMString>> getChildren(DOMString path);

Example

OS.File

for await (const { path } of new OS.FileDirectoryIterator(dirName)) {</p>
  ...
}

IOUtils

for (const path of await IOUtils.getChildren(dirName)) {
  ...
}

Check if a file exists

IOUtils provides the following method analogous to the OS.File method of the same name.

Promise<boolean> exists(DOMString path);

Example

OS.File

if (await OS.File.exists(path)) {
  ...
}

IOUtils

if (await IOUtils.exists(path)) {
  ...
}

Set the permissions of a file

IOUtils provides the following method analogous to the OS.File method of the same name.

Promise<void> setPermissions(DOMString path, unsigned long permissions, optional boolean honorUmask = true);

Options

`OS.File` option	`IOUtils` option	Description
unixMode: number	permissions: unsigned long	The UNIX file mode representing the permissions. Required in IOUtils.
unixHonorUmask: boolean	honorUmask: boolean	If omitted or true, any UNIX file mode is modified by the permissions. Otherwise the exact value of the permissions will be applied.

Example

OS.File

await OS.File.setPermissions(path, { unixMode: 0o600 });

IOUtils

await IOUtils.setPermissions(path, 0o600);

FAQs

Why should I use IOUtils instead of OS.File?

Bug 1231711 provides some good context, but some reasons include:

reduced cache-contention,
faster startup, and
less memory usage.

Additionally, IOUtils benefits from a native implementation, which assists in performance-related work for Project Fission.

We are actively working to migrate old code usages of OS.File to analogous IOUtils calls, so new usages of OS.File should not be introduced at this time.

Do I need to import anything to use this API?

Nope! It’s available via the IOUtils global in JavaScript (ChromeOnly context).

Can I use this API from C++ or Rust?

Currently usage is geared exclusively towards JavaScript callers, and all C++ methods are private except for the Web IDL bindings. However given sufficient interest, it should be easy to expose ergonomic public methods for C++ and/or Rust.

Why isn’t IOUtils written in Rust?

At the time of writing, support for Web IDL bindings was more mature for C++ oriented tooling than it was for Rust.

Is IOUtils feature complete? When will it be available?

IOUtils is considered feature complete as of Firefox 83.

IOUtils Migration Guide

What is IOUtils?

Differences from OS.File

No File instances (except SyncReadFile in workers)

WebIDL has no Date type

Some methods are not implemented

Errors are reported as DOMExceptions

IOUtils is mostly async-only

OS.File vs IOUtils

Reading a file

Options

Examples

Read raw (unsigned) byte values

Read UTF-8 encoded text

Read JSON file

Read LZ4 compressed file contents

Synchronously read a fragment of a file into a buffer, from a worker

Writing to a file

Options

Examples

Write raw (unsigned) byte values

Write UTF-8 encoded text

Write A JSON object

Write with LZ4 compression

Move a file

Options

Example

Remove a file

Options

Examples

Remove a file

Remove a directory and all its contents

Remove an empty directory

Make a directory

Options

Example

Update a file’s modification time

Example

Get file metadata

Example

Copy a file

Options

Examples

Copy a file

Copy a directory recursively

Iterate a directory

Example

Check if a file exists

Example

Set the permissions of a file

Options

Example

FAQs

Differences from `OS.File`

No `File` instances (except `SyncReadFile` in workers)

WebIDL has no `Date` type

Errors are reported as `DOMException`s

`IOUtils` is mostly async-only

`OS.File` vs `IOUtils`