MAR Files

MAR files, short for Mozilla ARchive files, are integral to Firefox Application Update, so this page will document what they are and go into their structure in some detail.

Overview

A MAR file consists of headers, signatures, multiple files including at least one manifest, and an index. Only one manifest, of the newest supported version, will be used; others will be ignored.

The manifest must be a complete file, but others may be patch files. Patch files are used to transform the old version of the file to the new version. Currently supported patch algorithms are bspatch and Zucchini.

Files are optionally compressed. Currently supported compression algorithms are bzip2 and xz.

Types of MARs

There are two types of MAR files: partial MARs and complete MARs.

A complete MAR contains an entire installation. It does not require any particular files to be on disk in order to update. Theoretically, it can be used to update from any version, but Watershed Updates complicate this somewhat.

A partial MAR contains only the changes between one version and another. While they may contain some files in their entirety, they mostly have binary patches that can be used to update an existing file to the new version. The advantage of this is that partial MARs can be much smaller than complete MARs. But the disadvantage is that a partial MAR must target a specific version of Firefox so that we can provide the correct patches to update that version of the files. And we just cannot provide a partial targeting every version of Firefox. We currently generate partials for the previous four versions released (including point versions). When Firefox is more out-of-date than that, a complete MAR must be used. Another limitation is that the files on the disk must not have been changed. We verify this by including headers in the patch file with the expected size and CRC of the existing file. If these do not match the file on disk, the update fails and we fall back to using a complete MAR.

Manifest

The manifest is a special file included in the MAR that describes the contents of the MAR and how to install it. It is identified by its filename: updatev3.manifest.

Empty lines and #-prefixed comment lines are allowed. Otherwise, each line of the manifest will consist of a method name followed by variable number of quoted parameters. These are separated by an arbitrary amount of space (0x20) and/or tab (0x09) characters. The lines must end with a Windows/CRLF newline (0x0D0A).

Here is an example manifest containing each of the possible method types:

type "partial"
add "2/20/20text0"
add-if "distribution/extensions/extensions0" "distribution/extensions/extensions0/extensions0text0"
add-if-not "defaults/pref/channel-prefs.js" "defaults/pref/channel-prefs.js"
patch "0/0exe0.exe.patch" "0/0exe0.exe"
patch-if "distribution/extensions/extensions1" "distribution/extensions/extensions1/extensions1png1.png.patch" "distribution/extensions/extensions1/extensions1png1.png"
remove "1/10/10text0"
rmdir "9/99/"
rmrfdir "9/98/"

type is special method that should be partial or complete depending on what kind of MAR this is. If included, it must be the first method specified. The rest are implemented by classes deriving from Action in /toolkit/mozapps/update/updater/updater.cpp.

Manipulation

The recommended way to manually manipulate (create, extract, sign, etc) MAR files is with the tool located here.

MAR Structure

This section was almost directly copied from the wiki in order to consolidate our documentation into a place where it is more likely to be updated. This document should be considered more up-to-date than that one.

Why the bespoke structure?

This question was given a fair amount of consideration. Ultimately, we decided to go with a custom file format because using libjar would have required a fair bit of hacking. Writing custom code was a simpler option, and it resulted in less code (mar_read.c is less than 300 lines of code). Moreover, the update system does not need a standard file format. The elements stored in the archive are bzip2 compressed binary diffs, generated using a variation of bsdiff. So, being able to unpack the archive file using standard tools wouldn’t be very useful in and of itself.

Byte Ordering

All fields are in big-endian format. The signatures are in NSS / OpenSSL / big-endian order and not CryptoAPI order. If CryptoAPI is used to check a signature, the bytes of the signature must be reversed before verifying the signature using CryptVerifySignature.

Constraints

To protect against invalid inputs the following constraints are in place:

  • There are at most 8 signatures.

  • The file size of the MAR file is at most 500MB.

  • No signature is more than 2048 bytes long.

File Layout

The following sections, in the given order, make up the structure of a MAR file. Some of these sections have been added over time and thus may not be present in sufficiently old MAR files. Old parsers will completely ignore these sections as they will simply appear to be unaddressed space in the “Files” section. Modern updater binaries will, however, reject MAR files without signatures.

Signatures

This section will be repeated as many times as indicated by the relevant header. Note that these signatures must sign all bytes of the MAR file excluding the signatures themselves.

  1. (4 bytes) ID representing the type of signature algorithm.

  2. (4 bytes) Size in bytes of the signature that follows.

  3. The signature itself.

Additional Sections Header

This consists of 4 bytes representing the number of additional sections.

Additional Sections

There will be as many of these sections as indicated in the Additional Sections Header, immediately prior.

  1. (4 bytes) Section size, in bytes. Note that, unlike the signature size, this is the size of the section and its metadata (the size and identifier).

  2. (4 bytes) Section identifier.

  3. The rest of the section. Consists of 8 bytes fewer than the section size given above.

Files

All file data. Must be addressed by the index, below. The length will be the sum of content sizes of all files in the index.

Index header

This consists of 4 bytes representing the size of the index in bytes.

Index entry

This section will be repeated to fill the size indicated by the index header, describing all files contained in the MAR.

  1. (4 bytes) Byte offset of the file, relative to the start of the MAR.

  2. (4 bytes) The content size in bytes.

  3. (4 bytes) File permission bits, in standard unix-style format.

  4. Variable length filename

  5. (1 byte) Null terminator

Additional Sections

These are meant to be flexible, so more could potentially be added later.

Product Information

The product information section identifies the product and channel this MAR file applies to. It also includes the new version number to avoid downgrades.

This section has this structure:

  1. (4 bytes) The size of the product information block.

  2. (4 bytes) The section ID, 1 in this case.

  3. (<64 bytes) The product and channel (such as from MAR_CHANNEL_ID). Example: mozilla-central

  4. (1 byte) Null terminator.

  5. (<32 bytes) The product version string (such as from MOZ_APP_VERSION) Examples: 12.0.1.5371, 12.0a1

  6. (1 byte) Null terminator.

Modifying Test MARs

Occasionally we need to modify a MAR file that we use in testing. These are some modifications that have been needed before and the steps to accomplish them.

Rename Files

In this example, we will rename some of the files in partial_mac.mar.

Regarding signing, note the following useful things: (a) The test MAR signatures successfully verify against toolkit/mozapps/update/updater/xpcshellCertificate.der and (b) according to this comment, that certificate was generated from mycert in modules/libmar/tests/unit/data.

  1. ./mach build so that <obj>/dist/bin/signmar is available.

  2. Install the MAR manipulation utility, if necessary: pip install mar.

  3. Make a temporary working directory: cd /path/to/mozilla/repo && mkdir temp && cd temp

  4. Extracted the MAR to be changed: mar -J -x ../toolkit/mozapps/update/tests/data/partial_mac.mar. The -J specifies a compression type of xz. You can instead specify --auto to automatically detect the compression type (though you may want to know the original compression later for recompression). You can check the compression type by running mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar and looking for the Compression type: line.

  5. Rename extracted files, as necessary.

  6. Edit updatev2.manifest and updatev3.manifest to update the changed paths.

  7. Run mar -T ../toolkit/mozapps/update/tests/data/partial_mac.mar to get a complete list of the files originally in that MAR as well as the product/version and channel strings (in this case xpcshell-test and * respectively).

  8. Create the new MAR: mar -J -c partial_mac_unsigned.mar -V '*' -H xpcshell-test <file1> <file2> ..., individually specifying each file path listed in by mar -T, substituting with renamed paths as necessary.

  9. Sign the MAR: ../<obj>/dist/bin/signmar -d ../modules/libmar/tests/unit/data -n mycert -s partial_mac_unsigned.mar partial_mac.mar.

  10. To verify signing: ../<obj>/dist/bin/signmar -D ../toolkit/mozapps/update/updater/xpcshellCertificate.der -v partial_mac.mar. This appears to output nothing on success, but it’s probably good to check to make sure echo $? displays 0. I also compared the output of mar -T partial_mac.mar to that of the original.

  11. To verify files: Extract the new MAR with mkdir cmp && cd cmp && mar -J -x ../partial_mac.mar && cd .. and verify the files match the originals.

  12. Overwrite the original MAR with the new one and remove the temp directory: cd .. && mv -f temp/partial_mac.mar toolkit/mozapps/update/tests/data/partial_mac.mar && rm -rf temp

Generate Zucchini Partials

In this example, we will generate partial MARs using Zucchini rather than bspatch.

First, note that in partial.mar, all exe patches are the same patch (turning complete.exe into partial.exe), and all png patches are the same patch (turning complete.png into partial.png):

$ mar -J -x ../partial.mar

$ grep -r MBDIFF10 .
Binary file ./0/00/00png0.png.patch matches
Binary file ./0/0exe0.exe.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
Binary file ./exe0.exe.patch matches
Binary file ./searchplugins/searchpluginspng0.png.patch matches
Binary file ./searchplugins/searchpluginspng1.png.patch matches

$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./0/00/00png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions0/extensions0png1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./distribution/extensions/extensions1/extensions1png1.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng0.png.patch
69ab8dc8614e6bb154c029b12ca8e3e9 *./searchplugins/searchpluginspng1.png.patch

$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
bd6e413d3248cfbeff65e104b6c4cd39 *./0/0exe0.exe.patch
bd6e413d3248cfbeff65e104b6c4cd39 *./exe0.exe.patch

Generate the equivalent zucchini patches:

$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.exe toolkit/mozapps/update/tests/data/partial.exe exe0.exe.patch -keep
$ ./obj-x86_64-pc-windows-msvc/dist/bin/zucchini.exe -gen toolkit/mozapps/update/tests/data/complete.png toolkit/mozapps/update/tests/data/partial.png 00png0.png.patch -keep

Replace all patch files on disk by their Zucchini equivalent, then double check by rerunning the md5sum commands:

$ grep -r Zucc .
Binary file ./0/00/00png0.png.patch matches
Binary file ./0/0exe0.exe.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png0.png.patch matches
Binary file ./distribution/extensions/extensions0/extensions0png1.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png0.png.patch matches
Binary file ./distribution/extensions/extensions1/extensions1png1.png.patch matches
Binary file ./exe0.exe.patch matches
Binary file ./searchplugins/searchpluginspng0.png.patch matches
Binary file ./searchplugins/searchpluginspng1.png.patch matches

$ md5sum ./0/00/00png0.png.patch ./distribution/extensions/extensions0/extensions0png0.png.patch ./distribution/extensions/extensions0/extensions0png1.png.patch ./distribution/extensions/extensions1/extensions1png0.png.patch ./distribution/extensions/extensions1/extensions1png1.png.patch ./searchplugins/searchpluginspng0.png.patch ./searchplugins/searchpluginspng1.png.patch
958bae1b40904145959ba45f988a7156 *./0/00/00png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions0/extensions0png1.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png0.png.patch
958bae1b40904145959ba45f988a7156 *./distribution/extensions/extensions1/extensions1png1.png.patch
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng0.png.patch
958bae1b40904145959ba45f988a7156 *./searchplugins/searchpluginspng1.png.patch

$ md5sum ./0/0exe0.exe.patch ./exe0.exe.patch
7750e88e0c1d006710abbd625710748b *./0/0exe0.exe.patch
7750e88e0c1d006710abbd625710748b *./exe0.exe.patch

Generate an unsigned mar file. Unfortunately on Windows this doesn’t preserve the permissions from the original mar. The simplest solution is to do this part on a UNIX (file)system.

$ mar -J -c ../partial_zucchini_unsigned.mar -H xpcshell-test -V '*' 2/20/20png0.png 0/0exe0.exe.patch 0/00/00text2 distribution/extensions/extensions0/extensions0text0 distribution/extensions/extensions0/extensions0png1.png.patch 0/00/00text0 distribution/extensions/extensions0/extensions0png0.png.patch searchplugins/searchpluginspng1.png.patch distribution/extensions/extensions1/extensions1png0.png.patch distribution/extensions/extensions1/extensions1text0 searchplugins/searchpluginstext0 distribution/extensions/extensions1/extensions1png1.png.patch defaults/pref/channel-prefs.js update-settings.ini updatev2.manifest searchplugins/searchpluginspng0.png.patch precomplete 0/00/00png0.png.patch updatev3.manifest exe0.exe.patch 2/20/20text0

Generate a signed mar from the unsigned mar:

$ /d/mozilla-source/firefox/obj-x86_64-pc-windows-msvc/dist/bin/signmar.exe -d /d/mozilla-source/firefox/modules/libmar/tests/unit/data/ -n mycert -s ../partial_zucchini_unsigned.mar ../partial_zucchini.mar