Security for software update in Clear Linux* OS

04 Oct, 2016


Securing an OS is a difficult thing. Attacks can be old or new, with some being well-understood and others yet undiscovered. The Clear Linux* project recently undertook the task of securing our “swupd” software update tooling against one of the primary means of attack. This document details steps taken to secure the update process, presenting an overview of how a Clear user can utilize this new security, and covers some details on our use of OpenSSL and other tools.

Signatures and signature verification

Signing and signature verification allows the Clear Linux team to create content and meta-data which enables the user to verify that the content came from us.  This is accomplished through public key cryptography, where a key pair, one private and one public, are used to encrypt a hash of the MoM, generating a signature file. This signature then provides two assurances - integrity and non-repudiation.

Done correctly, this prevents an attacker from modifying files on your system by hacking the system update process. Securing the update process can prevent a broad class of attacks from succeeding, while also providing the user a means of:

  1. Recognizing that their system has been hacked: 
    Use "$ swupd verify", which relies on a signed/verified manifest of files, to identify modified files.
  2. Providing a possible mechanism for recovery:
    $ swupd verify --fix" can, for certain attacks, restore corrupted, modified and missing files

Together these capabilities make it much more difficult for an attacker to land content on a system, and easier to recognize when or if such a hack takes place.


Pre-install: How to verify an OS image

Verification of downloaded images is a manual process performed by a human via the following steps:

  1. Download the current ClearLinuxRoot.pem certificate; this is provided with the release being downloaded. For example, to verify the 8970 release, obtain the certificate from

  2. Download the desired OS image, as well as the [image]-SHA512SUMS.sig file.

  3. Validate the release’s ClearLinuxRoot.pem certificate:

    • Validate the certificate by comparing the downloaded certificate’s sha256sum hash with what is published here:

      You should see this (accurate as of 2016-06-16 00:00 UTC):

$ sha256sum ClearLinuxRoot.pem

4b0ca67300727477913c331ff124928a98bcf2fb12c011a855f17cd73137a890  ClearLinuxRoot.pem

    • Now verify that the signature file is valid, which also proves the OS image tarball is as trusted as the ClearLinuxRoot certificate. To do this, create the SHA512SUMS file of the tarball. This is the content which is actually signed by the release team.

$ sha512sum [image].tar.xz > sha512sum.out

    • Finally, we can use openssl to validate the signed SHA512SUMS.sig was signed by the ClearLinuxRoot certificate:

$ openssl smime -verify -in [image]-SHA512SUMS.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem -out /dev/null

Verification successful

 After running this, you should see: “Verification successful” as shown. If you do not see this, you cannot be certain the OS you downloaded can be trusted.


How SWUPD verifies content before install or update

Figure 1 provides an overview of the update delivery and signing and verification process. The Build and Release process builds the update content, signs the Manifest of Manifests (MoM), and pushes that content to an external web server.  Collectively, the MoM, MoM signature, and known-good certificate (obtained with the initial OS image) are used to ensure files being installed or updated are the artifacts actually generated and published by the Clear Linux OS build and release process.


Overview of the update delivery and signing and verification process

 Figure 1


Creating a signature file for a clear update: Manifests

As noted above, the build and release process is responsible for signing the manifest. This statement embodies a lot of hidden nuance, so we will delve into that a bit shortly.  But first, some comments on manifests.

There are several different types of manifests for Clean Linux, which are a listing of filesystem artifacts and meta-data related to those artifacts.  Every bundle in Clear Linux has a manifest describing its contents.  The Manifest of Manifests (MoM) is a listing of all the valid bundle manifests and checksums of each Manifest.  Manifests are valid for transitions from a specific release to a specific release.

To take one example, here is an excerpt of an actual Clear Linux manifest:

version:    9840
previous:    9800
filecount:    15679
timestamp:    1470700089
contentsize:    225381381
includes:    os-core

L...    c8ecd198cd3e49343fb07cfd09b6f3e4de878e39f54681ad8330b5ab55f28e1d    9740    /usr/bin/cc
F...    1176e336f546949485f0bdccf82fd78da99f87f4fdb335b9bfab5c103afe53f7    9740    /usr/bin/callgraph
L...    4e34c5837a64d98dddb6a237a456026a75cfce2d81134388858de06f3714c4b5    9740    /usr/bin/c++
F...    92056e06ce2c2b791ef5f1961659c2dba2af9d3cd09e3faaf99655b8066c9052    9740    /usr/bin/benchcmp
F...    f056759a4fc3d09e0774b715fd946e47b01f01c7296c0dc0bce0254bddc314e4    9770    /usr/share/man/man3/lib.3


As we can see, the header describes the manifest format, the contents, "from" and "to" release versions and a few other fields we won't go into here.

Following the header is a list of the bundle contents and associated checksums:

L...    c8ecd198cd3e49343fb07cfd09b6f3e4de878e39f54681ad8330b5ab55f28e1d    9740    /usr/bin/cc

The leading four bytes describe what the entry is (file, directory, symLink or subManifest).

The 64-character hash is a checksum of file contents and file meta-data such as file permissions.

Taken collectively, the top level manifest provides a listing of the sub-manifests (one per bundle), with checksums for each. Those in turn provide checksums for the individual files, directories, and sym-links.

Thus that top-level manifest enables us to verify the contents of each and every file installed by Clear Linux OS onto a system, simply by walking through it and the installed sub-manifests, and by calculating hashes for installed files and comparing them to those listed.  If an attacker tries to change a bundle manifest and/or the associated MoM entry for that manifest, calculating the hash of hashes will yield a result different from the signed hash distributed by the Clear Linux* project, and will be detected.

So, how do we sign the manifest, thus tying the manifest contents to an unalterable identity rooted in the Clear Linux Certificate?

First, we want to sign the minimal amount of bits possible, since large files take large amounts of time to sign. So we generate a checksum for the full manifest and sign that.

$ sha256sum Manifest.MoM > manifest.sum

Within Intel, and for official Clear Linux releases, we use an enterprise signing service.

For Clear Linux custom OS mixes, you would use OpenSSL from the command line to sign a custom Manifest.sum, using a command such as:

$ openssl dgst -sha256 -sign “your private key” -out manifest.sig manifest.sum

Either way, you now have a manifest.sum, which can be used to verify the Manifest.MoM is unaltered.

The SWUPD implementation

Part 2 of this article, written by Tudor Marcu, will go into greater depth on our usage of the OpenSSL API.  But first, we will offer an overview of how swupd implements signature checking and manifest content verification. The following steps include some sample equivalent commands which can be done on the command line, but ultimately the implementation is done in C using the OpenSSLAPI.  See the “swupd-client” source on GitHub for full details.

  1. A trusted certificate is distributed with all Clear Linux OS releases in /usr/share/clear/update-ca/ClearLinuxRoot.pem.

  2. swupd downloads the top-level manifest (MoM), as well as the signed MoM.sig for the currently-installed image, and for the target release in the case of an update.

  3. swupd generates a sha256sum of the MoM.

  4. swupd uses the MoM.sig downloaded in step 1, as well as the sha256sum; and, using the OpensSSL API, it makes a call equivalent to the following verification command:

$ openssl smime -verify -in sha256sums.sig -inform der -content sha512sum.out -CAfile ClearLinuxRoot.pem

5. Note that the actual API call is to PKCS7_verify()).

6. With a successful verification, we can proceed to trust this MoM and its contents.

    • Success: When a successful signature verification occurs, you should see the following message as part of the swupd output:

Signature check succeeded

    • Failure: Should verification fail, you will see:


7. As swupd then uses or installs bundle manifests, that bundle manifest hash is compared to the trusted MoM, extending the chain of trust from the MoM to the bundle manifests and out to every file installed.