mixer is the tool used by the Clear Linux* OS team to generate official update content and releases. The update content generated by mixer is then consumed by swupd on a downstream client. The same mixer tool is available as part of Clear Linux OS to create your own customized update content and releases.

Description

mixer uses the following sources as inputs to generate update content:

  • Upstream Clear Linux OS bundles with their corresponding RPM packages
  • Locally-defined bundles with their corresponding local RPM packages
  • Locally-defined bundles with upstream RPM packages

Using the mixer tool, you select which set of content from these sources will be part of your update. You can select content from each of these sources to make a unique combination of functionality for your custom update content (known as a mix).

The update content that mixer generates consists of various pieces of OS content, update metadata, as well as a complete image. The OS content includes all files in an update, as well as zero- and delta-packs for improved update performance. The metadata, stored as manifests, describes all of the bundle information for the update. Update content produced by mixer is then published to a web server and consumed by clients via swupd. Refer to swupd for additional information regarding updates and update content.

How it works

Learn the mixer tool set up and workflow.

Prerequisites

  • mixer bundle

    Add the mixer tool with the mixer bundle. Refer to Install a bundle for more details.

  • Docker container

    mixer by default runs all build commands in a Docker container to make sure the correct tool versions are used. This also allows custom mixes to automatically perform downstream format bumps when the upstream releases a format bump. See Format version for additional information regarding format bumps.

    Refer to Configure and enable Docker for instruction.

  • Docker proxy (optional)

    If you use a proxy server, you must set your proxy environment variables and create a proxy configuration file for the Docker daemon and container.

    Consult your IT department if you are behind a corporate proxy for the correct values.

    Refer to Configure Docker proxy info for instruction.

  • Location to host the update content and images

    In order for swupd to make use of your mix, the update content for your mix must be hosted on a web server. Your mix will be configured with an update location URL, which swupd will use to pull down updates.

    Refer to Set up a nginx web server for mixer for an simple example of setting up an update location.

Mix setup

Follow these steps to create and initialize the mixer workspace. Complete setup before you create a mix.

  1. Create workspace.

    The mixer tool uses a simple workspace to contain all input and output in a basic directory structure. The workspace is simply an empty folder that you will execute the mixer commands from. Each mix will use its own separate workspace.

  2. Initialize the workspace and mix.

    Before you create a mix, you must explicitly initialize the mixer workspace. During initialization, the mixer workspace is configured and the base for your mix is defined. By default, your mix will be based on the latest upstream version and start with the minimum set of bundles. Your first custom mix version number will start at 10. You can alternately select other versions or bundle sets to start from.

    Initialization creates the directory structure within the workspace and adds the builder.conf file, which is used to configure the mixer tool.

    View the mixer.init man page for more information on mixer initialization.

    View the list of suitable versions to mix from.

  3. Edit builder.conf.

    builder.conf tells the mixer tool how to configure the mix. For example, it allows you to configure where mixer output is located and where swupd update content will be located.

    At minimum, set the URL of your update server so your custom OS knows where to get update content.

    Refer to the builder.conf section for more information.

Create a mix

A mix is created with the following steps:

  1. Add custom RPMs and set up local repo (optional).

    If you are adding custom RPMs to your mix, you will need to add the RPMs to your mix workspace and set up a corresponding local repository.

    Go to the autospec guide to learn to build RPMs from scratch. If the RPMs are not built on Clear Linux OS, make sure your configuration and toolchain builds them correctly for Clear Linux OS. Otherwise there is no guarantee they will be compatible.

    Refer to the autospec guide for more information on using autospec to build RPMs.

  2. Update and build bundles.

    Add, edit, or remove bundles that will be part of your content and build them. mixer will automatically update the mixbundles file when you update the bundles in your mix.

    View the mixer.bundle man page for more information on configuring bundles in a mix.

    View the mixer.build man page for more information on building bundles.

    View the Bundles section for more information on how mixer manages bundles.

  3. Create the update content.

    mixer creates update content with this step. Zero-packs are created automatically, and delta-packs can be optionally created at the same time (for all builds after version 0).

    A zero-pack is the full set of content needed to go from mix version 0 (nothing) to the mix version you just built content for.

    A delta-pack provides the content delta between a PAST_VERSION to a MIX_VERSION which allows the transition from one mix version to another.

    View swupd for more information on update content.

  4. Create image.

    mixer creates a bootable image from your updated content using the ister tool. In this step you can specify which bundles you want preinstalled in the image. Users can later install other bundles available in your mix.

  5. Make update available.

    Deploy update content and images to your update server.

    View the Example 3: Deploy updates to target for a simple deployment scenario.

Maintain or modify mix

Update or modify your content to a new version by following the same steps to create a mix. Increment the mix version number for the next mix.

Examples

The following examples are designed to work together and in order. The examples use:

  • A stock installation of Clear Linux OS.
  • A web server that comes with Clear Linux OS to host the content updates.
  • A simple VM that will update against the locally produced content created in Example 2.

Complete all Prerequisites before using these examples.

Example 1: Mix set up

This example shows the basic steps for first time setup of mixer for a new mix.

  1. Create an empty directory to use as a workspace for mixer:

    mkdir ~/mixer
    
  2. In your mixer workspace, generate an initial mix based on the latest upstream Clear Linux OS version, with minimum bundles:

    cd ~/mixer
    mixer init
    

    Note in the initialization output, that your initial mix version is set to 10 and that the minimum bundles have been added.

  3. Edit builder.conf to set the value of CONTENTURL and VERSIONURL to the IP address of the nginx server you set up in the prerequisite Set up a nginx web server for mixer. For example:

    CONTENTURL="http://192.168.25.52"
    VERSIONURL="http://192.168.25.52"
    

Example 2: Create a simple mix

This example shows how to create a simple custom mix using upstream content. We’ll create an image for a QEMU virtual machine which we can later use to test our mix.

We can use the default bundles that were added during intialization, but these include the native-kernel bundle which is intended to be used on a bare metal system instead of a VM. So we will modify the default bundle set to get a smaller kernel image, which will also be faster to load.

  1. Update bundles in mix:

    mixer bundle remove kernel-native
    mixer bundle add kernel-kvm
    
  2. Build bundles:

    mixer build bundles
    

    Look in ~/mixer/update/image/<mix version>/full for the full chroot after the build command completes.

  3. Build update content. Browse to your http://localhost site and you’ll see the web page is now up, but with no update content. Build the update content:

    mixer build update
    

    Refresh your http://localhost site and now you can see the update content for mix version 10.

    Look in ~/mixer/update/www/<mix version> to see the update content in your workspace.

  4. Configure image. Edit the ister configuration file for your image to include all of the bundles you want preinstalled in the image. If this is the first time creating an image, first get a copy of the release-image-config.json template file:

    curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json
    

    For this example, edit release-image-config.json so that the root partition size is “5G” and replace the “kernel-native” bundle with “kernel-kvm”.

    {
      "DestinationType" : "virtual",
      "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" },
                            { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" },
                            { "disk" : "release.img", "partition" : 3, "size" : "5G", "type" : "linux" } ],
      "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" },
                            { "disk" : "release.img", "partition" : 2, "type" : "swap" },
                            { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ],
      "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" },
                                 { "disk" : "release.img", "partition" : 3, "mount" : "/" } ],
      "Version": "latest",
      "Bundles": ["kernel-kvm", "os-core", "os-core-update"]
    }
    
  5. Build the image.

    sudo mixer build image
    

    The output from this step will be release.img, which is a live image.

  6. Make the next mix. Create a new version of your mix, for the live image to update to. Increment your mix version by 10:

    mixer versions update
    

    Repeat steps 1-3 to add the upstream curl bundle to the mix:

    mixer bundle add curl
    mixer build bundles
    mixer build update
    

    And build optional delta-packs, which will help reduce client update time:

    mixer build delta-packs --from 10 --to 20
    

    Refresh your http://localhost site and now you can see the update content for mix version 20.

    Look in ~/mixer/update/www/<mix version> to see the update content in your workspace.

Example 3: Deploy updates to target

The image created in Example 2 is directly bootable in QEMU. In this example, we’ll boot the image from Example 2 to verify it, and update the image from mix version 10 (which the image was built from), to mix version 20.

  1. Set up the QEMU environment.

    Install the kvm-host bundle to your Clear Linux OS:

    sudo swupd bundle-add kvm-host
    

    Get the virtual EFI firmware, download the image launch script, and make it executable:

    curl -O https://download.clearlinux.org/image/OVMF.fd
    curl -O https://download.clearlinux.org/image/start_qemu.sh
    chmod +x start_qemu.sh
    
  2. Start your VM image (created in Example 2):

    sudo ./start_qemu.sh release.img
    
  3. Log in as root and set a password

  4. Try out your mix.

    Take a look at the default bundles installed in your mix:

    swupd info
    swupd bundle-list
    swupd bundle-list -a
    

    Note that you cannot see the curl bundle that you added in Example 2 because your mix is still on version 10.

    Check for updates. You should see that version 20 is available. Use swupd to update your mix:

    swupd check-update
    swupd update
    swupd bundle-list -a
    

    Now your mix should be at version 20 and curl is now available. Try using curl. This will fail as curl is not yet installed:

    curl: command not found
    To install curl use: swupd bundle-add curl
    

    Add the new bundle from your update server to your VM. Retry curl. It works!

    swupd bundle-add curl
    curl -O https://download.clearlinux.org/image/start_qemu.sh
    

    And shutdown your VM:

    poweroff
    

References

Reference the mixer man page for details regarding mixer commands and options.

builder.conf

mixer initialization creates a builder.conf that stores the basic configuration for the mixer tool. The items of primary interest are CONTENTURL and VERSIONURL, which will be used by systems updating against your custom content.

#builder.conf

#VERSION 1.0

[Builder]
  CERT = "/home/clr/mix/Swupd_Root.pem"
  SERVER_STATE_DIR = "/home/clr/mix/update"
  VERSIONS_PATH = "/home/clr/mix"
  YUM_CONF = "/home/clr/mix/.yum-mix.conf"

[Swupd]
  BUNDLE = "os-core-update"
  CONTENTURL = "<URL where the content will be hosted>"
  VERSIONURL = "<URL where the version of the mix will be hosted>"

[Server]
  DEBUG_INFO_BANNED = "true"
  DEBUG_INFO_LIB = "/usr/lib/debug"
  DEBUG_INFO_SRC = "/usr/src/debug"

[Mixer]
  LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
  LOCAL_REPO_DIR = ""
  LOCAL_RPM_DIR = ""
  DOCKER_IMAGE_PATH = "clearlinux/mixer"

Additional explanation of variables in builder.conf is provided in Table 1.

Variable Explanation
CERT

Sets the path where mixer stores the certificate file used to sign content for verification. mixer automatically generates the certificate if you do not provide the path to an existing one, and signs the Manifest.MoM file to provide security for the updated content you create.

The chroot-builder uses the certificate file to sign the root Manifest.MoM file, to provide security for content verification.

The swupd uses this certificate to verify the Manifest.MoM file’s signature.

For now, we strongly recommend that you do not modify this variable, as swupd expects a certificate with a very specific configuration to sign and verify properly.

CONTENTURL and VERSIONURL

Set these variables to the IP address of the web server hosting the update content.

VERSIONURL is the IP address where the swupd client looks to determine if a new version is available.

CONTENTURL is the location where swupd will pull content updates from.

If the web server is on the same machine as the SERVER_STATE_DIR directory, you can create a symlink to the directory in your web server’s document root to easily host the content.

These URLs are embedded in the images created by mixer.

DOCKER_IMAGE_PATH Sets the base name of the docker image mixer will pull down in order to run builds in the proper container.
LOCAL_BUNDLE_DIR Sets the path where mixer stores the local bundle definition files. The bundle definition files include any new, original bundles you create, along with any edited versions of upstream bundles.
SERVER_STATE_DIR Sets the path for where mixer outputs content. By default, mixer will automatically set the path.
VERSIONS_PATH Sets the path for the mix version and upstream version’s two state files: mixversion and upstreamversion. mixer creates both files for you when you set up the workspace.
YUM_CONF

Sets the path where mixer automatically generates the .yum-mix.conf file.

The yum configuration file points the chroot-builder to where the RPMs are stored.

Table 1: Variables in builder.conf

Format version

Compatible versions of an OS are tracked with an OS compatibility epoch. Versions of an OS within an epoch are fully compatible and can update to any other version within that epoch. The compatibility epoch is set as the Format variable in the mixer.state file. Variables in the mixer.state are used by mixer between executions and should not be manually changed.

If Format increments to a new epoch (a “format bump”), the OS has changed in such a way that updating from build M in format X, to build N in format Y will not work. Generally, this scenario occurs when the software updater/the software has a change such that it is no longer compatible with the previous update scheme, or when a package is removed from the update stream and the update must ensure the files associated with that package are removed from the system.

Using a format increment, we make sure pre- and co-requisite changes flow out with proper ordering. The updated client will only update to the latest release in its respective format version, unless overridden by command line flags. This way we can guarantee that all clients update to the final version in their given format.

The given format must contain all the changes needed to understand the content built in the next format. Only after reaching the final release in the old format can a client continue to update to releases in the new format.

The format version is incremented only when a compatibility breakage is introduced. Normal updates, like updating a software package, do not require a format increment.

Bundles

mixer stores information about the bundles included in a mix in a flat file called mixbundles, located in the path set by the VERSIONS_PATH variable in builder.conf. mixbundles is automatically created when the mix is initiated. mixer will refresh the file each time you change the bundles in the mix.

Bundles can include other bundles. Nested bundles can themselves include other bundles. If you see an unexpected bundle in your mix, it is likely a nested bundle in one of the bundles you explicitley added.

A bundle will fill into one of two categoris: upstream or local. Upstream bundles are those provided by Clear Linux OS. Local bundles are either modified upstream bundles or new local bundles.

Upstream bundles

mixer automatically downloads and caches upstream bundle definition files. These definition files are stored in the upstream-bundles directory in the workspace. Do not modify the files in this directory. This directory is simply a mirror for mixer to use. mixer will automatically delete the contents of this directory before repopulating it on-the-fly if a new version must be downloaded.

The mixer tool automatically caches the bundles for the Clear Linux OS version configured in the upstreamversion file. mixer also cleans up old versions once they are no longer needed.

Local bundles

Local bundles are bundles that you create, or are edited versions of upstream bundles. Local bundle definition files are stored in the local-bundles directory in the workspace. The LOCAL_BUNDLE_DIR variable sets the path of this directory in the builder.conf file.

mixer always checks for local bundles first and the upstream bundles second. So bundles in the local-bundles directory will always take precedence over any upstream bundles that have the same name. This precedence enables you to copy upstream bundles locally, and edit into a local variation.

Bundle configuration

mixer provides commands to configure the bundles for a mix, for example to add a bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a mix. View the mixer.bundle man page for a full list of commands and more information on configuring bundles in a mix.

Editing an existing local bundle is as simple as opening the bundle definition file in your favorite editor, making the desired edits, and saving your changes.

A note on removing bundles from a mix: By default, removing a bundle will only remove the bundle from the mix. The local bundle defintion file will still remain. To completely remove a bundle, including its local bundle definition file, use the –local flag.

If you remove the bundle definition file for a local, edited version of an upstream bundle in a mix, the mix will revert to reference the original upstream version of the bundle.

Configure and enable Docker

Use these steps to enable Docker for the mixer tool. Make sure to Configure Docker proxy info first if needed.

  1. Start the Docker daemon:

    sudo systemctl start docker
    sudo chmod 777 /var/run/docker.sock
    sudo docker info
    
  2. Add user to the docker group

    sudo usermod -G docker -a <username>
    

Pull Docker container manually (optional)

By default, mixer will automatically pull a Docker container for mixing if one does not already exist. If you need to troubleshoot the mixer container, it may be useful to manually pull a mixer Docker container.

Versions of the mixer Docker container are available under the tags for the clearlinux/mixer repo on Docker Hub. Each version of the mixer Docker container is named after the associated Clear Linux OS upstream format version. Refer to Format version for additional information on upstream format versions.

Use the following steps to manually pull a mixer Docker container:

  1. Find the version of the container you need by viewing the tags for the clearlinux/mixer repo on Docker Hub.

  2. Pull the latest container version:

    docker pull clearlinux/mixer:<upstream-format-version>
    
  3. View local docker images:

    docker images
    

Configure Docker proxy info

If needed, use these steps to configure the Docker proxy information.

Configure the Docker daemon proxies:

  1. Create the Docker daemon proxy config directory:

    sudo mkdir -p /etc/systemd/system/docker.service.d
    

    Create /etc/systemd/system/docker.service.d/http-proxy.conf and add the following using your own proxy values:

    [Service]
    Environment="HTTP_PROXY=<HTTP proxy URL>:<port number>"
    Environment="HTTPS_PROXY=<HTTPS proxy URL>:<port number>"
    
  2. Reload the Docker daemon:

    sudo systemctl daemon-reload
    

Configure the Docker container proxies, in order to pass proxy settings to containers:

  1. Create a directory for your container config:

    mkdir ~/.docker
    
  2. Create the config file ~/.docker/config.json and add the following entries, using your own proxy values:

    {
      "proxies":
      {
        "default":
        {
          "httpProxy": "<proxy-url>:<port>",
          "httpsProxy": "<proxy-url>:<port>"
        }
      }
    }
    
  3. Set ownership and permission on the docker config directory:

    sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
    sudo chmod g+rwx "$HOME/.docker" -R
    

Lastly, configure proxies to allow mixer to access upstream content from behind a firewall. For example:

  1. Open your $HOME/.bashrc file and add proxy and port values for the following:

    export http_proxy="<proxy-url>:<port>"
    export https_proxy="<proxy-url>:<port>"
    export HTTP_PROXY="<proxy-url>:<port>"
    export HTTPS_PROXY="<proxy-url>:<port>"
    export no_proxy="<...>"
    
  2. Log out and log back in for the proxies to take effect.

Set up a nginx web server for mixer

A web server is needed to host your update content. In this example, we use the nginx web server, which comes with Clear Linux OS.

Set up a nginx web server for mixer with the following steps:

  1. Install the nginx bundle:

    sudo swupd bundle-add nginx
    
  2. Make the directory where mixer updates will reside:

    sudo mkdir -p /var/www
    
  3. Create a symbolic link between your workspace updates and the updates on the local nginx web server. In this example, $HOME/mixer is the workspace for the mix.

    sudo ln -sf $HOME/mixer/update/www /var/www/mixer
    
  4. Set up nginx configuration:

    sudo mkdir -p  /etc/nginx/conf.d
    
  5. Copy the default example configuration file:

    sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf
    
  6. Configure the mixer update server. Create and add the following server configuration content to /etc/nginx/conf.d/mixer.conf (sudo required):

    server {
         server_name localhost;
         location / {
                   root /var/www/mixer;
                   autoindex on;
         }
    }
    
  7. Restart the daemon, enable nginx on boot, and start the service.

    sudo systemctl daemon-reload
    
    sudo systemctl enable nginx
    
    sudo systemctl start nginx
    
  8. Verify the web server is running at http://localhost. At this point you should no longer see a “404 Not Found” message.