Tackling upgrade nightmares
by Alberto Murillo and Obed N Munoz
Upgrades are not an easy task for any project, but Clear Linux* OS made it possible with its software update feature.
Software Update provides fast upgrades with the implementation of binary deltas calculation; this means that only the binary differences that changed for your specific software components are downloaded.
When upgrading a large project like OpenStack*, which contains hundreds of Python* and non-Python dependencies, having a good mechanism to upgrade software without breaking the full project release is a must. In Clear Linux*, it took us only one day to do the merge and have the new OpenStack Mitaka release ready for use.
In the past, when we migrated from OpenStack Kilo to Liberty, we had to deal with issues that were directly impacting in production. In some cases, we noticed failures as we waited for the new OS release to be generated. This required quick fixes and patience while we waited until the next release was out, allowing us to validate once again.
Zero regressions in migration? Mixer, Autospec and Software Update made it so
We call regressions to those tests in our BAT (Basic Acceptance Test) framework that fail after changes in software components. In the past, we had regressions that needed to be solved quickly to avoid impacting the next OS release. This time we were able to prevent such regressions by using features in Clear Linux* OS that allowed us to validate new software before merging it into production build system.
Zero regression testing on OpenStack's upgrade was possible with the use of the Mixer, which provided a development software update server where developers put the latest changes. Then development machines performed software update against that server.
It couldn't be easier...
We started the migration taskforce during OpenStack RC versions. Once we received the notification of a new RC version, we just auto-generated the new RPM package with the help of autospec project, and then we moved the RPMs to our Mixer server where the mixing magic was happening.
Once the new release was generated in our alternate Software Update server, we simply ran:
# swupd update --url=http://mixer-server.example.com -F staging
The above command was executed in our development servers. In a few seconds, we had the new updated version of OpenStack packages. It accelerated the old process of generating a full OS image for every change in OpenStack or dependencies packages.
It doesn't matter how big or small the software is! Mixer provides the capabilities of upgrading a development system against a sandbox update server. And once you are ready to merge into production, you simply make the merge and update against the production server. That's all you need! No regressions happen because you did a trustworthy validation previously.