Spectre and Meltdown - We got your back

If you've not been living in a cave these last days, the words Spectre and Meltdown should sound familiar to you. This weeks, two CPU chips design flaws have been revealed to the public leading to possible data theft across application in a same system and even across virtual machines in a virtualized environment through the host.

To work around this hardware flaw, softwares have to be updated. Almost everyone is impacted: desktops, laptops, servers, you should update your workstation, as we are upgrading our infrastructure.

Maintenance operations on the platform

Our infrastructure contains hardware servers used as hypervisors, and Linux guest virtual machines dispatched on the different hosts running your app and database containers. To cover the Meltdown and Spectre vulnerabilites, two maintenance operations have to be done from our side. The first one is to upgrade the hypervisors running the virtual machines to protect the infrastructure against cross-VM side-channel attacks. This operation has already been achieved by our underlying provider. Thanks to the live-migration feature of our hypervisors (VMware), it did not have any impact on your applications or addons availability, communication has been done through our status page.

The second required maintenance operation is to update the Linux kernel of the virtual machines in our infrastructure. This upgrade is required to protect the infrastructure against cross-process information leak. By its nature, this kernel upgrade is not achievable without rebooting each operating system.

As soon as Canonical will release its patched version of the Linux kernel, we'll start this second maintenance operation:

• Application containers will be transparently migrated before a server is rebooted as the platform already does for self-healing operations or internal load balancing actions. This has no impact on applications availability.

• Single-node database addons will experience a short downtime during the reboot of the server, and their initialization duration.

• Multi-node databases addons will be restarted transparently, as only one of the nodes will be restarted at a given time, the cluster will remain available.

If you have any question concerning this maintenance operation, feel free to reach us through our support channels (in-app or email).

Performance Impact on the hosted applications and databases

The patchset added to the Linux kernel, named Kernel page-table isolation has a performance downside. Reseachers have observed a performance overhead of around 5% for most workload, and up to 30% in some more uncommon cases (application with intense memory usage).

Linux maintainers have decided to include KPTI in the mainline kernel first to protect against Spectre and Meltdown, then work will be done to improve performance and reduce the above-mentioned overhead. Our infrastructure will also be updated when this work will be released.

More about Spectre and Meltdown

If you wish to get more information about these security threats, here are interesting references about them:

• Summary about Spectre and Meltdown: causes, impacts and Q&A
• Google's What you need to know posting.
• A summary of performance impacts from the mitigations, posted by Red Hat.
• A vague statement from Intel that fixes are on the way. "Intel has already issued updates for the majority of processor products introduced within the past five years. By the end of next week, Intel expects to have issued updates for more than 90 percent of processor products introduced within the past five years."
• Intel has published a white paper [PDF] on the vulnerabilities.
• AMD's update and ARM's update on which processors are vulnerable.
• General distributor/project updates (other than specific package alerts): Chromium, Mozilla, Qubes, Red Hat, SUSE, Ubuntu, Xen, Fedora, Xen FAQ,
• LWN articles about Kernel patches: retpoline, IBRS control (for indirect branch speculation), speculative read inhibition.