GhostLock: How We Secured 6,000 Applications in 30 Hours

Context

On July 8, 2026, while we were all gathered in Strasbourg for our usual quarterly event, our security team received an alert. A new security vulnerability was making headlines: CVE-2026-43499, nicknamed “GhostLock”. It is a vulnerability affecting the Linux kernel, with a base severity rating from the vendor (CVSS score) of 7.8/10. In the specific context of the Scalingo platform, that score rises to 9.4/10.

The vulnerability was discovered as part of IonStack and, when combined with another vulnerability, can be used to gain full remote root access to an Android system with just a single user click.

This vulnerability immediately caught our attention, and the security team decided to investigate it. The researchers described an exploit chain capable of achieving privilege escalation with a reported reliability of around 97%. At the time of our analysis, the publicly available proof of concept (PoC) notably demonstrated a denial of service through a kernel panic. The vulnerability was introduced in Linux 2.6.39, released in 2011, and is therefore present in a very large number of systems. More concerning still, it relies entirely on standard Linux kernel features.

Since the beginning of the year, we had already encountered several high-severity vulnerabilities. Until then, however, they had affected kernel modules that were enabled by default but very rarely used in practice at Scalingo. This had allowed us to quickly implement simple, controlled mitigations: disabling the vulnerable modules without rebooting the machines, then rolling out the change in an automated way across several hundred servers.

GhostLock was different: it does not rely on any feature that can simply be disabled, as it exclusively uses core system functionality (pthread_mutex / futex operations).

At that point, we had only reproduced the denial-of-service condition demonstrated by the public PoC. However, the researchers described a more advanced chain allowing privilege escalation and container escape, a scenario we considered credible enough to take immediate action.

Action Plan and Setting Up the Incident Response Team

As soon as we became aware of this information, our first priority was to determine whether the Scalingo platform was vulnerable. The answer came quickly: within minutes, we managed to crash two out of three systems tested in our staging environment. In practical terms for Scalingo, this meant that a malicious user could potentially bring down enough servers to cause a platform-wide issue.

With the risk now confirmed, there was only one objective: eliminate it. The proof of concept (PoC) published by the security researchers offered little reassurance: there was no complete workaround. The only known fix at the time for Ubuntu systems was to upgrade the kernel to version 7.0.0-27.27, which had just been released by Canonical, the company behind Ubuntu.

However, this kernel was only available for the latest release, Ubuntu 26.04, which had been released in April. Like most of the industry, Scalingo was not yet running this version on its servers. Work to upgrade our infrastructure to Ubuntu 26.04 had recently begun, but was still at an early planning stage and had been paused for the duration of our company event.

Given the limited options available to us, we made a simple decision: dedicate the entire infrastructure team to the Ubuntu 26 project, complete all the necessary work, and deploy it to production as quickly as possible, prioritizing the servers hosting our customers' application containers.

At Scalingo, each server has a role that defines its configuration and all the services running on it. For this vulnerability, two types of servers were at risk:

• Servers hosting customer application containers: this represented the greatest risk. A malicious user could easily crash dozens of servers in succession, affecting a significant number of our customers.

• Servers responsible for building Docker images during deployments: the risk here was that new deployments could become unavailable. All applications already hosted on the platform would continue to operate normally, but it would no longer be possible to perform new deployments. While the impact would still be significant, these servers were less critical than those directly hosting customer applications.

We therefore decided to focus our efforts on the servers hosting customer application containers. We kept the image build servers in mind but planned to address them in a second phase.

At the same time, we set up an incident response team led by the CISO and including executive leadership, the CTO, the Technical Lead, and the infrastructure team. The CISO was responsible for monitoring developments around the vulnerability and handling internal and external communications, including preparing a security bulletin. The CTO and Technical Lead defined the technical plan and ensured it remained on track, while also supporting the infrastructure team responsible for executing it.

The infrastructure team was therefore relieved of its planned work for the week. Those responsibilities were taken over by other teams, particularly SRE, so that the infrastructure team could focus entirely on this effort.

Technical Implementation and Challenges

At Scalingo, when we want to move a server from one Ubuntu version to another, we don't log into the server and manually run an upgrade procedure as we might on our personal computers. In fact, we don't upgrade the server at all: we create a new virtual machine running the new version, drain the old machine, and then delete it. This is an approach widely used across the industry and is much more efficient than performing a manual upgrade.

Before we could boot a VM on a new Ubuntu release, there were two major steps:

• Create a custom Ubuntu 26.04 image (known as an OMI, or AMI in the AWS ecosystem);

• Get Chef, our tool for configuring our servers and deploying software to them, working on Ubuntu 26.04.

Because we were all physically present in Strasbourg at the time, we took advantage of the situation to gather in the same room and quickly divide the work among the infrastructure team. Although we're used to working remotely, being next to each other made it possible to see everyone's progress at a glance and resolve blockers more efficiently.

Our work therefore began with the creation of an OMI. This is a machine image that serves as the base for creating our virtual machines (VMs). An OMI defines the operating system version to boot — Ubuntu 26.04 in our case — the parameters passed to the kernel at startup, and other basic configuration settings. An OMI can also contain software and various configurations, making them immediately available when the VM first boots. At Scalingo, our OMIs are relatively simple, so we expected to have a working Ubuntu 26.04 version fairly quickly.

In parallel, we needed to get Chef working on Ubuntu 26.04. Chef is the software that runs on all our VMs and is responsible for configuring them and deploying all the software required according to each machine's role. We were more apprehensive about getting it to run on Ubuntu 26.04 for several reasons:

• Our version of Chef was not up to date because of the heterogeneous nature of our infrastructure. We needed to remain on an older version of Chef so that it would continue to work on machines running older versions of Ubuntu;

• Even if Chef itself worked, some of the cookbooks — dependencies in the Chef ecosystem — that we use might not support Ubuntu 26.04, which was still a relatively recent release;

• Some of the software required to operate the platform might need modifications to work on Ubuntu 26.04;

• Finally, moving to 26.04 meant that many components would move to new major versions, including Docker. Not everything might be compatible with those versions.

With these uncertainties in mind, we started running Chef on Ubuntu 26.04 in our development environment. Although we did encounter a few difficulties, they remained limited. In the end, only a small number of changes were necessary, particularly because the nodes hosting our customers' containers run only a limited subset of our software.

Deployment Plan

In parallel with this work, we began thinking about how to migrate all applications to up-to-date, non-vulnerable servers as quickly as possible and with minimal impact. Our existing orchestration tools make it easy to drain a machine by redistributing its containers across other servers. However, the challenge went far beyond that. We wanted to drain as many nodes as possible, as quickly as possible, without any visible impact on customer applications or excessive contention caused by migrating hundreds of containers simultaneously.

We therefore designed a two-part strategy:

• Create as many new application VMs as possible running Ubuntu 26.04;

• Migrate all containers from a vulnerable node A to a new, patched node B, taking advantage of the platform's container migration feature to do so without impacting customer applications.

This approach allowed us to work around the orchestration limitations we were already aware of:

• Our container migration algorithm was not originally designed to migrate so many applications in such a short period of time. During a migration, the scheduler normally processes each application and, for each of its containers, starts a new container on whichever machine it considers the most appropriate. This algorithm works well when draining a single node, but doesn't scale particularly well. When many containers are moved simultaneously, the scheduler tends to repeatedly select the same destination machine, creating a heavy load by asking it to start many containers at once. This behavior had already caused an incident during a previous maintenance operation. That's why we decided to write a script capable of “transferring” containers from server A to server B.

  The idea was fairly simple: for each vulnerable application server, we would create its Ubuntu 26.04 patched counterpart. Once the new server was ready, the script would take the containers from the original server one by one and move them to the Ubuntu 26.04 server. By leaving enough time between containers, we could maintain a steady flow of startups on the new servers without any visible impact, since no more than three or four containers would ever be starting on the same node at the same time.

  
  

• Creating new Ubuntu 26.04 VMs presented another technical limitation. In the osc-fr1 region, the virtual network containing all the application machines has a limited size, meaning we couldn't simply double the number of available machines. There would not be enough IP addresses in the network to address them all. This was a known limitation, and structural infrastructure changes had already been underway for several months to remove it. In this situation, however, we had to work within that constraint.

  We therefore adopted a batch-based approach. The idea was to create as many new Ubuntu 26.04 VMs as possible. Once they were ready, we would migrate all application containers from the first set of unpatched VMs to the Ubuntu 26.04 VMs. As soon as the unpatched VMs had been completely drained, we would delete them, freeing their IP addresses, and continue with a second batch. We would repeat the process until only patched Ubuntu 26.04 machines remained. Given the current size of osc-fr1, we would need four batches to migrate the entire region.

  In our second region, osc-secnum-fr1, the problem was simpler. Because the region is smaller, we could create enough VMs in one go to temporarily double its size. We would then simply migrate all containers from unpatched nodes to patched ones.

Pausing for the Night

By the evening of that first day, we had a working Ubuntu 26.04 AMI, and Chef was running in our local environment on an application-type machine. However, we still needed to create a VM under real-world conditions in our staging infrastructure and, most importantly, complete the full QA (quality assurance) process before considering a production rollout. This step could not be sacrificed for the sake of urgency, as doing so could have caused an incident more severe than the original problem.

Knowing that this work would take several more hours and that, even with considerable effort, we would not be able to secure the platform by the following morning, we decided to stop at this point for the night.

To reduce the attack surface, we temporarily suspended new account registrations and increased monitoring of the platform until the deployment resumed.

QA and Deployment

The following morning, we created the first VM running Ubuntu 26.04 in our staging environment. We spent the entire morning and early afternoon going through our verification procedures (QA) and fixing the few minor issues we identified. We also confirmed that it was no longer possible to crash the system, which remained the main reason we had undertaken this work with such urgency.

At 5:00 PM, we began the production rollout in the osc-secnum-fr1 region. Within an hour, the size of the application fleet had doubled, and we ran one final round of checks on the new nodes directly in production. Once that validation was complete, all the old nodes were marked as ineligible to receive new containers, leaving only patched nodes available.

At 6:15 PM, each VM began migrating all its containers to an Ubuntu 26.04 machine according to the procedure defined the previous day, transparently for customers. The migration took several hours and finished later that evening. The only visible trace was an event in each application's Scalingo timeline indicating that its containers had been restarted for “load balancing” purposes.

The rollout to osc-fr1 began somewhat later for two reasons. First, our quotas needed to be increased to allow us to create that many VMs at once. In addition, our infrastructure provider was experiencing an incident at the time, unrelated to our own operations, so we waited for the service to fully recover and stabilize before launching the first batch.

At 9:30 PM, the first batch began, and a group of unpatched machines started moving their containers to up-to-date servers.

By the end of the evening, a large proportion of the nodes in the first batch had been drained. The day ended at around 12:20 AM, as the last servers in the first batch finished moving their containers. New account registrations were disabled again overnight.

The following day began before 8:00 AM as we prepared to launch the second batch. Before we could do so, we first had to deal with the applications that had refused to migrate automatically to the new servers. If our algorithm is unable to start a working container on a new node, it deliberately leaves the existing container running on its current node rather than risk making the application unavailable.

There are several reasons why a container might fail to start. The most obvious is excessive memory consumption during startup, causing the container to crash shortly after launch even though its memory requirements during normal operation are lower. Because there is no universal way to handle these situations, every container that refused to start had to be quickly investigated manually so that we could complete its migration. Even with only a small number of containers exhibiting this behavior on each node, the process quickly became time-consuming.

Once this manual work was complete, we moved directly on to the second batch. By noon, 60% of the nodes in the osc-fr1 region were running the patched Ubuntu 26.04 release.

At 8:30 PM, a third migration wave was launched. Finally, at 11:15 PM, every node capable of hosting application containers was running Ubuntu 26.04, with the last applications still being migrated. At that point, the platform was fully secured because there was no longer any way to start an application container on a node affected by the vulnerability.

The migration of the remaining applications finished overnight, while our work ended there.

In around 30 hours, we migrated more than 6,000 applications and 19,000 containers, representing more than 5 PB of RAM, across our two regions. These migrations were carried out transparently and without any impact on users.

Work on Deployment Servers

The following Monday, on the eve of the national holiday, we decided to begin work on the second type of server still exposed to the vulnerability: the servers responsible for deployments.

Work began early in the day and proceeded smoothly, as the only difficulties had already been identified during the previous week's work on the application infrastructure. By early afternoon, the first node was available in our staging environment. QA uncovered no issues, allowing the production rollout to begin at 4:00 PM.

These nodes did not require a complex batch system. We could simply temporarily double the number of VMs and prevent new deployments from starting on the old ones, ensuring that every new deployment would run on a patched server. Deployments already in progress would finish on vulnerable servers, but they would be the last deployments performed on those machines.

By 5:00 PM, all new deployments launched in the osc-secnum-fr1 region were running on patched nodes. The osc-fr1 region followed at 6:00 PM. By the end of the day, the entire pipeline, from deployment through execution, was therefore secured against GhostLock.

These servers were the last to be patched through our emergency process. All other machines, whether internal systems or database nodes, would follow Scalingo's standard node replacement policy, as described in the security bulletin published at the same time.

Taking a Step Back: Lessons Learned

This security event gave us an opportunity to put the agility of both our platform and our teams to the test in a crisis situation. With well-established procedures and a clear-headed plan, it took less than three days to secure our entire application fleet.

The accelerating development of AI is forcing Security and Operations teams to respond faster than ever, sometimes to major vulnerabilities that have remained hidden for decades. At this increased pace, automation is more essential than ever. Even a small number of manual actions can have a major impact on execution speed when carrying out large-scale operations under severe time constraints.

It is also essential to have reliable internal tools. In our case, they allowed us to carry out large-scale changes across the platform while maintaining control over the operation.

More than ever, SaaS, IaaS, and PaaS providers must adapt to a world in which threats are intensifying and the need for security continues to grow.