[Obsolete - See update below]
A critical vulnerability was discovered last week in the Apache Log4j library. This impacts a wide range of software in the Java ecosystem. This article will briefly summarize the impact it has on Scalingo and give some advices on how you should act if you are using a Java app.
[Update 17/12/2021] As you have probably heard, several critical vulnerabilities (CVE-2021-44228 and CVE-2021-45046) were discovered in the last days in the Apache Log4j2 library. This impacts a wide range of software in the Java ecosystem.
[Update 20/12/2021] A less critical vulnerability referenced as CVE-2021-45105 has been reported. It is now advised to upgrade to Log4j2 2.17. We report it here for the sake of completeness, but as it doesn’t affect directly or indirectly our services, we will stop our Incident Response Team’s work on this family of vulnerabilities unless there is again a critical vulnerability reported. Please use https://logging.apache.org/log4j/2.x/security.html as a reference for Log4j2 patches and mitigations.
[Update 29/12/2021] CVE-2021-44832 has been reported. Please read the available information and apply the patches at your discretion.
This article will briefly summarize the impact they have on Scalingo and give some advice on how you should act if you are using a Java app. We are trying to update it daily based on our knowledge and the current situation development. Last article update: 20/12/2021 18:15
Last week a critical vulnerability CVE-2021-44228 was discovered in the Apache Log4j2 library.
The Apache foundation has published this documentation on the subject if you want to read it.
Log4j2 versions between 2.0-beta9 and 2.14.1 are impacted. The first patched version is 2.15.0 and workaround is available for previous versions (see below).
[Update 17/12/2021] As CVE-2021-45046 has been reported, the upgrade to Log4j2 ≥ 2.16.0 (for Java 8 users) or Log4j2 2.12.2 (for Java 7 users) is mandatory to fix the vulnerability.
[Update 20/12/2021] As CVE-2021-45105 has been reported, the upgrade to Log4j2 ≥ 2.17.0 (for Java 8 users) or Log4j2 2.12.3 (for Java 7 users) is mandatory to fix the vulnerability.
[Update 29/12/2021] As CVE-2021-44832 has been reported, you may want to upgrade to Log4j2 ≥ 2.17.1 (for Java 8 users) or Log4j2 ≥ 2.12.4 (for Java 7 users) or Log4j2 ≥ 2.3.2 (for Java 6 users) to fix the vulnerability.
At Scalingo, we are not heavy users of Java-based apps, nevertheless the open-source tool Metabase is used to do some analytics.
At the date of the incident, our Metabase instance was potentially affected by the vulnerability, but being run with an up-to-date JRE (Java Runtime Engine) that mitigated the issue, it wasn't possible to exploit it.
But we didn't wait to do an in-depth analysis and had already patched it this week-end.
We have also scanned our Metabase application logs and internal router and didn't find any trace of exploitation tentative prior to the application being patched.
FYI : we are now seeing what seems to be automated intrusion tentatives using the exploit on our infrastructure but thanks to the application being up to date, they won't work.
[Update 17/12/2021] We patched our internal services again for CVE-2021-45046
[Update 20/12/2021] There is no new impact on Scalingo
[Update 29/12/2021] There is no new impact on Scalingo
At Scalingo we are offering Elasticsearch to our customers as a database addon, and it was a priority to ensure the data of our customers were safe. Elasticsearch's official communication is stating that because they use the Java Security Manager feature, the impact of the CVE was not as high as other tools, but still a risk exists.
Two layers of protection are setup by default at Scalingo that mitigate further this issue:
Considering these measures, the impact of the CVE becomes lower on our Elasticsearch deployments.
As security is an accumulation of layers though, the version 6.8.21 of Elasticsearch released in reaction of the CVE has been made available. You can upgrade your ES instance to this new version on Scalingo immediately using your database addon management dashboard.
[Update 17/12/2021] Elasticsearch does not need a new update for CVE-2021-45046 according this Security Announcement
[Update 20/12/2021] Elasticsearch does not need a new update for CVE-2021-45105 according this Security Announcement
Our "one-click deploy" feature is a quick-way to deploy third-party code as Scalingo applications.
To do so, the projects have their own GitHub repositories and can be offered by anybody, without any support from Scalingo.
Nevertheless, there are a few "one-click deploy" projects that we historically host or use internally. We updated them and you must trigger a new deployment.
Important: You MUST trigger a new deployment to get the latest patched version, using the instructions in the README.
If you are using a third-party "one-click deploy" please contact the original maintainer to check if an updated version has been deployed.
If you are using Datadog in your application via our multi-buildpack support your application is vulnerable and you must trigger a deployment either via the Dashboard or the CLI.
WARNING: If you pinned your version with the
DD_AGENT_VERSION environment variable, you must ensure that this variable should be ≥ 6.32.3 (or 7.32.3)
[Update 20/12/2021] Please read the updated article from Datadog about the impact of CVE-2021-45105
This paragraph is here to help you based on our experience if you are hosting a Java app (on Scalingo or anywhere else).
You can also find a list of Log4Shell vulnerable software here.
An attacker crafting the correct string can execute any code in the context of your application thus leading to potential:
First question you have to answer: "Does your application use Java?"
If the answer is NO, you're safe.
Second question: "If you use Java, is your application or one of its dependencies use Log4j2 <=
2.14.1?" [Update 17/12/2021] Versions before 2.16.0 are potentially vulnerable. [Update 20/12/2021] Versions before 2.17.0 are potentially vulnerable. [Update 29/12/2021] Versions before 2.17.1 are potentially vulnerable.
If the answer is NO, you're safe. But it seems this kind of answer is terribly hard to answer due to the loads of dependencies Java apps use.
You have several quick changes that you can put in practice.
Option 1: Update the Log4j2 dependency
You don't control the source code, update the software. Example for Metabase:
scalingo --app my-app deploy <https://github.com/Scalingo/metabase-scalingo/archive/refs/heads/master.tar.gz>
Option 2: If you can't update Log4j2, you can set this environment variable
LOG4J_FORMAT_MSG_NO_LOOKUPS=true to patch the vulnerability.
This can be done via our dashboard or using this command
scalingo -a my-app env-set LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Please note that this only works for log4j2 > 2.10 and not earlier versions.
[Update 17/12/2021] This fix can be bypassed in extremely rare configurations, but is still a valid first response.
[Update 17/12/2021] Option 3: If you really are stuck and can’t update, but want to eliminate the possibility of doing JNDI Lookups, you could use the Method 2 given by JFrog that eliminates the problematic class from jour environment.
An easy way to implement this on Scalingo is to use a profile script.
Scalingo has patched all affected internal services by the vulnerability. No data breach was possible nor detected.
The impacted Scalingo product is Elasticsearch, the vulnerability is not exploitable. You can update it for added security using the database dashboard.
For any other Java Apps hosted on Scalingo, please update it or ensure you are already using a patched Log4j2 version.
Also if you are a Scalingo customer, you can come to our support for any doubt or questions. Our team will come back to you and make their best effort to answer your questions.