Security Bulletin: "Log4Shell" Apache Log4j2 vulnerability (CVE-2021-44228 & CVE-2021-45046)

[Obsolete - See update below] A critical vulnerability was discovered last week in the Apache Log4j library. This impacts a wide range of software in the Java ecosystem. This article will briefly summarize the impact it has on Scalingo and give some advices on how you should act if you are using a Java app.

[Update 17/12/2021] As you have probably heard, several critical vulnerabilities (CVE-2021-44228 and CVE-2021-45046) were discovered in the last days in the Apache Log4j2 library. This impacts a wide range of software in the Java ecosystem.

[Update 20/12/2021] A less critical vulnerability referenced as CVE-2021-45105 has been reported. It is now advised to upgrade to Log4j2 2.17. We report it here for the sake of completeness, but as it doesn’t affect directly or indirectly our services, we will stop our Incident Response Team’s work on this family of vulnerabilities unless there is again a critical vulnerability reported. Please use https://logging.apache.org/log4j/2.x/security.html as a reference for Log4j2 patches and mitigations.

[Update 29/12/2021] CVE-2021-44832 has been reported. Please read the available information and apply the patches at your discretion.

This article will briefly summarize the impact they have on Scalingo and give some advice on how you should act if you are using a Java app. We are trying to update it daily based on our knowledge and the current situation development. Last article update: 20/12/2021 18:15

What happened?

Last week a critical vulnerability CVE-2021-44228 was discovered in the Apache Log4j2 library.

The Apache foundation has published this documentation on the subject if you want to read it.

Log4j2 versions between 2.0-beta9 and 2.14.1 are impacted. The first patched version is 2.15.0 and workaround is available for previous versions (see below).

[Update 17/12/2021] As CVE-2021-45046 has been reported, the upgrade to Log4j2 ≥ 2.16.0 (for Java 8 users) or Log4j2 2.12.2 (for Java 7 users) is mandatory to fix the vulnerability.
[Update 20/12/2021] As CVE-2021-45105 has been reported, the upgrade to Log4j2 ≥ 2.17.0 (for Java 8 users) or Log4j2 2.12.3 (for Java 7 users) is mandatory to fix the vulnerability.

[Update 29/12/2021] As CVE-2021-44832 has been reported, you may want to upgrade to Log4j2 ≥ 2.17.1 (for Java 8 users) or Log4j2 ≥ 2.12.4 (for Java 7 users) or Log4j2 ≥ 2.3.2 (for Java 6 users) to fix the vulnerability.

Impact on Scalingo and what we are doing towards this

Internal Services

At Scalingo, we are not heavy users of Java-based apps, nevertheless the open-source tool Metabase is used to do some analytics.

At the date of the incident, our Metabase instance was potentially affected by the vulnerability, but being run with an up-to-date JRE (Java Runtime Engine) that mitigated the issue, it wasn't possible to exploit it.

But we didn't wait to do an in-depth analysis and had already patched it this week-end.

We have also scanned our Metabase application logs and internal router and didn't find any trace of exploitation tentative prior to the application being patched.

FYI : we are now seeing what seems to be automated intrusion tentatives using the exploit on our infrastructure but thanks to the application being up to date, they won't work.

[Update 17/12/2021] We patched our internal services again for CVE-2021-45046

[Update 20/12/2021] There is no new impact on Scalingo

[Update 29/12/2021] There is no new impact on Scalingo

Elasticsearch as a Service

At Scalingo we are offering Elasticsearch to our customers as a database addon, and it was a priority to ensure the data of our customers were safe. Elasticsearch's official communication is stating that because they use the Java Security Manager feature, the impact of the CVE was not as high as other tools, but still a risk exists.

Two layers of protection are setup by default at Scalingo that mitigate further this issue:

• Elasticsearch instances are never available directly to users and applications. Indeed each deployment is done inside a private network which only entrypoint for clients is a HAProxy gateway handling authentication. It means only authenticated requests are forwarded to Elasticsearch instances themselves. (If your password is available to an attacker, they already have access to your data)
• By default, our databases are not accessible on the internet. This help reduces the attack surface, as automated entities trying to exploit the CVE won’t find any Elasticsearch instance. Although this is not the case if the instance was explicitly exposed publicly.

Considering these measures, the impact of the CVE becomes lower on our Elasticsearch deployments.

As security is an accumulation of layers though, the version 6.8.21 of Elasticsearch released in reaction of the CVE has been made available. You can upgrade your ES instance to this new version on Scalingo immediately using your database addon management dashboard.

[Update 17/12/2021] Elasticsearch does not need a new update for CVE-2021-45046 according this Security Announcement

[Update 20/12/2021] Elasticsearch does not need a new update for CVE-2021-45105 according this Security Announcement

Third-party deployed via "one-click deploy"

Our "one-click deploy" feature is a quick-way to deploy third-party code as Scalingo applications.

To do so, the projects have their own GitHub repositories and can be offered by anybody, without any support from Scalingo.

Nevertheless, there are a few "one-click deploy" projects that we historically host or use internally. We updated them and you must trigger a new deployment.

• Metabase
  • [Update 17/12/2021] you need to trigger a deployment again to update to 0.41.5
  • [Update 20/12/2021] Metabase is not vulnerable to CVE-2021-45105
• Sonarqube
  • [Update 17/12/2021] you need to trigger a deployment again to update to 8.9.5 or 9.2.3. Be careful to not have set SONARQUBE_VERSION variable (or set it to something greater than 8.9.5 or 9.2.3)
  • [Update 20/12/2021] Sonarqube seems to be vulnerable to CVE-2021-45105, as the time of this update there is no available fix. When a fix is released, please trigger a deployment. Follow any updates on Sonarqube here.
• Logstash
  • [Update 17/12/2021] not affected by CVE-2021-45046, but be sure to have at least version 6.8.21 or 7.16.1)
  • [Update 20/12/2021] Logstash does not need a new update for CVE-2021-45105 according this Security Announcement
• Kibana
  • [Update 20/12/2021] Added for clarity: Kibana does not need any update regarding this vulnerabilities according this Security Announcement

Important: You MUST trigger a new deployment to get the latest patched version, using the instructions in the README.

If you are using a third-party "one-click deploy" please contact the original maintainer to check if an updated version has been deployed.

[Update 17/12/2021] Third-party buildpacks

If you are using Datadog in your application via our multi-buildpack support your application is vulnerable and you must trigger a deployment either via the Dashboard or the CLI.

WARNING: If you pinned your version with the DD_AGENT_VERSION environment variable, you must ensure that this variable should be ≥ 6.32.3 (or 7.32.3)

Source: https://www.datadoghq.com/log4j-vulnerability/

[Update 20/12/2021] Please read the updated article from Datadog about the impact of CVE-2021-45105

Other Java applications

This paragraph is here to help you based on our experience if you are hosting a Java app (on Scalingo or anywhere else).

You can also find a list of Log4Shell vulnerable software here.

What's the risk ?

An attacker crafting the correct string can execute any code in the context of your application thus leading to potential:

• Confidentiality: exfiltrating information from your application (and indirectly your database)
• Availability: disrupting the functionality of your application
• Integrity: inserting malformed or incorrect data in your database

How to know if your app is impacted

First question you have to answer: "Does your application use Java?"

If the answer is NO, you're safe.

Second question: "If you use Java, is your application or one of its dependencies use Log4j2 <= 2.14.1?" [Update 17/12/2021] Versions before 2.16.0 are potentially vulnerable. [Update 20/12/2021] Versions before 2.17.0 are potentially vulnerable. [Update 29/12/2021] Versions before 2.17.1 are potentially vulnerable.

If the answer is NO, you're safe. But it seems this kind of answer is terribly hard to answer due to the loads of dependencies Java apps use.

Possible corrections

You have several quick changes that you can put in practice.

Option 1: Update the Log4j2 dependency

• You control the code of your app, use your build system (Maven, Gradle, ...) to update Log4j2 to the latest version
  • [Update 17/12/2021]
    • Java 8 (or later) users should upgrade to release 2.16.0
    • Java 7 users should upgrade to release 2.12.2
  • [Update 20/12/2021]
    • Java 8 (or later) users should upgrade to release 2.17.0
    • Java 7 users should upgrade to release 2.12.3
  • [Update 29/12/2021]
    • Java 8 (or later) users should upgrade to release 2.17.1
    • Java 7 users should upgrade to release 2.12.4
    • Java 6 users should upgrade to release 2.3.2

• You don't control the source code, update the software. Example for Metabase:

  scalingo --app my-app deploy <https://github.com/Scalingo/metabase-scalingo/archive/refs/heads/master.tar.gz>

Option 2: If you can't update Log4j2, you can set this environment variable

LOG4J_FORMAT_MSG_NO_LOOKUPS=true to patch the vulnerability.

This can be done via our dashboard or using this command scalingo -a my-app env-set LOG4J_FORMAT_MSG_NO_LOOKUPS=true

Please note that this only works for log4j2 > 2.10 and not earlier versions.

[Update 17/12/2021] This fix can be bypassed in extremely rare configurations, but is still a valid first response.

[Update 17/12/2021] Option 3: If you really are stuck and can’t update, but want to eliminate the possibility of doing JNDI Lookups, you could use the Method 2 given by JFrog that eliminates the problematic class from jour environment.

An easy way to implement this on Scalingo is to use a profile script.

Conclusion

Scalingo has patched all affected internal services by the vulnerability. No data breach was possible nor detected.

The impacted Scalingo product is Elasticsearch, the vulnerability is not exploitable. You can update it for added security using the database dashboard.

For any other Java Apps hosted on Scalingo, please update it or ensure you are already using a patched Log4j2 version.

Also if you are a Scalingo customer, you can come to our support for any doubt or questions. Our team will come back to you and make their best effort to answer your questions.