corretto / hotpatch-for-apache-log4j2 Goto Github PK
View Code? Open in Web Editor NEWAn agent to hotpatch the log4j RCE from CVE-2021-44228.
License: Other
An agent to hotpatch the log4j RCE from CVE-2021-44228.
License: Other
Apache Oozie job intermittently fails after applying the hotpatch. This is because Oozie will list and the print the file and directory name in the current directory. On the other hand, the hotpatch creates attach_pid file in the current directory to attach the JVM and then delete it. When the attach_pid file is deleted between the listing and the printing, the Oozie job fails by NoSuchFileException. The below is the stacktrace:
java.nio.file.NoSuchFileException: ./.attach_pid1517
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
at java.nio.file.Files.walkFileTree(Files.java:2706)
at org.apache.oozie.action.hadoop.LocalFsOperations.printContentsOfDir(LocalFsOperations.java:59)
at org.apache.oozie.action.hadoop.LauncherAM.printDebugInfo(LauncherAM.java:293)
at org.apache.oozie.action.hadoop.LauncherAM.access$100(LauncherAM.java:55)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:221)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
The mitigation is to disable the hotpatch on the Oozie server.
I plan to provide a fix to Oozie to retry the directory listing, however, it is worth creating the issue here for searchability. Thanks.
Build and publish the current jar output and validate it can be used with at least 8, 11, 15 and 17
Steps to reproduce :
If you try to install any one of these procps-ng/net-tools/iputils before installing corretto, then log4j hotpatch will report following error while installation, however installation completed.
Dockerfile :
FROM amazonlinux:2
RUN yum update -y
RUN yum install -y procps-ng net-tools iputils
RUN amazon-linux-extras enable corretto8
RUN yum install -y java-1.8.0-amazon-corretto-devel
$ docker build -t log4j .
Installation of log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch
reports following error :
Failed to get D-Bus connection: Operation not permitted
Created symlink /etc/systemd/system/multi-user.target.wants/log4j-cve-2021-44228-hotpatch.service, pointing to /usr/lib/systemd/system/log4j-cve-2021-44228-hotpatch.service.
Failed to get D-Bus connection: Operation not permitted
warning: %post(log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch
As a workaround, you can try installing procps-ng/net-tools/iputils these after corretto.
If I understand the code correctly, package name is included in checking the impacted class. As shading would change the package name, this implies that the solution doesn't work with uber jars where log4j has been shaded to avoid version conflicts.
Can you confirm this?
Note: All instructions here assume Java 11 (OpenJDK), but I was also able to reproduce the issue with Java 8 (OpenJDK).
.war
).java -jar jenkins.war
.Log4jHotPatch
and run java -Dlog4jFixerVerbose=true -cp Log4jHotPatch.jar Log4jHotPatch ${JENKINS_PID}
.Note: These are the actual results with a local (non-official) build of Jenkins:
The Log4jHotPatch
agent is attached successfully.
The Log4jHotPatch
agent cannot be attached:
com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:165)
at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:228)
at Log4jHotPatch.main(Log4jHotPatch.java:301)
Error: couldn't loaded the agent into JVM process 546677
The Jenkins logs show:
Exception in thread "Attach Listener" java.lang.SecurityException: class "Log4jHotPatch"'s signer information does not match signer information of other classes in the same package
at java.base/java.lang.ClassLoader.checkCerts(ClassLoader.java:1151)
at java.base/java.lang.ClassLoader.preDefineClass(ClassLoader.java:906)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1015)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:431)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Agent failed to start!
Stepping through the problematic frames in the debugger, I do not think Jenkins is at fault here. Jenkins contains some classes in the unnamed package namespace:
$ jar tf jenkins.war
[…]
ColorFormatter.class
JNLPMain.class
LogFileOutputStream$1.class
LogFileOutputStream.class
Main$FileAndDescription.class
Main.class
MainDialog.class
[…]
META-INF/JENKINS.RSA
META-INF/JENKINS.SF
META-INF/MANIFEST.MF
[…]
$
Note that META-INF/JENKINS.RSA
and META-INF/JENKINS.SF
are only present in our signed official releases; a local build will not contain these. The presence of these classes and signatures means that the system class loader associates the signatures with the unnamed package namespace. Then, when Log4jHotPatch
attempts to load a class into the same unnamed package namespace without a signature, class loading fails.
I do not think Log4jHotPatch
should make any assumptions about whether or not it is free to load unsigned classes into the unnamed package namespace. Perhaps Log4jHotPatch
should use its own package namespace instead.
Amazon Linux 1 instances running a variety of Java versions.
Our security log watcher was detecting a user logging in which is a service account and should not log in. The timestamps on these events are within 1-3 seconds of the log4jhotpach firing and logging to /var/log/messages
However, it was impossible for us to see what the process was as we have several short-lived processes which are gone by the time we receive the alert.
I modified the log4j-cve-2021-44228-hotpatch script to log more information so we can better identify they detected process. I added this line of code
log "PID Info $(ps -o user:12,pid,ppid,pgid,sid,stat,time,lstart,cmd f ${pid})"
at line 114 of the 1.1.12 version of the patch
This helped us find the misbehaving processes.
Please consider adding something like this to the hotpatch
When trying to run
java -cp <java-home>/lib/tools.jar:Log4jHotPatch.jar <java-pid>
with JDK 8, I only get the message
Error: Could not find or load main class <PID>
what could i be doing wrong?
Thank you for posting this patch to the public. I fixed the issue locally by simply deleting the unicode characters, would still like to report it. The issue occurred in the License section of comments.
javac -XDignore.symbol.file=true -cp %JAVA_HOME%/lib/tools.jar Log4jHotPatch.java
Log4jHotPatch.java:4: error: unmappable character for encoding Cp1252
Using recent Gradle or Maven with plugins on some older distro builds systems is problematic.
Create a parallel system that builds the same artifacts
Import ASM source and update license, 3p attributions
Shade package name and imports in source
Update gradle to use source build of ASM
I seen this issue, I'm using java8
com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:121) at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:234) at Log4jHotPatch.main(Log4jHotPatch.java:298) at Log4jHotPatch17.main(Log4jHotPatch17.java:18) Error: couldn't loaded the agent into JVM process 1234 Are you running as a different user (including root) than process 1234? Errors occurred deploying hot patch. If you are using java 8 to run this tool against JVM 11 or later, the target JVM may still be patched. Please look for a message like 'Loading Java Agent (using ASM 6).' in stdout of the target JVM. Also note that JVM 17+ are not supported.
The current tests are based on a simple shell script. Move those tests into gradle and stop including a log4j jar for testing.
The command gives me the following error:
# ./gradlew build --warning-mode all
> Configure project :
The AbstractArchiveTask.version property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveVersion property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:version for more details.
at build_34dfj8owb22lztkrt6bemzzq5$_run_closure2.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:17)
(Run with --stacktrace to get the full stack trace of this deprecation warning.)
The AbstractArchiveTask.archiveName property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveFileName property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:archiveName for more details.
at build_34dfj8owb22lztkrt6bemzzq5$_run_closure3.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:34)
(Run with --stacktrace to get the full stack trace of this deprecation warning.)
FAILURE: Build failed with an exception.
* Where:
Build file '/root/hotpatch-for-apache-log4j2/build.gradle' line: 46
* What went wrong:
A problem occurred evaluating root project 'hotpatch-for-apache-log4j2'.
> java.lang.NullPointerException (no error message)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1s
Should I add debug output as well?
We need to include a proper CHANGELOG.md
with all the changes we have added since the initial release.
I'm curious if you can make something more generic with gdb that will mitigate future exploits that leak environment variables?
Also for that matter, use something at the Linux level to sanitize a process's network connection from doing the JDNI lookups?
Use and improve the current tests and setup github actions for them
Environment:
macOs Monterey 12.1
java -version
openjdk version "11.0.9.1" 2020-11-04 LTS
OpenJDK Runtime Environment Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS, mixed mode)
I tried to run the below command :
jar -cfm Log4jHotPatch.jar Manifest.mf *.class
Got the error below:
java.io.FileNotFoundException: Manifest.mf (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at jdk.jartool/sun.tools.jar.Main.run(Main.java:267)
at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)
Am i missing something.
With CVE-2021-45105 a malicious user can cause a DoS which in most scenarios will lead to JVM restart.
After restart there's a time window when an attack against CVE-2021-45046 or CVE-2021-44228 could be launched before the agent has attached to the JVM process.
Would it be possible to extend this solution to cover this scenario as well?
The hot patch created a lot of /tmp mount points. Since my customer collects metrics on mount points, these /tmp mount points cost customer a lot on these unwanted metrics. Could these mount points be disabled ?
Hi,
The patch has been deployed in AL1 on one of our server via "yum update --security", and we had a problem with it.
We have the following error in the tomcat log avoiding any web applications to start on this server :
java.io.FileNotFoundException: /tmp/agent693101429317028124.jar (Aucun fichier ou dossier de ce type)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:228)
at java.util.zip.ZipFile.(ZipFile.java:157)
at java.util.jar.JarFile.(JarFile.java:169)
at java.util.jar.JarFile.(JarFile.java:106)
at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
at org.apache.tomcat.util.scan.FileUrlJar.(FileUrlJar.java:48)
at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:34)
at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1957)
at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1932)
at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1917)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1322)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5566)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1017)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:993)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
In fact, this server has a specifiticy : there are 20 concurrent JVM / Tomcat instances running and launched via chkconfig services... I'm not sure but this could be the root cause of the pb...
We applied the "sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill" to deactivate the HotPatch, and it worked.
So it confirms that the issue is due to the Log4j HotPatch.
I think you should publish a fix for this because I assume we won't be the only impacted customers.
A precision : we use amazon-corretto-8.302.08.1-linux-x64 under AL1
Best Regards.
We are seeing logging to to stdout which affects parsing of our outputs.
I have tried:
<sysproperty key="log4jFixerVerbose" value="false" escape="false"/>
However, we are still getting the log messages like:
Loading Java Agent version 1 (using ASM6).
Patching class org.apache.logging.log4j.core.lookup.JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)
Any idea why the sysproperty might be ignored? Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.