takashi-ishio / selogger Goto Github PK
View Code? Open in Web Editor NEW(Near-)omniscient debugging/tracing/logging tool for Java
License: Other
(Near-)omniscient debugging/tracing/logging tool for Java
License: Other
I realized that selogger. reader
does work for .slg
file.
It seems there are two problems with this issue, describe details below.
selogger. reader
can not handle the header.Since v0.5.0, selogger produce header informatin for CSV in the nearomni
mode, but also it seems selogger produces header imformation in omnibinary
and omni
mode.
Hence, it throws an exception when DataIdMap
function try to parse header information.
StackTrace:
Exception in thread "main" java.util.InputMismatchException
at java.util.Scanner.throwFor(Scanner.java:864)
at java.util.Scanner.next(Scanner.java:1485)
at java.util.Scanner.nextInt(Scanner.java:2117)
at java.util.Scanner.nextInt(Scanner.java:2076)
at selogger.weaver.ClassInfo.parse(ClassInfo.java:165)
at selogger.reader.DataIdMap.loadClassEntryFile(DataIdMap.java:76)
at selogger.reader.DataIdMap.<init>(DataIdMap.java:33)
at selogger.reader.LogPrinter.main(LogPrinter.java:54)`
selogger.reader
throw exception when try to parse EventType which has single Attributes.In selogger.weaver.DataInfo.parse()
, it tries to split key values by separator, but some EvenType
has not a separator.
For Instance , "0,0,0,4,-1,RESERVED,V,"null""
StackTrace:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at selogger.weaver.DataInfo.parse(DataInfo.java:196)
at selogger.reader.DataIdMap.loadDataIdEntryFile(DataIdMap.java:111)
at selogger.reader.DataIdMap.<init>(DataIdMap.java:35)
at selogger.reader.LogPrinter.main(LogPrinter.java:54)
selogger version: 62063ed
Java env
openjdk version "1.8.0_342"
OpenJDK Runtime Environment (build 1.8.0_342-b07)
OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode)
To reproduce this issue, I created the branch on my repository.
https://github.com/yebityon/POLDAM/tree/feat/selogger/omni_binary
The sample Programs are located on Data/java8/src/omni_binary, I used selogger with Main.java and customClass/myMath.java.
selogger output directory is Data/java8/src/omni_binary/origin/selogger_out
.
The procedure to get selogger output directory is recorded in build.sh
Note: you need to git clone selogger on Data/java8/selogger
to run build.sh
.
latest-simple option (LatestEventLogger
) and latest (nearomni) option (LatestTimeEventLogger
) have almost the same feature, while the latter one records thread id and timestamps.
I believe that the latest-simple option is no longer necessary, LatestEventLogger
does not have a performance advantage.
When maven-surefire-plugin uses parallel testing (e.g. <forkCount>2</forkCount>
), Maven may report the following error message.
Corrupted channel by directly writing to native stream in forked JVM
This is because SELogger used STDOUT to report some error messages at runtime.
The problem is reproducible with Maven 3.6.
Building apache/commons-Lang repo with the following configuration for maven-surefire-plugin.
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${commons.surefire.version}</version>
<configuration>
<argLine>-javaagent:/home/ishio/selogger/target/selogger-0.4.0.jar=output=/</argLine>
<forkCount>2</forkCount>
</configuration>
In this example, /
is selected as an output directory.
SELogger writes an error message / is not writable directory
to STDOUT; it resulted in the warning.
SELogger tries to open a log file before creating an output directory.
Hence, a new output directory cannot have the weavelog.txt
file.
The Logger Agent works fine on a simple Hello World Programm.
When I run it against another more complex Jar this Error Happens
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.launcher.LauncherHelper$FXHelper.main(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: selogger/logging/Logging
at javafx.beans.property.SimpleBooleanProperty.<clinit>(SimpleBooleanProperty.java:38)
at com.sun.javafx.application.PlatformImpl.<clinit>(PlatformImpl.java:89)
at com.sun.javafx.application.LauncherImpl.startToolkit(LauncherImpl.java:675)
at com.sun.javafx.application.LauncherImpl.launchApplicationWithArgs(LauncherImpl.java:337)
at com.sun.javafx.application.LauncherImpl.launchApplication(LauncherImpl.java:328)
... 5 more
There is a risk of infinite loop if logging code is injected into checkPermission
method for SecurityManager classes as described in README.
Hi,
I am running JUnit test cases in Maven with SELogger on open-source projects. I can successfully run the test cases in most cases. But sometimes I get NullPointerException when using SELogger and the test case passes when I am not using SELogger.
I am using selogger-0.5.0.jar
, Maven 3.8.6
, and OpenJDK 1.8.0_302
.
Here is an example in the apache/shardingsphere project when checking out commit 43254af:
The Maven command:
mvn test -pl shardingsphere-features/shardingsphere-sharding/shardingsphere-sharding-route/pom.xml --also-make -Dsurefire.failIfNoSpecifiedTests=false -Dtest="ShardingUpdateStatementValidatorTest#assertValidateUpdateWithoutShardingKey" -Dcheckstyle.skip=true -Dspotless.apply.skip=true --batch-mode -DargLine="-javaagent:/path/to/selogger.jar=format=nearomni,exlocation=.m2,output=/path/to/trace"
The Maven log:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.862 s <<< FAILURE! - in org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest
[ERROR] assertValidateUpdateWithoutShardingKey(org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest) Time elapsed: 0.005 s <<< ERROR!
java.lang.NullPointerException
at org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest.assertValidateUpdateWithoutShardingKey(ShardingUpdateStatementValidatorTest.java:71)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ShardingUpdateStatementValidatorTest.assertValidateUpdateWithoutShardingKey:71 » NullPointer
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
But when I run this command:
mvn test -pl shardingsphere-features/shardingsphere-sharding/shardingsphere-sharding-route/pom.xml --also-make -Dsurefire.failIfNoSpecifiedTests=false -Dtest="ShardingUpdateStatementValidatorTest#assertValidateUpdateWithoutShardingKey" -Dcheckstyle.skip=true -Dspotless.apply.skip=true --batch-mode
The test case passes:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.076 s - in org.apache.shardingsphere.sharding.route.engine.validator.impl.ShardingUpdateStatementValidatorTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
JUnit test generates empty log files log-00001.slg
and log-00001.txt
in selogger-output
directory.
selogger version : 53ecf87
Following How to Build for JDK7
in README.md
instruction, build is a failure.
Note : In JDK8, the build is successful.
root@03972ec91840:~/usr/selogger# mvn package -Dmaven.test.skip=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building selogger 0.2.2
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ selogger ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /root/usr/selogger/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ selogger ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 46 source files to /root/usr/selogger/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /root/usr/selogger/src/selogger/weaver/method/MethodTransformer.java:[141,45] cannot find symbol
symbol: method toUnsignedInt(byte)
location: class java.lang.Byte
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.687s
[INFO] Finished at: Thu Jan 28 03:08:31 UTC 2021
[INFO] Final Memory: 12M/188M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project selogger: Compilation failure
[ERROR] /root/usr/selogger/src/selogger/weaver/method/MethodTransformer.java:[141,45] cannot find symbol
[ERROR] symbol: method toUnsignedInt(byte)
[ERROR] location: class java.lang.Byte
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Any invalid option like weave=INVALID
causes a NPE at the program termination.
Exception in thread "Thread-0" java.lang.NullPointerException: Cannot invoke "selogger.logging.IEventLogger.close()" because "this.logger" is null
at selogger.weaver.RuntimeWeaver.close(RuntimeWeaver.java:183)
at selogger.weaver.RuntimeWeaver$1.run(RuntimeWeaver.java:44)
at java.base/java.lang.Thread.run(Thread.java:832)
As 0.3.2 removed an internal buffer, 0.3.2 is much slower than 0.3.1.
A new buffering mechanism might be needed.
A user told me that a test execution of Apache Commons-Lang with format=nearomni,size=60000,weave=LINE
does not complete.
I could reproduce the problem. The problem does not occur when the buffer is the default size.
The problem is likely caused by the current implementation allocating a fixed-size buffer for each program location.
Each buffer size requires 20 bytes x the value of size=
option. When size=60000
, 1.2MB per location is required.
In case of Commons-Lang, more than 50000 locations exist. I estimated that the required memory is around 60GB.
The memory consumption may cause the following behavior.
SELogger needs a special procedure (at least terminating the execution) when reached OutOfMemory.
While mode=omni records object IDs, mode=nearomni does not.
Hi @takashi-ishio
I am trying to other feature on the project and run a project with this agent!
Do you gave a setup for debugging the project by running it and attaching a debugger ?
Is there a faster SDLC for this which you use on the project ?
Thanks
LogPrinter does not work in JDK 8 because it uses a method ByteBuffer.clear() available since Java 9.
Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at selogger.reader.EventReader.load(EventReader.java:76)
at selogger.reader.EventReader.<init>(EventReader.java:49)
at selogger.reader.LogPrinter.main(LogPrinter.java:55)
Log files are overwritten by another execution.
This feature is important to avoid an accidental overwrite.
An execution of selogger.testdata.SimpleTarget with -javaagent:${project_loc:selogger}\target\selogger-0.6.0.jar=output=selogger-output
resulted in the following error.
This is thrown when a weavelog file is closed. The elapsed time is not recorded in the log file.
Exception in thread "Thread-0" java.lang.ClassCircularityError: java/io/InterruptedIOException
at java.base/java.io.PrintStream.writeln(PrintStream.java:726)
at java.base/java.io.PrintStream.println(PrintStream.java:1028)
at selogger.weaver.LogMessageFile.log(LogMessageFile.java:28)
at selogger.weaver.RuntimeWeaver.close(RuntimeWeaver.java:191)
at selogger.weaver.RuntimeWeaver$1.run(RuntimeWeaver.java:44)
at java.base/java.lang.Thread.run(Thread.java:833)
k-shimari reported that a test org/apache/commons/math3/optimization/direct/BOBYQAOptimizerTest
in Defects4J 2.0 Math8b might execute some instructions more than 2G times.
The recendtata.txt file is like this:
20612,-821594356,-821594356,
defects4j test
does not finish for Mockito-2b in a practical time because test/org/mockitousage/performance/StubOnlyAvoidMemoryConsumptionTest.java
takes too much time.
This problem is observed only when format=omni
is specified.
The problem is not observed for format=nearomni
and format=freq
.
Executing defects4j test
for Jackson DataBind 92b resulted in an infinite loop.
When a checkPermission
method is called, logging code tries to record logging information. The step triggers an additional checkPermission
call.
Workaround: Excluding a security manager from bytecode weaving.
<jvmarg value="-javaagent:/data/selogger/target/selogger-0.2.3.jar=format=omni,e=com/fasterxml/jackson/databind/misc/AccessFixTest$CauseBlockingSecurityManager" />
In the current version, a weaving log file includes only woven classes.
A full list of loaded classes would be useful to check the correctness of -e
and -exlocation
options.
Update is needed to support newer JDK, while it is not urgent.
Weaving failed for a very simple class:
public class Main {
public static void main(String[] args) {
System.out.println("Hello");
}
}
Stack Trace:
Weave: file:/C:/Users/ishio/eclipse-workspace/HelloWorld/bin//Main
Failed to weave Main in file:/C:/Users/ishio/eclipse-workspace/HelloWorld/bin/
java.lang.IllegalArgumentException
at selogger.org.objectweb.asm.ClassReader.<init>(ClassReader.java:160)
at selogger.org.objectweb.asm.ClassReader.<init>(ClassReader.java:143)
at selogger.weaver.ClassTransformer.<init>(ClassTransformer.java:30)
at selogger.weaver.Weaver.weave(Weaver.java:155)
at selogger.weaver.RuntimeWeaver.transform(RuntimeWeaver.java:201)
at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:825)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:723)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:646)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:468)
at java.base/sun.launcher.LauncherHelper.loadMainClass(LauncherHelper.java:780)
at java.base/sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:675)
latest version(e81fe80) can not obtain objectType.
I confirmed that selogger which has the commit ( 27005c6 ) can obtain the objectType correctly and has no failed test.
However, seloggers who have after that commit are failed as NPE or can not obtain object type.
the description is available below.
$ git log
commit bb9d8ad (HEAD -> develop)
$ java selogger.reader.LogPrinter > ../../../../experiment/diff_script/binary_search_log.txt
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "selogger.reader.TypeList.getType(int)" because "this.typeList" is null
at selogger.reader.ObjectTypeMap.getObjectTypeName(ObjectTypeMap.java:89)
at selogger.reader.DataIdMap.getObjectType(DataIdMap.java:66)
at selogger.reader.Event.toString(Event.java:253)
at selogger.reader.LogPrinter.main(LogPrinter.java:76)
My environment is
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
Is there a way to get objectType with the latest version?
I will be grateful for any help you can provide.
selogger version: 14 May 2015 [9bf6fb1 @ master brunch]
target Java Application: jEdit (http://jedit.org/) version:5.3.0
I run the selogger on the following two OSs.
macOS:
$ uname -a
Darwin tklab143.sa.cs-private 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64
$ java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
Linux:
$uname -a
Linux rigatoni 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
On the macOS, I attemped to read execution traces with EventReader, but an assertion (objId == count) failed in ObjectTypeMap#register (line 52).
I found some lines in LOG$ObjectTypes01.txt were corrupted.
19499,24
19500,19501,10
19502,22
19503,22
19504,10
19505,10
19506,10
19507,10
356
19508,10
19509,24
19510,10
19511,172
19512,10
19513,10
19514,10
19515,351
19516,353
19517,354
19518,10
19519,24
19520,10
19521,24
19522,19523,10
19524,351
19525,22
19526,10
19527,22
19528,10
19529,10
19530,76
357
19531,10
On Ubuntu, I succeeded in reading the execution traces without the above error, and I confirmed LOG$ObjectTypes01.txt was not broken.
The types of the corrupted IDs (356 and 357) are as follows (extracted from LOG$Types.txt):
356,macosx.MacOSXPlugin,jeditresource:/MacOSX.jar!/macosx/MacOSXPlugin.class,355,-1
357,macosx.MacOSXPlugin$1,jeditresource:/MacOSX.jar!/macosx/MacOSXPlugin$1.class,9,-1
Here, I provide some debug information I found.
selogger/trunk/src/selogger/logging/TypeIdMap.java
Lines 45 to 63 in 0df8626
I commented out the line 56, and ran the woven application again. Then, the assertion error did not arise and I confirmed the LOG$ObjectTypes01.txt were not corrupted.
TypeIdMap#getClassLoacation(), which is invoked at line 56, calls some ClassLoader methods.
So I think there might be a problem with logging a class loader.
Actually, macosx.MacOSXPlugin was loaded by a custom class loader (org.gjt.sp.jedit.JARClassLoader) and the custom class loader was instrumented for logging.
# place the source codes of jEdit in the ${PRJ_DIR} directory
SELOGGER_DIR="/Users/shimastripe/selogger"
PRJ_DIR="/Users/shimastripe/jEdit"
cd ${PRJ_DIR}
ant retrieve
ant build
ant docs-html
java -cp\
${SELOGGER_DIR}/selogger-may-14-2015-9bf6fb1.jar:${SELOGGER_DIR}/asm-5.0.3.jar:${SELOGGER_DIR}/asm-analysis-5.0.3.jar:${SELOGGER_DIR}/asm-commons-5.0.3.jar:${SELOGGER_DIR}/asm-tree-5.0.3.jar:${SELOGGER_DIR}/asm-util-5.0.3.jar:${SELOGGER_DIR}/commons-lang3-3.1.jar:${SELOGGER_DIR}/trove-3.0.2.jar:${PRJ_DIR}/build/classes/core:${PRJ_DIR}/lib/compile/jsr305.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/lib/tools.jar\
selogger.weaver.TraceWeaver\
-jdk17\
-output=output/tracer/weaver_output\
-ignoreError\
-jardir ${PRJ_DIR}/build/classes/core
cd ${PRJ_DIR}/build
java -Xms2g -Xmx7g\
-cp ${SELOGGER_DIR}/selogger-may-14-2015-9bf6fb1.jar:${PRJ_DIR}/output/tracer/weaver_output:.:./jedit.jar \
-Dselogger.dir=${PRJ_DIR}/output/tracer/execution_trace \
-Dselogger.threads=4 \
-Dselogger.output=fixed-compress \
org.gjt.sp.jedit.jEdit \
-settings=settings
# After the jEdit application is launched, just quit the application by closing the application window.
The current version of SELogger uses ASM 9.4 that does not support the latest JDK.
Hence, an execution results in an empty trace file like this.
{ "format":"nearomni", "events": [
]}
A log file option (weaverlog=log.txt) results in ClassCircularityError
.
The log file records java.lang.IllegalArgumentException: Unsupported class file major version 66
.
int answer;
public void divided(int x) {
if (x == 0) {
return;
}
answer /= x;
}
weave=LABEL mode records a LABEL event inside the IF-THEN block even if the IF-THEN block is skipped by x!=0
.
SELogger creates selogger-output
directory by default if format=omni
option is specified.
However, the default output directory does not include data files such as methods.txt and dataids.txt to interpret the execution trace.
They are created if an output directory is explicitly specified by output=<dir>
.
eg.
Actual
: double a = 4614455732198220759L
Expected
: a = 3.23d
using Double.longBitsToDouble() while writing into serialised value can fix this;
The current trace file uses a type descriptor to indicate a variable type. For example, "int" is represented by "I".
I is not so understandable for users who are unfamiliar with JVM details.
DataFormat.md is inconsistent with actual behavior.
Weaving bytecode with weave=LINE option (that records only line number events) resulted in a bytecode verification error.
The current CSV format is difficult to manipulate in a text editor because each dataid (line) may have a different number of data values (columns).
The current version does not support static weaving, although the previous version (icpc2014 branch) has the feature.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.