high thread creation when executed by nagios nrpe. about jvmtop

[GoogleCodeExporter]

The first thing what makes me wonder is:
If you're providing the --once switch ( I can't see if this is the case when 
watching the video), jvmtop should exit after first iteration - the process 
should terminate after a few seconds at the latest.
However, in your case, the process remains alive.

Questions:
- Are you sure that jvmtop is called with the --once argument? (use jps -v to 
confirm)
- Can you provide a thread dump of the jvmtop process (jstack 24872)?
- How many jvms are running at your machine? (use jps | wc )
- Do you use an oracle jdk?

Original comment by [email protected] on 12 Aug 2013 at 9:10

[GoogleCodeExporter]

Thanks for your reply.
I forgot to mention previously that, i do see jvmtop.sh with --once switch 
invoking in htop and exiting after a couple of seconds later. when it exit, i 
think it adds a new thread process in queue with own pid. as u see in video.

Questions answers:-
jps -l

30943 org.jboss.Main
3680 /var/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/home/oc4j.jar
24872 com.jvmtop.JvmTop
7488 sun.tools.jps.Jps

Thread dump for pid 24872 is attached.

jps | wc  output:
4    8     47

Yes i use oracle JDK.

Original comment by [email protected] on 12 Aug 2013 at 10:21

Attachments:

• threaddump.log

[GoogleCodeExporter]

Can you please execute: jps -v    
(not: jps -l)

Original comment by [email protected] on 12 Aug 2013 at 11:01

[GoogleCodeExporter]

jps -v

32740 Jps -Dapplication.home=/usr/java/jdk1.6.0_25 -Xms8m
30943 Main -Dprogram.name=run.sh -Xms2G -Xmx4G -XX:MaxPermSize=256m 
-XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap2013-07-30_10:52.dump 
-verbose:gc -DCOMPIERE_HOME=/var/app/compiere/Compiere2 
-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true 
-Djava.endorsed.dirs=/var/app/compiere/Compiere2/jboss/lib/endorsed
3680 oc4j.jar -Xmx384M -XX:MaxPermSize=400M -XX:MinHeapFreeRatio=20 
-XX:MaxHeapFreeRatio=40 -DORACLE_HOME=/var/app/oracle/product/11.2.0/dbhome_1 
-Doracle.home=/var/app/oracle/product/11.2.0/dbhome_1/oc4j 
-Doracle.oc4j.localhome=/var/app/oracle/product/11.2.0/dbhome_1/xxxx/sysman 
-DEMSTATE=/var/app/oracle/product/11.2.0/dbhome_1/xxxx 
-Doracle.j2ee.dont.use.memory.archive=true 
-Djava.protocol.handler.pkgs=HTTPClient 
-Doracle.security.jazn.config=/var/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/
OC4J_DBConsole_xxxx/config/jazn.xml 
-Djava.security.policy=/var/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DB
Console_xxxx/config/java2.policy 
-Djavax.net.ssl.KeyStore=/var/app/oracle/product/11.2.0/dbhome_1/sysman/config/O
CMTrustedCerts.txt-Djava.security.properties=/var/app/oracle/product/11.2.0/dbho
me_1/oc4j/j2ee/home/config/jazn.security.props 
-DEMDROOT=/var/app/oracle/product/11.2.0/db
24872 JvmTop

Original comment by [email protected] on 12 Aug 2013 at 12:10

[GoogleCodeExporter]

Thanks.
As you can see, you invoked jvmtop (pid 24872) without --once
Please make sure that if you call jvmtop from nagios that this argument has 
been set.


Beside that, the massive amount of RMI threads which can be seen is a separate 
issue.
It seems that if a monitored jvm dies the associated rmi renew thread is kept 
alive for some reason.

This might be a jvm bug - but I have to investigate.
Comments or other help is welcome.

Maybe related:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5053529
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6199638

Original comment by [email protected] on 12 Aug 2013 at 12:21

[GoogleCodeExporter]

i guess i might have invoked jvmtop without --once argument during testing. i 
plan to reboot the server and see what happens when nagios nrpe daemon run my 
wrapper script, which invoke jvmtop.sh --once.

I would be greateful to your investigate. Without this work around, i could not 
post my nagios plugin on nagios website.  I will also need your permission to 
do so.

Thanks.

Original comment by [email protected] on 12 Aug 2013 at 12:37

[GoogleCodeExporter]

Ah, good to know.

Of course you're allowed to publish your nagios plugin for jvmtop.
It would be very cool if you share the URL to your nagios plugin, I'm sure 
other users are interested in this as well.

PS:
In order to avoid confusion, I'll create another issue for the rmi renew thread 
issue
- because it has nothing to do with the --once switch and/or nagios.

Original comment by [email protected] on 12 Aug 2013 at 12:49

[GoogleCodeExporter]

Ya, after reboot, things works as expected. No more massive threads in waiting 
state.

Just to clarrify, If the jvm found a deadlock, will it indicate with  !D  ? 

I have found one bug in my wrapper script and i will publish to nagios when it 
get resolve. jvmtop.sh gives "connection refused" error msg next to the process 
name in output.  I have to alter my script to go into CRICTICAL notify state 
rather than OK state if jvm process didn't returned values. possible there 
might be issue with jboss AS or backend database. I found connection refused 
when i ran an sql query to invoke deadlock from the DB.

I will come back to your when i ready to post my nagios plugin to community.
If you interested you can send me your email ID to communicate other than 
issues with jvmtop.sh

I really appreciate you and thank you. :-)

Original comment by [email protected] on 12 Aug 2013 at 1:51

[GoogleCodeExporter]

hi, One small request, is it possible to make jvmtop.sh executes like
./jvmtop.sh --once <jvmprocess name to monitor>

This way, user can define which jvm to monitor rather than whole full jvm.  can 
we have a argument to pass that will do this. It will be useful for nagios 
plugin which execute jvmtop.sh every 2 min just to find only 1 jvm process 
values.

Hope you get my point.

Original comment by [email protected] on 13 Aug 2013 at 8:20

[GoogleCodeExporter]

Hi,

my answers to your questions:
- Yes, if the jvm found a (java threading) deadlock it's indicated with !D.
- I'll send you an email (alternatively, you can just click on my name here to 
see my full email address)
- regarding your comment #9:
You need a filter for the overview mode (which is displayed when providing no 
PID) to only display a specific jvm?

Original comment by [email protected] on 13 Aug 2013 at 9:51

[GoogleCodeExporter]

yes, i need filter for the overview mode.. but if possible filter as  JVM 
process exact name..

eg:  ./jvmtop.sh --once  org.jboss.Main

OUTPUT:-
JvmTop 0.7.1 alpha - 06:13:38,  amd64,  4 cpus, Linux 2.6.18-19, load avg 0.00
 http://code.google.com/p/jvmtop

  PID MAIN-CLASS      HPCUR HPMAX NHCUR NHMAX    CPU     GC    VM USERNAME   #T DL
11228 org.jboss.Main   549m 3993m   70m  304m  0.00%  0.00% S6U25 compiere   74 


Instead of :

JvmTop 0.7.1 alpha - 06:13:38,  amd64,  4 cpus, Linux 2.6.18-19, load avg 0.00
 http://code.google.com/p/jvmtop

  PID MAIN-CLASS      HPCUR HPMAX NHCUR NHMAX    CPU     GC    VM USERNAME   #T DL
21665 m.jvmtop.JvmTop   12m 1820m    9m  130m 19.53%  0.00% S6U25     root   14 

 3596 oc4j.jar        [ERROR: Connection refused/access denied] 
11228 org.jboss.Main   549m 3993m   70m  304m  0.00%  0.00% S6U25 compiere   74 


Thanks.

Original comment by [email protected] on 13 Aug 2013 at 10:15

[GoogleCodeExporter]

Thanks for the clarification.

Maybe you can archive this with a simple grep, e.g.
jvmtop.sh --once | grep ^\s*11228'

Original comment by [email protected] on 13 Aug 2013 at 10:42

[GoogleCodeExporter]

Hi,

I already using grep on my nagios plugin.
My intention was to limit what jvmtop search and query for. making it less CPU 
intensive if possible with an argument.  

Generally if a process query lot of jvm then the time to process and CPU usage 
will be high. my idea was to minimise this with an argument.

Never mind if that requires a lot of work for u. I am happy with what you 
provide in overview mode.

Original comment by [email protected] on 13 Aug 2013 at 10:54

[GoogleCodeExporter]

>My intention was to limit what jvmtop search and query for. making it less CPU 
>intensive if possible with an argument.  

That makes sense.
Would you create a separate enhancement ticket for this?

Original comment by [email protected] on 13 Aug 2013 at 11:09

[GoogleCodeExporter]

Done. please change it as enhancement request.

Original comment by [email protected] on 13 Aug 2013 at 11:19

[GoogleCodeExporter]

Original comment by [email protected] on 19 Aug 2014 at 8:34

• Changed state: Done