stagemonitor / stagemonitor

an open source solution to application performance monitoring for java server applications

---

⚠️ This project is not maintained anymore

A good alternative to stagemonitor that offers a similar feature set, except for the in-browser-widget, is the Elastic APM Java Agent.

---

Stagemonitor is a Java monitoring agent that tightly integrates with time series databases like Elasticsearch, Graphite and InfluxDB to analyze graphed metrics and Kibana to analyze requests and call stacks. It includes preconfigured Grafana and Kibana dashboards that can be customized.

More Information

For more information about the project, please see http://www.stagemonitor.org

Live Demo

http://stagemonitor-demo.isys-software.de

Getting Started

Check the Installation site of the wiki

Issues

If you encounter any issues or if you have a question, don't hesitate to create an issue.

Mailing List

The mailing list can be found at https://github.com/stagemonitor/stagemonitor-mailinglist

Snapshots

Use JitPack to get the latest snapshots from master. When using gradle, add the --refresh-dependencies flag to ensure you always check for an updated snapshot version.

Build Locally

To build this project locally, clone the repo and execute ./gradlew install (Linux) gradlew.bat install (Windows) to install stagemonitor to your local maven repo (~/.m2/repository/org/stagemonitor/).

Contributing

We love contributions from the community! Please read CONTRIBUTING.md before creating a pull request.

stagemonitor's Issues

Display SQL in Call Stacks

Display SQL queries incl. parameters in call stacks like Spring Insight does.

Metrics tab for stagemonitor widget

Render the metrics exposed through metrics-servlets with some charting library.
Preferably group metrics by stagemonitor plugin.

Implement aspects for metrics-annotation

https://github.com/dropwizard/metrics/tree/master/metrics-annotation
Examples:

@Counted void timed() {}

Switch from caliper to jmh for benchmarks

JMH seems to be more reliable and mature. See also github.com/ReactiveX/RxJava/issues/776, http://stackoverflow.com/questions/20443997/how-to-use-jmh-with-gradle, https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ

Measure CPU time of requests and methods

Display in request table and call stacks

Support for request tables in grafana

relates to #2, github.com/torkelo/grafana/issues/78

Verbose option is causing AspectJ to log too much information

The -verbose weaver option generates unnecessary log messages for every request. See the file stagemonitor-core/src/main/resources/META-INF/aop.xml

Provide content and documentation for http://stagemonitor.github.io/

Monitor and display performance of SQL queries

Include SQL in stack traces

Uplift Metrics version to 3.1.1 for improved Graphite connectivity

Related to dropwizard/metrics#694
3.1.1 of Metrics corrects a problem with losing connectivity to Graphite and being unable to reconnect until the application is restarted.

Publish to maven central

Completely Disable Stage Monitor

When doing local development I'm unable to completely disable stagemonitor. Is there some way to do this? My current development configuration in eclipse sets the following command line argument:

-Dstagemonitor.property.overrides=stagemonitor-local.properties

And within that property file I have the following settings:

stagemonitor.instanceName=local
stagemonitor.applicationName=My Server
stagemonitor.active=false
stagemonitor.requestmonitor.collectRequestStats=false
stagemonitor.profiler.logCallStacks=false
stagemonitor.requestmonitor.http.collectHeaders=false
stagemonitor.requestmonitor.http.parseUserAgent=false
stagemonitor.web.rum.enabled=false
stagemonitor.web.widget.enabled=false
stagemonitor.profiler.logCallStacks=false
stagemonitor.reporting.interval.console=0

I see the following in the log file:

2015-Feb-13 20:15:26  INFO [localhost-startStop-1] StagemonitorCoreConfigurationSourceInitializer:28 - try loading of default property overrides: 'stagemonitor-local.properties'
2015-Feb-13 20:15:26  INFO [localhost-startStop-1] Stagemonitor:36 - stagemonitor is deactivated

Upon hitting a method that is marked to with @Timer or the like I get the following error:

[application=null] [instance=null] [host=null]
Feb 13, 2015 8:22:23 PM org.apache.catalina.core.StandardContext fireRequestInitEvent
SEVERE: Exception sending request initialized lifecycle event to listener instance of class org.stagemonitor.web.logging.MDCListener
java.lang.NullPointerException
    at java.util.Hashtable.put(Hashtable.java:514)
    at org.apache.log4j.MDC.put0(MDC.java:150)
    at org.apache.log4j.MDC.put(MDC.java:85)
    at org.slf4j.impl.Log4jMDCAdapter.put(Log4jMDCAdapter.java:59)
    at org.slf4j.MDC.put(MDC.java:113)
    at org.stagemonitor.web.logging.MDCListener.requestInitialized(MDCListener.java:27)
    at org.apache.catalina.core.StandardContext.fireRequestInitEvent(StandardContext.java:6129)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:166)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

Am I going about disabling this wrong? This isn't a server I want to track (my local instance running in tomcat / eclipse).

Evaluate Apdex, maybe use as source of alerting

https://docs.newrelic.com/docs/site/apdex-measuring-user-satisfaction

Improved call stacks view

Render call stacks with javascript to support expanding/collapsing parts of the tree. Integrate as grafana panel.
Only expand subtrees with >5% execution time.
mandatory for #2

Small typo

Hi,
I am not sure, what I saw is that the Boomerang Injector potentially needs to be renamed:

https://github.com/stagemonitor/stagemonitor/blob/584f8fe0b99ec09e3a5f988d01c1f0680b52c3b1/stagemonitor-web/src/main/java/org/stagemonitor/web/monitor/rum/BommerangJsHtmlInjector.java

to BoomerangJsHtmlInjector.java?

Support for "unsafe" HTML in grafana

To display arbitrary HTML and Javascript in grafana's HTML panel, the ng-bind-html-unsafe angularjs directive has to be used.
mandatory for #5

Measure execution time of the different application layers

Wish stagemonitor.property.overrides allows external filesystem property file

Hello,

Currently the PropertyFileConfigurationSource file is restrict to callpath location for stagemonitor.properties file.

Could you support as failover of getResourceAsStream() request an alternative file open on filesystem.

Regards

zucher

Collect and display host metrics

Options

• http://collectd.org/
• Own implementation based on http://www.hyperic.com/products/sigar (like elasticsearch did)

End user monitoring

Via javascript, that sends browser rendering metrics back.

Monitor and display Ehcache metrics

see also http://metrics.codahale.com/manual/ehcache/

Make configuration parameters dynamic

Add the possibility to change parameters at runtime by providing query parameters.

• Call Stacks on demand
• password protection (additional query param)

Logging and displaying exceptions

Datastore for metrics

Hi,
me again, having questions. The background of my questions is, that I don't want to use Grafana, but instead a combination of AngularJS and Highcharts with InfluxDB in the backend.

The question are:
Which time series database are you using in the backend in your Docker image?
How do you fill the database?

(Could not find both, perhaps I did not look properly). :-(

What I am currently building is something similar to the query and charting definition in grafana, but with different technologies and a pure Java/Angular/Bootstrap stack.

Fill wiki with documentation

Check licenses of dependencies

Check licenses of dependencies for incompatibilities with license for this project.

Measure number and size of created objects during a request

Only apply metrics-annotations to public methods

if applied to private methods, it can lead to strange npes when the method is also proxied e.g. because the method is annotated with Spring's @Scheduled.

Use grafana as dashboard system

Grafana https://github.com/torkelo/grafana/ has many more features compared to graphitus like creating and editing dashboards within the dashboard.

StringIndexOutOfBoundsException: String index out of range: -1 if only root CallStackElement exists

Hallo Everyone

Maybe I found a Bug?

It seems if given:

• StageMonitor v0.6 is active with default configuration from wiki
• Profiling the call-tree is active
• ConnectionMonitorAspect is active
• SpringHttpRequestMonitorFilter is active
• No matching project pointcut between RequestMonitorFilter and Log of DB-Query exists

Then:

• RequestMonitor#beforeExecution(..) creates the CallStackElement root with signature "total" => Profiler.activateProfiling("total");
• Later StagemonitorP6Logger.trackDbMetrics(..) calls CallStackElement.getShortSignature() that results in a StringIndexOutOfBoundsException because no opening round bracket exits

Part of a much longer stacktrace:
SCHWERWIEGEND: Servlet.service() for servlet [dwr-invoker] in context with path [/be_de_purchase] threw exception
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1954)
at org.stagemonitor.requestmonitor.profiler.CallStackElement.getShortSignature(CallStackElement.java:90)
at org.stagemonitor.jdbc.p6spy.StagemonitorP6Logger.trackDbMetrics(StagemonitorP6Logger.java:42)
at org.stagemonitor.jdbc.p6spy.StagemonitorP6Logger.logSQL(StagemonitorP6Logger.java:32)
at org.stagemonitor.jdbc.p6spy.P6SpyMultiLogger.logSQL(P6SpyMultiLogger.java:23)
at com.p6spy.engine.common.P6LogQuery.doLog(P6LogQuery.java:122)
at com.p6spy.engine.common.P6LogQuery.doLogElapsed(P6LogQuery.java:93)
at com.p6spy.engine.common.P6LogQuery.logElapsed(P6LogQuery.java:210)
at com.p6spy.engine.common.P6LogQuery.logElapsed(P6LogQuery.java:203)
at com.p6spy.engine.logging.P6LogPreparedStatementExecuteDelegate.invoke(P6LogPreparedStatementExecuteDelegate.java:51)
at com.p6spy.engine.proxy.GenericInvocationHandler.invoke(GenericInvocationHandler.java:103)
at org.p6spy.$java.sql.PreparedStatement$$EnhancerByCGLIB$$39f30543.executeQuery()
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1849)
at org.hibernate.loader.Loader.doQuery(Loader.java:718)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1933)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3270)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:269)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:152)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1080)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:997)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:990)
at org.springframework.orm.hibernate3.HibernateTemplate$1.doInHibernate(HibernateTemplate.java:520)
...

Improve test coverage

Catch Exceptions when initializing plugins

Add the possibility not monitor requests that have no Spring MVC mapping

Integrate in Spring Application

Hi,
I did take look at the [0] and tried to use it a Spring application. What I am using is Spring MVC and Spring Data Rest. Currently I just want to monitor the request execution time as defined in [1]. I defined the aop.xml as following:

<!DOCTYPE aspectj PUBLIC
        "-//AspectJ//DTD//EN" "http://www.eclipse.org/aspectj/dtd/aspectj.dtd">
<aspectj>
    <weaver options="-Xlint:ignore -Xset:weaveJavaxPackages=true">
        <!-- exclude generated proxies -->
        <exclude within="*..*CGLIB*" />
        <exclude within="*..*$$*" />
        <exclude within="*..access$*" />
        <exclude within="com.sun.proxy..*" />
    </weaver>

    <aspects>
        <concrete-aspect name="profiler"
                         extends="org.stagemonitor.requestmonitor.profiler.ProfilingAspect">
            <pointcut name="methodsToProfile" expression="execution(* de.project.web..*(..)) 
            AND !within(de.project.model..*)"/>
        </concrete-aspect>
    </aspects>
</aspectj>

In web there is a package called Controller with a home Controller. I set breakpoints in RequestMonitor, but could not track the execution. In my Spring Project (Java configured) I have enabled: @EnableLoadTimeWeaving and @EnableAspectJAutoProxy. What I am missing?

[0]https://github.com/stagemonitor/stagemonitor/tree/master/stagemonitor-requestmonitor
[1] https://github.com/stagemonitor/stagemonitor/blob/master/stagemonitor-requestmonitor/src/main/java/org/stagemonitor/requestmonitor/RequestMonitor.java

Where are stagemonitor internal metrics stored?

As described in docs, https://github.com/stagemonitor/stagemonitor/wiki/Configuration-Options#internal-monitoring , we have configured stagemonitor to collect internal performance data.
But where I can see this data? Neither in grafana nor in kibana I was able to see anything

Thanks

Fix travis build

Problem: dependency to snapshot caliper is needed.

HttpRequestTrace contains unsafe use of java.util.LinkedHashMap.

org.stagemonitor.web.monitor. HttpRequestTrace includes a following data structure, which is shared among all threads processing HTTP requests:

private static final Map<String, ReadableUserAgent> userAgentCache =
    new LinkedHashMap<String, ReadableUserAgent>(MAX_ELEMENTS + 1, 0.75f, true) {
        @Override
        protected boolean removeEldestEntry(Map.Entry eldest) {
            return size() > MAX_ELEMENTS;
        }
};

The javadoc of java.util.LinkedHashMap mentions "Note that this implementation is not synchronized. If multiple threads access a linked hash map concurrently, and at least one of the threads modifies the map structurally, it must be synchronized externally.".

If not, you might end up with several threads in infinite loop. Here's a stacktrace of one of the threads, but all of them were stuck in the same method.

java.lang.Thread.State: RUNNABLE
    at java.util.LinkedHashMap.transfer(LinkedHashMap.java:253)
    at java.util.HashMap.resize(HashMap.java:581)
    at java.util.HashMap.addEntry(HashMap.java:879)
    at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:427)
    at java.util.HashMap.put(HashMap.java:505)
    at org.stagemonitor.web.monitor.HttpRequestTrace.getUserAgentInformation(HttpRequestTrace.java:118)
    at org.stagemonitor.web.monitor.HttpRequestTrace.<init>(HttpRequestTrace.java:43)

To fix this the userAgentCache needs to be at least wrapped by Collections.synchronizedMap() :

private static final Map<String, ReadableUserAgent> userAgentCache =
    Collections.synchronizedMap(new LinkedHashMap<String, ReadableUserAgent>(MAX_ELEMENTS + 1, 0.75f, true) {
        @Override
        protected boolean removeEldestEntry(Map.Entry eldest) {
            return size() > MAX_ELEMENTS;
        }
});

info define aspect message upon startup

While attempting to use the ProfilerAspect, the message "info define aspect" is being written to log. I based my aop.xml configuration at the PetClinic demo aop.xml. The PetClinic demo exhibited the same message.

[AppClassLoader@126f304] info AspectJ Weaver Version 1.8.2 built on Thursday Aug 14, 2014 at 21:45:02 GMT
[AppClassLoader@126f304] info register classloader sun.misc.Launcher$AppClassLoader@126f304
[AppClassLoader@126f304] info using configuration /C:/Development/Tools/jersey-archive-1.5/META-INF%5cmyaop.xml
[AppClassLoader@126f304] info register aspect org.stagemonitor.core.metrics.aspects.TimedAspect
[AppClassLoader@126f304] info register aspect org.stagemonitor.core.metrics.aspects.MeteredAspect
[AppClassLoader@126f304] info register aspect org.stagemonitor.jdbc.ConnectionMonitorAspect
[AppClassLoader@126f304] info register aspect org.stagemonitor.logging.InformationAppendingLoggingAspect
[AppClassLoader@126f304] info register aspect org.stagemonitor.logging.MeterLoggingAspect
[AppClassLoader@126f304] info define aspect profiler

<aspects>
    <concrete-aspect name="profiler" extends="org.stagemonitor.requestmonitor.profiler.ProfilingAspect">
        <pointcut name="methodsToProfile" expression="execution(* ses..*(..))"/>
    </concrete-aspect>

    <aspect name="org.stagemonitor.core.metrics.aspects.TimedAspect"/>
    <aspect name="org.stagemonitor.core.metrics.aspects.MeteredAspect"/>
    <aspect name="org.stagemonitor.jdbc.ConnectionMonitorAspect"/>
    <aspect name="org.stagemonitor.logging.InformationAppendingLoggingAspect"/>
    <aspect name="org.stagemonitor.logging.MeterLoggingAspect"/>
</aspects>

Any assistance in resolving the problem or misunderstanding would be appreciated.

Create grafana dashboards

Dynamically load javaagent at runtime

Dynamically load javaagent at runtime, so that it's not mandatory to provide the -javaagent flag.

Resources:
http://dhruba.name/2010/02/07/creation-dynamic-loading-and-instrumentation-with-javaagents/
http://grepcode.com/file/repo1.maven.org/maven2/net.sf.ehcache/ehcache-core/2.6.6/net/sf/ehcache/pool/sizeof/AgentLoader.java/
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.openjpa/openjpa-kernel/2.3.0/org/apache/openjpa/enhance/InstrumentationFactory.java?av=f

Alerting based on thresholds

• which alerting backend?
  • https://github.com/scobal/seyren
  • http://www.shinken-monitoring.org/index.html#
• how to integrate into grafana?

see also: github.com/torkelo/grafana/issues/102

Monitoring of RMI calls

Problem: no equivalent to servlet filters available. Maybe one can create an aspect that intercepts some method in the RMI API where the calls are dispatched to the actual method.

Use elasticsearch instead of SQL database

Use elasticsearch instead of SQL database to store call stacks. stagemonitor-server is then not needed anymore.
Elasticsearch is also used by grafana to store and save dashboards.
Relates to #2

Choose license for this project

http://choosealicense.com/ could help

Android app that can receive notifications

steps to configure stagemonitor in wildFly8 (jboss8)

We are using wildFly 8(jboss 8) for java web application and not using Maven yet. Our application is becoming slow too frequently then we need to restart our server. Then everything works fine. Now I wan to use stagemonitor to identify the cause. Could you please let me know stagemonitor configuration steps in wildFly8

Collect and display metrics from elasticsearch

Contributing additional RequestTraceReporter implementations

I have a need to have a RequestTraceReporter that will report traces to Splunk. Currently there is not a way to contribute additional implementations of the RequestTraceReporter without changing stagemonitor itself. One way to contribute additional RequestTraceReporter implementations would be to use the ServiceLoader to look up additional implementations [1]. What are your thoughts on this approach?

Thanks for the great project!

[1]
From org.stagemonitor.requestmonitor.RequestMonitor

private static final List<RequestTraceReporter> requestTraceReporters = new CopyOnWriteArrayList<RequestTraceReporter>(){{
    add(new LogRequestTraceReporter());
    add(new ElasticsearchRequestTraceReporter());
    for (RequestTraceReporter requestTraceReporter : ServiceLoader.load(RequestTraceReporter.class)) {
        add(requestTraceReporter);
    }
}};

Alerts based on deviations of predictions

http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.holtWintersConfidenceBands

stagemonitor.property.overrides allow external filesystem property file

Hello,

Currently the PropertyFileConfigurationSource file is restrict to callpath location for stagemonitor.properties file.

Could you support as failover of getResourceAsStream() request an alternative file open on filesystem searching.

Regards

zucher