newrelic / nrjmx

Command line tool to connect to a JMX server and retrieve the MBeans it exposes.

New Relic JMX fetcher

A tool for extracting data out of any application exposing a JMX interface. It also provides support for golang.

Golang library

Starting from v2 you can call nrjmx from Go applications by using our gojmx module. If you want to read more about or to contribute you can check our develop docs.

Installation

Using a package manager

Common package managers can be used for this purpose: yum, apt, zypper.

For example: yum install nrjmx

Using the tarball

You can download and decompress the Java executable from the downloads page.

New Relic JMX integration (nri-jmx)

nrjmx is not bundled within the nri-jmx package. Still, it's declared as a dependency.

So if you have nrjmx already installed while installing nri-jmx, the installed version is kept. Otherwise, nri-jmx will try to get the latest nrjmx release.

Getting started

nrjmx expects the connection parameters to the JMX interface as command line arguments.

$ ./bin/nrjmx -hostname 127.0.0.1 -port 7199 -username user -password pwd

nrjmx reads lines from the standard input which should contain object name patterns for which we want to fetch their attributes. For each line, it gets the beans matching the pattern and outputs a JSON with all the attributes found.

For example, if you want to fetch some beans from Cassandra JMX metrics, you can execute:

$ echo
"org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=ReadLatency" | java -jar target/nrjmx-0.0.1-SNAPSHOT-jar-with-dependencies.jar -hostname 127.0.0.1 -port 7199 -username user -password pwd

or for Windows CMD:

$ echo | set /p="org.apache.cassandra.metrics:type=Table,keyspace=*,scope=*,name=ReadLatency" | java -jar target/nrjmx-0.0.1-SNAPSHOT-jar-with-dependencies.jar -hostname 127.0.0.1 -port 7199 -username user -password pwd

Usage

Additional options are listed below.

Custom protocols

JMX allows use of custom protocols to communicate with the application. To use a custom protocol you have to include the custom connectors in the nrjmx classpath.

By default, nrjmx will include the sub-folder connectors in its class path. If this folder does not exist create it under the folder where you have nrjmx installed.

For example, to add support for JBoss, create a folder connectors under the default (Linux) library path /usr/lib/nrjmx/ (/usr/lib/nrjmx/connectors) and copy the custom connector jar ($JBOSS_HOME/bin/client/jboss-cli-client.jar) into it. You can now execute JMX queries against JBoss.

Remote URL connection

If you want to use a remoting-jmx URL you can use the flag -remote. In this case it uses the remoting connection URL: service:jmx:remote://host:port instead of service:jmx:rmi:///jndi/rmi://host:port/jmxrmi.

This sets a URI ready for JBoss Domain mode.

You will need to add support for the custom JBoss protocol. See the previous section Custom protocols.

JBoss standalone mode

This is supported via -remoteJBossStandalone and sets a connection URL to service:jmx:remote+http://host:port.

Example of usage with remoting:

$ ./bin/nrjmx -hostname 127.0.0.1 -port 7199 -username user -password pwd -remote

You will need to add support for the custom JBoss protocol. See the previous section Custom protocols.

Non-Standard JMX Service URI

If your JMX provider uses a non-standard JMX service URI path (default path is jmxrmi), you can use the flag -uriPath to specify the path portion (without / prefix).

For example:

• A default URI path could be like: service:jmx:rmi:///jndi/rmi://localhost:1689/jmxrmi (path is last path of the URI without the prefix /)
• ForgeRock OpenDJ uses a JMX service URI like: service:jmx:rmi:///jndi/rmi://localhost:1689/org.opends.server.protocols.jmx.client-unknown

To extract data from this application:

$ ./bin/nrjmx -hostname localhost -port 1689 -uriPath "org.opends.server.protocols.jmx.client-unknown" -username user -password pwd

Troubleshooting

If you are having difficulties with nrjmx to get data out of your JMX service, You can check our troubleshoot guide.

Building

nrjmx uses Maven for generating the binaries:

$ mvn package

This creates the nrjmx.jar file under the ./bin/ directory. Copy the bin/nrjmx and bin/nrjmx.jar files to your preferred location. Both files must be located under the same folder.

It also creates DEB and RPM packages to automatically install nrjmx. If you want to skip the creation of DEB and RPM packages (for example, because your development machine does not provide the required tools), you can disable the deb and rpm Maven profiles from the command line:

mvn clean package -P \!deb,\!rpm,\!tarball,\!test

Support

New Relic hosts and moderates an online forum where customers can interact with New Relic employees as well as other customers to get help and share best practices. Like all official New Relic open source projects, there's a related Community topic in the New Relic Explorers Hub. You can find this project's topic/threads here:

https://discuss.newrelic.com/c/support-products-agents/new-relic-infrastructure

Contributing

We encourage your contributions to improve New Relic JMX fetcher! Keep in mind when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project. If you have any questions, or to execute our corporate CLA, required if your contribution is on behalf of a company, please drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today.

License

New Relic JMX fetcher is licensed under the Apache 2.0 License.

nrjmx's Issues

java.sql.Time dependency error in infra-bundle when running JMX queries

Latest infrastructure agent bundle image contains jlink version of nrjmx that does not seem to be working.

Steps to reproduce:

• Run any JMX query using the nri-jmx and nrjmx included in the latest infrastructure bundle.
• Error: Caused by: java.lang.NoClassDefFoundError: java/sql/Time

Context

[1/2d] Docker builder container for thrift

JMXFetcher does not support java.util.Date data type

Description

The nrjmx client does not support java.util.Date data type, which is causing issues when trying to retrieve mbean objects attribute information for diffusion license expiry
The error encountered is as follows

Feb 23, 2021 10:16:57 AM org.newrelic.nrjmx.JMXFetcher queryAttributes
FINE: Unsuported data type (class java.util.Date) for bean com.pushtechnology.diffusion:server="diffusion-0",type=Server,attr=LicenseExpiryDate
Feb 23, 2021 10:16:57 AM org.newrelic.nrjmx.JMXFetcher queryAttributes 

A link to the parsing method is below
https://github.com/newrelic/nrjmx/blob/master/src/main/java/org/newrelic/nrjmx/JMXFetcher.java#L353

Expected Behavior

Attribute values of type date to be returned during the jmx call

Steps to Reproduce

echo "com.pushtechnology.diffusion:type=Server,server=*" | /usr/bin/nrjmx -host <diffusion_host> -port <port_number>

Your Environment

EKS 1.19
nri-kubernetes 2.2.0
nri-jmx 2.4.5

gojmx incompatible with apache/thrift v0.15

👋 I was trying out gojmx for the first time, and per the README ran go get -u github.com/newrelic/nrjmx/gojmx. However, it seems that the latest version of apache/thrift now requires a context to be passed to iprot.ReadStructBegin and similar calls, and -u forces the upgrade. As a result, it's unable to compile this package:

ian@ian jmx % go get -u github.com/newrelic/nrjmx/gojmx
go: downloading github.com/newrelic/nrjmx v2.0.0+incompatible
go: downloading github.com/newrelic/nrjmx/gojmx v0.0.0-20211221135603-df3fa035adca
go: downloading github.com/apache/thrift v0.13.0
go: downloading github.com/apache/thrift v0.15.0
# github.com/newrelic/nrjmx/gojmx/internal/nrprotocol
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:182:37: not enough arguments in call to iprot.ReadStructBegin
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:188:57: not enough arguments in call to iprot.ReadFieldBegin
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:200:29: not enough arguments in call to iprot.Skip
	have (thrift.TType)
	want (context.Context, thrift.TType)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:339:33: not enough arguments in call to iprot.ReadFieldEnd
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:343:32: not enough arguments in call to iprot.ReadStructEnd
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:350:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:359:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:368:29: not enough arguments in call to iprot.ReadI32
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:377:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:386:32: not enough arguments in call to iprot.ReadString
	have ()
	want (context.Context)
../go/pkg/mod/github.com/newrelic/nrjmx/[email protected]/internal/nrprotocol/nrjmx.go:386:32: too many errors

Not updating the dependency (i.e. running only go get github.com/newrelic/nrjmx/gojmx) seems to temporarily fix it, but just wanted to flag it. Happy holidays!

MIgrating to Gradle

• Perform initial port via gradle init
• Auto-module legacy JARs
• Add JLink support
• Build testserver from within testcontainers rather than using separate task
• Check that generated POM is the same as what maven builds (Not needed as the old project did not actually publish a jar).
• Adapt functionality from maven-assembly-plugin if necessary
• Adapt functionality from copy-rename-maven-plugin if necessary
• Ensure generated tarball is close enough to what maven does
• Create deb from Gradle
• Create rpm from Gradle
• Delete Maven build files

Complete the nrjmx release pipeline

• sql.Time

• Check Pre-release pipeline (should be ready except for Windows)

• Add release pipeline for nrjmx (GHA)

• Integrate publishing action

• Validate we have the right package / structure published in linux & windows

• Validate the ARM64 release

Avoid nrjmx bash/bat and execute java from Go

We have problems with nrjmx because it's executed by a bash/bat script that cannot forward sigkill signal.

• We should try to make the already existing test for killing nrjmx process failing
• replace scripts with go code

nrjmx pipeline improvements

• Currently we don't perform the tests on Windows. I think at the moment we created the pipeline GHA runners was not supporting Linux Containers. Research if that can be fixed now. If not, at least add some sanity checks like building the code.
• the script on windows doesn't have feature parity with the linux one. e.g. env vars, connectors
• in all the integrations using gojmx on Windows we always bundle the latest version, we should make it consistent with the linux pipeline

Improve how shell wrapper executes the nr-jmx jar

The shell script nrjmx runs a naked exec of java without any error handling or proper discovery of the java binary. It would be helpful for an operator trying to use this script if it had some improved error handling

• if it attempted to use java binary in JAVA_HOME first since by convention that's where most applications e.g gradle springboot and tomcat look for java
• fail with a useful error message if it nots found in the path
• fail with a useful error message if the jar dis not found
• fail if there's an unitialized variable in the shell script

How to install this tool on Mac with apple chip?

I'm trying to use this tool on my mac (apple chip), but couldn't found how to install it

Your Environment

Macbook with M1 pro chip

Skip NaN values instead of assuming zeros

There’s an assumption in JMXFetcher.java that converting from NaN to zero is always a good idea. For any sort of error metric I believe it’s a bad idea, because graphs would show a zero errors where in reality the agent doesn’t know. I think it makes more sense to skip the event than assume it’s zero.

Real world example: we track message lag of a Kafka consumer via JMX. If the consumer stops, the lag metrics increase, so we know there’s a problem to investigate. However, eventually most of the JMX metrics change to NaN because the consumer isn’t updating them anymore. Rather than report no value, the infrastructure agent reports zero messages lag, which in this case would be confusing and potentially mask a real issue.

Scope:

• Consider a backward compatible setting to avoid sending the attribute if the value is NaN. In the agent for some samplers, if there are null attributes, we don’t include that attribute into the json serialization but we still submit the event.

Migrate to a Java 15 build to reduce footprint

The end result of building this project is a jlink'd binary, so there's no specific reason to stick to LTS releases.

The bundle size can be made slightly smaller (~5%) by upgrading the build to Java 15 (due to the removal of CMS and a few other bits). It is possible that startup time will also be improved, due to AppCDS, but this has not been tested yet.

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error	❌ Fail	⚠️ Warn	✅ Pass	Ignored	Total
0	3	1	3	0	7

• Fail
  • ❌ readme-starts-with-community-plus-header
  • ❌ readme-contains-link-to-security-policy
  • ❌ code-of-conduct-file-does-not-exist
• Warning
  • ⚠️ third-party-notices-file-exists
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-contains-discuss-topic

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend [![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus) to file

❌ readme-contains-link-to-security-policy #

Doesn't contain a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

❌ code-of-conduct-file-does-not-exist #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Found files. Below is a list of files or patterns that failed:

• CODE_OF_CONDUCT.md
  • 🔨 Suggested Fix: Remove file

Warning #

Click to see rules

⚠️ third-party-notices-file-exists #

A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. Below is a list of files or patterns that failed:

• THIRD_PARTY_NOTICES*
• THIRD-PARTY-NOTICES*
• THIRDPARTYNOTICES*

Passed #

Click to see rules

✅ license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

✅ readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

✅ readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

NRJMX Release Pipeline

remoteJBossStandalone flag is ignored if -r is not provided

Description

-remoteJBossStandalone flag is ignored if -r is not provided

Expected Behavior

Improve configuration option so that only one flag(remoteJBossStandalone) need to be specified

Add HashMap and ArrayList data support for JMX

Is your feature request related to a problem? Please describe.

Hashmaps and ArrayLists are not supported and customers are wasting time trying to instrument unsupported such environments:

} else if (value instanceof HashMap) {
            // TODO: Process hashmaps
            logger.fine("HashMaps are not supported yet: " + name);
        } else if (value instanceof ArrayList || value.getClass().isArray()) {
            // TODO: Process arrays
            logger.fine("Arrays are not supported yet: " + name);
        } else {
            throw new ValueError("Unsuported data type (" + value.getClass() + ") for bean " + name);
        }

Feature Description

Add support for Hashmaps and ArrayLists data types.

Additional context

We added a note in the documentation to alert users that these data types are not yet supported. See the Documentation PR #2372 for the changes.

Priority

Must Have

More Reporting On Number of Metrics Collected / Work Performed By Agent

A customer is running an analysis on the impact of the running nrjmx agent on their environemment and see request duration average creep up when running.

Now the performance team wants to find better ways to know how many metrics are being processed in each run. It's likely they may try to optimize the metric configurations based on these performance metrics.

• Can we configure debug log level and is it likely to show us the number of metrics processed per fetch
• Or can we consider having a rollup report on each run just saying how many metrics are gathered per each fetch per host

I will share two screenshots from the customer's benchmark environment:

WITH nrjmx running

WITHOUT nrjmx running