honeycombio / honeycomb-opentelemetry-java

Honeycomb's OpenTelemetry Java SDK distribution

Honeycomb OpenTelemetry Distro for Java

This is Honeycomb's distribution of OpenTelemetry for Java. It makes getting started with OpenTelemetry and Honeycomb easier!

Latest release built with:

• OpenTelemetry v1.27.0
• OpenTelemetry Java Agent v1.27.0

Getting Started

Honeycomb's OpenTelemetry Java Agent gives you all-in-one, easy-to-install auto-instrumentation for your Java application.

Honeycomb's OpenTelemetry Java SDK gives you the ability to add manual instrumentation.

• Documentation
• Examples

Why would I want to use this?

• Streamlined configuration for sending data to Honeycomb!
• Easy interop with existing instrumentation with OpenTelemetry!
• Deterministic sampling!
• Multi-span attributes!

License

Apache 2.0 License.

honeycomb-opentelemetry-java's Issues

SDK does not use additional span processors

SDK builder allows adding span processors via build.AddSpanProcessor() but those processors are not used when building the SDK.

Add publish task to push jars to maven

Publish artifacts to public maven repository

Add E2E test task [local]

One-Pager

The goal is to have a single command that can be run locally to execute the 2 E2E tests (agent path / SDK path).

• the task should launch each example app, hit the app endpoint, and use the validator to check that the traces ended up in Honeycomb
• it may be easier to dockerize the example apps, and use the images in E2E test
• the task should return a pass or fail result (could be exit code)
• this can be a gradle task (nice and idiomatic) or a script, if that's easier

Ability to add/set resource attributes in the Builder

This will make it easy to add things like build/version, environment, and other static fields to every span.

Builder.addResourceAttribute(key, value)
and/or
Builder.setResourceAttributes(map)

Differentiate between plain SDK and agent in the metadata

We cannot currently differentiate SDK-only users from agent users. Add/augment resource attributes to let us differentiate those. (We can use telemtry.sdk.* - though we don't technically provide our own SDK. Alternatively, we can use our own honeycomb.distro.* attributes)

add java docs to classes/methods

Flip repo to public

Do a quick scan of repository history and make any necessary cleanup tasks

• Git history reviewed
• Access Rights
• License
• Main branch
• Merge commits & deleting merged branches
• Branch protection
• codeowners
• changelog - create this with the first release
• .editorconfig files
• github actions
  • apply project labels
• circle ci
  • badge in readme
  • create GitHub release on version taggings (to be done in #20 ?)
  • publish release artifacts to GitHub release (to be done in #20 ?)
  • publish release artifacts to Maven (to be done in #20 ?)

grpc client from SDK can conflict with an existing client in the app

Customer was importing the SDK (using the agent for auto-instrumentation), and had an existing grpc client in their app. The grpc version in the SDK conflicted with the version in the app:

io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51).
java.lang.NoSuchMethodError: io/grpc/ClientStreamTracer$StreamInfo$Builder.setPreviousAttempts(I)Lio/grpc/ClientStreamTracer$StreamInfo$Builder; (loaded from file:/app.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51) called from class io.grpc.internal.GrpcUtil (loaded from file:/io.grpc/grpc-core/1.41.0/882b6572f7d805b9b32e3993b1d7d3e022791b3a/grpc-core-1.41.0.jar by jdk.internal.loader.ClassLoaders$AppClassLoader@193b9e51).

We have some docs in the repo about SDK containing the GRPC client, and how to exclude it.

• make the docs more discoverable (readme and public docs?)

React to java agent name change from upstream OTel

This is just a reminder issue to monitor java agent name change from upstream OTel - we'll likely need to update our dependency to remove "-all" in an upcoming release, once this change goes live and we update our dependencies.

open-telemetry/opentelemetry-java-instrumentation#3959 (comment)

Happy path integration test for traces [agent only]

One-Pager

The goal is to add a happy path integration (smoke) test that can be run locally. This is just the traces validation, for now.

• using the fake backend from #111 using collector
• using the example apps in the repo
• see prior art for smoke tests in the example distro (e.g. how to start docker containers from within java)
• ideally, a gradle command to run integration (smoke) tests make+docker+bats
• adding to CI is a separate task ✅

What we're validating:

• auto-instrumentation works
• custom (manual) instrumentation works
• baggage span processor works
• agent works alone sans SDK

Add smoke-tests backend

One-Pager

The goal is to have a fake backend that can receive OTLP/gRPC requests and report back sent requests.

Easiest but least flexible: Example smoke tests from OTEL -- the example distro relies on the OTEL docker images for the fake backend and the example app, so this route would tie us to that implementation.

Harder but more flexible: The main smoke-tests from OTEL -- we can follow their example for standing up a fake OTLP backend. They distribute everything as docker images, and the smoke test starts up containers and does the validation.

Hardest? but most flexible: roll our own fake backend. Most likely a go service that copies our OTLP ingest path (shared component would help here a lot).

Track unhappy integration test scenarios

One-Pager

The goal is to add integration/smoke-test coverage for some interesting edge cases.

Ideas:

• missing configuration (like API key)
• API not available
• multi-span attributes

add helper methods for something close to Beeline's "trace fields"?

With our default BaggageSpanProcessor enabled, an attribute in the current baggage context will be also be added to in-process child spans. The wrapper could make it easier to add attributes to both the current span and the current baggage to facilitate something similar to what the Beelines do with "trace fields."

An example of adding an attribute to baggage:

@RequestMapping("/greeting")
public String index() throws URISyntaxException, IOException, InterruptedException {
	// retrieve a name
	String name = nameService.getName();

	// add an app.username attribute to baggage
	Baggage.current()
		.toBuilder()
		.put("app.username", name)
		.build()
		.makeCurrent();

	// spans generated by this messageService and other services it calls will have
	// the app.username attribute propagated to them
	String message = messageService.getMessage();

	// this child span of the current method will also have the app.username attribute
	// added to it
	Span render_span = tracer.spanBuilder("🎨 render message ✨").startSpan();
	String greeting = String.format("Hello, %s, %s", name, message);
	render_span.end();

	return greeting;
}

If the developer wants the attribute added to all subsequent in-process child spans, propagated out to other services, and added to the current span, they would need to also add the attribute to the current span":

@RequestMapping("/greeting")
public String index() throws URISyntaxException, IOException, InterruptedException {
	// retrieve a name
	String name = nameService.getName();

	// add an app.username attribute to current span
	Span.current().setAttribute("app.username", name);

	// add an app.username attribute to baggage
	Baggage.current()
		.toBuilder()
		.put("app.username", name)
		.build()
		.makeCurrent();

	// spans generated by this messageService and other services it calls will have
	// the app.username attribute propagated to them
	String message = messageService.getMessage();

	// this child span of the current method will also have the app.username attribute
	// added to it
	Span render_span = tracer.spanBuilder("🎨 render message ✨").startSpan();
	String greeting = String.format("Hello, %s, %s", name, message);
	render_span.end();

	return greeting;
}

Set Java compatibility targets

https://docs.gradle.org/current/userguide/building_java_projects.html#sec:java_cross_compilation

https://docs.gradle.org/current/userguide/java_plugin.html#sec:java-extension

looks like the new hotness: https://blog.gradle.org/java-toolchains

we probably need to start using the gradle wrapper in circle: https://circleci.com/docs/2.0/language-java/

OTEL packages mostly target 1.8

Review SDK configuration for possible unit tests

One-Pager

Do a code review of the repo, and backfill any pieces that are not yet tested by the E2E/smoke tests and would be a good candidate for unit tests. (It's possible that there aren't any)

Add option to add span processors to the SDK builder

Currently there's no way to use the BaggageSpanProcessor (or add any span processor other than the defaulted BatchSpanProcessor) when using the Honey SDK.

Enable console output

Is your feature request related to a problem? Please describe.

To help to debug it would be nice to enable console output. However, that doesn't seem possible with the agent today.

Describe the solution you'd like

A config flag that enables console output.

Describe alternatives you've considered

Stand up a local otel collector configured with the console output and point the agent at that. This works but is a lot of effort.

Additional context

make agent and sdk jars have a honey-flavored name

List this project in the OTel Registry

https://github.com/open-telemetry/opentelemetry.io#adding-a-project-to-the-opentelemetry-registry

Make Honey SDK a pure configurer

The (currently known as) Honeycomb SDK should return an instance of OpenTelemetry class, instead of our own "SDK" class. Similarly to the .NET distro, we should simply configure the OTEL instances, and allow users to interact with OTEL interfaces from then on.

• this is a breaking change
• do we keep the module/artifact name? ("SDK") - changing the name might make it less confusing to users

Automate Nexus publish

Currently there's a manual step in our release process (need to close and release staged repositories in Sonatype UI). Automate that last bit as described here using the publish plugin

`0.4.0` agent distro fails to load

Versions

• Java: 11
• Honeycomb OpenTelemetry Distribution: 0.4.0

Create a Dockerfile with this base image, COPY and CMD:

FROM adoptopenjdk/openjdk11:alpine-jre

COPY honeycomb-opentelemetry-javaagent-0.4.0-all.jar  honeycomb-opentelemetry-javaagent-0.4.0-all.jar 

CMD java \
    -javaagent:honeycomb-opentelemetry-javaagent-0.4.0-all.jar  \
    -jar yourApp.jar

Exception in thread "main" java.lang.reflect.InvocationTargetException
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source)
      at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source)
Caused by: java.lang.NoSuchMethodError: 'boolean io.honeycomb.opentelemetry.EnvironmentConfiguration.isPresent(java.lang.String)'
      at io.honeycomb.opentelemetry.HoneycombAgent.configureEnvironment(HoneycombAgent.java:32)
      at io.honeycomb.opentelemetry.HoneycombAgent.premain(HoneycombAgent.java:17)
      ... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f07e40bc93b, pid=1, tid=11
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.12+7 (11.0.12+7) (build 11.0.12+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.12+7 (11.0.12+7, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x2693b]  abort+0x1ed
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#
exited with code 127

Share honey constants between modules

DRY up SDK and agent/custom modules, things like x-honeycomb-team and default honey API

Create an E2E validator

One-Pager

The validator will be used to run the E2E tests in the Java and dotnet distros. Given an expected result (such as an expected trace), it needs to determine if the trace exists in Honeycomb, and return a true/false. The outcome of this issue is having a validator available (can be run locally), not adding it to CI, yet.

• see e2e tests in hound for similar prior art
• we can use the query API
• we will likely want a new (private?) repo for the validator, since it's shared across distros
• we need to build in some latency resilience, like retries (can be a follow-up issue)
• we will likely want the validator available as a docker image (can be a follow-up issue)

Set telemetry.sdk.* in Java Beestro *SDK* (not agent)

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/README.md#telemetry-sdk

Set the telemetry.sdk.* attributes in the SDK produced by our configurer

• if we make it something specific to us (e.g. opentelemetry-honeycomb), we don't need to do anything else for #131
• but is that accurate? do we actually produce our own "SDK"?

Revert removal of runtime metadata

Is your feature request related to a problem? Please describe.

In 0.5.0 we incorrectly removed sending runtime metadata. This issue is to revert that change.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Fix maven publish: POM validation

Sonatype validation failed upon "closing" the staged repo:

typeId	pom-staging
failureMessage	Invalid POM: /io/honeycomb/honeycomb-opentelemetry-javaagent/0.2.0/honeycomb-opentelemetry-javaagent-0.2.0.pom: Project description missing
failureMessage	Invalid POM: /io/honeycomb/honeycomb-opentelemetry-sdk/0.2.0/honeycomb-opentelemetry-sdk-0.2.0.pom: Project description missing
failureMessage	Invalid POM: /io/honeycomb/honeycomb-opentelemetry-common/0.2.0/honeycomb-opentelemetry-common-0.2.0.pom: Project description missing

Update exposed OTEL packages in the SDK

Currently Honey SDK requires the OTEL SDK (which includes the OTEL API), and exposes the entire OTEL SDK as an api (makes it available to consumers).

Do Honey SDK consumers need access to the entire OTEL SDK? Can we just expose the OTEL API?

Also: add the extensions-annotations as an exposed dependency.

Add support for metrics

Add configuration to set metrics endpoint and/or dataset.

Open Telemetry Java now supports sending metrics to a specific endpoint. Metrics were temporarily disabled on this distribution because it was enabled by default as of Otel 1.1.0.

Add publish task to create a Github release

On new version tag, publish a Github release with the agent all-jar

APIKey and Dataset are Required but should be Optional

Right now we throw an exception if apikey or dataset are not set. There are situations when you may not want to set them such as when sending to an opentelemetry collectors. It would be better to throw a warning message instead.

Reëvaluate Java SDK interface

• Can we adjust our Java SDK/API to configure and return a vanilla OTel SDK instance pre-configured instead of having our own SDK class?
• ensure telemetry.sdk.* is set to our identifiers?

Allow TRACES and METRICS to go to different endpoints and datasets

A user wishes to send traces through Refinery and to send metrics directly to Honeycomb. We generally recommend that traces and metrics be sent to separate datasets. They have set the following environment variables:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "<refinery>"
OTEL_EXPORTER_OTLP_TRACES_HEADERS: "X-Honeycomb-Team=<traces-key>,X-Honeycomb-Dataset=<traces-dataset>"
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: "<honeycomb-api>"
OTEL_EXPORTER_OTLP_METRICS_HEADERS: "X-Honeycomb-Team=<metrics-key>,X-Honeycomb-Dataset=<metrics-dataset>"
HONEYCOMB_API_KEY: <key>
HONEYCOMB_DATASET: <dataset>

The "generic" endpoint settings—HONEYCOMB_API_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT—are not set. They report that all traces and metrics are sent to their Refinery URL.

Distribute sampler outside the SDK?

Potentially: provide the sampler in a standalone artifact, if there's a use case for people using vanilla OTEL that want the deterministic sampler.

Add example apps

One-Pager

Add example apps inline with the repo:

1. Spring Boot app instrumented with the agent. The app should exercise at least one auto-instrumentation (e.g. DB or network request), and have at least one manually created span.
2. Spring Boot app instrumented with the SDK. The app should have at least one manually created span (no auto-instrumentation in the SDK path).

Both apps should have a single curl-able endpoint to exercise the above instrumentations.
Both apps should use latest code version of the agent and SDK available in the repo (local dependency as opposed to pulling from maven central)

Add baggage propagator to SDK by default

Currently only adding W3CTraceContextPropagator, which means when users do getPropagators and inject/extract context, they will only see trace context, not baggage.

Make it more clear what version of OTel this is built against

Since we will be versioning independently of OTel due to the additional functionality we are adding we want to make it more easily discoverable what version of OTel we are building against.

Maybe at the top of the README? Maybe a badge?

Allow setting resource attributes when using the Honey Agent

Currently, when the Honey Agent metadata resource configuration overwrites user-defined otel.resource.attributes. (The environment service.name translation also overwrites the attributes)

Make it so you can set additional resource attributes using the Honey Agent.

We should be implementing the ResourceProvider to configure the resource attributes in the Agent wrapper. This may be sufficient to not clobber user-defined resource attributes. If it's not, we may be able to use the config argument to merge the user attributes with ours. We should also remove the service.name configuration from the Honey Agent environment re-write path.

Add Multi-Span attributes example to docs

We have the Multi-Span attributes plumbing (the SpanProcessor) but we haven't documented how to invoke it in the docs.

clean up sdk/build.gradle

Does it need all the things, now that it's a module? (like repositories)

Allow vars to be specified via a prop file?

Looks like OTEL has an undocumented support for this: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/ConfigInitializer.java#L26

Precedence goes system properties >> env vars >> prop file

Document Debugging

Folks are confused when it comes to using vanilla OTEL env vars/system properties with the beestro. Add something to the docs/readme about how those interact with the beestro (for example, you should be able to enable agent debug with otel.javaagent.debug but you might override honeycomb api key, if you set otel.exporter.otlp.headers)

slack thread: https://houndsh.slack.com/archives/C8RP24A6Q/p1627944971012200

change 1.0 versioning to 0.1

For an initial experimental release, we'll start at v0.1 (0.1.0?). Run through the hard-coded 1.0s and update them to 0.1.

Don't emit metrics by default

Currently, metrics are enabled by default, however they end up in the same dataset as traces. This is confusing, and not very useful. In the future, we can add actual support for metrics (e.g. add configuration to set metrics endpoint/dataset). In the meantime, we should not emit metrics by default.

Add integration tests to CI

One-Pager

Exercise the integration (smoke) tests during CI build (commits + nightly).

fix circleci

add some smoke tests

We need some tests to confirm our customizations of OpenTelemetry.

Some prior art:

• AWS' distribution

Add E2E test to CI

One-Pager

The goal is to have the E2E test run on every commit/PR as well as nightly.

Add WithSpan example to docs

When using the Java auto-instrumentation agent, a convenient way to get a custom span is to decorate a method with @WithSpan. Maybe add an example of that to the Creating New Spans section to contrast with the example already there for getting the tracer and creating a span from within a method.

v0.5.0 agent isn't working on its own

Versions

• Java: 11
• Honeycomb OpenTelemetry Distribution: v0.5.0

Steps to reproduce

Run agent alongside app with no SDK

Additional context

With same setup, vanilla otel agent 1.6.1 and java distro 0.4.0 shows some trace data in honeycomb. Nothing shows up with 0.5.0 agent.