Application Insights runtime attach agent metric-filtering not working about applicationinsights-java

[heyams]

@tamasvincze i will try it myself and let you know.

[heyams]

@tamasvincze APPLICATIONINSIGHTS_CONFIGURATION_CONTENT has higher precedence over the json config. that is why when this is present and you only set connection string in it, nothing gets applied.

APPLICATIONINSIGHTS_CONFIGURATION_CONTENT is equivalent to the json config. For connection string, you can use this env var called APPLICATIONINSIGHTS_CONNECTION_STRING or specify it in the json config.

It's great to hear that it's working for you. "Loaded Class Count" can't be filtered out. Please let me know if i can close this issue.

[heyams]

@tamasvincze your config looks good to me. it didn't work probably because config didn't get picked up.
Since you enabled App Insights Java programmatically, the configuration file will be read from the classpath ((src/main/resources, src/test/resources). or use system property applicationinsights.runtime-attach.configuration.classpath.file to specify the json config file path.

https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-spring-boot#configuration-file-location

[heyams]

@tamasvincze we GA sampling overrides since 3.5.0 and the latest is 3.5.1 GA.

[tamasvincze]

Hey @heyams, thanks for the responses.
My applicationinsights.json file is on the classpath: C:\Dev\redacted-app\src\main\resources\applicationinsights.json

I upgraded to 3.5.1:

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-core</artifactId>
    <version>3.5.1</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-runtime-attach</artifactId>
    <version>3.5.1</version>
</dependency>

My current config file (on the same location:

{
  "sampling": {
    "requestsPerSecond": 0.5
  },
  "preview": {
    "sampling": {
      "requestsPerSecond": 0.5,
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        }
      ]
    }
  },
  "processors": [
    {
      "type": "metric-filter",
      "exclude": {
        "matchType": "strict",
        "metricNames": [
          "process_cpu_usage",
          "log4j2_events",
          "jvm_threads_states",
          "executor_pool_core",
          "executor_pool_max",
          "jvm_gc_overhead",
          "% Of Max Heap Memory Used",
          "Loaded Class Count"
        ]
      }
    }
  ]
}

In the log I see the newest version being used:
2024-03-13 17:46:52.829Z INFO c.m.applicationinsights.agent - Application Insights Java Agent 3.5.1 started successfully (PID 32080, JVM running for 5.752 s)

However on the azure portal, I still see the event that should have been excluded:

[heyams]

please try this instead without preview for sampling:

{
  "sampling": {
      "requestsPerSecond": 0.5,
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        }
      ]
   }, 
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

if you set the self-diagnostics log to debug level, it should output your json config. that is one way to verify that your config gets picked up.

[tamasvincze]

I'm added selfDiagnostics to the json:

{
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  }
}

The applicationinsights.log in the root of my project does not show any debug level info:

2024-03-13 18:09:31.441Z WARN  c.a.c.h.netty.implementation.Utility - The following Netty dependencies have versions that do not match the versions specified in the azure-core-http-netty pom.xml file. This may result in unexpected behavior. If your application runs without issue this message can be ignored, otherwise please update the Netty dependencies to match the versions specified in the pom.xml file. Versions found in runtime: 'io.netty:netty-common' version not found (expected: 4.1.101.Final),'io.netty:netty-handler' version not found (expected: 4.1.101.Final),'io.netty:netty-handler-proxy' version not found (expected: 4.1.101.Final),'io.netty:netty-buffer' version not found (expected: 4.1.101.Final),'io.netty:netty-codec' version not found (expected: 4.1.101.Final),'io.netty:netty-codec-http' version not found (expected: 4.1.101.Final),'io.netty:netty-codec-http2' version not found (expected: 4.1.101.Final)
2024-03-13 18:09:33.742Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.5.1 started successfully (PID 20072, JVM running for 7.776 s)
2024-03-13 18:09:33.745Z INFO  c.m.applicationinsights.agent - Java version: 17.0.10, vendor: Oracle Corporation, home: C:\Program Files\Java\jdk-17.0.10

However my application is logging out all properties through Spring's ConfigurableEnvironment, and I see this printed:

Mar 13 2024 18:09:48 main INFO  redacted.app.PropertySourceLogger [] - applicationinsights.internal.runtime.attached.json={  "sampling": {    "requestsPerSecond": 0.5,    "overrides": [      {        "telemetryType": "dependency",        "percentage": 0      }    ]  },  "preview": {    "processors": [      {        "type": "metric-filter",        "exclude": {          "matchType": "strict",          "metricNames": [            "process_cpu_usage",            "log4j2_events",            "jvm_threads_states",            "executor_pool_core",            "executor_pool_max",            "jvm_gc_overhead",            "% Of Max Heap Memory Used",            "Loaded Class Count"          ]        }      }    ]  },  "selfDiagnostics": {    "destination": "file+console",    "level": "DEBUG",    "file": {      "path": "applicationinsights.log",      "maxSizeMb": 5,      "maxHistory": 1    }  }}

Sadly, I still see the metrics being sent to azure:

[heyams]

@tamasvincze Here is my sample for metric-filter.
it works for me. i used the latest 3.5.1 GA.

We reserve certain default metrics not to be filtered out intentionally. You should be able to filter out any other custom metrics though.

In my applicationinsights.log, it showed clearly that json config was applied successfully:
DEBUG c.m.a.a.i.c.ConfigurationBuilder - configuration: {"connectionString":

[tamasvincze]

Your configuration made me try something.

Here is my current config:

{
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  },
  "instrumentation": {
    "logging": {
      "enabled": "true"
    }
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

And for my app, I have 2 environmental variables exported (in the IDEA runconfig):

APPLICATIONINSIGHTS_ROLE_NAME=test-vinctam12
APPLICATIONINSIGHTS_CONFIGURATION_CONTENT={"connectionString":"InstrumentationKey=REDACTED;IngestionEndpoint=https://northeurope-0.in.applicationinsights.azure.com/;LiveEndpoint=https://northeurope.livediagnostics.monitor.azure.com/"}

I don't see any DEBUG logs from the agent.

On Azure Portal I see the excluded metrics:

However if I delete the env vars and use role configuration through the json, like this:

{
  "connectionString": "InstrumentationKey=REDACTED;IngestionEndpoint=https://northeurope-0.in.applicationinsights.azure.com/;LiveEndpoint=https://northeurope.livediagnostics.monitor.azure.com/",
  "role": {
    "name": "test-vinctam13",
    "instance": "instance1"
  },
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  },
  "instrumentation": {
    "logging": {
      "enabled": "true"
    }
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

Now I see DEBUG logs, like:

2024-03-13 20:33:49.724Z DEBUG c.m.a.a.i.c.ConfigurationBuilder - read system property: applicationinsights.internal.runtime.attached.json=...
2024-03-13 20:33:49.728Z DEBUG c.m.a.a.i.c.ConfigurationBuilder - configuration: {"connectionString":"InstrumentationKey=...

And now I only see one of the excluded metrics:

Can you try exporting the rolename and connection string instead of having them in the json?

[tamasvincze]

Oh you're right, I've been using the wrong env var.
Thanks for the help!
I'll close the issue :)