newrelic / newrelic-dotnet-agent

The New Relic .NET language agent.

New Relic Monitoring for .NET

.NET Agent

New Relic's .NET agent monitors your .NET app, giving you an end-to-end view of your app's performance. It works with all .NET compatible languages, such as C#, VB.NET and CLI.

Installation

The latest released binaries for the .NET Agent can be found either here or on NuGet.org.

See our agent installation documentation to understand which of these files are appropriate for your application type and hosting environment.

Getting Started

• See our introduction to New Relic for .NET to learn how to use the .NET agent with your application.
• See our changelog for release notes.

Building and Testing

Get started by reviewing our instructions for how to build and test.

Support

Should you need assistance with New Relic products, you are in good hands with several support diagnostic tools and support channels.

This troubleshooting framework steps you through common troubleshooting questions. Additional troubleshooting scenarios can be found on the New Relic Docs site.

New Relic offers NRDiag, a client-side diagnostic utility that automatically detects common problems with New Relic agents. If NRDiag detects a problem, it suggests troubleshooting steps. NRDiag can also automatically attach troubleshooting data to a New Relic Support ticket.

If the issue has been confirmed as a bug or is a Feature request, please file a Github issue.

Support Channels

• New Relic Documentation: Comprehensive guidance for using our agent
• New Relic Community: The best place to engage in troubleshooting questions
• New Relic Developer: Resources for building a custom observability applications
• New Relic University: A range of online training for New Relic users of every level
• New Relic Technical Support 24/7/365 ticketed support. Read more about our Technical Support Offerings.

Privacy

At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.

We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address and email address.

Please review New Relic’s General Data Privacy Notice for more information.

Contributing

We encourage your contributions to improve New Relic's .NET monitoring products! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project. To execute our corporate CLA, which is required if your contribution is on behalf of a company, or if you have any questions, please drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, please review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today. We also host a community project page dedicated to the New Relic .NET agent.

License

The .NET Agent is licensed under the Apache 2.0 License. The .NET Agent also uses source code from third-party libraries. You can find full details on which libraries are used and the terms under which they are licensed in the third-party notices document.

newrelic-dotnet-agent's Issues

Support MySqlConnector

MySqlConnector is an advanced ADO.NET library for MySQL that supports asynchronous operations. (The "standard" MySQL Connector/NET package does not.)

Feature Description

Supporting this library out-of-the-box is a popular feature request for New Relic customers.

There is custom instrumentation XML for MySqlConnector, but as discussed in in NR Support case 373822 (NOTE: private link) and demonstrated in this repo, when it's used, database timings are doubled:

This trace shows a definite problem with the database timings. Specifically, almost all database calls are logged as extending to the end of the transaction, e.g., 164.66ms then 162.23ms, then 160.97ms, then 158.94ms, etc. (They should be something more like 2.4ms, 1.3ms, 2.0ms, etc.)

Summing up all these incorrect timings gives an overall incorrect picture of how long the API is spending in database transactions.

According to that support ticket:

So, after diving more deeply into the code and your repro our dev team has determined the agent code behind the SqlCommandTracerAsync tracer does not work correctly with the MySqlConnector driver and proper support will require additional investigation, development, testing, etc. There isn't a quick fix here :-(

Now that newrelic-dotnet-agent is Open Source, is it possible the community could help develop this fix? What are the specifics of how the "tracer does not work correctly", and what would be the correct approach to fixing it?

Priority

This is a "Must Have" for us. We use MySqlConnector for its async I/O capabilities, but all database timings are greatly inflated in the NR UI, making it useless for actually diagnosing DB issues.

appsettings.json file not found for .NET Core app hosted in Windows Service or InProcess IIS

Description
When using InProcess IIS hosting or Windows Service, the Agent looks for the appsettings.json using GetCurrentDirectory(). For InProcess IIS or a Windows Service, GetCurrentDirectory() returns the path of the IIS app pool process or - in the case of Windows Service - C:\Windows\system32\), not the path of the application.
This is a known issue in AspNetCore.

This Issue is to change the Agent to use a different way to get to the appsettings.json file.

One alternative is suggested in the AspNetCore Issue.
Another alternative may be to use the AppContext.BaseDirectory property instead of GetCurrentDirectory().

Expected Behavior
The Agent should find and use appsettings.json if it is located in the current working directory of the application.

Steps to Reproduce

• Create a .NET Core application to be hosted in a Windows Service or InProcess IIS.
• Install the Agent.
• In appsettings.json, set configuration for app name:

  {
      "NewRelic.AppName": "Descriptive Name"
  }
  

• Run the application with the Agent engaged.
• Verify via agent log or New Relic UI that the application name is not what was specified in appsettings.json.

Your Environment
.NET Core application, hosted in InProcess IIS or Windows Service.

Additional context
Workarounds:

1. IIS apps can be set to run out of process via the web.config which allows the agent to find the correct appsettings.json
   <aspNetCore processPath="dotnet" arguments=".\MVCcore.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="OutOfProcess" />
2. Use app-local newrelic.config file for supported Agent configuration instead of appsettings.json.

Internal NR only: ref1 ref2 ref3

Note about development environment
N/A

Scriptable installer not configuring the installed agent correctly

Customer Issue:
Zendesk 423907

Description
If you use the scriptable installer to install the framework agent, when the agent starts up you get the following error, preventing the agent from working.

NewRelic ERROR: [pid: 4684, tid: 7] There was an error initializing the agent: System.ArgumentNullException: Value cannot be null.
Parameter name: path1
at System.IO.Path.Combine(String path1, String path2)
at NewRelic.Agent.Core.Utilities.ExtensionsLoader.Initialize(String installPathExtensionsDirectory)
at NewRelic.Agent.Core.AgentManager..ctor()
at NewRelic.Agent.Core.AgentManager.AgentSingleton.CreateInstance()

Expected Behavior
The agent should not get the above error when starting up.

Troubleshooting or NR Diag results
N/A

Steps to Reproduce

1. Start with a fresh system that doesn't have New Relic installed yet.
2. Run the scriptable installer to install the agent.
3. Start an application that should be instrumented by the agent.
4. View the managed agent log and see the above error.

Your Environment
N/A

Additional context
The scriptable installer sets the NEWRELIC_INSTALL_PATH environment variable to be "C:\Program Files\New Relic.NET Agent". It also places the agent's files into that directory.

When the agent starts up it using the following logic to determine the agent's installed directory.

        private static string GetNewRelicInstallPath()
        {
            var newRelicInstallPath = System.Environment.GetEnvironmentVariable(NewRelicInstallPathEnvironmentVariable);
            if (newRelicInstallPath != null)
            {
                newRelicInstallPath = Path.Combine(newRelicInstallPath, RuntimeDirectoryName);
                if (Directory.Exists(newRelicInstallPath)) return newRelicInstallPath;
            }

            newRelicInstallPath = System.Environment.GetEnvironmentVariable(NewRelicHomeEnvironmentVariable);
            return newRelicInstallPath;
        }

The problem is occurring because when the NEWRELIC_INSTALL_PATH environment variable is used, it will look for the agent in netframework subdirectory within NEWRELIC_INSTALL_PATH. But the scriptable installer does not place the agent bits into that subdirectory, so the agent will check the NEWRELIC_HOME environment variable next. However, that environment variable is not defined, so the agent's install directory will be set to null, which is what's causing the exception.

This has likely been broken for a while.

Agent decrypts encrypted proxy password configuration

Agent will support obscured proxy password in the proxy configuration

MsiInstaller tests need to get a valid license key from a secret store

The MsiInstaller tests have two test cases which require a valid license key to work. They are failing because they are attempting to use a hardcoded, invalid license key. These tests should be modified to use the same secrets store mechanism as the integration tests to get the license key.

Test Coverage for Span Events category 'http'

Feature Description

We have test coverage for Span Events with category 'generic' and category 'datastore'. We should add coverage for Span Events with category 'http'. See spec: https://source.datanerd.us/agents/agent-specs/blob/master/Span-Events.md

See file DistributedTracingTests.cs in CompositeTestAgent.

Describe Alternatives

N/A

Additional context

Originated from DOTNET-3389
The reporter of this issue is Lynice S.

Priority

Nice to Have Really Want, Must Have, Blocker

Documentation: Run UnboundedIntegrationTests locally

The process for running UnboundedIntegrationTests locally will change with this Milestone. Formerly, services required for running these tests were maintained on a server(s) in Jenkins. With this Milestone, scripts will create local Docker containers for the various services required.

New scripts and Docker files have been added to the repo. Documentation will describe how to use these scripts to set up the local environment for running these tests.

One thing that will be useful to document is how to run each service individually instead of having to launch them all and wait 10 minutes if they're testing something that only needs one service: https://docs.docker.com/compose/reference/run/ (especially the --service-ports option)

Modify Couchbase unbounded integration tests to use a local Docker container

Currently, some of these tests rely on a Couchbase server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Internally we already figured out how to run the Couchbase server using the following:
docker run -d --restart always --name CouchbaseServer -p 8091-8094:8091-8094 -p 11210:11210 couchbase/server:enterprise-4.5.1 The only catch is that some initial setup steps need to be performed before the necessary travel-sample database exists and can be used by the tests, so this will have to be addressed somehow.

New ignore error configuration support

Feature Description

The existing ignore error configuration is deprecated according to the agent specs. We need to support the new configurations.

Modify MS SQL Server integration tests to use a local Docker-hosted service

Currently, some of these tests rely on a MS SQL server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Note: I think our current MS SQL Server unbounded integration tests expect certain data to already exist in the DB (unlike a lot of the other tests, which set up any data needed to run queries against when the test fixture is initialized) so part of this story will involve modifying the tests to perform their own setup.

https://hub.docker.com/_/microsoft-mssql-server

Assembly mismatch causing instrumentation to shutdown

Description
The NewRelic.Providers.Wrapper.Mvc3.AsyncBeginInvokeActionWrapper wrapper is shutting down because of a error caused by different assembly versions.

2020-07-24 10:41:10,656 NewRelic  DEBUG: [pid: 5812, tid: 14] Wrapper "NewRelic.Providers.Wrapper.Mvc3.AsyncBeginInvokeActionWrapper" will be used for instrumented method "System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction"
2020-07-24 10:41:10,668 NewRelic  ERROR: [pid: 5812, tid: 14] Tracer invocation error: System.InvalidCastException: Expected argument 0 to be of type System.Web.Mvc.ControllerContext (v3.0.0.1), but was of type System.Web.Mvc.ControllerContext (v4.0.0.1)
   at NewRelic.SystemExtensions.ObjectArrayExtensions.ExtractAs[T](Object[] objects, Int32 index)
   at NewRelic.SystemExtensions.ObjectArrayExtensions.ExtractNotNullAs[T](Object[] objects, Int32 index)
   at NewRelic.Providers.Wrapper.Mvc3.AsyncBeginInvokeActionWrapper.BeforeWrappedMethod(InstrumentedMethodCall instrumentedMethodCall, IAgent agent, ITransaction transaction)
   at NewRelic.Agent.Core.Wrapper.WrapperService.BeforeWrappedMethod(Type type, String methodName, String argumentSignature, Object invocationTarget, Object[] methodArguments, String tracerFactoryName, String metricName, UInt32 tracerArguments, UInt64 functionId)
   at NewRelic.Agent.Core.AgentManager.GetTracerImpl(String tracerFactoryName, UInt32 tracerArguments, String metricName, String assemblyName, Type type, String typeName, String methodName, String argumentSignature, Object invocationTarget, Object[] arguments, UInt64 functionId)

Expected Behavior
We should not get this error even when the versions are different.

Troubleshooting or NR Diag results
N/A

Steps to Reproduce
N/A

Your Environment
N/A

Additional context
This was discovered in an internal support ticket.

Modify Redis integration tests to use a local Docker-hosted service

Currently, some of these tests rely on a Redis server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

https://hub.docker.com/_/redis

Stop relying on dependency injection containers

Summary

The agent uses dependency injection containers to manage the dependencies for all of our objects, as well as starting up our agent. This has lead to multiple bugs during agent startup, it makes it really hard to reason about which things are created when, it makes it hard to know which long running services are started at which point in time, and it's also lead us to create extra code to handle dependency injection (see #26 as an example).

Desired Behaviour

The agent doesn't use a dependency injection container, and the agent startup is more declarative so that we can see exactly what happens and in what order it happens.

Possible Solution

Additional context

Internal NR only: slack convo with some more context

GC Performance Counters for Windows - Unable to obtain Perf Counter Instance Name in IIS until GC has occurred

When collecting Garbage Collection performance counters for Windows, the agent, in part, uses the application's Instance Name to disambiguate different instances of the same running application. For example, two instances of an application named "Test" may appear as Test and Test#1.

To identify which Instance name belongs to the currently executing process, a reverse lookup is made to the performance counter Process ID and is matched to the Process ID (PID) of the current process. With this instance name, Garbage Collection Performance Counters are created and a reference to each of them is maintained in memory.

Description
In IIS Apps, unless a garbage collection has occurred, the reverse lookup does not work. The PID reported

/// <summary>
/// The instance name of a performance counter is a combination of the process Name and an index id to 
/// uniquely identify it when multiple of the same process are running.  For example "TestApp #1" pid=82, "TestApp #2" pid=134
/// are two instances of the same executable, TestApp.  In order to determine if an instance matches a process,
/// an additional performance counter "Process ID" must be used which returns the PID of the process.
/// </summary>
/// <param name="perfCategoryName"></param>
/// <param name="processName"></param>
/// <param name="pid"></param>
/// <returns>The instance name that will be used to collect performance counter data for the process</returns>
private string GetInstanceNameForProcessAndCategory(string perfCategoryName, string processName, int pid)
{
	var performanceCategory = _createPerformanceCounterCategory(perfCategoryName);

	var instanceNames = performanceCategory.GetInstanceNames()
		.Where(x => x.StartsWith(processName, StringComparison.OrdinalIgnoreCase));

	foreach (var instanceName in instanceNames)
	{
		try
		{
			using (var pcPid = _createPerformanceCounter(perfCategoryName, ProcessIdCounterName, instanceName))
			{
				if ((int)pcPid.NextValue() == pid)
				{
					return instanceName;
				}
			}
		}
		catch (Exception ex)
		{
			//Log a message here and continue because this instance may not be relevant to the process that we are looking for.
			Log.Finest($"Error determining if instance '{instanceName}' is process '{processName}'({pid}). Error: {ex}");
		}
	}

        //At this point a match was not found. The caller should handle accordingly
	return null;
}

In the above code which performs this reverse lookup. Unless a GC Has occurred, (int)pcPid.NextValue() reports 0 as the PID.

Expected Behavior
When GC Stats become available, the agent should collect them. Perhaps it should retry.

Troubleshooting or NR Diag results
[NOTE]: # ( Provide any other relevant log data such as the agent and/or profiler logs, preferably at DEBUG level or higher. )
[TIP]: # ( Scrub logs and diagnostic information for sensitive information )

Steps to Reproduce (this may be intermittent)

• Build a APP and host it in IIS
• Ensure that app-pool user has appropriate permissions to collect Performance Counters (Performance User) or is an administrator.
• Start the app
  --> Notice the error in the log

Your Environment
Windows, .Net Framework 4.7 (both local development server and Azure VM)

Remove localhost from secrets

Description

With the move of services for UnboundedIntegrationTests to local Docker containers, references to the Jenkins database server ip addrs have been replaced with 127.0.0.1. These host addresses are no longer 'secrets' and can be moved back into the configuration in the solution.

This change implies changes to test configuration code and changes to the secrets stored in Jenkins (and LastPass).

Modify MongoDB_2.6 unbounded integration tests to use a local Docker-hosted service

Currently, some of these tests rely on a MongoDB server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Internally we already run the necessary service with:
docker run -d --restart always --name MongoDBServer -p 27017:27017 mongo:3.6.4

Note: we have two different groups of MongoDB integration tests, which use different services - because we have two different instrumentation wrappers for MongoDB for different client libraries. If we have time let's investigate whether we can combine the services.

Expected Errors Support

Add support for Expected Errors. Expected errors will be reported to APM and available for viewing, but they won't affect the Apdex or error rate (or alert conditions based on error rate).

Nullability Support for APIs [.NET 5]

Microsoft's guidance is to have nullability support put into all APIs by the GA release of .NET 5.

This will apply to our Agent API, Logs in Context, Telemetry SDK, and Lambda products.

Aha! Link: https://newrelic.aha.io/features/DOTNET-236

Modify DB2 unbounded integration tests to use a local Docker container

Currently, some of these tests rely on a DB2 server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

DB2 docker hub entry: https://hub.docker.com/r/ibmcom/db2

Modify RabbitMq unbounded integration tests to use a local Docker container

Currently, some of these tests rely on a RabbitMQ service hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Internally we already run the RabbitMQ service by doing this: docker run -d --restart always --hostname my-rabbit --name RabbitmqServer -p 4369:4369 -p 5671-5672:5671-5672 -p 25672:25672 rabbitmq:3.5

Dotnet core agent - old installation binary is removed from download site

Description
New relic download site only has the latest version of deb file for dotnet agent installation. We have the docker build that download and install the the agent using the file. Old version installation files are removed upon release of new version of the installation file and that breaks our docker build.

Expected Behavior
Keep the old version of dotnet agent installation file on download site.

Steps to Reproduce
As of now I see newrelic-netcore20-agent_8.30.0.0_amd64.deb listed on the download site. It will disappear once a new version of agent is released.

Integration Tests - for Infinite Tracing (8T)

Is your feature request related to a problem? Please describe.

No

Feature Description

Add integration test for 8T

Describe Alternatives

N/A

Additional context

Add to CI
Originated from DOTNET4618
Reporter: Jason Feingold

Priority

[Nice to Have, Really Want, Must Have, Blocker]

Modify PostgreSQL integration tests to use a local Docker-hosted service

Currently, some of these tests rely on a PostgreSQL server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

https://hub.docker.com/_/postgres

Add Lambda Example

This issue is to create an example app demonstrating the Agent's support of Lambda.

Additional context

The example app should be located in the examples folder at the top of the repo. At the time of this writing, examples folder does not exist so this Issue would include creating it.

Priority

An example would help the Lambda community understand how to use the Agent for such applications. New Relic welcomes contributions from the community for this purpose and looks to an interested contributor to give it a go!

Proxy Config Encryption Support

Passwords should be encrypted within the proxy configuration file to prevent unauthorized capture of account details.

Aha! Link: https://newrelic.aha.io/features/DOTNET-272

Clean up unused AWS assets

We have unused AWS assets that need cleaning up.

New Relic CLI Utility to encode/obfuscate Proxy Password

Add a utility to the New Relic CLI that allows a user to encode their proxy password using their key. It is being added to the New Relic CLI so that other agent teams may take advantage of it.

Feature Description

Add a utility to the New Relic CLI that allows a user to encode their proxy password using their key. It is being added to the New Relic CLI so that other agent teams may take advantage of it.

Describe Alternatives

• Adding a CLI to the .NET Agent
• Building a Nerdlet for this task

Additional context

Customers want to encrypt/obfuscate the proxy password so that it cannot be discovered from the config file.

MVC Invalid Action for valid Controller can cause MGI

Description
If an invalid method (controller action) is specified in a route, but the controller is valid, the system records a transaction. This can be problematic when there are phishing attempts and/or security audits of sites looking for vulnerabilities.

Expected Behavior
This is something that still needs to be decided. Should we just not apply the MVC controller-action naming, or should we provide some sort of "invalid" controller-action name that allows these types of requests to be rolled up. I do not think that these types of requests should be marked as ignored.

Troubleshooting or NR Diag results
N/A

Steps to Reproduce
Simply make a request that is routed to a valid controller but doesn't have a method for the action that's requested.

Your Environment
N/A

Additional context
See internal link for the original issue.

For example:
Account/Edit/32, Valid controller, valid action

Account/XYZABDCDRE, Valid controller, invalid action → still

CONT1234/Edit, invalid controller → Should double check, but likely covered by current agent behavior

Impact

While this does not happen very often, there isn’t really a workaround to help customer experiencing issue.

InstrumentationWatcher can cause ThreadAbortException

InstrumentationWatcher uses a worker thread to monitor for instrumentation configuration changes. It uses SignableAction to request a Rejit when a change is detected. The worker thread gets aborted on Dispose.

Description
This exception creates an annoying distraction in the Agent log, and Exceptions are a performance hit. In this case, the work is already done, the worker thread is no longer needed, and the exception is moot. Dispose() should not abort the worker thread.

Suggested implementation: Instead of _worker.Abort(); in Dispose() do something like

_doShutdown = true;
Monitor.Pulse(_lock);

then check _doShutdown in the Action's while-loop.

Expected Behavior
Dispose() shuts down worker thread gracefully; no ThreadAbortException appears in Agent log.

Steps to Reproduce

• Create Agent-monitored app and add custom instrumentation using the 'XML custom instrumentation' method. You may want to have at least 2 methods in the app for which you will add custom instrumentation because you will need to add and/or remove instrumentation while it is running.
• Run the app. While it is running, change the custom instrumentation to add or remove an instrumented method by editing your .xml file.
• The ThreadAbortException may be intermittent and depend on timing, so if you don't see it in the Agent log, try changing custom instrumentation several times while the app is running to have a better chance of generating the exception.

Your Environment
This issue can occur with any instrumented application utilizing custom instrumentation regardless of framework (.NET or .NET Core) and app type (Console, Web, Mvc, etc.)

Additional context
Testing: Add unit test in SignalableActionTests.

NewRelic.AppName not being read from the correct appsettings.{environment}.json file

Description

Setting the Microsoft recommended ASPNETCORE_ENVIRONMENT system environment variable doesn't work for the NewRelic .Net Core Agent v 8.29.0.0 in a .Net Core 3.x project.

It appears that the New Relic Agent looks at a different EnvironmentName environment variable to determine which overriding appsettings.{environment}.json file to use for the NewRelic.AppName configuration setting.

Reading the Environment System Environment Variables

newrelic-dotnet-agent/src/Agent/NewRelic/Agent/Core/Configuration/AppSettingsConfigResolveWhenUsed.cs

Line 29 in 0dcbe00

var environment = env.GetEnvironmentVariable("EnvironmentName");

Overridding the appsettings.json Settings per Environment

newrelic-dotnet-agent/src/Agent/NewRelic/Agent/Core/Configuration/AppSettingsConfigResolveWhenUsed.cs

Line 35 in b4918c4

if (!string.IsNullOrEmpty(environment)) builder.AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: false);

Steps to Reproduce

In a .Net Core 3.x application using appsettings.*.json files to specify the NewRelic.AppName:

In appsettings.json specify:

{
  "NewRelic.AppName": "Local"
}

In appsettings.Staging.json specify:

{
  "NewRelic.AppName": "Staging"
}

Deploy to an environment with the Environment variable ASPNETCORE_ENVIRONMENT = Staging set.

See that the value for NewRelic.AppName will be Local instead of the correct Staging value.

Expected Behaviour

I would expect that the Microsoft recommended ASPNETCORE_ENVIRONMENT would define the environment instead of the current EnvironmentName being used.

            // Something like this should provide backwards compatibility for people using EnvironmentName
            var environment = env.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
            if (string.IsNullOrEmpty(environment))
                environment = env.GetEnvironmentVariable("EnvironmentName");

Your Environment

Running a .Net Core 3.x Web Application on a Windows Server using IIS as an InProc site.

GitHub Actions Migration

To improve the experience for external contributors and standardize engineering processes across the New Relic instrumentation agents we will be making improvements to our CI and migrating to GitHub Actions.

Aha! Link: https://newrelic.aha.io/features/DOTNET-270

GC Performance Counters for Windows improperly assumes Instance Name Can't change

When collecting Garbage Collection performance counters for Windows, the agent, in part, uses the application's Instance Name to disambiguate different instances of the same running application. For example, two instances of an application named "Test" may appear as Test and Test#1.

To identify which Instance name belongs to the currently executing process, a reverse lookup is made to the performance counter Process ID and is matched to the Process ID (PID) of the current process. With this instance name, Garbage Collection Performance Counters are created and a reference to each of them is maintained in memory.

On regular intervals, these performance counter references are used to read the current values.

Description

• The Instance Name is dynamic and can change throughout the life cycle of an application. If the instance name changes, two one of two things can occur: (A) Most likely, if the instance name no longer exists, an error will occur; (B) The perf counters for the wrong instance of the application will be read.

Expected Behavior

• Agent always collects the perf counters for the current app and does not ever fail when multiple instances of the same app are running.

Steps to Reproduce

• Start two instances of this program.
  • You should get TestProgram and TestProgram #1 as instance names.
• Terminate instance called TestProgram
• Notice TestProgram#1 will experience an error because the performance counter can no longer be fetched.

class Program
{
	static void Main(string[] args)
	{
		while (true)
		{
			var perfCounters = CreateCounters();
			perfCounters = perfCounters.ToList();
			while (true)
			{
				ReadValues(perfCounters);
				Thread.Sleep(TimeSpan.FromSeconds(2));
			}
		}
	}

	private static readonly string[] _perfCounterNames = new[]
	{
		"Gen 0 heap size",			
		"Promoted Memory from Gen 0",
		"Gen 1 heap size",			
		"Promoted Memory from Gen 1",
		"Gen 2 heap size",			
		"Large Object Heap size",	
		"# GC Handles",				
		"# Induced GC",				
		"% Time in GC",				
		"# Gen 0 Collections",		
		"# Gen 1 Collections",		
		"# Gen 2 Collections"		
	};

	private const string perfCatName = ".NET CLR Memory";

	private static IEnumerable<PerformanceCounter> CreateCounters()
	{
		var process = Process.GetCurrentProcess();
		var processName = process.ProcessName;
		var pid = process.Id;
		var perfCat = new PerformanceCounterCategory(perfCatName);

		var instanceNames = perfCat.GetInstanceNames().Where(x => x.StartsWith(processName));
		var instanceName = default(string);

		foreach (var instNm in instanceNames)
		{
			try
			{
				using (var pcPid = new PerformanceCounter(perfCatName, "Process ID", instNm))
				{
					if ((int)pcPid.NextValue() == pid)
					{
						instanceName = instNm;
						break;
					}
				}
			}
			catch (Exception ex)
			{
				//Log a message here and continue because this instance may not be relevant to the process that we are looking for.
				Console.WriteLine($"Error determining if instance '{instNm}' is process '{processName}'({pid}). Error: {ex}");
			}
		}
	
		if (instanceName == null)
		{
			throw new Exception("Unable to obtain instance name");
		}

		Console.WriteLine($"InstanceName: {instanceName ?? "NULL"}");

		foreach (var perfCounterName in _perfCounterNames)
		{
			yield return new PerformanceCounter(perfCatName, perfCounterName, instanceName);
		}
	}

	private static void ReadValues(IEnumerable<PerformanceCounter> perfCounters)
	{
		Console.WriteLine();

		foreach (var perfCounter in perfCounters)
		{
			var val = perfCounter.NextValue();
			Console.WriteLine($"{perfCounter.CounterName,-50}{val}");
		}
	}	
}

Your Environment
Windows, .NET Framework 4.7

Stop using Exceptions for flow control

Using exceptions for flow control is an anti-pattern.

Description
This Issue is to refactor MetricNameService to remove the use of IgnoreTransactionException in RenameUsingRegexRules().

Expected Behavior
Behavior should not change.

Your Environment
This issue can occur with any instrumented application regardless of framework (.NET or .NET Core) and app type (Console, Web, Mvc, etc.)

Additional context
It is possible the IgnoreTransactionException can be eliminated completely, it appears to only be used in this context.
Testing: Existing MetricNameServiceTest will need to be adjusted.

AwsLambda tracer platform tests fail consistently.

Description
AwsLambda related platform tests has been failing consistently. These should be fixed.

Modify MySQL integration tests to use a local Docker-hosted service

Currently, some of these tests rely on a MySQL server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

https://hub.docker.com/_/mysql

External Calls create MGI

Description

External/12000etreshumains-rh.profils.org/Stream/GET
ExternalApp/webhelp.talent-soft.com/252541#17467371/all
ExternalTransaction/zebulon.talent-soft.com/252541#17467271/WebTransaction/WebAPI/Statistics/GetUiCultureOfUsers

->

External/*/Stream/GET
ExternalApp/*/252541#17467371/all
ExternalTransaction/*/252541#17467271/WebTransaction/WebAPI/Statistics/GetUiCultureOfUsers

Expected Behavior

**Troubleshooting **

Steps to Reproduce

Your Environment

Additional context

** MISC **
[Internal Process]
Zendesk

The initial investigation leads to this as more of an agent spec problem with CAT

Modify MongoDB unbounded integration tests to use a local Docker container

Currently, some of these tests rely on a MongoDB server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Docker Hub entry for Mongo: https://hub.docker.com/_/mongo

Linux systems do not report total physical memory on agent connect

On agent connect the following utilization hash (example below) is included in the connect payload. On linux systems total_ram_mib is always null. This is because the code that pulls this information only works on Windows.

See:

• https://github.com/newrelic/newrelic-dotnet-agent/blob/main/src/Agent/NewRelic/Agent/Core/Utilities/SystemInfo.cs#L29-L46
• https://github.com/newrelic/newrelic-dotnet-agent/blob/main/src/Agent/NewRelic/Agent/Core/Utilities/SystemInfo.cs#L123-L124

I have not reviewed this discussion very closely, yet, though it appears there is an open discussion around an API to retrieve this data: dotnet/runtime#22948. To my knowledge, there is nothing on System.Diagnostics.Process that gets us physical memory, though this should be confirmed.

However, even if there was an .NET API to retrieve this information, containerized environments may cause us additional problems.

Example utilization hash:

"utilization": {
    "metadata_version": 5,
    "logical_processors": 1,
    "total_ram_mib": null,
    "hostname": "...",
    "full_hostname": "...",
    "ip_address": [
        "..."
    ],
    "boot_id": "...",
    "vendors": {
        "azure": {
            "location": "...",
            "name": "...",
            "vmId": "...",
            "vmSize": "Standard_D1_v2"
        },
        "docker": {
            "id": "..."
        },
        "kubernetes": {
            "kubernetes_service_host": "..."
        }
    }
},

YUM Upgrade does not maintain newrelic.config or setenv.sh files

Description
When upgrading agent via YUM, it does not maintain the existing global newrelic.config. This is problematic as customers may use the global newrelic.config to set their license key, and or to configure options like Distributed Tracing.

Expected Behavior
When upgrading using YUM, if there is an existing newrelic.config file, it would be maintained, not overwritten.

Steps to Reproduce

• Create a CentOS VM and/or container
• Using these instructions as a guide, install an older version of the agent. You'll need to install an older version so that you can upgrade it.

cat << REPO | sudo tee "/etc/yum.repos.d/newrelic-netcore20-agent.repo"
[newrelic-netcore20-agent-repo]
name=New Relic .NET Core packages for Enterprise Linux
baseurl=http://yum.newrelic.com/pub/newrelic/el7/\$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-NewRelic
REPO

In this example, install the v8.28 version of the agent (current release is v8.30).

sudo yum install newrelic-netcore20-agent-8.28.0.0-1

• Modify the newrelic.config file. In this example, the licence key was modified, distributed tracing service was enabled, and Cross Application Tracing was disabled.

<!-- For more information see: https://newrelic.com/docs/dotnet/dotnet-agent-configuration -->
<configuration xmlns="urn:newrelic-config" agentEnabled="true">
        <service licenseKey="HERES MY NEW LICENCE KEY" />
        <application>
                <name>My Application</name>
        </application>
        <log level="info"/>
        <transactionTracer enabled="true"
                transactionThreshold="apdex_f"
                stackTraceThreshold="500"
                recordSql="obfuscated"
                explainEnabled="false"
                explainThreshold="500"/>
        <crossApplicationTracer enabled="false"/>
        <distributedTracing enabled="true"/>
        <errorCollector enabled="true">
                <ignoreErrors>
                        <exception>System.IO.FileNotFoundException</exception>
                        <exception>System.Threading.ThreadAbortException</exception>
                </ignoreErrors>
                <ignoreStatusCodes>
                        <code>401</code>
                        <code>404</code>
                </ignoreStatusCodes>
        </errorCollector>
        <browserMonitoring autoInstrument="true" />
        <threadProfiling>
                <ignoreMethod>System.Threading.WaitHandle:InternalWaitOne</ignoreMethod>
                <ignoreMethod>System.Threading.WaitHandle:WaitAny</ignoreMethod>
        </threadProfiling>
</configuration>

• Using these instructions, update the agent to the current version

sudo yum update newrelic-netcore20-agent

• Re-open the system wide newrelic.config file. Notice that it has been overwritten.

Your Environment
CentOS virtual machine.

OpenTelemetry Exporter (Metrics)

OpenTelemetry provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. New Relic's OpenTelemetry exporters send your OpenTelemetry data to your New Relic account.

Our existing exporter currently only supports traces so we will add support for Metrics in the exporter.

Add password obfuscation functionality to the newrelic-cli.

Feature Description

Users will use the newrelic-cli to obfuscate a proxy password which the agent knows how to deobfuscate.

Describe Alternatives

Proposed command newrelic agent obfuscate-config-value --value pa$$word --obscuring-key 'this is my key'.
Writing the obscured value to agent configuration file is optional.

Logging response headers from New Relic endpoint to DEBUG logs.

Feature Description

When we send data up to New Relic, the response headers that we get back from New Relic have been useful for debugging issues like missing data due to 503s, it's useful to know if the issues were isolated to a particular geo-location. Additionally, if the failure is Cloudflare related, Cloudflare is going to need a ray ID and request ID to debug these issues.

Currently, for the .NET agent, we only logged the response body.

Priority:
Low - Nice to have (visited 9/13/2022)

Update Async Timing Calculations

Description
The agent does not deduct the duration of asynchronous methods from parent span/segment/methods' exclusive duration, which leads to misleading durations reported when those asynchronous child methods are called in a synchronous manner.

One such scenario is ASP.NET Web API. We instrument ExecuteRequestHandler that calls System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync which we also instrument. InvokeActionAsync is an async method, so the agent does not deduct its time from the time of ExecuteRequestHandler. Since we know ExecuteRequestHandler actually does block on InvokeActionAsync it probably makes sense to special case these types of scenarios.

Expected Behavior
The reported exclusive duration of common ASP.NET middleware pipeline steps like ExecuteRequestHandler should not include the time spent in blocking async child methods like InvokeActionAsync.

Internal NR only:
ref1
ref2
ref3

Reduce string allocations in BrowserMonitoringWriter

String allocations are expensive. There are ways to reduce this performance hit when working with strings in .NET. This issue is to examine and improve use of strings in a specific place in the Agent code - BrowserMonitoringWriter.

Description
Examples of inefficient use of strings:

• This method sets a string bodyWithJsHeader to String.Empty at the start, but it gets replaced later if method returns early. An improvement would be to return String.Empty from the early return and return constructed string otherwise, avoiding allocation of String.Empty.
• In same method, rather than allocating jsScriptWithBodyPrefix and using content.Replace, use string interpolation and a const.
• In same method, return the constructed string from above directly instead of storing it in a local string variable.
• Where possible, use consts (which occur at compile-time instead of run-time) for strings that will not change.

Expected Behavior
Current functioning should not change. Ideally, these changes would improve performance for apps using this feature of the Agent. See "Additional context: Testing" below.

Steps to Reproduce
N/A

Your Environment
Instrumented app using Agent's browser injection.

Additional context
Testing: Existing unit tests should be sufficient for this Issue to find regressions. Additionally, a test app making heavy use of the browser monitoring feature of the agent could be used to compare before and after scenarios for memory use and garbage collection.

Unexpected thread usage increase

Description
Something is causing web applications instrumented by the agent to periodically use roughly 500 threads instead of around 50-60 threads.

Expected Behavior
We expect thread usage to increase when the agent is running because of the native threads that are required for certain .NET profiling API calls, metric sampling (CPU, Memory, Garbage Collection, Thread Info), sending data to New Relic, and continuation based async timing, but it should not increase the thread count by a large amount.

Troubleshooting or NR Diag results
See this internal link for some of the troubleshooting that has been done previously to try discover what's causing the spike in thread usage. The link details which agent features and instrumentation that were disabled, as well as the agent settings that were changed, to try to minimize the increase in thread usage.

Steps to Reproduce
N/A

Your Environment
N/A

Additional context
I've seen this thread explosion from time to time with some test applications during performance testing. However, I do not yet understand what is exactly triggering this explosion. Is it just a combination of circumstances such as: 1. A certain amount of load within the application, 2. A certain amount of load on the system running the application, 3. Samplers, Harvests, transactions completing, and async continuations needing to execute around the same time, 4. Is this happening after a bunch of work was delayed due to a blocking GC?

Exception when distributed tracing is enabled and using ASP.NET Core RequestLocalization middleware

Description

Agent causes exception when distributed tracing is enabled and ASP.NET Core application uses the RequestLocalization middleware. Appears to affect ASP.NET Core apps running on Linux in a docker container.

Steps to Reproduce

Create new Web API

mkdir locale
cd locale
dotnet new webapi

Add the RequestLocalization middleware in Startup.cs

var supportedCultures = new[]
{
    new CultureInfo("en-US"),
    new CultureInfo("ar"),
};

app.UseRequestLocalization(new RequestLocalizationOptions
{
    DefaultRequestCulture = new RequestCulture("en-US"),
    SupportedCultures = supportedCultures
});

Add the following controller

using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;

namespace RequestLocalization.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class TestController : ControllerBase
    {
        [HttpGet]
        public async Task<string> Get()
        {
            using (var client = new System.Net.Http.HttpClient())
            {
                return await client.GetStringAsync("https://newrelic.com");
            }
        }
    }
}

Create a Dockerfile and replace put_license_key_here

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1.0

RUN curl -O https://download.newrelic.com/dot_net_agent/latest_release/newrelic-netcore20-agent_8.29.0.0_amd64.deb
RUN dpkg -i ./newrelic-netcore20-agent*.deb

ENV CORECLR_NEWRELIC_HOME /usr/local/newrelic-netcore20-agent
ENV NEW_RELIC_LICENSE_KEY put_license_key_here

WORKDIR /app
COPY ./bin/Debug/netcoreapp3.1/publish .

EXPOSE 5000

ENTRYPOINT ["/bin/bash", "-c", "dotnet locale.dll"]

Create a docker-compose.yml file

version: "3"

services:
  locale:
    image: locale:latest
    ports:
      - 5000:5000
    environment:
      ASPNETCORE_URLS: http://*:5000
      ASPNETCORE_ENVIRONMENT: Development
      CORECLR_ENABLE_PROFILING: 1
      CORECLR_PROFILER: "{36032161-FFC0-4B61-B559-F6C5D41BAE5A}"
      CORECLR_PROFILER_PATH: /usr/local/newrelic-netcore20-agent/libNewRelicProfiler.so
      NEW_RELIC_DISTRIBUTED_TRACING_ENABLED: "true"

Publish and run the application

dotnet publish
docker build -t locale .
docker-compose up

Hit the endpoint

curl -s -H 'Accept-Language: ar' -XGET http://localhost:5000/Test

You will get the following exception:

System.Net.Http.HttpRequestException: Request headers must contain only ASCII characters.

Modify Oracle integration tests to use a local Docker-hosted service

Currently, some of these tests rely on an Oracle server hosted inside New Relic's network.

In order to enable community contributors to run these tests, the plan is to provide a Dockerfile/docker-compose solution for spinning up the necessary external dependencies on a developer's system before running the tests.

Oracle does not make a freely-available docker image available on Docker Hub. However, they do have a GitHub repo with scripts to help you build a Docker image if you already have the install binaries for the database server: https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md

The idea is that we'll figure out how to build a container image that works for our use case and then figure out how to host it somewhere it can be pulled from easily.

• Tried using GitHub Packages to host, but pulling requires being logged in with a GitHub access token which is too much work to ask community contributors to go through
• Will look into hosting on Docker Hub with New Relic's account: https://hub.docker.com/u/newrelic

Sub-tasks:
[X] Figure out how to build a container for Oracle DB using Oracle's scripts
[X] Make the container work for our tests
[] Figure out how to host the container publicly

.NET Open Source Agent - Milestone 2 - Integration tests can be run by external contributors

Ensure that an external contributor can run integration tests on a local machine.

Aha! Link: https://newrelic.aha.io/features/DOTNET-276

Postgres Instrumentation - Instrument NpgSqlCmd.ExecuteReaderAsync

Is your feature request related to a problem? Please describe.

Need to instrument an additional path for NpgSql 4.1.3.1

Feature Description

NpgSql 4.1.3.1

using (var connection = new NpgsqlConnection(_connectionString))
           using (var command = new NpgsqlCommand("SELECT * FROM public.TestGTSE1877(:arg0,:arg1,:arg2);", connection))
			{
				connection.Open();
				command.Parameters.AddWithValue(":arg0", "Jason");
				command.Parameters.AddWithValue(":arg1", "Jason");
				command.Parameters.AddWithValue(":arg2", 32);

				using (var reader = command.ExecuteReader())
				{

The above sample code did not get instrumented by default.
Upon following the execution path, it seems that

NpgSqlCmd.ExecuteReader() 
   NpgSqlCmd.ExecuteReader(CmdBehavior)
      NpgSqlCmd.ExecuteReaderAsync(CmdBehavior, bool, CancellationToken)

This path is not being instrumented.

Describe Alternatives

N/A

Additional context

Originated from DOTNET 4668
The reporter was Jason F.

Priority

Please help us better understand this feature request by choosing a priority from the following options:
[Really Want]

Make Spans More useful - Parent Attributes

Is your feature request related to a problem? Please describe.

A clear and concise description of the problem. For example, "I'm always frustrated when [...]".

Feature Description

Describe Alternatives

Additional context

Originated from DOTNET-4755
The reporter of the problem was Jason F.

Priority

[Nice to Have]