Description

During Soak Tests execution, a performance degradation was revealed for commit 259b8ce of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Description

During Soak Tests execution, a performance degradation was revealed for commit c73cd98 of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Description

During Soak Tests execution, a performance degradation was revealed for commit c73cd98 of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Currently, ADOT .NET SDK initializes Random instance on each thread to generate random number, which is not the best practice of what Microsoft suggests.

Description

During Soak Tests execution, a performance degradation was revealed for commit a6c4dca of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Description

During Soak Tests execution, a performance degradation was revealed for commit 8f0b2dd of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

I'm using ADOT .NET SDK and the ADOT Collector.

I can view traces in X-Ray, however related logs aren't shown.

Logs
All logs for this trace

No data found.
Try adjusting the dashboard time range or log query.

If I tap 'View in CloudWatch Insights' it'll materialize a query which returns no logs:

fields @log, @timestamp, @message
| filter @message like "1-620ca1d6-07f7494e51e410242c9d67f3"
| sort @timestamp, @message desc

however the correct query, fixing the trace ID to remove the leading digit and hyphens, returns related logs:

fields @log, @timestamp, @message
| filter @message like "620ca1d607f7494e51e410242c9d67f3"
| sort @timestamp, @message desc

Is this expected behaviour?

System info

    <PackageVersion Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.1.0" />
    <PackageVersion Include="OpenTelemetry.Contrib.Instrumentation.AWS" Version="1.0.1" />
    <PackageVersion Include="OpenTelemetry.Exporter.Console" Version="1.1.0" />
    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.1.0" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9" />

amazon/aws-otel-collector:latest

Hi Team,

We have implemented tracing with AWS Distro for OpenTelemetry .Net SDK following this instruction .
Our service is running on AWS EKS and one thing we noticed is there is no resource information collected after we added AWSEKSResourceDetector to our code. See below code:

           Sdk.CreateTracerProviderBuilder()
                .AddSource("psgrpcsvc")
                .SetResourceBuilder(
                    ResourceBuilder.CreateDefault()
                        .AddService("psgrpcsvc")
                        .AddDetector(new AWSEKSResourceDetector()))
                .AddXRayTraceId()
                .AddAspNetCoreInstrumentation()
                .AddNpgsql()
                .AddOtlpExporter()
                .Build();
            Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

Package versions:

    <PackageReference Include="Npgsql.OpenTelemetry" Version="7.0.1" />
    <PackageReference Include="OpenTelemetry" Version="1.3.2" />
    <PackageReference Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.2.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.3.2" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc7" />

Are we missing anything in our code or is there anything needed on AWS EKS infrastructure side we may miss(for example allowing EKS metadata to be read) in order to use the AWSEKSResourceDetector?

When .NET SDK goes GA for metrics make sure that:

1. The sample and tests are up to date.
2. The relevant parts of the ADOT docs are updated.

Note: this SDK is a dependency of our ADOT Lambda offering.

Description

During Soak Tests execution, a performance degradation was revealed for commit 259b8ce of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Description

During Soak Tests execution, a performance degradation was revealed for commit 8f0b2dd of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

I've created an AWS support ticket under enterprise support as well, but that's having troubles so figured I'd also open an issue here since it more directly relates to this OTEL library.

We have a .NET application using OTEL, including this package. We originally just had SSM plugged in and that was working fine - we recently added ASG calls to the application and immediately ran into issues. With the AWS instrumentation enabled, the constructor for the ASG client immediately throws an exception.

Exception message:
"Cannot find a handler of type EndpointResolver"

Stack Trace:
at Amazon.Runtime.Internal.RuntimePipeline.AddHandlerAfter[T](IPipelineHandler handler)
at Amazon.Runtime.Internal.RuntimePipelineCustomizerRegistry.ApplyCustomizations(Type type, RuntimePipeline pipeline)
----Removed line referencing the method call in Program.cs since this has personal information in it----

Steps to Reproduce:
It's extremely simple to reproduce and doesn't require any credentials/IAM role/etc:

1. Create a blank console .NET app, I did .NET 6.
2. Add AWSSDK, AWSSDK.AutoScaling, core OTEL packages, and the AWS OTEL package:
   
3. Add initialization code for OTEL, include the AWS instrumentation.
4. Attempt to create an AmazonAutoScalingClient
5. Exception is thrown
6. Comment out line for AWS instrumentation
7. Rerun
8. Exception is no longer thrown (need credentials at this point).

This is what an entire test app looks like:

I've attached a ZIP here containing an extremely simple console app that has all of the above - I understand the security around arbitrarily running code uploaded by someone random, but it can be used as a reference for creating your own console app. Commenting out line 15 (.AddAWSInstrumentation()) makes the ASG client work fine.

AWSSDKTest - minimal repo.zip

Let me know if I can provide any other information!

I have well maintained OTEL providers for just every service and package in my application, save for the most obvious one AWS. The published SDK relies on far outdated AWS packages so we've had to fork & maintain our own just to track SQS & S3 operations on the worlds biggest & most widely used cloud. While I get that everyone at AWS is busy with things, the idea that AWS observability on dotnet is so outdated is a really bad look. It implies that dotnet developers are third-class citizens on the only cloud they might consider beyond Azure.

Description

During Soak Tests execution, a performance degradation was revealed for commit dc8a7d2 of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

Update the .NET sample app to have metric parity with the Java spark sample app.

Description

After the Soak Tests completed, a performance degradation was revealed for commit b9ca772 of the refs/heads/main branch for the (microsoft_dotnet, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

The threshold violation should also be noticeable on our graph of Soak Test average results per commit.