aws-observability / aws-otel-go Goto Github PK

View Code? Open in Web Editor NEW

34.0 34.0 22.0 10.48 MB

AWS Distro for OpenTelemetry Go

Home Page: https://aws-otel.github.io/

License: Apache License 2.0

Go 93.10% Dockerfile 6.90%

observability opensource opentelemetry opentelemetry-api

aws-otel-go's People

Contributors

Stargazers

Watchers

aws-otel-go's Issues

Test Template

Performance Threshold breached AFTER Soak Tests completed for the (golang_alpine, manual) Sample App

Description

After the Soak Tests completed, a performance degradation was revealed for commit 030cda7 of the refs/heads/main branch for the (golang_alpine, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

The threshold violation should also be noticeable on our graph of Soak Test average results per commit.

Performance Threshold breached AFTER Soak Tests completed for the (golang_alpine, manual) Sample App

Description

After the Soak Tests completed, a performance degradation was revealed for commit 5f23ffa of the refs/heads/main branch for the (golang_alpine, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

The threshold violation should also be noticeable on our graph of Soak Test average results per commit.

Remote resources not visible on X-Ray map

Hi !

I have an application connecting to a SQL database, and since there's not yet support for database/sql auto-instrumentation, I did a custom tracing implementation.

tracer := otel.GetTracerProvider().Tracer("petlistadoptions")
_, span := tracer.Start(ctx, "mssql query")

span.SetAttributes(
  label.String("sql", sql),
  label.String("url", r.safeConnStr),
)

rows, err := r.db.Query(sql)
if err != nil {
  logger.Log("error", err)
  return nil, err
}
span.End()

The link to the database do not appear as a remote resource and doesn't show up in the trace map

Environment details :

Go 1.15
Running on AWS Fargate, with amazon/aws-otel-collector:latest as sidecar
go.opentelemetry.io/otel v0.15.0

Raw trace data

{
    "Id": "1-5ffc45c2-4b5e80c8040c405153b5c4ec",
    "Duration": 0.034,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "7af254d953cd2c47",
            "Document": {
                "id": "7af254d953cd2c47",
                "name": "PetSearch",
                "start_time": 1610368450.014082,
                "trace_id": "1-5ffc45c2-4b5e80c8040c405153b5c4ec",
                "end_time": 1610368450.025903,
                "parent_id": "2a898f4841386dfe",
                "http": {
                    "request": {
                        "url": "http://servi-searc-8256xjtfxppf-1893318472.eu-west-1.elb.amazonaws.com/api/search?petid=005",
                        "method": "GET",
                        "user_agent": "Go-http-client/1.1",
                        "client_ip": "54.78.128.12",
                        "x_forwarded_for": true
                    },
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "xray": {
                        "sdk_version": "2.6.2",
                        "sdk": "X-Ray for .NET Core"
                    }
                },
                "metadata": {
                    "default": {
                        "Pets": "[{\"pettype\":\"puppy\",\"petid\":\"005\",\"price\":\"59\",\"petcolor\":\"brown\",\"cuteness_rate\":\"3\",\"availability\":\"no\",\"image\":null,\"peturl\":\"https://services-s3bucketpetadoptioncb20dce5-u0k3dtb800rk.s3.eu-west-1.amazonaws.com/puppies/p3.jpg?X-Amz-Expires=300\\u0026x-amz-security-token=IQoJb3JpZ2luX2VjENz%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCWV1LXdlc3QtMSJGMEQCIA0s%2FJhni7lUj8SmNHE%2BwqlgZx9s%2F1f2F6amHUEZMajnAiAjiyECmqKZ%2B94PPaKVh8kyZrB9jh3M1W3IrCrR%2BvLG8iroAwil%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAEaDDMzOTc0MzEwMzcxNyIMUN8WYjKrVgBInf6PKrwDj%2BcHeNIr1tJ9WQkMqYrl5ytkFx3x%2FXDpz718AQLFLcnqj79gHuO%2BhQrBlLp%2BZooIGDCJzoNIzAQ%2FfZVBQ5Wdi3%2F0yYBtmme7rQ3Z6nXiihjYV2W8OPWGeTR1ANx6E4zM%2BQ%2BBPUMyWkzj542aazM9CE5TWqDmIaNw%2BkvRi7X6MoguGiQ8olXzovQCi5ugDZLxwsyi5rOEkqJahnTmQovTLETJHzAndRYGl6eapW9xRKmFCBpbrOt2tVnlpdovdCHhAvSSKAAbd1i3MHvnkCDksN0cUGYtnNo%2FKN%2FNQn0GNsZlAlyftPNQPBJXoOrCN7mtXrRksjya2GWLN9tMk2%2BeYUvEfaz7%2FhkCOnO6uXgAsmb4cP0ldOFskPyLAJ%2FyHXbpbOH35vOde31uXygvx1eZVRqIihNv6JKGMcWSGg82pqqtRpmEDddgyxNwdwTgUczn26FGWURFR%2Bo6rWCK1SN2K65Bc3oR9FJ1OpMDMGhA35phBDdbOBtI38qPGkgTjlbhywAgNvKn9aMJK6Ypy1cr%2FkH2eipzi%2FcDW6leCNsqbv8d5JCMnRbkXE7%2FuNWZlTbx1P2I2b2wQdITRvnYMNT48P8FOu0BUH8hri3555CU8KXxKl9%2BCFUKvEod0Sy4CJx6pbIpo2e2gjdLVCvG5w2LzJOsPlBK%2Fmz77tdDiRvEeXBQpPGrl0gyBs1PSusjbu73vG2xW2lpEGsaeCXLFxl1YsenyAFAvoXsg69pstGBUDFYcYfMIE8q4dNt1Rxsyjl3p3mtOR4eM5js%2FtzvYy%2F56EsEJ06cf3bggfOY03Kr3cyB08A%2BHHPPGkX72QWEkEjOirYeU51M7UasRcn%2FlMgrZgYwCg8oY1%2FX5zr9Cx6NvDQt5tnCFkCrrg7X0ejXcJFe6iQXPakT6MKbQQ%2BpKP4dDU5k\\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\\u0026X-Amz-Credential=ASIAU6GSELLS6NTJ45P5/20210111/eu-west-1/s3/aws4_request\\u0026X-Amz-Date=20210111T123410Z\\u0026X-Amz-SignedHeaders=host;x-amz-security-token\\u0026X-Amz-Signature=aea598ef6e8b4f26e6cd76e26934196c2805a84d19ac552bc03e666ce068f34c\"}]"
                    }
                },
                "service": {
                    "runtime": ".NET Core Framework",
                    "runtime_version": "3.1.10"
                },
                "subsegments": [
                    {
                        "id": "f82385679a9ae1f3",
                        "name": "Scanning DynamoDB Table",
                        "start_time": 1610368450.017318,
                        "end_time": 1610368450.022804,
                        "annotations": {
                            "Query": "petcolor:-pettype:-petid:005"
                        },
                        "subsegments": [
                            {
                                "id": "be7d4a3b16ec486a",
                                "name": "DynamoDB",
                                "start_time": 1610368450.017891,
                                "end_time": 1610368450.02277,
                                "http": {
                                    "response": {
                                        "status": 200,
                                        "content_length": 200
                                    }
                                },
                                "aws": {
                                    "select": null,
                                    "count": 1,
                                    "table_name": "Services-ddbpetadoption7B7CFEC9-P94QSTUX451H",
                                    "resource_names": [
                                        "Services-ddbpetadoption7B7CFEC9-P94QSTUX451H"
                                    ],
                                    "consistent_read": false,
                                    "consumed_capacity": null,
                                    "scanned_count": 26,
                                    "segment": 0,
                                    "limit": 0,
                                    "attributes_to_get": [],
                                    "projection_expression": null,
                                    "region": "eu-west-1",
                                    "operation": "Scan",
                                    "request_id": "Q5BOGNJR23DJ6UQSC71MV3RGV7VV4KQNSO5AEMVJF66Q9ASUAAJG",
                                    "total_segments": 0,
                                    "index_name": null
                                },
                                "namespace": "aws"
                            }
                        ]
                    }
                ]
            }
        },
        {
            "Id": "507aa886d62e8df8",
            "Document": {
                "id": "507aa886d62e8df8",
                "name": "servi-lista-pwhg5v3y9jol-1655753230.eu-west-1.elb.amazonaws.com",
                "start_time": 1610368450.0068173,
                "trace_id": "1-5ffc45c2-4b5e80c8040c405153b5c4ec",
                "end_time": 1610368450.0271156,
                "parent_id": "1adf1081d3fbd287",
                "fault": false,
                "error": false,
                "http": {
                    "request": {
                        "url": "http://servi-lista-pwhg5v3y9jol-1655753230.eu-west-1.elb.amazonaws.com/api/adoptionlist/",
                        "method": "GET",
                        "client_ip": "54.78.128.12",
                        "x_forwarded_for": true
                    },
                    "response": {
                        "status": 200,
                        "content_length": 0
                    }
                },
                "aws": {
                    "xray": {
                        "auto_instrumentation": false
                    }
                },
                "metadata": {
                    "default": {
                        "net.transport": "IP.TCP",
                        "resultCount": "",
                        "http.flavor": "1.1",
                        "http.route": "/api/adoptionlist/",
                        "timeTakenSeconds": ""
                    }
                },
                "subsegments": [
                    {
                        "id": "79e5dddc6e67ba9e",
                        "name": "mssql query",
                        "start_time": 1610368450.0068529,
                        "end_time": 1610368450.0116804,
                        "fault": false,
                        "error": false,
                        "aws": {
                            "xray": {
                                "auto_instrumentation": false
                            }
                        },
                        "metadata": {
                            "default": {
                                "url": "sqlserver://xxx:@si5zoymobm0wws.xxx.eu-west-1.rds.amazonaws.com:1433?database=adoptions",
                                "sql": "SELECT TOP 25 PetId, Transaction_Id, Adoption_Date FROM dbo.transactions"
                            }
                        }
                    }
                ]
            }
        },
        {
            "Id": "a91bb14030a5daed",
            "Document": {
                "id": "a91bb14030a5daed",
                "name": "PetSite",
                "start_time": 1610368450.001682,
                "trace_id": "1-5ffc45c2-4b5e80c8040c405153b5c4ec",
                "end_time": 1610368450.035526,
                "http": {
                    "request": {
                        "url": "http://servi-petsi-ga3fszv5gykz-1180376689.eu-west-1.elb.amazonaws.com/PetListAdoptions",
                        "method": "GET",
                        "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
                        "client_ip": "90.79.92.140",
                        "x_forwarded_for": true
                    },
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "ecs": {
                        "container": "ip-11-0-248-241.eu-west-1.compute.internal"
                    },
                    "xray": {
                        "sampling_rule_name": "Default",
                        "sdk_version": "2.9.0",
                        "sdk": "X-Ray for .NET Core"
                    }
                },
                "service": {
                    "runtime": ".NET Core Framework",
                    "runtime_version": "3.1.10"
                },
                "origin": "AWS::ECS::Container",
                "subsegments": [
                    {
                        "id": "d732014dd9edb9e7",
                        "name": "Calling PetListAdoptions",
                        "start_time": 1610368450.001949,
                        "end_time": 1610368450.029522,
                        "subsegments": [
                            {
                                "id": "1adf1081d3fbd287",
                                "name": "servi-lista-pwhg5v3y9jol-1655753230.eu-west-1.elb.amazonaws.com",
                                "start_time": 1610368450.002071,
                                "end_time": 1610368450.028931,
                                "http": {
                                    "request": {
                                        "url": "http://servi-lista-pwhg5v3y9jol-1655753230.eu-west-1.elb.amazonaws.com/api/adoptionlist/",
                                        "method": "GET"
                                    },
                                    "response": {
                                        "status": 200,
                                        "content_length": 1873
                                    }
                                },
                                "namespace": "remote"
                            }
                        ]
                    }
                ]
            }
        },
        {
            "Id": "02875c972c787511",
            "Document": {
                "id": "02875c972c787511",
                "name": "DynamoDB",
                "start_time": 1610368450.017891,
                "trace_id": "1-5ffc45c2-4b5e80c8040c405153b5c4ec",
                "end_time": 1610368450.02277,
                "parent_id": "be7d4a3b16ec486a",
                "inferred": true,
                "http": {
                    "response": {
                        "status": 200,
                        "content_length": 200
                    }
                },
                "aws": {
                    "select": null,
                    "count": 1,
                    "table_name": "Services-ddbpetadoption7B7CFEC9-P94QSTUX451H",
                    "resource_names": [
                        "Services-ddbpetadoption7B7CFEC9-P94QSTUX451H"
                    ],
                    "consistent_read": false,
                    "consumed_capacity": null,
                    "scanned_count": 26,
                    "segment": 0,
                    "limit": 0,
                    "attributes_to_get": [],
                    "projection_expression": null,
                    "region": "eu-west-1",
                    "operation": "Scan",
                    "request_id": "Q5BOGNJR23DJ6UQSC71MV3RGV7VV4KQNSO5AEMVJF66Q9ASUAAJG",
                    "total_segments": 0,
                    "index_name": null
                },
                "origin": "AWS::DynamoDB::Table"
            }
        }
    ]
}

Performance Threshold breached AFTER Soak Tests completed for the (golang_alpine, manual) Sample App

Description

After the Soak Tests completed, a performance degradation was revealed for commit e12fdf3 of the refs/heads/main branch for the (golang_alpine, manual) Sample App. Check out the Action Logs from the Soak Testing workflow run on GitHub to view the threshold violation.

Useful Links

Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:

The threshold violation should also be noticeable on our graph of Soak Test average results per commit.

Go / Mux context propagation issues with X-Ray

Hi there 👋

I’ve written two HTTP servers in Golang, instrumented with otelmux, deployed to EKS with the aws-otel-collector sidecar. In X-Ray, I see the traces coming in and the services within the service map, but the graph doesn’t look as expected.

The call flow of my two services is:

Client -> service a -> service b

service-a injects it's request context into the outbound request to service-b before making the request.

req, _ := http.NewRequest(http.MethodGet, "http://service-b.aws-otel-eks.svc.cluster.local/work", nil)
ctx, req := otelhttptrace.W3C(r.Context(), req)
otelhttptrace.Inject(ctx, req)

Is this the correct way of handling context propagation?

When I initialize my tracer I also do

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
  propagation.TraceContext{},
  propagation.Baggage{},
  xray.Propagator{},
))

In the X-Ray service map, things don't link up as expected.

service-a shows two arrows to two different service-b nodes, along with nodes for all the HTTP headers.

These service-b nodes that are children of service-a are yellow indicating an error (no errors occur in making the actual request, returned 200) while a green service-b node sits alone without a parent.

As an aside, it's strange that service-b.aws-otel-eks.svc.cluster.local and service-b.aws-otel-eks.svc.cluster.local:80 are separate nodes.

Here's a gist of one of my servers if it's helpful.

https://gist.github.com/christopher-wong/b1b3f149a4ec45196f7b85cbd760b52f

Move benchmark upstream

Currently, this repository contains benchmarks for idgenerator and propagator and has a Github workflow which run benchmarks on every PR. Ideally, we should add these benchmark upstream with the actual code of idgenerator and propagator.

Using the EKS Resource detector

Hi there 👋

I'm following the docs https://aws-otel.github.io/docs/getting-started/go-sdk#using-aws-eks-resource-detector to setup the EKS Resource detector but am running into some issues.

// Instantiate a new EKS Resource detector
eksResourceDetector := new(eks.ResourceDetector)

seems to return nil.

What's the proper way of instantiating this?

ECS resource detector crash

Hello !

This is similar to #11. Using the ecs detector like this:

ecsResourceDetector := new(ecs.ResourceDetector)
resource, err := ecsResourceDetector.Detect(ctx)

if err != nil {
	fmt.Println("ECS Resource detection error:", err)
}

So, running on EC2 returns an error as expected, process is not on ECS, cannot detect environment variables from ECS.
However on ECS, there is no error, and the application crash with the following logs

Tracing AWS SDK client calls

Hi !

Moving from instrumenting with X-Ray, I was wondering if there was any support for automatic tracing around AWS SDK like the following example

svc := secretsmanager.New(session.New(&aws.Config{Region: aws.String(region)}))
xray.AWS(svc.Client)
ctx, seg := xray.BeginSegment(context.Background(), "payforadoptions")

res, err := svc.GetSecretValueWithContext(ctx, &secretsmanager.GetSecretValueInput{
	SecretId: aws.String(secretID),
})
seg.Close(nil)

Thanks

Outdated pieces of documentation

Hello !

Thanks for the great documentation to get started with Open Telemetry on AWS. However I've noticed few bits of the documentation being slightly outdated.

Go context missing
The first example is missing context.Context as the first argument. https://aws-otel.github.io/docs/getting-started/go-sdk#installation-and-configuration

// Create new OTLP Exporter
exporter, err := otlp.NewExporter(
    otlp.WithTLSCredentials(creds), 
    otlp.WithAddress(<INSERT LISTEN ADDRESS HERE>),
)

Packages path

The ECS, and EKS detectors in the sections here and here are using a package which has been moved from:

go.opentelemetry.io/contrib/detectors/ecs to I believe go.opentelemetry.io/contrib/detectors/aws/ecs
go.opentelemetry.io/contrib/detectors/eks to I believe go.opentelemetry.io/contrib/detectors/aws/eks

Thanks

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.