traceloop / openllmetry-js Goto Github PK

Write tests to cover our OpenAI / Azure OpenAI instrumentations, similar to other instrumentations in this repo.

Following #56, we switched to mocked tests based on pre-recording of HTTP calls.
While this method worked well for OpenAI and other HTTP calls, I had some issues with the way our Pinecone tests are built.
This needs to be fixed and re-enabled.

Which component is this feature for?

All Packages

🔖 Feature description

Instrument calls to Chroma, including adding attributes, similar to our Chroma on Python instrumentation. The instrumentation should support all types of calls - sync, async, etc.

🎤 Why is this feature needed?

Completeness of OpenLLMetry

✌️ How do you aim to achieve this?

Similar to other instrumentations, we have in this repo.

🔄️ Additional Information

• Make sure to initialize this in Traceloop SDK.
• Add proper tests.
• Add a sample in the sample app.
• Add a screenshot from Traceloop showing the instrumentation in action.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Steps reproduce

Run following file

import * as traceloop from "@traceloop/node-server-sdk";
import OpenAI from "openai";

traceloop.initialize({
  appName: "sample_openai",
  apiKey: process.env.TRACELOOP_API_KEY,
  disableBatch: true,
});
const openai = new OpenAI();

class SampleOpenAI {
  @traceloop.workflow("sample_completion")
  async completion() {
    const completion = await openai.completions.create({
      prompt: "Tell me a joke about TypeScript",
      model: "gpt-3.5-turbo-instruct",
      stream: true
    });

    return completion;
  }
}

traceloop.withAssociationProperties(
  { user_id: "12345", chat_id: "789" },
  async () => {
    const sampleOpenAI = new SampleOpenAI();

    const completion = await sampleOpenAI.completion();
    console.log(completion);

    await traceloop.reportScore({ chat_id: "789" }, 1);
  },
);

Actual

Traceloop exporting traces to https://api.traceloop.com
/Users/k/personal/openllmetry-js/packages/instrumentation-openai/dist/src/instrumentation.js:211
                result.choices.forEach((choice, index) => {
                               ^

TypeError: Cannot read properties of undefined (reading 'forEach')
    at OpenAIInstrumentation._endSpan (/Users/k/personal/openllmetry-js/packages/instrumentation-openai/dist/src/instrumentation.js:211:32)
    at /Users/k/personal/openllmetry-js/packages/instrumentation-openai/dist/src/instrumentation.js:171:30
    at new Promise (<anonymous>)
    at /Users/k/personal/openllmetry-js/packages/instrumentation-openai/dist/src/instrumentation.js:149:20
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.completion (/Users/k/personal/openllmetry-js/packages/sample-app/dist/src/sample_streaming.js:19:28)
    at async /Users/k/personal/openllmetry-js/packages/traceloop-sdk/dist/src/lib/tracing/decorators.js:22:25
    at async descriptor.value (/Users/k/personal/openllmetry-js/packages/traceloop-sdk/dist/src/lib/tracing/decorators.js:60:24)
    at async /Users/k/personal/openllmetry-js/packages/sample-app/dist/src/sample_streaming.js:32:24

Node.js v18.19.0

Expected

Program should not crash and should send correct spans to traceloop.

Which component is this feature for?

All Packages

🔖 Feature description

Instrument LlamaIndex apps, including adding attributes, similarly to our instrumentation in Python. The instrumentation should support all types of calls - streaming, non streaming, async, etc.

🎤 Why is this feature needed ?

Completness of OpenLLMetry-JS

✌️ How do you aim to achieve this?

Similarily to other instrumentations we have in this repo.

🔄️ Additional Information

This should instrument all typescript-equivalent APIs that we instrument in Python.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Following #56, we switched to mocked tests based on pre-recording of HTTP calls.
While this method worked well for OpenAI and other HTTP calls, this doesn't work for GRPC which vertex / google is using.

We need to figure out how to mock out the requests there.

Right now, the only way to create spans of prompts and completions is to use one of the ready-made instrumentations.

We should have an easy to use API in the SDK to manually create a span with the appropriate attributes for cases where the current instrumentations are not enough. Similar to the APIs we currently have on Go and Ruby.

Hi,

I'm running otel from cloudflare workers using the various http libs and different exporters. Will the instrumentation libs @traceloop/instrumentation-openai etc with @opentelemetry/instrumentation-http?

Which component is this feature for?

All Packages

🔖 Feature description

Instrument calls to Pinecone, including adding attributes, similarly to our Pinecone on Python instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.

🎤 Why is this feature needed?

Completeness of OpenLLMetry

✌️ How do you aim to achieve this?

Similar to other instrumentations, we have in this repo.

🔄️ Additional Information

• Make sure to initialize this in Traceloop SDK.
• Add proper tests.
• Add a sample in the sample app.
• Add a screenshot from Traceloop showing the instrumentation in action.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Which component is this feature for?

LlamaIndex Instrumentation

🔖 Feature description

Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. LlamaIndex has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.

🎤 Why is this feature needed ?

Support LlamaIndex entirely and be future-proof to internal API changes

✌️ How do you aim to achieve this?

Look into LlamaIndex callback_manager and how other frameworks are using it.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Which component is this feature for?

Langchain Instrumentation

🔖 Feature description

Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. Langchain has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.

🎤 Why is this feature needed ?

Support Langchain entirely and be future-proof to internal API changes

✌️ How do you aim to achieve this?

Look into Langchain callbacks and how other frameworks are using it.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Description

Add a new OpenTelemetry auto-instrumentation for Langchain Library. This instrumentation will greatly improve the observability and tracing capabilities of the Langchain based applications.

Details

• Instrumentation Name: instrumentation-langchain
• Description: This instrumentation will provide automatic tracing and observability for the Langchain Library.
• Language: TypeScript

Motivation

Adding this instrumentation will allow seamless integration of OpenTelemetry with Langchain enabling better visibility into chains which will be represented as workflows/traces in tracing data and Traceloop dashoard.

High Level Instructions

1. Add a new package in this monorepo which should called instrumentation-langchain, with the same structure/boilerplate of instrumentation-openai.
2. Implement the auto instrumentation such that (quite similar to the instrumentation logic in the python sdk):

• Every chain in Langchain will produce a span with the name <chain class name>.task
• Traceloop Span Kind attribute from the semantic-convetions should be set to task and the span name is <chain class name>.task.
• Propagate also the Workflow Name and Correlation Ids as well.
• Produce workflow span For SequentialChain & RetrievalQA chains.

3. Integrate the above implemented instrumentation within traceloop-sdk
4. Add tests

Checklist

Please make sure the following criteria are met:

• I have searched the existing issues and confirmed that this request is not a duplicate.
• I have provided a clear and concise description of the requested instrumentation.
• I have explained the motivation and benefits of adding this instrumentation.
• I have included any additional information or resources that may be relevant.

Thank you!

Which component is this feature for?

All Packages

🔖 Feature description

Instrument calls to AWS's [Bedrock(https://www.npmjs.com/package/@aws-sdk/client-bedrock-runtime), including adding attributes for input parameters, model, etc. - similarly to our OpenAI instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.

Instrumentation should support all models supported in our Python instrumentation.

This should specifically work with Google's new Gemini model.

🎤 Why is this feature needed ?

Completeness of OpenLLMetry

✌️ How do you aim to achieve this?

Similar to other instrumentations we have in this repo.

🔄️ Additional Information

• Make sure to initialize this in Traceloop SDK.
• Add proper tests.
• Add a sample in the sample app.
• Upload a screenshot from Traceloop with how your instrumentation works.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Which component is this feature for?

All Packages

🔖 Feature description

Instrument calls to Google's Vertex AI, including adding attributes for input parameters, model, etc. - similarly to our OpenAI instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.

Instrumentation should support all models supported in our Python instrumentation including the new Gemini AI model.

This should specifically work with Google's new Gemini model.

🎤 Why is this feature needed ?

Completeness of OpenLLMetry

✌️ How do you aim to achieve this?

Similar to other instrumentations we have in this repo.

🔄️ Additional Information

• Make sure to initialize this in Traceloop SDK.
• Add proper tests.
• Add a sample in the sample app.
• Upload a screenshot from Traceloop with how your instrumentation works.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

Which component is this feature for?

All Packages

🔖 Feature description

Instrument calls to Cohere, including adding attributes for input parameters, model, etc. - similarly to our OpenAI instrumentation. The instrumentation should support all types of calls - streaming, non streaming, async, etc.

Instrumentation should support all models supported in our Python instrumentation.

This should specifically work with Google's new Gemini model.

🎤 Why is this feature needed ?

Completeness of OpenLLMetry

✌️ How do you aim to achieve this?

Similar to other instrumentations we have in this repo.

🔄️ Additional Information

• Make sure to initialize this in Traceloop SDK.
• Add proper tests.
• Add a sample in the sample app.
• Upload a screenshot from Traceloop with how your instrumentation works.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None