🚀 Feature: re-write Langchain instrumentation to use Langchain Callbacks about openllmetry-js

[Xabilahu]

Hi, I will try to solve this issue

[Xabilahu]

It makes much more sense :) I will give it a try this weekend!

[Xabilahu]

Looking at the documentation about Langchain callbacks, I can see that there are 2 places where the callbacks argument is accepted:

1. Constructor callback
2. Request callback (a.k.a on call method)

I don't see a way on how this mechanism could replace monkey patching, as for 1 we would need to instantiate the objects for consumers, and for 2 we would need to find a way to intercept / wrap calls.

An idea would be to extend the classes that we want to instrument, and have users consume those instead of the ones from langchain, but I guess that defeats the purpose of auto-instrumentation...

Am I missing something?

[nirga]

Thanks @Xabilahu! So what I had in mind is to monkey patch the constructors of objects like LLMChain, etc. and add our own callback there, and then build a callback handler that creates/ends spans accordingly. The benefit is that instead of being tightly coupled with all of Langchain implementation (which changes quite often), we need to know only the names of classes with callbacks - not that many. WDYT?

[Xabilahu]

Small status update. I've been trying to make this integration work, but I'm not able to. The issue is that the snippet below never gets executed (and thus, the spans are not being reflected in Traceloop), so I still need some more deep dive.

I don't know if this integration has ever been tested, if so, let me know, maybe I'm missing something.

openllmetry-js/packages/instrumentation-langchain/src/instrumentation.ts

Lines 81 to 102 in 3688da7

	private patchChainModule(
	moduleExports: typeof ChainsModule,
	moduleVersion?: string,
	) {
	this._diag.debug(`Patching langchain/chains.cjs@${moduleVersion}`);
	this._wrap(
	moduleExports.RetrievalQAChain.prototype,
	"_call",
	workflowWrapper(
	() => this.tracer,
	this._shouldSendPrompts(),
	"retrieval_qa.workflow",
	),
	);
	this._wrap(
	moduleExports.BaseChain.prototype,
	"call",
	taskWrapper(() => this.tracer, this._shouldSendPrompts()),
	);
	return moduleExports;
	}

[nirga]

@Xabilahu - weird, have you change the log level to debug?
Want to join slack to discuss?

[Xabilahu]

Submitted the WIP PR #176, the missing piece is being able to monkey-patch JS class constructors via shimmer (as it's the library being used by OTel under the hood, source) so that we can inject the CallbackHandler in constructor params. For the moment, I've not been able to.