feat: accept metadata transformer for LangChain callback handler

[gustavocidornelas]

Pull Request

Summary

Introduces an optional kwarg metadata_transformer to the LangChain callback handler.

metadata_transformer should be a transformation function (Callable[[Dict[str, Any]], Dict[str, Any]]) that is applied to the metadata object currently logged by the callback handler. This gives the user fine-grained control about the metadata logged.

Correctly parses usage information when streaming responses

The previous version of the callback handler didn't have the on_new_token method implemented. The current implementation handles the usage information, which is streamed at the very end when stream_usage=True.

Context logging

Now, we're correctly logging the context on retrieval steps as well as on chains that contain source_documents. This enables context-dependent metrics on the platform

Usage

For example, if the user wants to exclude metadata fields that start with langgraph, a possible metadata transformer would be:

def metadata_filter(metadata: Dict[str, Any]) -> Dict[str, Any]:
    return {k: v for k, v in metadata.items() if not k.startswith("langgraph")}


from openlayer.lib.integrations import langchain_callback

openlayer_handler = langchain_callback.OpenlayerHandler(metadata_transformer=metadata_filter)

For streaming, the usage is:

chat = AzureChatOpenAI(...)
for chunk in chat.stream("What's the meaning of life?", stream_usage=True):
    print(chunk.text(), end="")