feat: add msgspec openai adapter, adapter protocol

[viraatc]

What does this PR do?

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• [] Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Summary of Changes

Hello @viraatc, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant architectural improvement by implementing a generic adapter protocol for handling HTTP request and response serialization/deserialization. It also provides a high-performance msgspec-based adapter for OpenAI API interactions, enhancing flexibility and efficiency in communicating with various inference endpoints. The core worker logic has been updated to use this new, pluggable adapter system.

Highlights

• New Adapter Protocol: Introduced a new abstract base class, HttpRequestAdapter, to standardize the conversion between internal Query/QueryResult types and external HTTP endpoint-specific formats. This provides a flexible interface for different API integrations.
• Msgspec-Optimized OpenAI Adapter: Added OpenAIMsgspecAdapter, a new implementation of the HttpRequestAdapter that leverages msgspec for highly efficient serialization and deserialization of OpenAI API requests and responses. This is designed for improved performance.
• Refactored Worker Logic: The core worker (worker.py) has been refactored to utilize the new HttpRequestAdapter protocol. This change abstracts away the specifics of JSON handling and API-specific conversions, making the worker more generic and maintainable.
• Updated OpenAI Adapter: The existing OpenAIAdapter now conforms to the HttpRequestAdapter protocol, implementing the required abstract methods. Its internal methods have been renamed for clarity (e.g., to_openai_request to to_endpoint_request) to align with the new protocol.
• Configuration for Adapters: The HTTPClientConfig now includes an adapter_type field, allowing the client to specify which HttpRequestAdapter implementation to use, with OpenAIMsgspecAdapter set as the default.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull Request Overview

This PR introduces an adapter protocol for HTTP request handling and implements a msgspec-based OpenAI adapter for improved serialization performance. The changes refactor existing OpenAI adapter methods to follow a consistent naming convention (to_endpoint_request, from_endpoint_response, etc.) and establish a protocol-based architecture for supporting multiple endpoint formats.

Key changes:

• Introduces HttpRequestAdapter protocol defining the interface for request/response adapters
• Adds OpenAIMsgspecAdapter using msgspec for fast serialization/deserialization
• Refactors existing OpenAIAdapter to implement the protocol with orjson-based encoding
• Updates worker logic to use configurable adapter types instead of hardcoded OpenAI implementation

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/inference_endpoint/endpoint_client/adapter_protocol.py	Defines abstract base class for HTTP request adapters with SSE parsing support
src/inference_endpoint/openai/openai_msgspec_adapter.py	New msgspec-based OpenAI adapter implementation for performance
src/inference_endpoint/openai/openai_adapter.py	Refactored to implement HttpRequestAdapter protocol with renamed methods
src/inference_endpoint/endpoint_client/configs.py	Added adapter_type configuration with default to OpenAIMsgspecAdapter
src/inference_endpoint/endpoint_client/worker.py	Updated to use configurable adapter instead of hardcoded OpenAIAdapter
tests/unit/test_http_mock_fixtures.py	Updated method calls to use new naming convention
tests/unit/test_core_types.py	Updated method calls to use new naming convention
tests/unit/openai/test_openai_types.py	Updated method calls to use new naming convention
tests/integration/test_server_roundtrip.py	Updated method calls to use new naming convention
tests/integration/endpoint_client/test_worker_errors.py	Added msgspec error message to assertion
tests/integration/endpoint_client/test_http_client_core.py	Improved error handling test to accept both exception and error result

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Code Review

This pull request introduces a well-designed HttpRequestAdapter protocol to abstract away the specifics of different API formats, which is a great step for modularity. The addition of a msgspec-based adapter for OpenAI is a solid performance enhancement. The implementation is clean and the refactoring has been correctly applied across the worker and configuration files. I have a few suggestions to improve robustness, particularly around error handling and input validation.

[arekay-nv]

Looks much better and we can also try out other variations with this change.

[Copilot]

Pull Request Overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.