Skip to content

Commit c8174f6

Browse files
gustavocidornelasgustavocidornelas
committed
fix(closes OPEN-9306): oCI tracer issue serializing GenericChatResponse
1 parent 2ae4b8b commit c8174f6

File tree

3 files changed

+119
-173
lines changed

3 files changed

+119
-173
lines changed

‎examples/tracing/oci/oci_genai_tracing.ipynb‎

Lines changed: 85 additions & 170 deletions
Original file line numberDiff line numberDiff line change
@@ -8,34 +8,11 @@
88
}
99
},
1010
"source": [
11-
"# Oracle OCI Generative AI Tracing with Openlayer\n",
11+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openlayer-ai/openlayer-python/blob/main/examples/tracing/oci/oci_genai_tracing.ipynb)\n",
1212
"\n",
13-
"This notebook demonstrates how to use Openlayer tracing with Oracle Cloud Infrastructure (OCI) Generative AI service.\n",
13+
"# <a id=\"top\">Oracle OCI Generative AI Tracing with Openlayer</a>\n",
1414
"\n",
15-
"## Setup\n",
16-
"\n",
17-
"Before running this notebook, ensure you have:\n",
18-
"1. An OCI account with access to Generative AI service\n",
19-
"2. OCI CLI configured or OCI config file set up\n",
20-
"3. An Openlayer account with API key and inference pipeline ID\n",
21-
"4. The required packages installed:\n",
22-
" - `pip install oci`\n",
23-
" - `pip install openlayer`\n",
24-
"\n",
25-
"## Configuration\n",
26-
"\n",
27-
"### Openlayer Setup\n",
28-
"Set these environment variables before running:\n",
29-
"```bash\n",
30-
"export OPENLAYER_API_KEY=\"your-api-key\"\n",
31-
"export OPENLAYER_INFERENCE_PIPELINE_ID=\"your-pipeline-id\"\n",
32-
"```\n",
33-
"\n",
34-
"### OCI Setup\n",
35-
"Make sure your OCI configuration is properly set up. You can either:\n",
36-
"- Use the default OCI config file (`~/.oci/config`)\n",
37-
"- Set up environment variables\n",
38-
"- Use instance principal authentication (when running on OCI compute)\n"
15+
"This notebook demonstrates how to use Openlayer tracing with Oracle Cloud Infrastructure (OCI) Generative AI service.\n"
3916
]
4017
},
4118
{
@@ -44,44 +21,14 @@
4421
"metadata": {},
4522
"outputs": [],
4623
"source": [
47-
"# Install required packages (uncomment if needed)\n",
48-
"# !pip install oci openlayer\n",
49-
"\n",
50-
"# Set up Openlayer environment variables\n",
51-
"import os\n",
52-
"\n",
53-
"# Configure Openlayer API credentials\n",
54-
"os.environ[\"OPENLAYER_API_KEY\"] = \"your-openlayer-api-key-here\"\n",
55-
"os.environ[\"OPENLAYER_INFERENCE_PIPELINE_ID\"] = \"your-inference-pipeline-id-here\"\n",
56-
"\n",
57-
"# NOTE: Remember to set your actual Openlayer API key and inference pipeline ID!"
58-
]
59-
},
60-
{
61-
"cell_type": "code",
62-
"execution_count": null,
63-
"metadata": {},
64-
"outputs": [],
65-
"source": [
66-
"import oci\n",
67-
"from oci.generative_ai_inference import GenerativeAiInferenceClient\n",
68-
"from oci.generative_ai_inference.models import Message, ChatDetails, GenericChatRequest\n",
69-
"\n",
70-
"# Import the Openlayer tracer\n",
71-
"from openlayer.lib.integrations import trace_oci_genai"
24+
"!pip install openlayer oci"
7225
]
7326
},
7427
{
7528
"cell_type": "markdown",
76-
"metadata": {
77-
"vscode": {
78-
"languageId": "raw"
79-
}
80-
},
29+
"metadata": {},
8130
"source": [
82-
"## Initialize OCI Client\n",
83-
"\n",
84-
"Set up the OCI Generative AI client with your configuration.\n"
31+
"## 1. Set the environment variables"
8532
]
8633
},
8734
{
@@ -90,36 +37,18 @@
9037
"metadata": {},
9138
"outputs": [],
9239
"source": [
93-
"# Configuration - Update these values for your environment\n",
94-
"COMPARTMENT_ID = \"your-compartment-ocid-here\" # Replace with your compartment OCID\n",
95-
"ENDPOINT = \"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\" # Replace with your region's endpoint\n",
96-
"\n",
97-
"# Load OCI configuration\n",
98-
"config = oci.config.from_file() # Uses default config file location\n",
99-
"# Alternatively, you can specify a custom config file:\n",
100-
"# config = oci.config.from_file(\"~/.oci/config\", \"DEFAULT\")\n",
40+
"import os\n",
10141
"\n",
102-
"# Create the OCI Generative AI client\n",
103-
"client = GenerativeAiInferenceClient(config=config, service_endpoint=ENDPOINT)\n"
42+
"# Configure Openlayer API credentials\n",
43+
"os.environ[\"OPENLAYER_API_KEY\"] = \"YOUR_OPENLAYER_API_KEY_HERE\"\n",
44+
"os.environ[\"OPENLAYER_INFERENCE_PIPELINE_ID\"] = \"YOUR_INFERENCE_PIPELINE_ID_HERE\""
10445
]
10546
},
10647
{
10748
"cell_type": "markdown",
108-
"metadata": {
109-
"vscode": {
110-
"languageId": "raw"
111-
}
112-
},
49+
"metadata": {},
11350
"source": [
114-
"## Apply Openlayer Tracing\n",
115-
"\n",
116-
"Wrap the OCI client with Openlayer tracing to automatically capture all interactions.\n",
117-
"\n",
118-
"The `trace_oci_genai()` function accepts an optional `estimate_tokens` parameter:\n",
119-
"- `estimate_tokens=True` (default): Estimates token counts when not provided by OCI response\n",
120-
"- `estimate_tokens=False`: Returns None for token fields when not available in the response\n",
121-
"\n",
122-
"OCI responses can be either CohereChatResponse or GenericChatResponse, both containing usage information when available.\n"
51+
"## 2. Wrap the OCI Generative AI client with the `trace_oci_genai` function"
12352
]
12453
},
12554
{
@@ -128,25 +57,28 @@
12857
"metadata": {},
12958
"outputs": [],
13059
"source": [
131-
"# Apply Openlayer tracing to the OCI client\n",
132-
"# With token estimation enabled (default)\n",
133-
"traced_client = trace_oci_genai(client, estimate_tokens=True)\n",
60+
"import oci\n",
61+
"from oci.generative_ai_inference import GenerativeAiInferenceClient\n",
62+
"from oci.generative_ai_inference.models import (\n",
63+
" Message,\n",
64+
" ChatDetails,\n",
65+
" TextContent,\n",
66+
" BaseChatRequest,\n",
67+
" GenericChatRequest,\n",
68+
" OnDemandServingMode,\n",
69+
")\n",
13470
"\n",
135-
"# Alternative: Disable token estimation to get None values when tokens are not available\n",
136-
"# traced_client = trace_oci_genai(client, estimate_tokens=False)"
137-
]
138-
},
139-
{
140-
"cell_type": "markdown",
141-
"metadata": {
142-
"vscode": {
143-
"languageId": "raw"
144-
}
145-
},
146-
"source": [
147-
"## Example 1: Non-Streaming Chat Completion\n",
71+
"# Point to your real files\n",
72+
"OCI_CONFIG_PATH = \"YOUR_OCI_CONFIG_PATH_HERE\"\n",
73+
"OCI_PROFILE = \"DEFAULT\"\n",
74+
"\n",
75+
"ENDPOINT = \"YOUR_ENDPOINT_HERE\"\n",
76+
"COMPARTMENT_ID = \"YOUR_COMPARTMENT_OCID_HERE\"\n",
77+
"MODEL_ID = \"YOUR_MODEL_OCID_HERE\"\n",
14878
"\n",
149-
"Simple chat completion without streaming.\n"
79+
"\n",
80+
"config = oci.config.from_file(OCI_CONFIG_PATH, OCI_PROFILE)\n",
81+
"oci.config.validate_config(config)\n"
15082
]
15183
},
15284
{
@@ -155,33 +87,24 @@
15587
"metadata": {},
15688
"outputs": [],
15789
"source": [
158-
"# Create a chat request\n",
159-
"chat_request = GenericChatRequest(\n",
160-
" messages=[Message(role=\"user\", content=\"Hello! Can you explain what Oracle Cloud Infrastructure is?\")],\n",
161-
" model_id=\"cohere.command-r-plus\",\n",
162-
" max_tokens=200,\n",
163-
" temperature=0.7,\n",
164-
" is_stream=False, # Non-streaming\n",
165-
")\n",
166-
"\n",
167-
"chat_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=chat_request)\n",
168-
"\n",
169-
"# Make the request - the tracer will automatically capture it\n",
170-
"response = traced_client.chat(chat_details)\n",
171-
"response"
90+
"# Import the Openlayer tracer\n",
91+
"from openlayer.lib.integrations import trace_oci_genai\n",
92+
"\n",
93+
"client = trace_oci_genai(GenerativeAiInferenceClient(\n",
94+
" config=config,\n",
95+
" service_endpoint=ENDPOINT,\n",
96+
" retry_strategy=oci.retry.NoneRetryStrategy(),\n",
97+
" timeout=(10, 120),\n",
98+
"), estimate_tokens=True)"
17299
]
173100
},
174101
{
175102
"cell_type": "markdown",
176-
"metadata": {
177-
"vscode": {
178-
"languageId": "raw"
179-
}
180-
},
103+
"metadata": {},
181104
"source": [
182-
"## Example 2: Streaming Chat Completion\n",
105+
"## 3. Use the traced OCI Generative AI client normally\n",
183106
"\n",
184-
"Chat completion with streaming enabled to see tokens as they're generated.\n"
107+
"That's it! Now you can continue using the traced OpenAI client normally. The data is automatically published to Openlayer and you can start creating tests around it!"
185108
]
186109
},
187110
{
@@ -190,46 +113,40 @@
190113
"metadata": {},
191114
"outputs": [],
192115
"source": [
193-
"# Create a streaming chat request\n",
194-
"streaming_chat_request = GenericChatRequest(\n",
116+
"chat_request = GenericChatRequest(\n",
117+
" api_format=BaseChatRequest.API_FORMAT_GENERIC,\n",
195118
" messages=[\n",
196-
" Message(role=\"system\", content=\"You are a helpful AI assistant that provides concise, informative answers.\"),\n",
197-
" Message(role=\"user\", content=\"Tell me a short story about cloud computing and AI working together.\"),\n",
119+
" Message(\n",
120+
" role=\"USER\",\n",
121+
" content=[TextContent(text=\"What is OCI in one sentence?\")]\n",
122+
" )\n",
198123
" ],\n",
199-
" model_id=\"meta.llama-3.1-70b-instruct\",\n",
200-
" max_tokens=300,\n",
201-
" temperature=0.8,\n",
202-
" is_stream=True, # Enable streaming\n",
124+
" max_tokens=100,\n",
125+
" temperature=0.2,\n",
126+
" is_stream=False,\n",
203127
")\n",
204128
"\n",
205-
"streaming_chat_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=streaming_chat_request)\n",
206-
"\n",
207-
"# Make the streaming request\n",
208-
"streaming_response = traced_client.chat(streaming_chat_details)\n",
209-
"\n",
210-
"# Process the streaming response\n",
211-
"full_content = \"\"\n",
212-
"for chunk in streaming_response:\n",
213-
" if hasattr(chunk, \"data\") and hastr(chunk.data, \"choices\"):\n",
214-
" if chunk.data.choices and hasattr(chunk.data.choices[0], \"delta\"):\n",
215-
" delta = chunk.data.choices[0].delta\n",
216-
" if hasattr(delta, \"content\") and delta.content:\n",
217-
" full_content += delta.content\n",
218-
"\n",
219-
"full_content"
129+
"chat_details = ChatDetails(\n",
130+
" serving_mode=OnDemandServingMode(model_id=MODEL_ID),\n",
131+
" chat_request=chat_request,\n",
132+
" compartment_id=COMPARTMENT_ID,\n",
133+
")\n",
134+
"\n"
220135
]
221136
},
222137
{
223-
"cell_type": "markdown",
224-
"metadata": {
225-
"vscode": {
226-
"languageId": "raw"
227-
}
228-
},
138+
"cell_type": "code",
139+
"execution_count": null,
140+
"metadata": {},
141+
"outputs": [],
229142
"source": [
230-
"## Example 3: Custom Parameters and Error Handling\n",
143+
"resp = client.chat(chat_details)\n",
231144
"\n",
232-
"Demonstrate various model parameters and how tracing works with different scenarios.\n"
145+
"answer = \"\"\n",
146+
"choice = resp.data.chat_response.choices[0]\n",
147+
"for part in choice.message.content:\n",
148+
" if hasattr(part, \"text\"):\n",
149+
" answer += part.text"
233150
]
234151
},
235152
{
@@ -238,29 +155,27 @@
238155
"metadata": {},
239156
"outputs": [],
240157
"source": [
241-
"# Advanced parameters example\n",
242-
"advanced_request = GenericChatRequest(\n",
243-
" messages=[Message(role=\"user\", content=\"Write a creative haiku about artificial intelligence.\")],\n",
244-
" model_id=\"meta.llama-3.1-70b-instruct\",\n",
245-
" max_tokens=100,\n",
246-
" temperature=0.9, # High creativity\n",
247-
" top_p=0.8,\n",
248-
" frequency_penalty=0.2, # Reduce repetition\n",
249-
" presence_penalty=0.1,\n",
250-
" stop=[\"\\n\\n\"], # Stop at double newline\n",
251-
" is_stream=False,\n",
252-
")\n",
253-
"\n",
254-
"advanced_details = ChatDetails(compartment_id=COMPARTMENT_ID, chat_request=advanced_request)\n",
255-
"\n",
256-
"response = traced_client.chat(advanced_details)\n",
257-
"response"
158+
"answer"
258159
]
259160
}
260161
],
261162
"metadata": {
163+
"kernelspec": {
164+
"display_name": "oracle",
165+
"language": "python",
166+
"name": "python3"
167+
},
262168
"language_info": {
263-
"name": "python"
169+
"codemirror_mode": {
170+
"name": "ipython",
171+
"version": 3
172+
},
173+
"file_extension": ".py",
174+
"mimetype": "text/x-python",
175+
"name": "python",
176+
"nbconvert_exporter": "python",
177+
"pygments_lexer": "ipython3",
178+
"version": "3.12.12"
264179
}
265180
},
266181
"nbformat": 4,

‎src/openlayer/lib/integrations/oci_tracer.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
from oci.generative_ai_inference import GenerativeAiInferenceClient
1919

2020
from ..tracing import tracer
21+
from ..utils import safe_serialize_raw_output
2122

2223
logger = logging.getLogger(__name__)
2324

@@ -126,8 +127,16 @@ def handle_streaming_chat(
126127
Iterator[Any]
127128
A generator that yields the chunks of the completion.
128129
"""
130+
# OCI SDK versions may expose stream events differently; support both wrapped
131+
# responses (`response.data.events()`) and direct iterators.
132+
chunks = response
133+
if hasattr(response, "data") and hasattr(response.data, "events") and callable(response.data.events):
134+
chunks = response.data.events()
135+
elif hasattr(response, "events") and callable(response.events):
136+
chunks = response.events()
137+
129138
return stream_chunks(
130-
chunks=response.data.events(),
139+
chunks=chunks,
131140
chat_details=chat_details,
132141
kwargs=kwargs,
133142
start_time=start_time,
@@ -202,6 +211,7 @@ def stream_chunks(
202211

203212
except Exception as e:
204213
logger.error("Failed yield chunk. %s", e)
214+
raise
205215
finally:
206216
# Try to add step to the trace
207217
try:
@@ -306,7 +316,7 @@ def handle_non_streaming_chat(
306316
completion_tokens=tokens_info.get("output_tokens"),
307317
model=model_id,
308318
model_parameters=get_model_parameters(chat_details),
309-
raw_output=response.data.__dict__ if hasattr(response, "data") else response.__dict__,
319+
raw_output=safe_serialize_raw_output(response),
310320
id=None,
311321
metadata=additional_metadata,
312322
)

0 commit comments

Comments
 (0)