Skip to content

Commit 6020f64

Browse files
jimbobbennettclaude
jimbobbennett
and
claude
committed
Add 9 Arize LLM observability skills
Add skills for Arize AI platform covering trace export, instrumentation, datasets, experiments, evaluators, AI provider integrations, annotations, prompt optimization, and deep linking to the Arize UI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 35d73e0 commit 6020f64

26 files changed

Lines changed: 4175 additions & 0 deletions

File tree

‎docs/README.skills.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,15 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to
3434
| [apple-appstore-reviewer](../skills/apple-appstore-reviewer/SKILL.md) | Serves as a reviewer of the codebase with instructions on looking for Apple App Store optimizations or rejection reasons. | None |
3535
| [arch-linux-triage](../skills/arch-linux-triage/SKILL.md) | Triage and resolve Arch Linux issues with pacman, systemd, and rolling-release best practices. | None |
3636
| [architecture-blueprint-generator](../skills/architecture-blueprint-generator/SKILL.md) | Comprehensive project architecture blueprint generator that analyzes codebases to create detailed architectural documentation. Automatically detects technology stacks and architectural patterns, generates visual diagrams, documents implementation patterns, and provides extensible blueprints for maintaining architectural consistency and guiding new development. | None |
37+
| [arize-ai-provider-integration](../skills/arize-ai-provider-integration/SKILL.md) | INVOKE THIS SKILL when creating, reading, updating, or deleting Arize AI integrations. Covers listing integrations, creating integrations for any supported LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM, custom), updating credentials or metadata, and deleting integrations using the ax CLI. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
38+
| [arize-annotation](../skills/arize-annotation/SKILL.md) | INVOKE THIS SKILL when creating, managing, or using annotation configs on Arize (categorical, continuous, freeform), or applying human annotations to project spans via the Python SDK. Configs are the label schema for human feedback on spans and other surfaces in the Arize UI. Triggers: annotation config, label schema, human feedback schema, bulk annotate spans, update_annotations. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
39+
| [arize-dataset](../skills/arize-dataset/SKILL.md) | INVOKE THIS SKILL when creating, managing, or querying Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
40+
| [arize-evaluator](../skills/arize-evaluator/SKILL.md) | INVOKE THIS SKILL for LLM-as-judge evaluation workflows on Arize: creating/updating evaluators, running evaluations on spans or experiments, tasks, trigger-run, column mapping, and continuous monitoring. Use when the user says: create an evaluator, LLM judge, hallucination/faithfulness/correctness/relevance, run eval, score my spans or experiment, ax tasks, trigger-run, trigger eval, column mapping, continuous monitoring, query filter for evals, evaluator version, or improve an evaluator prompt. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
41+
| [arize-experiment](../skills/arize-experiment/SKILL.md) | INVOKE THIS SKILL when creating, running, or analyzing Arize experiments. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
42+
| [arize-instrumentation](../skills/arize-instrumentation/SKILL.md) | INVOKE THIS SKILL when adding Arize AX tracing to an application. Follow the Agent-Assisted Tracing two-phase flow: analyze the codebase (read-only), then implement instrumentation after user confirmation. When the app uses LLM tool/function calling, add manual CHAIN + TOOL spans so traces show each tool's input and output. Leverages https://arize.com/docs/ax/alyx/tracing-assistant and https://arize.com/docs/PROMPT.md. | `references/ax-profiles.md` |
43+
| [arize-link](../skills/arize-link/SKILL.md) | Generate deep links to the Arize UI. Use when the user wants a clickable URL to open a specific trace, span, session, dataset, labeling queue, evaluator, or annotation config. | `references/EXAMPLES.md` |
44+
| [arize-prompt-optimization](../skills/arize-prompt-optimization/SKILL.md) | INVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data, evaluations, and annotations. Covers extracting prompts from spans, gathering performance signal, and running a data-driven optimization loop using the ax CLI. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
45+
| [arize-trace](../skills/arize-trace/SKILL.md) | INVOKE THIS SKILL when downloading or exporting Arize traces and spans. Covers exporting traces by ID, sessions by ID, and debugging LLM application issues using the ax CLI. | `references/ax-profiles.md`<br />`references/ax-setup.md` |
3746
| [aspire](../skills/aspire/SKILL.md) | Aspire skill covering the Aspire CLI, AppHost orchestration, service discovery, integrations, MCP server, VS Code extension, Dev Containers, GitHub Codespaces, templates, dashboard, and deployment. Use when the user asks to create, run, debug, configure, deploy, or troubleshoot an Aspire distributed application. | `references/architecture.md`<br />`references/cli-reference.md`<br />`references/dashboard.md`<br />`references/deployment.md`<br />`references/integrations-catalog.md`<br />`references/mcp-server.md`<br />`references/polyglot-apis.md`<br />`references/testing.md`<br />`references/troubleshooting.md` |
3847
| [aspnet-minimal-api-openapi](../skills/aspnet-minimal-api-openapi/SKILL.md) | Create ASP.NET Minimal API endpoints with proper OpenAPI documentation | None |
3948
| [automate-this](../skills/automate-this/SKILL.md) | Analyze a screen recording of a manual process and produce targeted, working automation scripts. Extracts frames and audio narration from video files, reconstructs the step-by-step workflow, and proposes automation at multiple complexity levels using tools already installed on the user machine. | None |

‎skills/arize-ai-provider-integration/SKILL.md‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
---
2+
name: arize-ai-provider-integration
3+
description: "INVOKE THIS SKILL when creating, reading, updating, or deleting Arize AI integrations. Covers listing integrations, creating integrations for any supported LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM, custom), updating credentials or metadata, and deleting integrations using the ax CLI."
4+
---
5+
6+
# Arize AI Integration Skill
7+
8+
## Concepts
9+
10+
- **AI Integration** = stored LLM provider credentials registered in Arize; used by evaluators to call a judge model and by other Arize features that need to invoke an LLM on your behalf
11+
- **Provider** = the LLM service backing the integration (e.g., `openAI`, `anthropic`, `awsBedrock`)
12+
- **Integration ID** = a base64-encoded global identifier for an integration (e.g., `TGxtSW50ZWdyYXRpb246MTI6YUJjRA==`); required for evaluator creation and other downstream operations
13+
- **Scoping** = visibility rules controlling which spaces or users can use an integration
14+
- **Auth type** = how Arize authenticates with the provider: `default` (provider API key), `proxy_with_headers` (proxy via custom headers), or `bearer_token` (bearer token auth)
15+
16+
## Prerequisites
17+
18+
Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
19+
20+
If an `ax` command fails, troubleshoot based on the error:
21+
- `command not found` or version error → see ax-setup.md
22+
- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong: check `.env` for `ARIZE_API_KEY` and use it to create/update the profile via ax-profiles.md. If `.env` has no key either, ask the user for their Arize API key (https://app.arize.com/admin > API Keys)
23+
- Space ID unknown → check `.env` for `ARIZE_SPACE_ID`, or run `ax spaces list -o json`, or ask the user
24+
- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → check `.env`, load if present, otherwise ask the user
25+
26+
---
27+
28+
## List AI Integrations
29+
30+
List all integrations accessible in a space:
31+
32+
```bash
33+
ax ai-integrations list --space-id SPACE_ID
34+
```
35+
36+
Filter by name (case-insensitive substring match):
37+
38+
```bash
39+
ax ai-integrations list --space-id SPACE_ID --name "openai"
40+
```
41+
42+
Paginate large result sets:
43+
44+
```bash
45+
# Get first page
46+
ax ai-integrations list --space-id SPACE_ID --limit 20 -o json
47+
48+
# Get next page using cursor from previous response
49+
ax ai-integrations list --space-id SPACE_ID --limit 20 --cursor CURSOR_TOKEN -o json
50+
```
51+
52+
**Key flags:**
53+
54+
| Flag | Description |
55+
|------|-------------|
56+
| `--space-id` | Space to list integrations in |
57+
| `--name` | Case-insensitive substring filter on integration name |
58+
| `--limit` | Max results (1–100, default 50) |
59+
| `--cursor` | Pagination token from a previous response |
60+
| `-o, --output` | Output format: `table` (default) or `json` |
61+
62+
**Response fields:**
63+
64+
| Field | Description |
65+
|-------|-------------|
66+
| `id` | Base64 integration ID — copy this for downstream commands |
67+
| `name` | Human-readable name |
68+
| `provider` | LLM provider enum (see Supported Providers below) |
69+
| `has_api_key` | `true` if credentials are stored |
70+
| `model_names` | Allowed model list, or `null` if all models are enabled |
71+
| `enable_default_models` | Whether default models for this provider are allowed |
72+
| `function_calling_enabled` | Whether tool/function calling is enabled |
73+
| `auth_type` | Authentication method: `default`, `proxy_with_headers`, or `bearer_token` |
74+
75+
---
76+
77+
## Get a Specific Integration
78+
79+
```bash
80+
ax ai-integrations get INT_ID
81+
ax ai-integrations get INT_ID -o json
82+
```
83+
84+
Use this to inspect an integration's full configuration or to confirm its ID after creation.
85+
86+
---
87+
88+
## Create an AI Integration
89+
90+
Before creating, always list integrations first — the user may already have a suitable one:
91+
92+
```bash
93+
ax ai-integrations list --space-id SPACE_ID
94+
```
95+
96+
If no suitable integration exists, create one. The required flags depend on the provider.
97+
98+
### OpenAI
99+
100+
```bash
101+
ax ai-integrations create \
102+
--name "My OpenAI Integration" \
103+
--provider openAI \
104+
--api-key $OPENAI_API_KEY
105+
```
106+
107+
### Anthropic
108+
109+
```bash
110+
ax ai-integrations create \
111+
--name "My Anthropic Integration" \
112+
--provider anthropic \
113+
--api-key $ANTHROPIC_API_KEY
114+
```
115+
116+
### Azure OpenAI
117+
118+
```bash
119+
ax ai-integrations create \
120+
--name "My Azure OpenAI Integration" \
121+
--provider azureOpenAI \
122+
--api-key $AZURE_OPENAI_API_KEY \
123+
--base-url "https://my-resource.openai.azure.com/"
124+
```
125+
126+
### AWS Bedrock
127+
128+
AWS Bedrock uses IAM role-based auth instead of an API key. Provide the ARN of the role Arize should assume:
129+
130+
```bash
131+
ax ai-integrations create \
132+
--name "My Bedrock Integration" \
133+
--provider awsBedrock \
134+
--role-arn "arn:aws:iam::123456789012:role/ArizeBedrockRole"
135+
```
136+
137+
### Vertex AI
138+
139+
Vertex AI uses GCP service account credentials. Provide the GCP project and region:
140+
141+
```bash
142+
ax ai-integrations create \
143+
--name "My Vertex AI Integration" \
144+
--provider vertexAI \
145+
--project-id "my-gcp-project" \
146+
--location "us-central1"
147+
```
148+
149+
### Gemini
150+
151+
```bash
152+
ax ai-integrations create \
153+
--name "My Gemini Integration" \
154+
--provider gemini \
155+
--api-key $GEMINI_API_KEY
156+
```
157+
158+
### NVIDIA NIM
159+
160+
```bash
161+
ax ai-integrations create \
162+
--name "My NVIDIA NIM Integration" \
163+
--provider nvidiaNim \
164+
--api-key $NVIDIA_API_KEY \
165+
--base-url "https://integrate.api.nvidia.com/v1"
166+
```
167+
168+
### Custom (OpenAI-compatible endpoint)
169+
170+
```bash
171+
ax ai-integrations create \
172+
--name "My Custom Integration" \
173+
--provider custom \
174+
--base-url "https://my-llm-proxy.example.com/v1" \
175+
--api-key $CUSTOM_LLM_API_KEY
176+
```
177+
178+
### Supported Providers
179+
180+
| Provider | Required extra flags |
181+
|----------|---------------------|
182+
| `openAI` | `--api-key <key>` |
183+
| `anthropic` | `--api-key <key>` |
184+
| `azureOpenAI` | `--api-key <key>`, `--base-url <azure-endpoint>` |
185+
| `awsBedrock` | `--role-arn <arn>` |
186+
| `vertexAI` | `--project-id <gcp-project>`, `--location <region>` |
187+
| `gemini` | `--api-key <key>` |
188+
| `nvidiaNim` | `--api-key <key>`, `--base-url <nim-endpoint>` |
189+
| `custom` | `--base-url <endpoint>` |
190+
191+
### Optional flags for any provider
192+
193+
| Flag | Description |
194+
|------|-------------|
195+
| `--model-names` | Comma-separated list of allowed model names; omit to allow all models |
196+
| `--enable-default-models` / `--no-default-models` | Enable or disable the provider's default model list |
197+
| `--function-calling` / `--no-function-calling` | Enable or disable tool/function calling support |
198+
199+
### After creation
200+
201+
Capture the returned integration ID (e.g., `TGxtSW50ZWdyYXRpb246MTI6YUJjRA==`) — it is needed for evaluator creation and other downstream commands. If you missed it, retrieve it:
202+
203+
```bash
204+
ax ai-integrations list --space-id SPACE_ID -o json
205+
# or, if you know the ID:
206+
ax ai-integrations get INT_ID
207+
```
208+
209+
---
210+
211+
## Update an AI Integration
212+
213+
`update` is a partial update — only the flags you provide are changed. Omitted fields stay as-is.
214+
215+
```bash
216+
# Rename
217+
ax ai-integrations update INT_ID --name "New Name"
218+
219+
# Rotate the API key
220+
ax ai-integrations update INT_ID --api-key $OPENAI_API_KEY
221+
222+
# Change the model list
223+
ax ai-integrations update INT_ID --model-names "gpt-4o,gpt-4o-mini"
224+
225+
# Update base URL (for Azure, custom, or NIM)
226+
ax ai-integrations update INT_ID --base-url "https://new-endpoint.example.com/v1"
227+
```
228+
229+
Any flag accepted by `create` can be passed to `update`.
230+
231+
---
232+
233+
## Delete an AI Integration
234+
235+
**Warning:** Deletion is permanent. Evaluators that reference this integration will no longer be able to run.
236+
237+
```bash
238+
ax ai-integrations delete INT_ID --force
239+
```
240+
241+
Omit `--force` to get a confirmation prompt instead of deleting immediately.
242+
243+
---
244+
245+
## Troubleshooting
246+
247+
| Problem | Solution |
248+
|---------|----------|
249+
| `ax: command not found` | See ax-setup.md |
250+
| `401 Unauthorized` | API key may not have access to this space. Verify key and space ID at https://app.arize.com/admin > API Keys |
251+
| `No profile found` | Run `ax profiles show --expand`; set `ARIZE_API_KEY` env var or write `~/.arize/config.toml` |
252+
| `Integration not found` | Verify with `ax ai-integrations list --space-id SPACE_ID` |
253+
| `has_api_key: false` after create | Credentials were not saved — re-run `update` with the correct `--api-key` or `--role-arn` |
254+
| Evaluator runs fail with LLM errors | Check integration credentials with `ax ai-integrations get INT_ID`; rotate the API key if needed |
255+
| `provider` mismatch | Cannot change provider after creation — delete and recreate with the correct provider |
256+
257+
---
258+
259+
## Related Skills
260+
261+
- **arize-evaluator**: Create LLM-as-judge evaluators that use an AI integration → use `arize-evaluator`
262+
- **arize-experiment**: Run experiments that use evaluators backed by an AI integration → use `arize-experiment`
263+
264+
---
265+
266+
## Save Credentials for Future Use
267+
268+
See ax-profiles.md § Save Credentials for Future Use.

0 commit comments

Comments
 (0)