Skip to content

Commit 2a9c788

Browse files
Arize AI Devsfjcasti1
authored andcommitted
Project import generated by Copybara.
GitOrigin-RevId: 2e2b50eaff27d742032c5a33892eced342f06ab5
1 parent f2dc872 commit 2a9c788

115 files changed

Lines changed: 6389 additions & 1546 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
# Changelog
22

3+
## [8.23.0](https://github.com/Arize-ai/arize/compare/arize-python-sdk/v8.22.4...arize-python-sdk/v8.23.0) (2026-05-11)
4+
5+
6+
### 🎁 New Features
7+
8+
* **api-keys:** extend GET /v2/api-keys with space_id and user_id filters ([#70697](https://github.com/Arize-ai/arize/issues/70697)) ([06dfc73](https://github.com/Arize-ai/arize/commit/06dfc73f9f3d7eb08c9d3c9435ff17d3462fa5e3))
9+
* add run_experiment task type ([#70545](https://github.com/Arize-ai/arize/issues/70545)) ([2ed75b9](https://github.com/Arize-ai/arize/commit/2ed75b998fb90298575329f4a63c95435a9a74b2))
10+
311
## [8.22.4](https://github.com/Arize-ai/arize/compare/arize-python-sdk/v8.22.3...arize-python-sdk/v8.22.4) (2026-05-08)
412

513

README.md

Lines changed: 65 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -881,21 +881,59 @@ task_list = resp.tasks
881881

882882
### Create a Task
883883

884-
Tasks run evaluators against spans from a project or experiments from a dataset. Either `project` or `dataset` is required (not both). At least one evaluator is required.
884+
There are two task types for running evaluators, and one for server-side LLM experiments. Use the appropriate typed helper:
885+
886+
**Template or code evaluation task** — runs evaluators against spans from a project or experiments from a dataset.
885887

886888
```python
887-
from arize.tasks.types import TasksCreateRequestEvaluatorsInner
889+
from arize.tasks.types import BaseEvaluationTaskRequestEvaluatorsInner
888890

889-
task = client.tasks.create(
890-
name="<your-task-name>",
891+
# Project-based, continuous template evaluation task
892+
task = client.tasks.create_evaluation_task(
893+
name="Production Hallucination Check",
891894
task_type="template_evaluation",
892-
evaluators=[TasksCreateRequestEvaluatorsInner(...)],
893-
project="<project-id-or-name>", # Required if not using dataset
894-
# dataset="<dataset-id-or-name>", # Required if not using project
895-
space=..., # Optional, space ID or name
896-
sampling_rate=..., # Optional, fraction of data to evaluate (0.0–1.0)
897-
is_continuous=..., # Optional, run continuously on new data
898-
query_filter=..., # Optional, filter expression for spans
895+
evaluators=[BaseEvaluationTaskRequestEvaluatorsInner(
896+
evaluator_id="<evaluator-id>", # Required
897+
column_mappings={"input": "attributes.input.value"}, # Optional
898+
)],
899+
project="<project-id-or-name>", # Required if not using dataset
900+
space=..., # Optional, space ID or name
901+
sampling_rate=1.0, # Optional, fraction of data to evaluate (0.0–1.0)
902+
is_continuous=True, # Optional, run continuously on new data
903+
query_filter=..., # Optional, filter expression for spans
904+
)
905+
906+
# Dataset-based code evaluation task
907+
task = client.tasks.create_evaluation_task(
908+
name="My Code Evaluator Task",
909+
task_type="code_evaluation",
910+
evaluators=[BaseEvaluationTaskRequestEvaluatorsInner(evaluator_id="<evaluator-id>")],
911+
dataset="<dataset-id-or-name>", # Required if not using project
912+
experiment_ids=["<experiment-id>"], # Required when using dataset
913+
)
914+
```
915+
916+
**Run experiment task** — the server drives LLM calls using the configured AI integration. No local callable is required.
917+
918+
```python
919+
from arize.tasks.types import LlmGenerationRunConfig, RunConfiguration
920+
921+
task = client.tasks.create_run_experiment_task(
922+
name="GPT-4o Baseline Task",
923+
dataset="<dataset-id-or-name>",
924+
run_configuration=RunConfiguration(
925+
actual_instance=LlmGenerationRunConfig(
926+
experiment_type="llm_generation",
927+
ai_integration_id="<ai-integration-id>",
928+
model_name="gpt-4o",
929+
input_variable_format="f_string",
930+
messages=[
931+
{"role": "system", "content": "You are a helpful assistant."},
932+
{"role": "user", "content": "Answer: {question}"},
933+
],
934+
)
935+
),
936+
space=..., # Optional, space ID or name
899937
)
900938
```
901939

@@ -910,16 +948,26 @@ task = client.tasks.get(
910948

911949
### Trigger a Task Run
912950

913-
You can trigger an on-demand run of a task. The returned `TaskRun` will initially have `"pending"` status.
951+
You can trigger an on-demand run of a task. The returned `TaskRun` will initially have `"pending"` status. The method automatically dispatches based on the task type.
914952

915953
```python
954+
# Evaluation task (template_evaluation / code_evaluation)
916955
run = client.tasks.trigger_run(
917956
task="<task-id-or-name>",
918-
space=..., # Optional
919-
data_start_time=..., # Optional, start of data window
920-
data_end_time=..., # Optional, end of data window
921-
max_spans=..., # Optional, maximum spans to evaluate
922-
override_evaluations=..., # Optional, re-evaluate already-evaluated spans
957+
space=..., # Optional
958+
data_start_time=..., # Optional, start of data window
959+
data_end_time=..., # Optional, end of data window
960+
max_spans=..., # Optional, maximum spans to evaluate
961+
override_evaluations=..., # Optional, re-evaluate already-evaluated spans
962+
)
963+
964+
# Run experiment task — experiment_name is required
965+
run = client.tasks.trigger_run(
966+
task="<task-id-or-name>",
967+
space=..., # Optional
968+
experiment_name="GPT-4o Baseline v2", # Required for run_experiment tasks
969+
max_examples=50, # Optional, limit number of dataset examples
970+
tracing_metadata={"source": "api"}, # Optional, metadata for traces
923971
)
924972
```
925973

docs/source/_static/switcher.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@
44
"url": "https://arize-client-python.readthedocs.io/en/latest/",
55
"preferred": true
66
},
7+
{
8+
"version": "v8.23.0",
9+
"url": "https://arize-client-python.readthedocs.io/en/v8.23.0/"
10+
},
711
{
812
"version": "v8.22.0",
913
"url": "https://arize-client-python.readthedocs.io/en/v8.22.0/"

docs/source/tasks.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,19 @@
2828
:members:
2929
:member-order: bysource
3030
31-
.. autoclass:: TasksCreateRequestEvaluatorsInner
31+
.. autoclass:: BaseEvaluationTaskRequestEvaluatorsInner
32+
:members:
33+
:member-order: bysource
34+
35+
.. autoclass:: LlmGenerationRunConfig
36+
:members:
37+
:member-order: bysource
38+
39+
.. autoclass:: TemplateEvaluationRunConfig
40+
:members:
41+
:member-order: bysource
42+
43+
.. autoclass:: RunConfiguration
3244
:members:
3345
:member-order: bysource
3446
```

src/.openapi-generator/FILES

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,8 @@ arize/_generated/api_client/docs/AssignAnnotationQueueRecordRequestBody.md
7373
arize/_generated/api_client/docs/AssignmentMethod.md
7474
arize/_generated/api_client/docs/AwsProviderMetadata.md
7575
arize/_generated/api_client/docs/AwsProviderMetadataKind.md
76+
arize/_generated/api_client/docs/BaseEvaluationTaskRequest.md
77+
arize/_generated/api_client/docs/BaseEvaluationTaskRequestEvaluatorsInner.md
7678
arize/_generated/api_client/docs/BuiltinRoleAssignment.md
7779
arize/_generated/api_client/docs/BuiltinUserRoleAssignment.md
7880
arize/_generated/api_client/docs/CategoricalAnnotationConfig.md
@@ -84,6 +86,9 @@ arize/_generated/api_client/docs/ContinuousAnnotationConfig.md
8486
arize/_generated/api_client/docs/ContinuousAnnotationConfigCreate.md
8587
arize/_generated/api_client/docs/CreateAnnotationConfigRequestBody.md
8688
arize/_generated/api_client/docs/CreateAnnotationQueueRequestBody.md
89+
arize/_generated/api_client/docs/CreateCodeEvaluationTaskRequest.md
90+
arize/_generated/api_client/docs/CreateRunExperimentTaskRequest.md
91+
arize/_generated/api_client/docs/CreateTemplateEvaluationTaskRequest.md
8792
arize/_generated/api_client/docs/CreateUserRequest.md
8893
arize/_generated/api_client/docs/CustomCodeConfig.md
8994
arize/_generated/api_client/docs/CustomRoleAssignment.md
@@ -132,12 +137,20 @@ arize/_generated/api_client/docs/InputVariableFormat.md
132137
arize/_generated/api_client/docs/InviteMode.md
133138
arize/_generated/api_client/docs/InvocationParams.md
134139
arize/_generated/api_client/docs/LLMMessage.md
140+
arize/_generated/api_client/docs/LlmGenerationRunConfig.md
135141
arize/_generated/api_client/docs/LlmProvider.md
136142
arize/_generated/api_client/docs/ManagedCodeConfig.md
137143
arize/_generated/api_client/docs/ManagedCodeEvaluator.md
138144
arize/_generated/api_client/docs/MessageRole.md
139145
arize/_generated/api_client/docs/OptimizationDirection.md
140146
arize/_generated/api_client/docs/Organization.md
147+
arize/_generated/api_client/docs/OrganizationBuiltinRoleAssignment.md
148+
arize/_generated/api_client/docs/OrganizationCustomRoleAssignment.md
149+
arize/_generated/api_client/docs/OrganizationMembership.md
150+
arize/_generated/api_client/docs/OrganizationMembershipInput.md
151+
arize/_generated/api_client/docs/OrganizationRole.md
152+
arize/_generated/api_client/docs/OrganizationRoleAssignment.md
153+
arize/_generated/api_client/docs/OrganizationRoleAssignmentType.md
141154
arize/_generated/api_client/docs/OrganizationsApi.md
142155
arize/_generated/api_client/docs/OrganizationsCreateRequest.md
143156
arize/_generated/api_client/docs/OrganizationsList200Response.md
@@ -182,6 +195,7 @@ arize/_generated/api_client/docs/RoleCreate.md
182195
arize/_generated/api_client/docs/RoleUpdate.md
183196
arize/_generated/api_client/docs/RolesApi.md
184197
arize/_generated/api_client/docs/RolesList200Response.md
198+
arize/_generated/api_client/docs/RunConfiguration.md
185199
arize/_generated/api_client/docs/Space.md
186200
arize/_generated/api_client/docs/SpaceMembership.md
187201
arize/_generated/api_client/docs/SpaceMembershipInput.md
@@ -207,17 +221,21 @@ arize/_generated/api_client/docs/TaskEvaluator.md
207221
arize/_generated/api_client/docs/TaskRun.md
208222
arize/_generated/api_client/docs/TasksApi.md
209223
arize/_generated/api_client/docs/TasksCreateRequest.md
210-
arize/_generated/api_client/docs/TasksCreateRequestEvaluatorsInner.md
211224
arize/_generated/api_client/docs/TasksList200Response.md
212225
arize/_generated/api_client/docs/TasksListRuns200Response.md
213226
arize/_generated/api_client/docs/TasksTriggerRunRequest.md
214227
arize/_generated/api_client/docs/TasksUpdateRequest.md
215228
arize/_generated/api_client/docs/TemplateConfig.md
229+
arize/_generated/api_client/docs/TemplateEvaluationRunConfig.md
216230
arize/_generated/api_client/docs/ToolCall.md
217231
arize/_generated/api_client/docs/ToolCallFunction.md
218232
arize/_generated/api_client/docs/ToolCallType.md
219233
arize/_generated/api_client/docs/ToolConfig.md
234+
arize/_generated/api_client/docs/TriggerEvaluationTaskRunRequest.md
235+
arize/_generated/api_client/docs/TriggerRunExperimentTaskRunRequest.md
220236
arize/_generated/api_client/docs/UpdateAnnotationQueueRequestBody.md
237+
arize/_generated/api_client/docs/UpdateEvaluationTaskRequest.md
238+
arize/_generated/api_client/docs/UpdateRunExperimentTaskRequest.md
221239
arize/_generated/api_client/docs/User.md
222240
arize/_generated/api_client/docs/UserCreatedResponse.md
223241
arize/_generated/api_client/docs/UserRole.md
@@ -278,6 +296,8 @@ arize/_generated/api_client/models/assign_annotation_queue_record_request_body.p
278296
arize/_generated/api_client/models/assignment_method.py
279297
arize/_generated/api_client/models/aws_provider_metadata.py
280298
arize/_generated/api_client/models/aws_provider_metadata_kind.py
299+
arize/_generated/api_client/models/base_evaluation_task_request.py
300+
arize/_generated/api_client/models/base_evaluation_task_request_evaluators_inner.py
281301
arize/_generated/api_client/models/builtin_role_assignment.py
282302
arize/_generated/api_client/models/builtin_user_role_assignment.py
283303
arize/_generated/api_client/models/categorical_annotation_config.py
@@ -289,6 +309,9 @@ arize/_generated/api_client/models/continuous_annotation_config.py
289309
arize/_generated/api_client/models/continuous_annotation_config_create.py
290310
arize/_generated/api_client/models/create_annotation_config_request_body.py
291311
arize/_generated/api_client/models/create_annotation_queue_request_body.py
312+
arize/_generated/api_client/models/create_code_evaluation_task_request.py
313+
arize/_generated/api_client/models/create_run_experiment_task_request.py
314+
arize/_generated/api_client/models/create_template_evaluation_task_request.py
292315
arize/_generated/api_client/models/create_user_request.py
293316
arize/_generated/api_client/models/custom_code_config.py
294317
arize/_generated/api_client/models/custom_role_assignment.py
@@ -333,13 +356,21 @@ arize/_generated/api_client/models/gcp_provider_metadata_kind.py
333356
arize/_generated/api_client/models/input_variable_format.py
334357
arize/_generated/api_client/models/invite_mode.py
335358
arize/_generated/api_client/models/invocation_params.py
359+
arize/_generated/api_client/models/llm_generation_run_config.py
336360
arize/_generated/api_client/models/llm_message.py
337361
arize/_generated/api_client/models/llm_provider.py
338362
arize/_generated/api_client/models/managed_code_config.py
339363
arize/_generated/api_client/models/managed_code_evaluator.py
340364
arize/_generated/api_client/models/message_role.py
341365
arize/_generated/api_client/models/optimization_direction.py
342366
arize/_generated/api_client/models/organization.py
367+
arize/_generated/api_client/models/organization_builtin_role_assignment.py
368+
arize/_generated/api_client/models/organization_custom_role_assignment.py
369+
arize/_generated/api_client/models/organization_membership.py
370+
arize/_generated/api_client/models/organization_membership_input.py
371+
arize/_generated/api_client/models/organization_role.py
372+
arize/_generated/api_client/models/organization_role_assignment.py
373+
arize/_generated/api_client/models/organization_role_assignment_type.py
343374
arize/_generated/api_client/models/organizations_create_request.py
344375
arize/_generated/api_client/models/organizations_list200_response.py
345376
arize/_generated/api_client/models/organizations_update_request.py
@@ -378,6 +409,7 @@ arize/_generated/api_client/models/role_binding_update.py
378409
arize/_generated/api_client/models/role_create.py
379410
arize/_generated/api_client/models/role_update.py
380411
arize/_generated/api_client/models/roles_list200_response.py
412+
arize/_generated/api_client/models/run_configuration.py
381413
arize/_generated/api_client/models/space.py
382414
arize/_generated/api_client/models/space_membership.py
383415
arize/_generated/api_client/models/space_membership_input.py
@@ -400,17 +432,21 @@ arize/_generated/api_client/models/task.py
400432
arize/_generated/api_client/models/task_evaluator.py
401433
arize/_generated/api_client/models/task_run.py
402434
arize/_generated/api_client/models/tasks_create_request.py
403-
arize/_generated/api_client/models/tasks_create_request_evaluators_inner.py
404435
arize/_generated/api_client/models/tasks_list200_response.py
405436
arize/_generated/api_client/models/tasks_list_runs200_response.py
406437
arize/_generated/api_client/models/tasks_trigger_run_request.py
407438
arize/_generated/api_client/models/tasks_update_request.py
408439
arize/_generated/api_client/models/template_config.py
440+
arize/_generated/api_client/models/template_evaluation_run_config.py
409441
arize/_generated/api_client/models/tool_call.py
410442
arize/_generated/api_client/models/tool_call_function.py
411443
arize/_generated/api_client/models/tool_call_type.py
412444
arize/_generated/api_client/models/tool_config.py
445+
arize/_generated/api_client/models/trigger_evaluation_task_run_request.py
446+
arize/_generated/api_client/models/trigger_run_experiment_task_run_request.py
413447
arize/_generated/api_client/models/update_annotation_queue_request_body.py
448+
arize/_generated/api_client/models/update_evaluation_task_request.py
449+
arize/_generated/api_client/models/update_run_experiment_task_request.py
414450
arize/_generated/api_client/models/user.py
415451
arize/_generated/api_client/models/user_created_response.py
416452
arize/_generated/api_client/models/user_role.py
@@ -465,8 +501,6 @@ arize/_generated/api_client/test/test_api_key_status.py
465501
arize/_generated/api_client/test/test_assign_annotation_queue_record_request_body.py
466502
arize/_generated/api_client/test/test_assignment_method.py
467503
arize/_generated/api_client/test/test_aws_provider_metadata_kind.py
468-
arize/_generated/api_client/test/test_builtin_role_assignment.py
469-
arize/_generated/api_client/test/test_builtin_user_role_assignment.py
470504
arize/_generated/api_client/test/test_categorical_annotation_config.py
471505
arize/_generated/api_client/test/test_categorical_annotation_config_create.py
472506
arize/_generated/api_client/test/test_categorical_annotation_value.py
@@ -477,8 +511,6 @@ arize/_generated/api_client/test/test_continuous_annotation_config_create.py
477511
arize/_generated/api_client/test/test_create_annotation_config_request_body.py
478512
arize/_generated/api_client/test/test_create_annotation_queue_request_body.py
479513
arize/_generated/api_client/test/test_custom_code_config.py
480-
arize/_generated/api_client/test/test_custom_role_assignment.py
481-
arize/_generated/api_client/test/test_custom_user_role_assignment.py
482514
arize/_generated/api_client/test/test_dataset_example.py
483515
arize/_generated/api_client/test/test_dataset_example_update.py
484516
arize/_generated/api_client/test/test_dataset_version_with_example_ids.py
@@ -560,10 +592,6 @@ arize/_generated/api_client/test/test_role_create.py
560592
arize/_generated/api_client/test/test_role_update.py
561593
arize/_generated/api_client/test/test_roles_api.py
562594
arize/_generated/api_client/test/test_roles_list200_response.py
563-
arize/_generated/api_client/test/test_space_membership.py
564-
arize/_generated/api_client/test/test_space_membership_input.py
565-
arize/_generated/api_client/test/test_space_role_assignment.py
566-
arize/_generated/api_client/test/test_space_role_assignment_type.py
567595
arize/_generated/api_client/test/test_spaces_create_request.py
568596
arize/_generated/api_client/test/test_spaces_update_request.py
569597
arize/_generated/api_client/test/test_spans_delete200_response.py
@@ -576,7 +604,6 @@ arize/_generated/api_client/test/test_task_evaluator.py
576604
arize/_generated/api_client/test/test_task_run.py
577605
arize/_generated/api_client/test/test_tasks_api.py
578606
arize/_generated/api_client/test/test_tasks_create_request.py
579-
arize/_generated/api_client/test/test_tasks_create_request_evaluators_inner.py
580607
arize/_generated/api_client/test/test_tasks_list200_response.py
581608
arize/_generated/api_client/test/test_tasks_list_runs200_response.py
582609
arize/_generated/api_client/test/test_tasks_trigger_run_request.py
@@ -587,9 +614,6 @@ arize/_generated/api_client/test/test_tool_call_function.py
587614
arize/_generated/api_client/test/test_tool_call_type.py
588615
arize/_generated/api_client/test/test_tool_config.py
589616
arize/_generated/api_client/test/test_user.py
590-
arize/_generated/api_client/test/test_user_role_assignment.py
591-
arize/_generated/api_client/test/test_user_role_assignment_type.py
592-
arize/_generated/api_client/test/test_user_space_role.py
593617
arize/_generated/api_client/test/test_user_status.py
594618
arize/_generated/api_client/test/test_user_update.py
595619
arize/_generated/api_client/test/test_users_api.py

0 commit comments

Comments
 (0)