Skip to content

Commit 06355dc

Browse files
faymarieMarie-Luise KlauswochingeArzelaAscoIiagnieszka-m
authored
feat: enable publishing pipelines and indexes to deepset AI platfrom (#256)
* enable publish * fixes * fixes * fixes * format fixes * format fixes * format fixes * format fixes * format fixes * format fixes * fix sorting * optimize input/ouput validations if else statements * early return in Pipeline import * change to debug log status * Add an SDK class with init method * rephrase pydoc style * rephrase pydoc style * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Tobias Wochinger <tobias.wochinger@deepset.ai> * support sync and async publishing * use yaml parsing * fix docstrings * fix docstrings * fix test interference * fix format * use separate configs for Index and Pipeline * add docstrings directly to code * add docstrings directly to code * formatting * docs fixes * revert local version * close event loop if it was newly created * test exception * use Mock object * remove changed stout * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: ArzelaAscoIi <37148029+ArzelaAscoIi@users.noreply.github.com> * reset rate limiter in tests * fix import * re-add log configs * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: ArzelaAscoIi <37148029+ArzelaAscoIi@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: ArzelaAscoIi <37148029+ArzelaAscoIi@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: ArzelaAscoIi <37148029+ArzelaAscoIi@users.noreply.github.com> * move _convert_to_yaml_dict to models * rename to _from_haystack_pipelin * haystack version * rename to import instead of publish * revert python version * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * fix dics * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * fix docs * fix docs * fix docs * fix docs * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/models.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * fix docs * fix docs * add fullstops * from future import annothations * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/sdk.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * from future import annothations * update docstring to use paramater * remove initilized * disable too few public methods * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * Update deepset_cloud_sdk/workflows/pipeline_client/pipeline_service.py Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com> * fix docstring * fix docstring * fix docstring * fix docstring * fix docstring * fix docstring * fix docstring * fix docstring --------- Co-authored-by: Marie-Luise Klaus <marieluiseklauscode@gmail.com> Co-authored-by: Tobias Wochinger <tobias.wochinger@deepset.ai> Co-authored-by: ArzelaAscoIi <37148029+ArzelaAscoIi@users.noreply.github.com> Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
1 parent d5dcd98 commit 06355dc

10 files changed

Lines changed: 1232 additions & 9 deletions

File tree

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,21 @@
1-
"""Workflows for the deepset cloud SDK."""
1+
"""Workflows for deepset AI platform SDK."""
2+
3+
from deepset_cloud_sdk.workflows.pipeline_client.models import (
4+
IndexConfig,
5+
IndexInputs,
6+
IndexOutputs,
7+
PipelineConfig,
8+
PipelineInputs,
9+
PipelineOutputs,
10+
)
11+
from deepset_cloud_sdk.workflows.sdk import DeepsetSDK
12+
13+
__all__ = [
14+
"PipelineInputs",
15+
"IndexInputs",
16+
"IndexOutputs",
17+
"PipelineOutputs",
18+
"IndexConfig",
19+
"PipelineConfig",
20+
"DeepsetSDK",
21+
]
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Package to enable importing pipelines and indexes to deepset AI platform."""
Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
"""Models for the pipeline service."""
2+
from typing import List
3+
4+
from pydantic import BaseModel, Field, model_validator
5+
6+
7+
class InputOutputBaseModel(BaseModel):
8+
"""Base model for input and output configurations.
9+
10+
This class provides common functionality for input and output models, such as YAML conversion.
11+
"""
12+
13+
def to_yaml_dict(self) -> dict:
14+
"""Convert the model to a YAML-compatible dictionary.
15+
16+
Clears empty values from the dictionary.
17+
18+
:return: Dictionary ready for YAML serialization.
19+
"""
20+
fields = self.model_dump(exclude_none=True)
21+
# Remove empty values
22+
return {k: v for k, v in fields.items() if v}
23+
24+
25+
class PipelineInputs(InputOutputBaseModel):
26+
"""Pipeline input configuration.
27+
28+
Defines the components that should receive the Query input and any filters that apply to it.
29+
30+
:param query: List of components that will receive the `query` input.
31+
Specify each component in the format: '<component-name>.<run-method-parameter-name>', for example: 'retriever.query'
32+
:param filters: Optional list of components that will receive the filters input.
33+
Specify each component using the format: '<component-name>.<run-method-parameter-name>', for example: 'retriever.filters'.
34+
"""
35+
36+
model_config = {"extra": "allow"} # Allow additional fields in inputs
37+
38+
query: List[str] = Field(
39+
...,
40+
description=(
41+
"List of components and parameters that will receive the `query` input when they are executed. "
42+
"Use the format: '<component-name>.<run-method-parameter-name>', for example: 'retriever.query'."
43+
),
44+
min_items=1,
45+
)
46+
filters: List[str] = Field(
47+
default_factory=list,
48+
description=(
49+
"List of components and parameters that will receive the `filters` input when they are executed. "
50+
"Use the format: '<component-name>.<run-method-parameter-name>', for example: 'retriever.filters'."
51+
),
52+
)
53+
54+
55+
class PipelineOutputs(InputOutputBaseModel):
56+
"""Pipeline output configuration.
57+
58+
Specify the components that will output `documents`, `answers`, or both.
59+
You must include at least one. The outputs of these components become the final output of the pipeline.
60+
61+
:param documents: Name of the component and parameter that will provide `documents` as output.
62+
Use the format '<component-name>.<output-parameter>', for example: 'retriever.documents'.
63+
:param answers: Name of the component and parameter that will provide `answers` as output.
64+
Use the format '<component-name>.<output-parameter>', for example: 'reader.answers'.
65+
"""
66+
67+
model_config = {"extra": "allow"} # Allow additional fields in outputs
68+
69+
documents: str | None = Field(
70+
default=None,
71+
description="Name of the component that will provide `documents` as output. "
72+
"Format: '<component-name>.<output-parameter>', for example: 'meta_ranker.documents'",
73+
)
74+
answers: str | None = Field(
75+
default=None,
76+
description="Name of the component that will provide `answers` as output. "
77+
"Format: '<component-name>.<output-parameter>', for example: 'answers_builder.answers'",
78+
)
79+
80+
@model_validator(mode="after")
81+
def validate_documents_xor_answers(self) -> "PipelineOutputs":
82+
"""Validate that either `documents`, `answers`, or both are defined."""
83+
if not self.documents and not self.answers:
84+
raise ValueError("Define at least one pipeline output, either 'documents, 'answers' or both.")
85+
return self
86+
87+
88+
class IndexOutputs(InputOutputBaseModel):
89+
"""Output configuration for the index.
90+
91+
Index outputs are optional.
92+
"""
93+
94+
model_config = {"extra": "allow"} # Allow additional fields in outputs
95+
96+
97+
class PipelineConfig(BaseModel):
98+
"""Configuration required to import the pipeline into deepset AI Platform.
99+
100+
:param name: Name of the pipeline to be imported
101+
:param inputs: Pipeline input configuration. Use `PipelineInputs` model to define the inputs.
102+
:param outputs: Pipeline output configuration. Use `PipelineOutputs` model to define the outputs.
103+
"""
104+
105+
model_config = {"extra": "forbid"}
106+
107+
name: str = Field(..., description="The name of the pipeline to be imported", min_length=1)
108+
inputs: PipelineInputs = Field(
109+
default_factory=PipelineInputs,
110+
description=("Pipeline input configuration. Use `PipelineInputs` model to define the inputs."),
111+
)
112+
outputs: PipelineOutputs = Field(
113+
default_factory=PipelineOutputs,
114+
description=("Pipeline output configuration. Use `PipelineOutputs` model to define the outputs."),
115+
)
116+
117+
118+
class IndexInputs(InputOutputBaseModel):
119+
"""Configuration required to import an index into deepset AI Platform.
120+
121+
Defines the index components that should receive the `Files` input.
122+
123+
:param files: List of components and parameters that should receive files as input.
124+
Specify the components using the format: '<component-name>.<run-method-parameter-name>', for example: 'file_type_router.sources'.
125+
"""
126+
127+
model_config = {"extra": "allow"} # Allow additional fields in inputs
128+
129+
files: List[str] = Field(
130+
default_factory=list,
131+
description=(
132+
"List of components and parameters that will receive files as input when they're executed. "
133+
"Format: '<component-name>.<run-parameter-name>', for example: 'file_type_router.sources'."
134+
),
135+
)
136+
137+
138+
class IndexConfig(BaseModel):
139+
"""Index configuration for importing an index to deepset AI platform.
140+
141+
:param name: Name of the index to be imported.
142+
:param inputs: Index input configuration. Use `IndexInputs` model to define the inputs.
143+
:param outputs: Index output configuration. Optional. Use `IndexOutputs` model to define the outputs.
144+
"""
145+
146+
model_config = {"extra": "forbid"}
147+
148+
name: str = Field(..., description="Name of the index to be imported.", min_length=1)
149+
inputs: IndexInputs = Field(
150+
default_factory=IndexInputs,
151+
description=("Input configuration for the index. Use `IndexInputs` model to define the inputs."),
152+
)
153+
outputs: IndexOutputs | None = Field(
154+
default_factory=IndexOutputs,
155+
description=("Optional output configuration for the index. Use `IndexOutputs` model to define the outputs."),
156+
)

0 commit comments

Comments
 (0)