Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/compute/pipelines/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Explore a collection of ready-to-use ML pipeline templates for training, evaluat

* **Schedule training on your inference clusters.** Training, evaluation, and inference workloads share the same Compute Orchestration clusters — no separate cluster to provision and pay for. Run a training job overnight on the same nodepool that serves your daytime inference traffic.
* **Outputs land in the model registry.** Trained and quantized models from pipeline runs are registered directly into Clarifai's model registry, ready to serve or evaluate further without an export step.
* **Python-first authoring.** Define pipelines as decorated Python functions with the [DSL](dsl-reference.md) — no parallel YAML to keep in sync.
* **Python or YAML — both first-class.** Author pipelines as decorated Python functions with the [DSL](dsl-reference.md), or as YAML config with [scaffold templates](create-api.md). Same engine, same templates, same upload path — pick whichever fits your team.

## Use Cases for Pipelines

Expand Down Expand Up @@ -89,15 +89,15 @@ The pipeline fine-tunes the model on a single-GPU instance (the template's compu

That's it — you have a fine-tuned model registered in your Clarifai model registry, ready to serve, evaluate, or refine further.

:::tip Python-first alternative
:::tip Two upload paths

The LoRA template ships with both a YAML scaffold (`config.yaml`) and a Python DSL form (`dsl.py`). To upload directly from the DSL Python file instead of the scaffold:
The LoRA template ships with both a YAML scaffold (`config.yaml`) and a Python DSL form (`dsl.py`). The default `clarifai pipeline upload` uses the YAML scaffold. To upload from the DSL Python file instead:

```bash
clarifai pipeline upload dsl.py
```

Both paths produce the same pipeline. The DSL form is the recommended starting point if you plan to author your own custom pipelines — see the [Pipeline DSL reference](dsl-reference.md) for the full API (`@step`, `step_ref`, `>>` composition, `base_image`, etc.).
Both paths produce the same pipeline. The [YAML config](create-api.md) is convenient for explicit, version-controlled configuration. The [Pipeline DSL](dsl-reference.md) is convenient for code-first authoring with type hints, `@step` decorators, and `>>` DAG composition. Use whichever fits your team.

:::

Expand Down Expand Up @@ -125,7 +125,7 @@ For custom workflows that don't fit a template, use the [Pipeline DSL](dsl-refer
* **[Pipeline DSL reference](dsl-reference.md)** — `@step`, `step_ref`, `>>` composition, `ComputeInfo`, secrets, codegen.
* **[Manage Pipelines](manage.md)** — list, validate, and inspect pipelines on the platform.
* **[Manage Pipeline Runs](manage-run.md)** — monitor, pause, resume, and cancel runs.
* **[Advanced: YAML / config-based pipelines](create-api.md)** — the scaffold-directory authoring flow underneath the templates, useful for existing pipelines or workflows that need explicit YAML control.
* **[YAML / config-based pipelines](create-api.md)** — the YAML config authoring path, useful for explicit version-controlled configuration, YAML-first workflows, or as a reference for the scaffold-directory file structure.

Comment on lines 126 to 129
### Pipeline outputs → Artifacts

Expand Down
8 changes: 4 additions & 4 deletions docs/compute/pipelines/create-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,16 @@ sidebar_position: 5
toc_max_heading_level: 5
---

# Advanced: YAML / Config-Based Pipelines
# YAML / Config-Based Pipelines

**YAML / config-based authoring flow for pipelines, with the full file structure reference**
<hr />

:::tip Looking for the Python-first path?
:::tip Prefer Python?

For new pipelines, we recommend the [Pipeline DSL](dsl-reference.md) — define steps as decorated Python functions, compose them with `>>`, and upload directly from a `.py` file with `clarifai pipeline upload my_pipeline.py`. No YAML to maintain.
Clarifai Pipelines support two authoring paths: this YAML / config-based flow, and the [Pipeline DSL](dsl-reference.md) for code-first authoring with `@step` decorators and `>>` DAG composition. Both paths reach the same engine and produce the same pipeline — pick whichever fits your team.

This page documents the **YAML / config-based** authoring flow: scaffold a directory of `config.yaml` + `pipeline_step.py` files with `clarifai pipeline init`, edit them, and upload the directory. Useful for existing pipelines, workflows that need explicit YAML control, or as a reference for the file structure that the DSL compiles down to.
This page documents the YAML path: scaffold a directory of `config.yaml` + `pipeline_step.py` files with `clarifai pipeline init`, edit them, and upload the directory.

:::

Expand Down