Skip to content

Commit 61ca332

Browse files
committed
Add Docker Model Runner documentation and compatibility tests
Add documentation for using Docker Model Runner with ModelPack models, including detection mechanism, field mappings, and media type conversions. Add downstream compatibility validation tests to prevent accidental breakage of fields and media types that downstream consumers depend on. Closes #151 Signed-off-by: Pradhyum Rajasekar <pradhyum314@gmail.com> Signed-off-by: pradhyum6144 <pradhyum314@gmail.com>
1 parent 5196234 commit 61ca332

3 files changed

Lines changed: 463 additions & 1 deletion

File tree

docs/docker-model-runner.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Using Docker Model Runner with ModelPack
2+
3+
This guide shows you how to use [Docker Model Runner](https://docs.docker.com/desktop/features/model-runner/) to pull and run AI models packaged using the ModelPack specification.
4+
5+
## What is Docker Model Runner?
6+
7+
Docker Model Runner is a built-in feature of Docker Desktop that enables pulling, managing, and running AI models directly from OCI registries. It natively supports the ModelPack specification format, allowing you to run ModelPack-packaged models without any additional tools.
8+
9+
## Prerequisites
10+
11+
- [Docker Desktop](https://docs.docker.com/get-docker/) 4.40 or later with Model Runner enabled
12+
- A ModelPack-compatible model pushed to an OCI registry (see [modctl](./modctl.md) or [AIKit](./aikit.md) for packaging)
13+
14+
## Enable Docker Model Runner
15+
16+
Docker Model Runner is available through Docker Desktop. Enable it in Docker Desktop settings:
17+
18+
1. Open Docker Desktop
19+
2. Go to **Settings** > **Features in development**
20+
3. Enable **Docker Model Runner**
21+
22+
You can verify it is enabled by running:
23+
24+
```bash
25+
docker model list
26+
```
27+
28+
## Pull a ModelPack Model
29+
30+
Docker Model Runner can pull models directly from OCI registries. When pulling a ModelPack-formatted artifact, Docker automatically detects the ModelPack config format and converts it for local use.
31+
32+
```bash
33+
# Pull a model from an OCI registry
34+
docker model pull myregistry.com/mymodel:v1.0
35+
```
36+
37+
## Run a Model
38+
39+
Once pulled, you can run inference using the model:
40+
41+
```bash
42+
# Run a model interactively
43+
docker model run myregistry.com/mymodel:v1.0
44+
45+
# Send a prompt to the model
46+
docker model run myregistry.com/mymodel:v1.0 "Explain cloud-native computing"
47+
```
48+
49+
## List and Manage Models
50+
51+
```bash
52+
# List all downloaded models
53+
docker model list
54+
55+
# Remove a model
56+
docker model rm myregistry.com/mymodel:v1.0
57+
```
58+
59+
## Use Models via the OpenAI-Compatible API
60+
61+
Docker Model Runner exposes an OpenAI-compatible API endpoint, enabling integration with existing tools and libraries:
62+
63+
```bash
64+
curl http://localhost:12434/engines/v1/chat/completions \
65+
-H "Content-Type: application/json" \
66+
-d '{
67+
"model": "myregistry.com/mymodel:v1.0",
68+
"messages": [{"role": "user", "content": "Hello!"}]
69+
}'
70+
```
71+
72+
## How ModelPack Format Is Detected
73+
74+
Docker Model Runner identifies a ModelPack artifact by checking the OCI config blob for any of the following fields:
75+
76+
- `config.paramSize` — the model parameter size
77+
- `descriptor.createdAt` — the model creation timestamp
78+
- `modelfs` — the model filesystem descriptor
79+
80+
If any of these fields are present, the artifact is recognized as a ModelPack-formatted model.
81+
82+
## Field Mapping: ModelPack to Docker
83+
84+
When Docker Model Runner pulls a ModelPack model, it converts the config fields to Docker's internal format:
85+
86+
| ModelPack Field | Docker Field | Description |
87+
|---|---|---|
88+
| `descriptor.createdAt` | `created` | Model creation timestamp |
89+
| `descriptor.name` | `descriptor.name` | Model name |
90+
| `descriptor.family` | `descriptor.family` | Model family |
91+
| `descriptor.description` | `descriptor.description` | Model description |
92+
| `descriptor.licenses` | `descriptor.licenses` | License information |
93+
| `config.paramSize` | `parameters` | Model parameter count |
94+
| `config.format` | `config.format` | Model format (e.g., GGUF) |
95+
| `config.quantization` | `config.quantization` | Quantization method |
96+
| `config.architecture` | `config.architecture` | Model architecture |
97+
| `modelfs` | `rootfs` | Layer content addresses |
98+
99+
## Media Type Mapping
100+
101+
ModelPack media types are converted to Docker's internal media types:
102+
103+
| ModelPack Media Type | Docker Media Type |
104+
|---|---|
105+
| `application/vnd.cncf.model.weight.v1.raw` | Mapped based on file extension (e.g., `.gguf``application/vnd.docker.ai.gguf.v3`) |
106+
| `application/vnd.cncf.model.weight.v1.tar+gzip` | `application/vnd.docker.ai.gguf.v3+gzip` |
107+
| `application/vnd.cncf.model.weight.config.v1.raw` | `application/vnd.docker.ai.config` |
108+
| `application/vnd.cncf.model.doc.v1.raw` | `application/vnd.docker.ai.doc` |
109+
110+
## Next Steps
111+
112+
- **Package models** using [modctl](./modctl.md) or [AIKit](./aikit.md) to create ModelPack artifacts
113+
- **Learn about the [Model CSI Driver](https://github.com/modelpack/model-csi-driver)** for Kubernetes integration
114+
- **Read the [full ModelPack specification](./spec.md)** for technical implementation details

docs/getting-started.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ This section lists the core infrastructure components that ModelPack is working
3535
- **[modctl](https://github.com/modelpack/modctl)**: CLI tool for building, pushing, pulling, and managing OCI model artifacts
3636
- **[KitOps](https://kitops.ml/)**: ModelKit packaging and deployment platform that supports the ModelPack specification
3737
- **[AIKit](https://kaito-project.github.io/aikit/docs/packaging)**: Package AI models as OCI artifacts from local, HTTP, or Hugging Face sources with extensible formats, including ModelPack specification
38+
- **[Docker Model Runner](https://docs.docker.com/desktop/features/model-runner/)**: Pull and run ModelPack models directly from OCI registries using Docker Desktop
3839

3940
### Kubernetes Integration
4041

@@ -65,6 +66,7 @@ The ModelPack specification can be used with different tools depending on your n
6566
- **[modctl](./modctl.md)**: CLI tool for building, pushing, pulling, and managing OCI model artifacts. Great for command-line workflows and CI/CD pipelines.
6667
- **[AIKit](./aikit.md)**: Package AI models as OCI artifacts from local, HTTP, or Hugging Face sources with extensible formats.
6768
- **[KitOps](https://kitops.ml/)**: ModelKit packaging and deployment platform that supports the ModelPack specification.
69+
- **[Docker Model Runner](./docker-model-runner.md)**: Pull and run ModelPack models directly from OCI registries using Docker Desktop.
6870

6971
### Install Model CSI Driver
7072

@@ -100,7 +102,7 @@ This example shows how to mount a model artifact directly into a Kubernetes pod
100102
101103
## Next Steps
102104
103-
1. **Get hands-on experience**: Follow the step-by-step guides for [modctl](./modctl.md) or [AIKit](./aikit.md)
105+
1. **Get hands-on experience**: Follow the step-by-step guides for [modctl](./modctl.md), [AIKit](./aikit.md), or [Docker Model Runner](./docker-model-runner.md)
104106
2. **Explore the [full ModelPack specification](./spec.md)** for technical implementation details
105107
3. **Join the community** on [CNCF Slack #modelpack](https://cloud-native.slack.com/archives/C07T0V480LF)
106108
4. **Contribute** to the ModelPack project - see our [contributing guidelines](../CONTRIBUTING.md)
@@ -114,5 +116,6 @@ This example shows how to mount a model artifact directly into a Kubernetes pod
114116
- [KitOps](https://kitops.ml/)
115117
- [Hugging Face](https://huggingface.co/)
116118
- [AIKit](https://github.com/kaito-project/aikit)
119+
- [Docker Model Runner](https://docs.docker.com/desktop/features/model-runner/)
117120
118121
The ModelPack specification represents the next evolution in infrastructure standardization, bringing the benefits of containerization to AI model management. Start with the basics, explore the ecosystem, and join our growing community of contributors and users building the future of cloud-native AI.

0 commit comments

Comments
 (0)