You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+86-45Lines changed: 86 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,7 @@
3
3
Finetune Controller is a robust and flexible system designed to manage and streamline the fine-tuning of machine learning models on Kubernetes, particularly within OpenShift clusters. This project leverages modern tools and workflows, enabling efficient development and deployment processes for AI-driven applications.
4
4
5
5
### Features
6
+
6
7
- Local Development: Get started quickly with a streamlined setup process using uv, a high-performance Python package and project manager.
7
8
- OpenShift Integration: Simplify deployment and scaling with OpenShift-specific configurations and GPU support for intensive workloads.
8
9
- MongoDB Backend: Seamlessly connect to a local or cluster-based MongoDB database.
@@ -11,12 +12,15 @@ Finetune Controller is a robust and flexible system designed to manage and strea
11
12
## Getting Started
12
13
13
14
If the cluster is already set up continue else follow the cluster setup instructions [here](#setup-openshift-cluster)
15
+
14
16
### Prereqs
15
-
1. Recommend using [uv](https://github.com/astral-sh/uv), *an extremely fast Python package and project manager*
16
17
17
-
```shell
18
-
pip install uv
19
-
```
18
+
1. Recommend using [uv](https://github.com/astral-sh/uv), _an extremely fast Python package and project manager_
19
+
20
+
```shell
21
+
pip install uv
22
+
```
23
+
20
24
2. A container engine such as Docker or Podman
21
25
22
26
<!-- ## Quick Setup using Compose
@@ -41,51 +45,59 @@ To stop the application:
41
45
docker compose down
42
46
``` -->
43
47
44
-
45
48
### Install
49
+
46
50
1. Create virtual environment and install dependencies
47
-
```shell
48
-
uv sync
49
-
```
50
51
51
-
2. Start a local developement mongo database *(or connect to one on cluster with port-forward)*
3. Connect to the Openshift cluster with the cli login command `oc login`. If cluster not already set up follow [these](#setup-openshift-cluster) steps
69
76
70
77
4. Create a project level `.env` file (see `.env.example`) and update the variables.
71
-
```shell
72
-
cp .env.example .env
73
-
```
78
+
79
+
```shell
80
+
cp .env.example .env
81
+
```
74
82
75
83
5. Make sure the virtual environment is activated and start the local finetuning controller application.
76
-
```shell
77
-
source .venv/bin/activate
78
84
79
-
uvicorn app.main:app --reload
80
-
```
85
+
```shell
86
+
source .venv/bin/activate
87
+
88
+
uvicorn app.main:app --reload
89
+
```
81
90
82
91
This will:
92
+
83
93
- Start MongoDB with the required configuration
84
94
- Build and start the FastAPI server
85
95
- Make the application available at http://localhost:8000
86
96
87
97
### Development and Contributing
98
+
88
99
Setup [pre-commit](https://pre-commit.com/#install) to keep linting and code styling up to standard.
100
+
89
101
```shell
90
102
uv sync
91
103
pre-commit install
@@ -94,57 +106,69 @@ pre-commit install
94
106
## Setup OpenShift Cluster Resources
95
107
96
108
### Create default project
109
+
97
110
Name can be descriptive for these examples we will use `finetune-controller`
First update the namepspace for the crd LocalQueue object in [default-user-queue.yaml](/crds/kueue/default-user-queue.yaml). default namepsace: "default"
157
+
136
158
```shell
137
159
yq e '.metadata.namespace = "finetune-controller"' -i crds/kueue/default-user-queue.yaml
138
160
```
139
161
140
162
Apply the default CRD config for Kueue or update by following their docs
163
+
141
164
```shell
142
165
kubectl apply -f crds/kueue/
143
166
```
144
167
145
168
### Install mongodb server
146
169
147
-
Example configuration. *do properly configure for production*
170
+
Example configuration. _do properly configure for production_
@@ -165,16 +189,19 @@ oc set env deployment/ml-pipeline-ui DISABLE_GKE_METADATA=true
165
189
``` -->
166
190
167
191
### Add GPU nodes to ROSA cluster
192
+
168
193
Go to your cluster on redhat console [admin dashboard](https://console.redhat.com/openshift/cluster-list). Add a machine pool of your choosing with the following configuration:
169
194
170
195
Taints
196
+
171
197
```
172
198
key: nvidia.com/gpu
173
199
value: <machine pool type or other>
174
200
effect: NoSchedule
175
201
```
176
202
177
203
Node Labels
204
+
178
205
```
179
206
Key: cluster-api/accelerator
180
207
Value: <gpu type e.g. V100 or empty>
@@ -183,6 +210,7 @@ Value: <gpu type e.g. V100 or empty>
1. Create a `.env.production` file and update the defaults. For this example set `MONGODB_URL=mongodb://mongodb-community-server.finetune-controller.svc.cluster.local:27017`
0 commit comments