Skip to content

Commit 954ca69

Browse files
lbeckman314lbeckman314
committed
feat: Add missing Funnel GCP Batch docs and update Kubernetes docs
Signed-off-by: Liam Beckman <lbeckman314@gmail.com>
1 parent 123383a commit 954ca69

6 files changed

Lines changed: 165 additions & 77 deletions

File tree

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
---
22
template: home.html
3+
body_class: home
34
hide:
45
- navigation
56
- toc

‎docs/stylesheets/extra.css‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ h3 {
2727
}
2828

2929
/* horizontal dividers */
30-
.md-content h2::before {
30+
body.home .md-content h2::before {
3131
content: "";
3232
display: block;
3333
width: 100%;

‎docs/tools/funnel/docs/compute/gcp-batch.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
title: GCP Batch
3+
menu:
4+
main:
5+
parent: Compute
6+
weight: 20
7+
---
8+
9+
!!! warning "Current Limitations ⚠️"
10+
- Latest Funnel release ([v0.11.11](https://github.com/ohsu-comp-bio/funnel/releases/tag/v0.11.11)) requires specific bucket prefixing in the inputs and outputs
11+
- e.g. `/mnt/disks/<BUCKET>/<FILE>` instead of `/<FILE>`
12+
- Nextflow workflows are currently not supported (as Nextflow expects root-level working directories → `/work`)
13+
14+
# Overview
15+
16+
The following steps illustrate how to run a TES tasks via GCP Batch utilizing Google Storage Buckets.
17+
18+
# Quick Start
19+
20+
```sh
21+
curl -fsSL https://calypr.github.io/funnel/install.sh | bash
22+
23+
funnel server run --Compute "gcp-batch" \
24+
--GCPBatch.Project "example-project" \
25+
--GCPBatch.Location "us-central1"
26+
```
27+
28+
## 3. Submit Task
29+
30+
<details>
31+
<summary><code>gcp-example.json</code></summary>
32+
33+
```json
34+
{
35+
"name": "Input/Output Test",
36+
"inputs": [
37+
{
38+
"url": "gs://tes-batch-integration/README.md",
39+
"path": "/mnt/disks/tes-batch-integration/README.md"
40+
}
41+
],
42+
"outputs": [
43+
{
44+
"url": "gs://tes-batch-integration/README.md.sha256",
45+
"path": "/mnt/disks/tes-batch-integration/README.md.sha256"
46+
}
47+
],
48+
"executors": [
49+
{
50+
"image": "alpine",
51+
"command": [
52+
"sha256sum",
53+
"/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
54+
]
55+
}
56+
]
57+
}
58+
```
59+
60+
</details>
61+
62+
```sh
63+
funnel task create gcp-example.json
64+
<TASK ID>
65+
```
66+
67+
## 4. Query Task
68+
69+
```sh
70+
funnel task get <TASK ID>
71+
```
72+
73+
```json
74+
{
75+
"executors": [
76+
{
77+
"command": [
78+
"sha256sum",
79+
"/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
80+
],
81+
"image": "alpine"
82+
}
83+
],
84+
"id": "d6f0tgpurbu7o23pgj20",
85+
"inputs": [
86+
{
87+
"path": "/mnt/disks/tes-batch-integration/README.md",
88+
"url": "gs://tes-batch-integration/README.md"
89+
}
90+
],
91+
"name": "GCP Batch Task Example",
92+
"outputs": [
93+
{
94+
"path": "/mnt/disks/tes-batch-integration/README.md.sha256",
95+
"url": "gs://tes-batch-integration/README.md.sha256"
96+
}
97+
],
98+
"state": "COMPLETE"
99+
}
100+
```
101+
102+
## 5. Verify Outputs
103+
104+
```sh
105+
gsutil cat gs://tes-batch-integration/README.md.sha256
106+
9b9916cea5348edd6ad78893231edb81fc96772d1dd99fae9c2a64f84646cb1c /mnt/disks/tes-batch-integration/README.md
107+
```
108+
109+
# Additional Resources
110+
111+
- [GCP Batch Documentation](https://docs.cloud.google.com/batch/docs)

‎docs/tools/funnel/docs/compute/k8s-pvc.png‎

40.5 KB
Loading

‎docs/tools/funnel/docs/compute/kubernetes.md‎

Lines changed: 47 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -6,116 +6,87 @@ menu:
66
weight: 20
77
---
88

9-
> Funnel on Kubernetes is in active development and may involve frequent updates
9+
!!! warning
10+
Funnel on Kubernetes is in active development and may involve frequent updates 🚧
1011

11-
# Quick Start
12+
# Kubernetes
1213

13-
## 1. Deploying with Helm
14+
## 1. Add Helm Repo
1415

15-
```sh
16-
helm repo add ohsu https://ohsu-comp-bio.github.io/helm-charts
17-
helm repo update
18-
helm upgrade --install ohsu funnel
19-
```
20-
21-
## Alternative: Deploying with `kubectl` ⚙️"
22-
23-
### 1. Create a Service:
24-
25-
Deploy it:
16+
!!! note
17+
See Funnel's [Helm Charts](https://github.com/calypr/helm-charts/tree/main/charts/funnel) for the latest configuration options
2618

2719
```sh
28-
kubectl apply -f funnel-service.yml
29-
```
30-
31-
### 2. Create Funnel config files
32-
33-
> *[funnel-server.yaml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-server.yaml)*
34-
35-
> *[funnel-worker.yaml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-worker.yaml)*
20+
helm repo add calypr https://calypr.org/helm-charts
3621

37-
Get the clusterIP:
22+
helm repo update calypr
3823

39-
```sh
40-
{% raw %}
41-
export HOSTNAME=$(kubectl get services funnel --output=jsonpath='{.spec.clusterIP}')
42-
43-
sed -i "s|\${HOSTNAME}|${HOSTNAME}|g" funnel-worker.yaml
44-
{% endraw %}
24+
helm search repo calypr/funnel
4525
```
4626

47-
### 3. Create a ConfigMap
27+
## 2. Deploy Funnel
4828

4929
```sh
50-
kubectl create configmap funnel-config --from-file=funnel-server.yaml --from-file=funnel-worker.yaml
51-
```
52-
53-
### 4. Create a Service Account for Funnel
54-
55-
Define a Role and RoleBinding:
30+
helm upgrade --install funnel calypr/funnel
5631

57-
> *[role.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/role.yml)*
32+
kubectl rollout status deployment/funnel-server
5833

59-
> *[role_binding.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/role_binding.yml)*
60-
61-
```sh
62-
kubectl create serviceaccount funnel-sa --namespace default
63-
kubectl apply -f role.yml
64-
kubectl apply -f role_binding.yml
34+
kubectl get deployments
35+
# NAME READY STATUS
36+
# funnel-server 1/1 Running
6537
```
6638

67-
### 5. Create a Persistent Volume Claim
68-
69-
> *[funnel-storage-pvc.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-storage-pvc.yml)*
39+
## 4. Port-Forward for Local Access
7040

7141
```sh
72-
kubectl apply -f funnel-storage-pvc.yml
73-
```
42+
kubectl port-forward svc/funnel 8000:8000
7443

75-
### 6. Create a Deployment
76-
77-
> *[funnel-deployment.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-deployment.yml)*
78-
79-
```sh
80-
kubectl apply -f funnel-deployment.yml
81-
```
82-
83-
{% raw %}{{< /details >}}{% endraw %}
84-
85-
# 2. Proxy the Service for local testing
86-
87-
```sh
88-
kubectl port-forward service/funnel 8000:8000
89-
```
90-
91-
Now the funnel server can be accessed as if it were running locally. This can be verified by listing all tasks, which will return an empty JSON list:
92-
93-
```sh
9444
funnel task list
9545
# {}
9646
```
9747

98-
A task can then be submitted following the [standard workflow](../tasks.md):
48+
## 4. Submit Example Task
9949

10050
```sh
10151
funnel examples hello-world > hello-world.json
10252

10353
funnel task create hello-world.json
10454
# <Task ID>
55+
56+
funnel task get <Task ID> --view MINIMAL
57+
# { "id": "...", "state": "COMPLETE" }
10558
```
10659

10760
# Storage Architecture
10861

109-
<a href="https://www.figma.com/board/bzgv8kVL2QKESU3Sqn7S1a/Funnel-%2B-Gen3?node-id=2-1059&t=9bcuG0bMAcxBLcRD-1">
110-
<img title="K8s Storage" src="/img/k8s-pvc.png" />
111-
</a>
62+
Funnel uses the [AWS S3 CSI Driver](https://github.com/awslabs/mountpoint-s3-csi-driver) to provision a per-task PersistentVolume (PV) and PersistentVolumeClaim (PVC) backed by S3. This gives Worker and Executor pods `ReadWriteMany` access across nodes without requiring them to co-schedule.
63+
64+
```mermaid
65+
sequenceDiagram
66+
participant Server as Funnel Server
67+
participant K8s as Kubernetes API
68+
participant CSI as S3 CSI Driver
69+
participant Worker as Worker Pod (Node A)
70+
participant Executor as Executor Pod (Node B)
71+
participant S3 as S3 Bucket
72+
73+
Server->>K8s: Create PV + PVC (s3://bucket/)
74+
K8s->>CSI: Mount bucket → funnel-worker-pv-<taskID>
75+
Server->>Worker: Launch Worker Job
76+
Worker->>S3: DownloadInputs (GenericS3)
77+
Worker->>Executor: Launch Executor Job (PVC subPath mount)
78+
Executor->>Executor: Run task command
79+
Executor-->>Worker: Task complete
80+
Worker->>S3: UploadOutputs (GenericS3)
81+
Worker->>K8s: Delete PVC + PV
82+
Worker->>Server: Report COMPLETE
83+
```
11284

11385
# Additional Resources 📚
11486

115-
- [Helm Repo](https://ohsu-comp-bio.github.io/helm-charts)
116-
87+
- [Helm Repo](https://calypr.org/helm-charts)
11788
- [Helm Repo Source](https://github.com/ohsu-comp-bio/helm-charts)
118-
119-
- [Helm Charts](https://github.com/ohsu-comp-bio/funnel/tree/develop/deployments/kubernetes/helm)
120-
89+
- [Helm Chart values reference](https://github.com/ohsu-comp-bio/helm-charts/tree/main/charts/funnel)
90+
- [AWS S3 CSI Driver](https://github.com/awslabs/mountpoint-s3-csi-driver)
12191
- [The Chart Best Practices Guide](https://helm.sh/docs/chart_best_practices/)
92+
- [GitHub issue #1412 — S3 Working Directory Behavior](https://github.com/calypr/funnel/issues/1412)

‎mkdocs.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,11 @@ markdown_extensions:
4040
anchor_linenums: false
4141
line_spans: __span
4242
pygments_lang_class: true
43+
- pymdownx.magiclink:
44+
normalize_issue_symbols: true
45+
repo_url_shorthand: true
46+
user: calypr
47+
repo: calypr.github.io
4348
- pymdownx.inlinehilite
4449
- pymdownx.snippets
4550
- pymdownx.superfences:

0 commit comments

Comments
 (0)