Skip to content

Commit 532cf06

Browse files
feat(portal): implement object download functionality (ceph) (#990)
* feat(portal): add object download BFF endpoint with progress streaming * docs(bff): document object download BFF procedures * feat(portal): add object download to ObjectsTableView * fix(portal): add PortalProvider to ObjectsTableView tests * feat(portal): add row-click preview and download for object files * fix(portal): update ObjectsTableView tests for anchor-based preview/download * docs(bff): document downloadObject content-type resolution * fix(portal): use plain identifiers in Lingui message interpolations * chore(core): add missing translations * test(portal): add schema validation tests for object storage types * chore(core): create a changeset * fix(portal): track downloads and previews per-row to support concurrent transfers * fix(portal): prevent version rows from using the object-only download path * fix(portal): emit mapped progress error for every download failure path * fix(portal): clear the 30s wait timer once progress arrives * fix(portal): restrict browser preview to non-scriptable content types * docs(bff): fix misleading resolveMimeFromKey comment * docs(bff): update preview whitelist doc to match tightened MIME set * chore(core): update changeset description * fix(portal): restore mocks between tests and cover progress subscription
1 parent f67c54f commit 532cf06

16 files changed

Lines changed: 2532 additions & 26 deletions

File tree

.changeset/eight-corners-fail.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
"@cobaltcore-dev/aurora": minor
3+
---
4+
5+
Add object download and preview for Ceph object storage, bringing Ceph to parity with the existing Swift capability.
6+
7+
- Stream downloads through the BFF (`downloadObject`) with live progress tracking via `watchDownloadProgress`. Multiple downloads/previews can be in flight at once, each tracked and reported independently.
8+
- Row-click on an object previews it in a new browser tab when the type is safely renderable (images excluding SVG, video, audio, PDF, and plain text); everything else downloads directly. Scriptable types (HTML, JSON, XML, SVG) are intentionally excluded from preview since they can execute active content when opened from a blob URL.
9+
- New context-menu **Download** action always forces a file save, regardless of type.
10+
- The BFF resolves a reliable MIME type from the object key extension when S3/Ceph reports a generic or incorrect `Content-Type` (e.g. the `binary/octet-stream` default, or values set by some upload tools), so files preview correctly even without proper upload metadata.

packages/aurora/docs/009_ceph_s3_bff.md

Lines changed: 138 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,9 @@ storage.ceph
165165
│ ├── createFolder() → { success: boolean }
166166
│ ├── copy() → { success: boolean, etag?: string }
167167
│ ├── move() → { success: boolean, etag?: string }
168-
│ └── updateMetadata() → { success: boolean }
168+
│ ├── updateMetadata() → { success: boolean }
169+
│ ├── downloadObject() → AsyncIterable<{ chunk, downloaded, total, contentType?, filename? }>
170+
│ └── watchDownloadProgress() → AsyncIterable<{ downloaded, total, percent }> (subscription)
169171
└── versioning
170172
├── getStatus() → { status: 'Enabled' | 'Suspended' | 'Unversioned' }
171173
├── setStatus() → { success: boolean }
@@ -854,6 +856,127 @@ await trpc.storage.ceph.objects.updateMetadata.mutate({
854856
855857
---
856858
859+
#### `downloadObject`
860+
861+
Downloads an object by streaming its content through the BFF as base64-encoded chunks via a tRPC async iterable (S3 `GetObjectCommand`). The server never buffers the whole object in memory, and per-chunk progress is published so a concurrent `watchDownloadProgress` subscription can drive a progress bar. This is a **direct stream** — no presigned/temporary URLs are issued.
862+
863+
**Input:**
864+
865+
```typescript
866+
{
867+
project_id: string,
868+
containerName: string,
869+
objectKey: string,
870+
filename: string, // Suggested download filename (echoed in the first chunk)
871+
downloadId: string // Client-computed "<bucket>:<objectKey>:<uuid>" correlating
872+
// the stream with a watchDownloadProgress subscription
873+
}
874+
```
875+
876+
**Output:** an async iterable yielding chunks:
877+
878+
```typescript
879+
{
880+
chunk: string, // base64-encoded Uint8Array slice of the object
881+
downloaded: number, // cumulative bytes streamed so far
882+
total: number, // total object size in bytes (0 if unknown)
883+
contentType?: string, // present only in the first chunk
884+
filename?: string // present only in the first chunk
885+
}
886+
```
887+
888+
**Example:**
889+
890+
```typescript
891+
const downloadId = `${bucket}:${objectKey}:${crypto.randomUUID()}`
892+
893+
const iterable = await trpc.storage.ceph.objects.downloadObject.mutate({
894+
project_id: "abc123",
895+
containerName: "my-bucket",
896+
objectKey: "documents/report.pdf",
897+
filename: "report.pdf",
898+
downloadId,
899+
})
900+
901+
const parts: Uint8Array[] = []
902+
let contentType = "application/octet-stream"
903+
for await (const { chunk, contentType: ct } of iterable) {
904+
if (ct) contentType = ct
905+
parts.push(Uint8Array.from(atob(chunk), (c) => c.charCodeAt(0)))
906+
}
907+
const blob = new Blob(parts, { type: contentType })
908+
// → wrap in an object URL and trigger an <a download>
909+
```
910+
911+
**Notes:**
912+
913+
- Each chunk is base64-encoded because tRPC's iterable transport is JSON/SSE-based and yielded values must be JSON-serializable.
914+
- `contentType` and `filename` are sent only in the first chunk to avoid repetition.
915+
- Progress is scoped by `project_id`, so a user can never observe another tenant's transfer.
916+
- An empty object yields no chunks (the loop simply never runs).
917+
918+
**Content-Type resolution:**
919+
920+
S3/Ceph often can't be trusted to report a useful `Content-Type` — RGW defaults to `binary/octet-stream` when none was set on upload, and some upload tools store an unrelated type (e.g. `application/x-www-form-urlencoded`). Since the raw stored value directly determines whether the frontend can preview a file inline vs. must force a download, `downloadObject` resolves the type as follows before yielding the first chunk:
921+
922+
1. If the object key has a recognized extension (`.jpg`, `.pdf`, `.txt`, `.png`, etc.), the MIME type is derived from that extension and takes priority over whatever S3 returned.
923+
2. Otherwise (unknown or missing extension — e.g. a UUID-style key), the `ContentType` from the S3 `GetObjectCommand` response is used as-is, falling back to `application/octet-stream` if absent.
924+
925+
This means the `contentType` yielded in the first chunk may differ from the object's stored `Content-Type` in S3. Consumers that need the _raw_ stored value (rather than the resolved/display value) should call `getDetails` instead, which returns the S3 metadata unmodified.
926+
927+
The Ceph frontend (`ObjectsTableView`) uses this resolved `contentType` to decide row-click behavior: safe browser-previewable types (passive media `image/*` excluding `image/svg+xml`, `video/*`, `audio/*`, plus `application/pdf` and `text/plain`) open in a new tab; everything else downloads. The context-menu **Download** action always forces a download regardless of type.
928+
929+
---
930+
931+
#### `watchDownloadProgress`
932+
933+
Subscribes to live progress for an in-flight download identified by `downloadId`. Open this **before** calling `downloadObject` so no early events are missed; a snapshot of current progress is emitted immediately for late subscribers.
934+
935+
**Input:**
936+
937+
```typescript
938+
{
939+
project_id: string,
940+
downloadId: string // Same id passed to downloadObject
941+
}
942+
```
943+
944+
**Output:** an async iterable (subscription) yielding progress:
945+
946+
```typescript
947+
{
948+
downloaded: number, // cumulative bytes streamed
949+
total: number, // total object size in bytes (0 if unknown)
950+
percent: number // 0–100 (0 when total is unknown)
951+
}
952+
```
953+
954+
**Example:**
955+
956+
```typescript
957+
const downloadId = `${bucket}:${objectKey}:${crypto.randomUUID()}`
958+
959+
trpc.storage.ceph.objects.watchDownloadProgress.subscribe(
960+
{ project_id: "abc123", downloadId },
961+
{ onData: ({ percent }) => setProgress(percent) }
962+
)
963+
964+
await trpc.storage.ceph.objects.downloadObject.mutate({
965+
project_id: "abc123",
966+
containerName: "my-bucket",
967+
objectKey: "documents/report.pdf",
968+
filename: "report.pdf",
969+
downloadId,
970+
})
971+
```
972+
973+
**Notes:**
974+
975+
- The subscription completes when the download finishes and re-throws if the download errors.
976+
- If no events arrive within 30 s (e.g. the download finished before the subscription opened), it ends gracefully rather than hanging.
977+
978+
---
979+
857980
### Versioning (`storage.ceph.versioning`)
858981
859982
Bucket versioning allows keeping multiple variants of objects in the same bucket, enabling recovery from accidental deletions and overwrites.
@@ -1531,17 +1654,17 @@ console.log("Old credential deleted")
15311654
15321655
## Comparison: Ceph S3 vs. Swift
15331656
1534-
| Feature | Ceph S3 (this BFF) | Swift BFF |
1535-
| -------------------- | --------------------------------- | -------------------------------------- |
1536-
| **Authentication** | EC2 credentials (access + secret) | Keystone token |
1537-
| **API Style** | S3-compatible (AWS SDK) | OpenStack Swift API |
1538-
| **Bucket/Container** | S3 buckets | Swift containers |
1539-
| **Object Hierarchy** | `delimiter` + `prefix` | `delimiter` + `prefix` |
1540-
| **Metadata** | User metadata via `x-amz-meta-` | Custom headers via `X-Object-Meta-` |
1541-
| **Large Objects** | Multipart uploads | Static/Dynamic Large Objects (SLO/DLO) |
1542-
| **Uploads** | (Not yet implemented) | `octetInputParser` streaming |
1543-
| **Downloads** | (Not yet implemented) | Async iterable base64 chunks |
1544-
| **Temporary URLs** | Presigned URLs (not yet impl) | HMAC-SHA256 signed temp URLs |
1657+
| Feature | Ceph S3 (this BFF) | Swift BFF |
1658+
| -------------------- | ----------------------------------------- | -------------------------------------- |
1659+
| **Authentication** | EC2 credentials (access + secret) | Keystone token |
1660+
| **API Style** | S3-compatible (AWS SDK) | OpenStack Swift API |
1661+
| **Bucket/Container** | S3 buckets | Swift containers |
1662+
| **Object Hierarchy** | `delimiter` + `prefix` | `delimiter` + `prefix` |
1663+
| **Metadata** | User metadata via `x-amz-meta-` | Custom headers via `X-Object-Meta-` |
1664+
| **Large Objects** | Multipart uploads | Static/Dynamic Large Objects (SLO/DLO) |
1665+
| **Uploads** | (Not yet implemented) | `octetInputParser` streaming |
1666+
| **Downloads** | Async iterable base64 chunks (+ progress) | Async iterable base64 chunks |
1667+
| **Temporary URLs** | Presigned URLs (not yet impl) | HMAC-SHA256 signed temp URLs |
15451668
15461669
**When to use which?**
15471670
@@ -1564,8 +1687,8 @@ Both can coexist — Ceph RGW supports **both Swift and S3 APIs** on the same cl
15641687
15651688
2. **Object Upload/Download**
15661689
- Upload object (`PutObjectCommand`)
1567-
- Download object (`GetObjectCommand`)
1568-
- Streaming upload/download via tRPC (similar to Swift BFF `uploadObject`/`downloadObject`)
1690+
- ~~Download object (`GetObjectCommand`)~~ ✅ Implemented — streamed through the BFF as a tRPC async iterable, with a `watchDownloadProgress` subscription for progress
1691+
- Streaming upload via tRPC (similar to Swift BFF `uploadObject`)
15691692
- Multipart uploads for large files
15701693
15711694
3. **Object Manipulation**
@@ -1790,7 +1913,7 @@ openstack endpoint list --service ceph
17901913
17911914
## Next Steps
17921915
1793-
1. **Implement object upload/download**streaming via `octetInputParser` / async iterables (similar to Swift BFF)
1916+
1. **Implement object upload** — streaming via `octetInputParser` (download already implemented via async-iterable streaming with progress tracking)
17941917
2. **Add presigned URL generation** — allow direct client → Ceph transfers for large files
17951918
3. ~~**Implement bucket management**~~ ✅ Completed — create, delete, empty buckets
17961919
4. **Add credential caching** — reduce Keystone API calls

packages/aurora/src/client/routes/_auth/projects/$projectId/storage/-components/Ceph/Objects/ObjectBrowserView.tsx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import {
3838
getObjectMoveErrorToast,
3939
getObjectMetadataUpdatedToast,
4040
getObjectMetadataUpdateErrorToast,
41+
getObjectDownloadErrorToast,
4142
} from "./ObjectToastNotifications"
4243
import { encodePrefix, decodePrefix } from "../../utils/prefixEncoding"
4344

@@ -564,6 +565,10 @@ export function ObjectBrowserView({ bucketName }: ObjectBrowserViewProps) {
564565
const { message, ...options } = getObjectMetadataUpdateErrorToast(objectKey, errorMessage)
565566
toast.error(message, options)
566567
}}
568+
onDownloadError={(objectKey, errorMessage) => {
569+
const { message, ...options } = getObjectDownloadErrorToast(objectKey, errorMessage)
570+
toast.error(message, options)
571+
}}
567572
/>
568573
)}
569574

packages/aurora/src/client/routes/_auth/projects/$projectId/storage/-components/Ceph/Objects/ObjectToastNotifications.test.tsx

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
getObjectMoveErrorToast,
1414
getObjectMetadataUpdatedToast,
1515
getObjectMetadataUpdateErrorToast,
16+
getObjectDownloadErrorToast,
1617
getVersionRestoredToast,
1718
getVersionRestoreErrorToast,
1819
getVersionDeletedToast,
@@ -156,6 +157,22 @@ describe("ObjectToastNotifications", () => {
156157
})
157158
})
158159

160+
// ── Object download ──────────────────────────────────────────────────────────
161+
162+
describe("getObjectDownloadErrorToast", () => {
163+
it("renders correct error content", () => {
164+
renderNotification(getObjectDownloadErrorToast("documents/file.txt", "Network error"))
165+
expect(screen.getByText("Failed to Download Object")).toBeInTheDocument()
166+
expect(screen.getByText(/Could not download/)).toBeInTheDocument()
167+
expect(screen.getByText(/Network error/)).toBeInTheDocument()
168+
})
169+
170+
it("extracts the display name from a nested object key", () => {
171+
renderNotification(getObjectDownloadErrorToast("documents/reports/Q1.pdf", "err"))
172+
expect(screen.getByText(/Q1\.pdf/)).toBeInTheDocument()
173+
})
174+
})
175+
159176
// ── Notification configuration ───────────────────────────────────────────────
160177

161178
describe("Notification configuration", () => {
@@ -171,6 +188,7 @@ describe("ObjectToastNotifications", () => {
171188
getObjectMoveErrorToast("a.txt", "err"),
172189
getObjectMetadataUpdatedToast("a.txt"),
173190
getObjectMetadataUpdateErrorToast("a.txt", "err"),
191+
getObjectDownloadErrorToast("a.txt", "err"),
174192
getVersionRestoredToast("a.txt"),
175193
getVersionRestoreErrorToast("a.txt", "err"),
176194
getVersionDeletedToast("a.txt"),

packages/aurora/src/client/routes/_auth/projects/$projectId/storage/-components/Ceph/Objects/ObjectToastNotifications.tsx

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,23 @@ export const getObjectMetadataUpdateErrorToast = (
144144
}
145145
}
146146

147+
// ── Object download ────────────────────────────────────────────────────────────
148+
149+
export const getObjectDownloadErrorToast = (
150+
objectKey: string,
151+
errorMessage: string
152+
): { message: ReactNode } & NotificationOptions => {
153+
const displayName = objectKey.split("/").filter(Boolean).pop() ?? objectKey
154+
return {
155+
message: <Trans>Failed to Download Object</Trans>,
156+
description: (
157+
<Trans>
158+
Could not download "{displayName}": {errorMessage}
159+
</Trans>
160+
),
161+
}
162+
}
163+
147164
// ── Version restore ────────────────────────────────────────────────────────
148165

149166
export const getVersionRestoredToast = (objectKey: string): { message: ReactNode } & NotificationOptions => {

0 commit comments

Comments
 (0)