|
| 1 | +# Persistent Storage |
| 2 | + |
| 3 | +This document describes Ocean Node **Persistent Storage** at a high level: what it is, how it is structured, how access control works, and how to use it via **P2P commands** and **HTTP endpoints**. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## What it is |
| 8 | + |
| 9 | +Persistent Storage is a simple bucket + file store intended for **long-lived artifacts** that Ocean Node needs to keep across requests (and potentially across restarts), and to reference later (e.g. as file objects for compute). |
| 10 | + |
| 11 | +Key primitives: |
| 12 | +- **Bucket**: a logical container for files. |
| 13 | +- **File**: binary content stored inside a bucket. |
| 14 | +- **Bucket registry**: a local SQLite table that stores bucket metadata (owner, access lists, createdAt). |
| 15 | + |
| 16 | +--- |
| 17 | + |
| 18 | +## Architecture (high level) |
| 19 | + |
| 20 | +### Components |
| 21 | + |
| 22 | +- **Handlers (protocol layer)**: `src/components/core/handler/persistentStorage.ts` |
| 23 | + - Implements protocol commands such as create bucket, list files, upload, delete, and get buckets. |
| 24 | + - Validates auth (token or signature) and applies high-level authorization checks. |
| 25 | + |
| 26 | +- **Persistent storage backends (storage layer)**: `src/components/persistentStorage/*` |
| 27 | + - `PersistentStorageFactory`: shared functionality (SQLite bucket registry, access list checks). |
| 28 | + - `PersistentStorageLocalFS`: local filesystem backend. |
| 29 | + - `PersistentStorageS3`: stub for future S3-compatible backend. |
| 30 | + |
| 31 | +- **HTTP routes (HTTP interface)**: `src/components/httpRoutes/persistentStorage.ts` |
| 32 | + - Exposes REST-ish endpoints under `/api/services/persistentStorage/...` that call the same handlers. |
| 33 | + |
| 34 | +### Data storage |
| 35 | + |
| 36 | +Persistent Storage uses two stores: |
| 37 | + |
| 38 | +1) **Bucket registry (SQLite)** |
| 39 | +- File: `databases/persistentStorage.sqlite` |
| 40 | +- Table: `persistent_storage_buckets` |
| 41 | +- Columns: |
| 42 | + - `bucketId` (primary key) |
| 43 | + - `owner` (address, stored as a string) |
| 44 | + - `accessListJson` (JSON-encoded access list array) |
| 45 | + - `createdAt` (unix timestamp) |
| 46 | + |
| 47 | +2) **Backend data** |
| 48 | +- `localfs`: writes file bytes to the configured folder under `buckets/<bucketId>/<fileName>`. |
| 49 | +- `s3`: not implemented yet. |
| 50 | + |
| 51 | +--- |
| 52 | + |
| 53 | +## Ownership and access control |
| 54 | + |
| 55 | +### Ownership |
| 56 | + |
| 57 | +Every bucket has a single **owner** address, stored in the bucket registry. |
| 58 | + |
| 59 | +- When a bucket is created, the node sets: |
| 60 | + - `owner = consumerAddress` (normalized via `ethers.getAddress`) |
| 61 | + |
| 62 | +### Bucket access list |
| 63 | + |
| 64 | +Each bucket stores an **AccessList[]** (per-chain list(s) of access list contract addresses): |
| 65 | + |
| 66 | +```ts |
| 67 | +export interface AccessList { |
| 68 | + [chainId: string]: string[] |
| 69 | +} |
| 70 | +``` |
| 71 | + |
| 72 | +This access list is used to decide whether a given `consumerAddress` is allowed to interact with a bucket. |
| 73 | + |
| 74 | +### Where checks happen |
| 75 | + |
| 76 | +Access checks happen at two levels: |
| 77 | + |
| 78 | +1) **Backend enforcement** (required) |
| 79 | +- Backend operations `listFiles`, `uploadFile`, `deleteFile`, and `getFileObject` all require `consumerAddress`. |
| 80 | +- The base class helper `assertConsumerAllowedForBucket(consumerAddress, bucketId)` loads the bucket ACL and throws `PersistentStorageAccessDeniedError` if the consumer is not allowed. |
| 81 | + |
| 82 | +2) **Handler enforcement** (command-specific) |
| 83 | +- `createBucket`: additionally checks the node-level allow list `config.persistentStorage.accessLists` (who can create buckets at all). |
| 84 | +- `getBuckets`: queries registry rows filtered by `owner` and then: |
| 85 | + - if `consumerAddress === owner`: returns all buckets for that owner |
| 86 | + - else: filters buckets by the bucket ACL |
| 87 | + |
| 88 | +### Error behavior |
| 89 | + |
| 90 | +- Backends throw `PersistentStorageAccessDeniedError` when forbidden. |
| 91 | +- Handlers translate that into **HTTP 403** / `status.httpStatus = 403`. |
| 92 | + |
| 93 | +--- |
| 94 | + |
| 95 | +## Features |
| 96 | + |
| 97 | +### Supported today |
| 98 | + |
| 99 | +- **Create bucket** |
| 100 | + - Creates a bucket id (UUID), persists it in SQLite with `owner` and `accessListJson`, and creates a local directory (localfs). |
| 101 | + |
| 102 | +- **List buckets (by owner)** |
| 103 | + - Returns buckets from the registry filtered by `owner` (mandatory arg). |
| 104 | + - Applies ACL filtering for non-owners. |
| 105 | + |
| 106 | +- **Upload file** |
| 107 | + - Writes a stream to the backend. |
| 108 | + - Enforces bucket ACL. |
| 109 | + |
| 110 | +- **List files** |
| 111 | + - Returns file metadata (`name`, `size`, `lastModified`) for a bucket. |
| 112 | + - Enforces bucket ACL. |
| 113 | + |
| 114 | +- **Delete file** |
| 115 | + - Deletes the named file from the bucket. |
| 116 | + - Enforces bucket ACL. |
| 117 | + |
| 118 | +### Not implemented yet |
| 119 | + |
| 120 | +- **S3 backend** |
| 121 | + - `PersistentStorageS3` exists as a placeholder and currently throws “not implemented”. |
| 122 | + |
| 123 | +--- |
| 124 | + |
| 125 | +## Configuration |
| 126 | + |
| 127 | +Persistent storage is controlled by `persistentStorage` in node config. |
| 128 | + |
| 129 | +Key fields: |
| 130 | +- `enabled`: boolean |
| 131 | +- `type`: `"localfs"` or `"s3"` |
| 132 | +- `accessLists`: AccessList[] — node-level allow list to create buckets |
| 133 | +- `options`: |
| 134 | + - localfs: `{ "folder": "/path/to/storage" }` |
| 135 | + - s3: `{ endpoint, objectKey, accessKeyId, secretAccessKey, ... }` (future) |
| 136 | + |
| 137 | +--- |
| 138 | + |
| 139 | +## Usage |
| 140 | + |
| 141 | +### P2P commands |
| 142 | + |
| 143 | +All persistent storage operations are implemented as protocol commands in the handler: |
| 144 | +- `persistentStorageCreateBucket` |
| 145 | +- `persistentStorageGetBuckets` |
| 146 | +- `persistentStorageListFiles` |
| 147 | +- `persistentStorageUploadFile` |
| 148 | +- `persistentStorageDeleteFile` |
| 149 | + |
| 150 | +Each command requires authentication (token or signature) based on Ocean Node’s auth configuration. |
| 151 | + |
| 152 | +### HTTP endpoints |
| 153 | + |
| 154 | +HTTP routes are available under `/api/services/persistentStorage/...` and call the same handlers. See `docs/API.md` for the full parameter lists and examples. |
| 155 | + |
| 156 | +At a glance: |
| 157 | +- `POST /api/services/persistentStorage/buckets` |
| 158 | +- `GET /api/services/persistentStorage/buckets` |
| 159 | +- `GET /api/services/persistentStorage/buckets/:bucketId/files` |
| 160 | +- `POST /api/services/persistentStorage/buckets/:bucketId/files/:fileName` |
| 161 | +- `DELETE /api/services/persistentStorage/buckets/:bucketId/files/:fileName` |
| 162 | + |
| 163 | +Upload uses the raw request body as bytes and forwards it to the handler as a stream. |
| 164 | + |
| 165 | +--- |
| 166 | + |
| 167 | +## Limitations and notes |
| 168 | + |
| 169 | +- The bucket registry is local to the node (SQLite file). If you run multiple nodes, each node’s registry is independent unless you externalize/replicate it. |
| 170 | +- `listBuckets(owner)` requires `owner` and only returns buckets that were created with that owner recorded. |
| 171 | +- Filenames in `localfs` are constrained (no path separators) to avoid path traversal. |
| 172 | + |
0 commit comments