You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/persistentStorage.md
+25-6Lines changed: 25 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,10 @@ This document describes Ocean Node **Persistent Storage** at a high level: what
6
6
7
7
## What it is
8
8
9
-
Persistent Storage is a simple bucket + file store intended for **long-lived artifacts** that Ocean Node needs to keep across requests (and potentially across restarts), and to reference later (e.g. as file objects for compute).
9
+
Persistent Storage is a simple bucket + file store intended for **long-lived artifacts** that Ocean Node needs to keep across requests and across restarts, and to reference later (e.g. as file objects for compute).
10
10
11
11
Key primitives:
12
+
12
13
-**Bucket**: a logical container for files.
13
14
-**File**: binary content stored inside a bucket.
14
15
-**Bucket registry**: a local SQLite table that stores bucket metadata (owner, access lists, createdAt).
@@ -35,7 +36,8 @@ Key primitives:
35
36
36
37
Persistent Storage uses two stores:
37
38
38
-
1)**Bucket registry (SQLite)**
39
+
1.**Bucket registry (SQLite)**
40
+
39
41
- File: `databases/persistentStorage.sqlite`
40
42
- Table: `persistent_storage_buckets`
41
43
- Columns:
@@ -44,7 +46,8 @@ Persistent Storage uses two stores:
44
46
-`accessListJson` (JSON-encoded access list array)
45
47
-`createdAt` (unix timestamp)
46
48
47
-
2)**Backend data**
49
+
2.**Backend data**
50
+
48
51
-`localfs`: writes file bytes to the configured folder under `buckets/<bucketId>/<fileName>`.
49
52
-`s3`: not implemented yet.
50
53
@@ -75,11 +78,13 @@ This access list is used to decide whether a given `consumerAddress` is allowed
75
78
76
79
Access checks happen at two levels:
77
80
78
-
1)**Backend enforcement** (required)
81
+
1.**Backend enforcement** (required)
82
+
79
83
- Backend operations `listFiles`, `uploadFile`, `deleteFile`, and `getFileObject` all require `consumerAddress`.
80
84
- The base class helper `assertConsumerAllowedForBucket(consumerAddress, bucketId)` loads the bucket ACL and throws `PersistentStorageAccessDeniedError` if the consumer is not allowed.
81
85
82
-
2)**Handler enforcement** (command-specific)
86
+
2.**Handler enforcement** (command-specific)
87
+
83
88
-`createBucket`: additionally checks the node-level allow list `config.persistentStorage.accessLists` (who can create buckets at all).
84
89
-`getBuckets`: queries registry rows filtered by `owner` and then:
85
90
- if `consumerAddress === owner`: returns all buckets for that owner
@@ -115,6 +120,10 @@ Access checks happen at two levels:
115
120
- Deletes the named file from the bucket.
116
121
- Enforces bucket ACL.
117
122
123
+
-**getFileObject**
124
+
- Returns fileObject format for c2d use
125
+
- Enforces bucket ACL.
126
+
118
127
### Not implemented yet
119
128
120
129
-**S3 backend**
@@ -127,6 +136,7 @@ Access checks happen at two levels:
127
136
Persistent storage is controlled by `persistentStorage` in node config.
128
137
129
138
Key fields:
139
+
130
140
-`enabled`: boolean
131
141
-`type`: `"localfs"` or `"s3"`
132
142
-`accessLists`: AccessList[] — node-level allow list to create buckets
@@ -138,9 +148,18 @@ Key fields:
138
148
139
149
## Usage
140
150
151
+
Flow is:
152
+
153
+
- create bucket (or use existing bucket)
154
+
- list files
155
+
- upload file if needed
156
+
- GetFileObject to get object needed for c2d reference
157
+
- start c2d job using fileObject for datasets
158
+
141
159
### P2P commands
142
160
143
161
All persistent storage operations are implemented as protocol commands in the handler:
162
+
144
163
-`persistentStorageCreateBucket`
145
164
-`persistentStorageGetBuckets`
146
165
-`persistentStorageListFiles`
@@ -155,6 +174,7 @@ Each command requires authentication (token or signature) based on Ocean Node’
155
174
HTTP routes are available under `/api/services/persistentStorage/...` and call the same handlers. See `docs/API.md` for the full parameter lists and examples.
@@ -171,4 +191,3 @@ Upload uses the raw request body as bytes and forwards it to the handler as a st
171
191
- The bucket registry is local to the node (SQLite file). If you run multiple nodes, each node’s registry is independent unless you externalize/replicate it.
172
192
-`listBuckets(owner)` requires `owner` and only returns buckets that were created with that owner recorded.
173
193
- Filenames in `localfs` are constrained (no path separators) to avoid path traversal.
0 commit comments