Skip to content

Commit df1347a

Browse files
authored
perf: Optimize metadata records processing in SqlStorageClient (#1551)
### Description - This PR adds new `buffer` tables to improve handling of `metadata` records. The key change is that updates to `metadata` are now accumulated in `buffer` and applied when `get_metadata` is called. With the old behavior, `metadata` records were updated instantly within a transaction. This led to waiting for locks to be released in high-concurrency situations. ### Issues - Closes: #1533
1 parent 230f0d7 commit df1347a

File tree

10 files changed

+611
-238
lines changed

10 files changed

+611
-238
lines changed

docs/guides/storage_clients.mdx

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,7 @@ class datasets {
199199
+ created_at
200200
+ modified_at
201201
+ item_count
202+
+ buffer_locked_until
202203
}
203204
204205
class dataset_records {
@@ -208,6 +209,15 @@ class dataset_records {
208209
+ data
209210
}
210211
212+
class dataset_metadata_buffer {
213+
<<table>>
214+
+ id (PK)
215+
+ accessed_at
216+
+ modified_at
217+
+ dataset_id (FK)
218+
+ delta_item_count
219+
}
220+
211221
%% ========================
212222
%% Key-Value Store Tables
213223
%% ========================
@@ -220,6 +230,7 @@ class key_value_stores {
220230
+ accessed_at
221231
+ created_at
222232
+ modified_at
233+
+ buffer_locked_until
223234
}
224235
225236
class key_value_store_records {
@@ -231,15 +242,25 @@ class key_value_store_records {
231242
+ size
232243
}
233244
245+
class key_value_store_metadata_buffer {
246+
<<table>>
247+
+ id (PK)
248+
+ accessed_at
249+
+ modified_at
250+
+ key_value_store_id (FK)
251+
}
252+
234253
%% ========================
235254
%% Client to Table arrows
236255
%% ========================
237256
238257
SqlDatasetClient --> datasets
239258
SqlDatasetClient --> dataset_records
259+
SqlDatasetClient --> dataset_metadata_buffer
240260
241261
SqlKeyValueStoreClient --> key_value_stores
242262
SqlKeyValueStoreClient --> key_value_store_records
263+
SqlKeyValueStoreClient --> key_value_store_metadata_buffer
243264
```
244265
```mermaid
245266
---
@@ -274,6 +295,7 @@ class request_queues {
274295
+ handled_request_count
275296
+ pending_request_count
276297
+ total_request_count
298+
+ buffer_locked_until
277299
}
278300
279301
class request_queue_records {
@@ -294,13 +316,27 @@ class request_queue_state {
294316
+ forefront_sequence_counter
295317
}
296318
319+
class request_queue_metadata_buffer {
320+
<<table>>
321+
+ id (PK)
322+
+ accessed_at
323+
+ modified_at
324+
+ request_queue_id (FK)
325+
+ client_id
326+
+ delta_handled_count
327+
+ delta_pending_count
328+
+ delta_total_count
329+
+ need_recalc
330+
}
331+
297332
%% ========================
298333
%% Client to Table arrows
299334
%% ========================
300335
301336
SqlRequestQueueClient --> request_queues
302337
SqlRequestQueueClient --> request_queue_records
303338
SqlRequestQueueClient --> request_queue_state
339+
SqlRequestQueueClient --> request_queue_metadata_buffer
304340
```
305341

306342
Configuration options for the <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> can be set through environment variables or the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class:

0 commit comments

Comments
 (0)