Commit df1347a
authored
perf: Optimize metadata records processing in
### Description
- This PR adds new `buffer` tables to improve handling of `metadata`
records. The key change is that updates to `metadata` are now
accumulated in `buffer` and applied when `get_metadata` is called. With
the old behavior, `metadata` records were updated instantly within a
transaction. This led to waiting for locks to be released in
high-concurrency situations.
### Issues
- Closes: #1533SqlStorageClient (#1551)1 parent 230f0d7 commit df1347a
File tree
10 files changed
+611
-238
lines changed- docs/guides
- src/crawlee/storage_clients/_sql
- tests/unit/storage_clients/_sql
10 files changed
+611
-238
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
199 | 199 | | |
200 | 200 | | |
201 | 201 | | |
| 202 | + | |
202 | 203 | | |
203 | 204 | | |
204 | 205 | | |
| |||
208 | 209 | | |
209 | 210 | | |
210 | 211 | | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
211 | 221 | | |
212 | 222 | | |
213 | 223 | | |
| |||
220 | 230 | | |
221 | 231 | | |
222 | 232 | | |
| 233 | + | |
223 | 234 | | |
224 | 235 | | |
225 | 236 | | |
| |||
231 | 242 | | |
232 | 243 | | |
233 | 244 | | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
234 | 253 | | |
235 | 254 | | |
236 | 255 | | |
237 | 256 | | |
238 | 257 | | |
239 | 258 | | |
| 259 | + | |
240 | 260 | | |
241 | 261 | | |
242 | 262 | | |
| 263 | + | |
243 | 264 | | |
244 | 265 | | |
245 | 266 | | |
| |||
274 | 295 | | |
275 | 296 | | |
276 | 297 | | |
| 298 | + | |
277 | 299 | | |
278 | 300 | | |
279 | 301 | | |
| |||
294 | 316 | | |
295 | 317 | | |
296 | 318 | | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
297 | 332 | | |
298 | 333 | | |
299 | 334 | | |
300 | 335 | | |
301 | 336 | | |
302 | 337 | | |
303 | 338 | | |
| 339 | + | |
304 | 340 | | |
305 | 341 | | |
306 | 342 | | |
| |||
0 commit comments