Fix cancellation safety of LruCachingStore::write_batch (using cache invalidation)

[ma2bd]

Motivation

The LruCachingStore::write_batch method was not cancellation-safe. It wrote to the DB first, then updated the LRU cache. If the future was cancelled between the two steps (e.g. by the RollbackGuard introduced in #5790), the DB would have the new data but the cache would retain stale entries. Subsequent reads would hit the stale cache, and the next save() would overwrite the DB with old data -- causing silent data loss.

This was identified as the likely root cause of the missing outbox bucket data on validator 4 (The front bucket is always loaded panic).

Proposal

Invalidate cache entries BEFORE writing to the DB, then repopulate the cache after success. If cancelled at any point after invalidation, subsequent reads bypass the cache and go directly to the DB, seeing the correct state.

New LruPrefixCache methods:

• invalidate_key / invalidate_prefix -- remove entries without recording negative results (DoesNotExist). Used for pre-write invalidation.
• record_key_deletion_unchecked / record_prefix_deletion_unchecked -- record deletion outcomes. Precondition: the corresponding invalidate_* was already called.
• delete_key / delete_prefix -- convenience methods that call both.

Note:

• This code was previously modified by Reorder operations in write_batch. #4954 to fix LRU cache discrepancies in case of errors, but we didn't think much about this corner case regarding cancelation safety.
• After this code if the DB returns an error while having succeeded, at least the LRU cache will be correct. However, the rest of code still rolls back. Next, for extra safety, we may want to poison the chain state on every write_batch error (Review impacts of potential error-throwing successful DB writes #6052).

Test Plan

• CI

Links

• Related: Remove chain actors (#5502, #5687) #5790 (Remove chain actors -- introduced RollbackGuard and cancellation)
• Related: Fix BucketQueueView::delete_front load failure handling #6015 (Fix BucketQueueView::delete_front load failure handling)

[afck]

Good catch!

(I approve, but would be good if @MathieuDutSik could also take a look.)

[deuszx]

Can we add a test for this?

1. Populate cache and db
2. try inserting but cancel before completing
3. confirm that cache is empty and db has the new value

[ma2bd]

Can we add a test for this?

1. Populate cache and db
2. try inserting but cancel before completing
3. confirm that cache is empty and db has the new value

This seems nontrivial. We could have unit tests for the new methods though.

[ma2bd]

After fixing some issues, the PR is way more complicated, and perhaps not as interesting any more compared to the obvious quick fix.

[ma2bd]

Closing in favor of #6056