Skip to content

feat(sdk): Implement persistent storage for ThreadEventCache#6317

Merged
Hywan merged 36 commits intomatrix-org:event-cache-refactoringfrom
Hywan:feat-sdk-event-cache-thread-persistent-storage
Apr 28, 2026
Merged

feat(sdk): Implement persistent storage for ThreadEventCache#6317
Hywan merged 36 commits intomatrix-org:event-cache-refactoringfrom
Hywan:feat-sdk-event-cache-thread-persistent-storage

Conversation

@Hywan
Copy link
Copy Markdown
Member

@Hywan Hywan commented Mar 19, 2026

This patch implement persistent storage for ThreadEventCache.

This contribution deserves a lot more tests but it will increase the diff size by a lot. The basic tests are present though (adding 800 LOC in this patch already!). The current strategy is to ensure all existing tests are passing. A test with a custom build of Element X confirms it works as expected. Another PR will add all the missing tests, along with modifying the test suite of the Event Cache to share many tests between the RoomEventCache and the ThreadEventCache.

This contribution makes it work. However, refactoring is now necessary as similar code exists between RoomEventCache and ThreadEventCache. That's for a next patch.

Best to review one patch at a time. I tried to keep patch as small as possible.



  • I've documented the public API Changes in the appropriate CHANGELOG.md files.
  • This PR was made with the help of AI.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 19, 2026

Merging this PR will not alter performance

✅ 50 untouched benchmarks


Comparing Hywan:feat-sdk-event-cache-thread-persistent-storage (f83fbc9) with main (6777907)1

Open in CodSpeed

Footnotes

  1. No successful run was found on event-cache-refactoring (6777907) during the generation of this report, so main (6777907) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@Hywan Hywan force-pushed the feat-sdk-event-cache-thread-persistent-storage branch 4 times, most recently from 942609f to 12a1ef0 Compare March 24, 2026 17:39
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 24, 2026

Codecov Report

❌ Patch coverage is 86.62885% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.85%. Comparing base (dab87b7) to head (f83fbc9).
⚠️ Report is 148 commits behind head on event-cache-refactoring.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
.../matrix-sdk/src/event_cache/caches/thread/state.rs 69.23% 40 Missing and 8 partials ⚠️
...es/matrix-sdk/src/event_cache/caches/thread/mod.rs 95.40% 14 Missing and 22 partials ⚠️
crates/matrix-sdk/src/event_cache/persistence.rs 53.84% 32 Missing and 4 partials ⚠️
...ix-sdk/src/event_cache/caches/thread/pagination.rs 72.80% 25 Missing and 6 partials ⚠️
...es/matrix-sdk/src/event_cache/caches/room/state.rs 85.41% 1 Missing and 6 partials ⚠️
...ix-sdk/src/event_cache/caches/pinned_events/mod.rs 0.00% 3 Missing ⚠️
...ates/matrix-sdk/src/event_cache/caches/room/mod.rs 81.25% 0 Missing and 3 partials ⚠️
...trix-sdk/src/event_cache/caches/room/pagination.rs 90.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                    Coverage Diff                    @@
##           event-cache-refactoring    #6317    +/-   ##
=========================================================
  Coverage                    89.84%   89.85%            
=========================================================
  Files                          378      379     +1     
  Lines                       104337   105292   +955     
  Branches                    104337   105292   +955     
=========================================================
+ Hits                         93746    94608   +862     
- Misses                        7000     7061    +61     
- Partials                      3591     3623    +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hywan added 22 commits April 15, 2026 11:34
…adEventCacheState`.

This patch checks if the `EventCacheStoreLock` is dirty when
constructing `LockedThreadEventCacheState`. This patch is the first
step, but it involved a bit of refactoring since `new` is now fallible
and async. It propagates up to the public API which were already
fallible and async fortunately for us: no breaking change.

This patch changes `get_or_reload` to not clone the room ID, the store
etc. every time, but rather only when the `ThreadEventCache` must be
created.
This patch moves the `load_linked_chunk_metadata` function from
`event_cache::caches::room::state` to `event_cache::persistence`. It's
going to be useful for `ThreadEventCache`.
This patch loads the LinkedChunk metadata, in addition to lazy load the
last chunk of the `ThreadEventCache`.
…ded.

This patch updates the `reload` method on the
`ThreadEventCacheStateLockWriteGuard` to shrink to the last chunk from
the store.
…he`.

This patch updates `ThreadEventCacheStateLockWriteGuard::remove_events`
to also remove events from the store (it was only removing events from
the in-memory LinkedChunk prior to this patch).
This patch simply inlines a variable definition to simplify the code.
This patch replaces
`ThreadEventCacheStateLockWriteGuard::filter_duplicate_events` by
`deduplicator::filter_duplicate_events`. First off, this function is
shared between multiple event caches. Second, it deduplicates events
regarding in-memory and in-store state.

The calls to `ThreadEventCacheStateLockWriteGuard::remove_events` have
been updated to pass the duplicated in-store events.
This patch forces to use of `use self::s` instead of `use s` to make it
clear it comes from `self`. It clarifies the code here.
This patch makes `RoomEventCacheInner::weak_room`, `state`,
`pagination_batch_token_notifier`, and `shared_pagination_status`
private.
This patch creates the `ThreadEventCacheUpdateSender` à la
`RoomEventCacheUpdateSender` to abstract the sender.

This patch uses `ThreadEventCacheUpdateSender` in
`ThreadEventCacheInner` (new) along with `ThreadEventCacheState`. The
use in the `ThreadEventCacheInner` type is necessary to be able to send
updates without reaching the state (which is behind an async faillible
lock). This is necessary for pagination, see next commit.
This patch adds the ability to paginate in the store for
`ThreadPagination`.
…EventCacheStateLockWriteGuard`.

This patch passes the `prev_batch_token` up to
`ThreadEventCacheStateLockWriteGuard::handle_sync`.
This patch stops erasing threads when a gap is encountered. Instead, it
will insert gaps as we do for rooms.
This patch renames the `test_thread_can_paginate_evn_if_seen_sync_event`
to `test_thread_contains_its_root_event`. It is what it tests.
This patch replaces the construction of an event by its equivalent in a
variable. The test uses `first_event` and `second_event`, but builds the
“`second_event`” manually for no reason, it's just confusing
…dated.

This patch fixes an invalid behaviour in a thread: when an in-thread
event was redacted, it was removed from the thread. This is inconsistent
regarding in-room event redaction where a redacted event is updated to
its redacted form and the redaction event is added to the timeline. This
patch makes the behaviour consistent by updating the redacted event.

Basically, it replaces `remove_if_present` (deleted) by
`replace_event_if_present` (new).
@Hywan Hywan force-pushed the feat-sdk-event-cache-thread-persistent-storage branch from 7fe2c70 to 9b69678 Compare April 15, 2026 09:39
No need to deserialise the messages. We can assert the event ID
directly.
Hywan added 7 commits April 17, 2026 14:42
…ericUpdate`.

This patch updates `ThreadEventCacheUpdateSender` to send
`RoomEventCacheGenericUpdate`. This is useful for `ThreadPagination`.
…Cache` is built.

This patch updates `ThreadEventCache::new` to emit a
`RoomEventCacheGenericUpdate` when it's built, similarly to
`RoomEventCache`. This is important for external API, like Latest Event.
@Hywan Hywan force-pushed the feat-sdk-event-cache-thread-persistent-storage branch from 938b267 to 4dd3e66 Compare April 17, 2026 15:58
Hywan added 3 commits April 20, 2026 09:46
`RoomEventCache` holds the states for the room cache, the thread caches,
and the pinned-events cache. When the cross-process lock is dirty, it
has to reload all these caches.

Ideally, we must move all the states in a single struct, behind the
cross-process lock, because right now, when the lock is dirty, only a
single room is reloaded, not all the rooms. This is a temporary fix.
@Hywan Hywan force-pushed the feat-sdk-event-cache-thread-persistent-storage branch from 5e22fa2 to 4d50324 Compare April 20, 2026 13:21
@Hywan Hywan marked this pull request as ready for review April 20, 2026 14:27
@Hywan Hywan requested a review from a team as a code owner April 20, 2026 14:27
@Hywan Hywan requested review from poljar and removed request for a team April 20, 2026 14:27
Copy link
Copy Markdown
Contributor

@poljar poljar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple of nits and questions, besides those this looks good to me.

Comment thread crates/matrix-sdk/src/event_cache/caches/thread/state.rs
Comment thread crates/matrix-sdk/src/event_cache/caches/thread/pagination.rs Outdated
Comment thread crates/matrix-sdk/src/event_cache/caches/thread/pagination.rs Outdated
Comment thread crates/matrix-sdk/src/event_cache/caches/thread/pagination.rs Outdated
Comment thread crates/matrix-sdk/src/event_cache/caches/thread/pagination.rs
Hywan added 3 commits April 28, 2026 10:50
This patch adds `LinkedChunk::first_chunk`, and exposes it to
`EventLinkedChunk` to replace the use of 2 iterators in `RoomPagination`
and `ThreadPagination`, and to remove 2 `unwrap`s.
@Hywan Hywan changed the base branch from main to event-cache-refactoring April 28, 2026 09:28
@Hywan Hywan merged commit 94716fd into matrix-org:event-cache-refactoring Apr 28, 2026
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants