erofs: add LRU dentry cache to bound memory usage#13199
Conversation
Reference counting: parent holds ref on each child in childMap. Lookup returns a ref'd dentry. Dentries with refs==1 (only parent's ref) and no children enter the LRU cache. Eviction removes from parent's childMap and DecRefs the dentry. Cache limit is 1000.
Could you explain this? benchmarks are regressing with caching? Or improving because lower-is-better for latency benchmarks. What are these microbenchmarks? |
Sorry for the unclear description. The baseline is the current code (no eviction - all dentries cached forever). Throughput drops because the patch adds ref counting on every path step and evicts dentries beyond the 1000 limit. Without this patch, memory grows unbounded. Microbenchmarks are find/read/stat workloads. Parallel find: -2%. Over-cache random read: -5%. In-cache stat: -8%. Over-cache stat: -22%. |
|
Also about memory (following up on my last comment): While this change introduces some performance overhead from refcounting and cache misses due to eviction, the main benefit is that memory usage is now bounded. I ran the repro from #13195 again with the patch applied: Memory only grew to ~7.7 MB. Before the fix it grew to 2.3 GB. The cache limit is currently a constant (1000 entries). I can make it tunable if preferred. |
2 participants
Fixes #13195
The erofs filesystem never evicts cached dentries, leading to unbounded memory growth (4 MB -> 2.3 GB after walking 1.5M files).
Add an LRU dentry cache modeled after fsimpl/gofer. Dentries with refs == 1 (only the parent's reference) and no children are placed on an LRU list and evicted when the cache exceeds 1000 entries. Path resolution functions now follow ref-on-return semantics.
Microbenchmarks show up to ~20% regression.