L1 Cache Admission/Replacement Policy

[rsalus]

Background

It's no secret that the IMemoryCache interface is rather limited. In the past few years, there have been rumblings of updating the implementation and even creating an entirely new one. Besides its counterintuitive design and lackluster performance characteristics, one major gap I've noticed with .NET's in-memory caching is the lack of a configurable cache replacement policy or admission policy.

Ask

I'm aware that this is quite a broad topic, but would it be possible for FusionCache to offer support for configuring L1 caching strategies? It may be significantly out of scope for FusionCache to offer the implementation(s) for this directly, but there are existing caching primitives available (the excellent BitFaster library comes to mind).

As far as I'm aware, there are two "tricky" aspects to this:

1. Whatever caching primitive utilized would need to provide an "adapter" class which integrates with IMemoryCache. This seems feasible but would require careful implementation, especially around handling ICacheEntry.
2. From FusionCache's POV, how would cache synchronization work between the L1 and L2 caches?

Why

I think there is significant value in tailoring in-memory caching strategies for specific workloads. Some examples off the top of my mind:

• Scenario 1: An application caches results from user searches or generates user-specific reports/views. Many of these items might be accessed only once ("one-hit wonders") and never needed again, while others might be revisited.

  • MemoryCache Limitation: MemoryCache generally admits any item you .Set(). Caching large numbers of unique, single-use items can "pollute" the cache, filling it with data that offers little future value and causing more useful, frequently accessed items to be evicted prematurely.
  • Tailored Strategy Gain: An admission policy could significantly improve hit rates. For instance, we could utilize a frequency threshold such that we only admit an item to the cache after it has been requested twice within a certain time window. This filters out one-hit wonders.

• Scenario 2: An application caches data that is core to many requests but changes infrequently – think product categories, country lists, global configuration settings, user roles/permissions, etc. Some of this data might be extremely popular, while other parts are accessed less often.

  • MemoryCache Limitation: Under memory pressure, MemoryCache might evict a very popular (high frequency) but Normal priority item if it hasn't been accessed very recently, especially if many less popular High priority items exist or were accessed more recently. Its recency bias (part of its internal heuristic) can work against keeping globally popular items.
  • Tailored Strategy Gain: A strict LFU policy would excel here. It prioritizes keeping items based on their total access count, ensuring that truly popular reference data stays cached, even if it wasn't the most recently touched item before an eviction cycle runs. This leads to a higher cache hit rate for the most common data paths.

Would love to hear your thoughts @jodydonetti!

[jodydonetti]

Hi @rsalus , thanks for the very thorough issue.

Short version: look at this item sitting in my FusionCache todo list for quite some time 😬

Longer version: just like with the IDistributedCache abstraction for L2 which allows multiple implementations to be used, I'm planning an ISomethingL1 (no name yet) which would allow different L1 implementations to be used, including of course BitFaster & friends.

So: no, there will be no need to go through the IMemoryCache abstraction which is, well, let's say suboptimal.

Thoughts?

[rsalus]

That makes sense and would certainly be the most straightforward option for FusionCache. I have two main thoughts:

1. It does put the impetus on the cache primitives library to integrate with FusionCache. In practice, this would likely mean that library having to maintain several integrations, presuming they want to provide support for the native HybridCache or other L1+L2 cache implementations. I do think the responsibility should be on the cache primitives library to offer this functionality, so this makes sense IMO.

2. I am curious if you have any concerns around cache synchronization in this scenario. For instance, my first worry would be a substantial increase in calls to the L2 cache based on evictions in the L1 cache.

Beyond those two things, it really comes down to how API shape looks like. Naturally, I'd hope there would be support for low allocations; I'd want to avoid closures (TState) and boxing if possible.

[jodydonetti]

1. It does put the impetus on the cache primitives library to integrate with FusionCache.

Kind of: yes, they would be free to create the adapter if they want, but I think I'll provide the main implementations myself, just like I did with the serializers. The door remains open for anyone, but the main ones will be taken care of.

In practice, this would likely mean that library having to maintain several integrations

Agree, kind of: since there's no Microsoft provided generic memory cache abstraction, like IDistributedCache for distributed caches, this means I need to come up with my own abstraction (again, like I did with IFusionCacheSerializer), and so library authors will need to either create their own adapter implementation or ask me to do it.

And if in the future Microsoft will come up with their own I can add support for it.

NOTE: yeah, in theory IMemoryCache is such abstraction, but as you correctly pointed out it's not that great and would not lead to performant implementations.

presuming they want to provide support for the native HybridCache or other L1+L2 cache implementations.

Agree, but HybridCache supports extensibility of L2 only (via IDistributedCache), not for L1 (unless you go through the IMemoryCache abstraction). As said, if in the future they'll come up with a new abstraction, like they were trying to do with RCache, I can add support for such abstraction.

I do think the responsibility should be on the cache primitives library to offer this functionality, so this makes sense IMO.

Agree, even though as I said for the main ones (think BitFaster) I can do it myself to ease the burden.

2. I am curious if you have any concerns around cache synchronization in this scenario. For instance, my first worry would be a substantial increase in calls to the L2 cache based on evictions in the L1 cache.

Honestly not, meaning: if you configure your L1 to cache stuff for less time, L2 calls will have a bump, bu that is what is already happening today if you use a low memory duration and a longer distributed duration, so I don't see any particular difference here.

What I'm saying is that you'll pick a different L1 impl and configure it to keep the memory usage very low, a direct consequence will be higher load on L2 or the database, depending on the specific scenario. But I think this is expected or, at least, the only thing take makes sense when you think about it.

Also, thanks to full support for observability (see here) which is something you should probably do anyway, I think users will be able to spot an increase in L2 hits very easily.

Beyond those two things, it really comes down to how API shape looks like. Naturally, I'd hope there would be support for low allocations; I'd want to avoid closures (TState) and boxing if possible.

That is something I'm looking into, paired with another item on my todo list which would drop closures for the most common scenarios.

Thoughts?

[rsalus]

Agree with all your points regarding 1; having FusionCache provide the integration itself would be awesome.

Thanks for the insights on 2, I'm glad to know that it doesn't seem like a big issue.

Regarding the API shape, I think most of the core methods outlined in the RCache proposal provided closure-less operations. If you're curious, I was tinkering with a similar implementation in my own project. I think the key is to provide explicit state objects with the GetOrSet methods.

I'd be happy to contribute in this space however you'd like.

[jodydonetti]

Ah, I see where you are going, but in my case is FusionCache itself that provides the cache stampede protection and all the other features: I'll basically need only Set, Get and Remove I think.

I would not be designing a new generic memory cache to use on its own, but only an "L1 for FusionCache" 😉