Custom serializer/codec for `redis-strings` - extension for compression/encryption

[m-sanders]

The Redis cache handler stores Next.js cache values as uncompressed JSON, wasting 2.5-10x more memory than necessary. Serialization is hardcoded in redis-strings.ts with no way to add compression, encryption, or custom formats.

My ideas is to:

Implement a pluggable serializer system with:

• Strategy pattern for serialization/deserialization
• defaultSerializer - no compression (backward compatible)
• gzipSerializer - 60-85% size reduction
• Extensible CacheSerializer interface for custom implementations

Impact

Reduces Redis memory usage and infrastructure costs while enabling customization for compression algorithms, encryption, or specialized formats.

I am happy to look at implementing.

(EDIT: I stupidly opened the wrong issue for this repo as I mixed up tabs 😬 )

[AyronK]

Great idea @m-sanders! I agree it should be implemented as an opt-in processor rather than a built in solution. Something like:

// New opt in config allowing any kind of post processing such as compression, encryption, serialization etc.
// By default just the usual json stringify/parse
const redisHandler = await createRedisHandler({
  client: createClient({
    url: process.env.REDIS_URL,
  }),
  keyPrefix: "myApp:",
  sharedTagsKey: "myTags",
  sharedTagsTtlKey: "myTagTtls",

  valueSerializer?: CacheValueSerializer // default -> JsonSerializer
});

// New opt in Serializer type
export interface CacheValueSerializer {
  serialize(value: CacheHandlerValue): string
  deserialize(value: string): CacheHandlerValue
}

// Default Serializer (current implementation)
const jsonValueSerializer: CacheValueSerializer = {
  serialize(value) {
    return JSON.stringify(value)
  },
  deserialize(value) {
    return JSON.parse(
      typeof value === "string" ? value : value.toString()
    )
  },
}

// Then someone can inject custom implementaiton
import { gzipSync, gunzipSync } from "node:zlib"

export const gzipJsonValueSerializer: CacheValueSerializer = {
  serialize(value) {
    return gzipSync(JSON.stringify(value))
  },
  deserialize(value) {
    const buffer =
      typeof value === "string" ? Buffer.from(value) : value

    return JSON.parse(gunzipSync(buffer).toString())
  },
}

Some project don't care about space, only about speed. Some projects will require extra encryption for sensitive data. Some as you mentioned would benefit from compression.

The only thing I am against is adding extra dependencies to the core package.

[gergokee]

Hi hi, this would be an awesome feature we started using compression, we use brotli, so i'd like to ask why not use brotli instead of gzip (seems more performant).

[AyronK]

@gergokee the idea is to expose abstraction so anyone can use whatever they like, I don't want to make any actual tight dependencies.

[m-sanders]

Apologies, I have yet to find time to pick this up.

I originally proposed gzip as it avoids an external dependency + suggesting the strategy pattern to support plug-and-play.

There are some interesting articles on this like:

https://careersatdoordash.com/blog/speeding-up-redis-with-compression/

https://medium.com/@vishalpriyadarshi/optimizing-redis-memory-usage-and-performance-with-encoding-and-compression-in-java-ce74a3c4fdb9

The key is to balance compression compute in the application server with the output size in redis. The balance you want to aim for is cheap non-blocking UV thread pool based compression at sensible payload sizes.

If someone else wants to action this, please do otherwise I'll try to find time in late March.

[AyronK]

Serialization/encryption are usually one liners, I don't think there's a need to put actual implementation strategies in the package. I would rather expose API so people can adjust the default json serialization to their needs and decorate it with any string processing they need.

[VladBo]

Added optional valueSerializer + exported jsonCacheValueSerializer, default unchanged - #211

[AyronK]

Done in 3.1.0. Thanks @VladBo!