perf(event-stream): improve reliability

[dargmuesli]

This pull request introduces Kafka event stream processing with dead letter queue handling and adds Redis-based email rate limiting to improve reliability and stability. It also refactors message processing logic for notifications and uploads, and adds new dependencies for Kafka and Redis clients.

[Copilot]

Pull request overview

This PR improves server-side event-stream reliability by introducing retry-with-backoff plus dead-letter-queue (DLQ) handling for Kafka consumers, and adds Redis-backed per-second email rate limiting to reduce spikes and improve stability.

Changes:

• Refactors Kafka event processing into per-topic handlers with retry + DLQ publishing/consumption.
• Adds Redis-based rate limiting to sendEmail and introduces a Redis dependency wrapper.
• Adds a generic retry-with-exponential-backoff utility and formalizes “raw message” handlers for notification/upload events.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/server/plugins/event-stream.ts	Adds retry/DLQ flow, splits topic handling, and introduces a dedicated DLQ consumer + producer.
src/server/utils/retry.ts	New retryWithBackoff helper used by the Kafka plugin.
src/server/utils/notification.ts	Adds processRawNotification to centralize parsing/guard logic for Kafka messages.
src/server/utils/upload.ts	Adds processRawUpload to centralize parsing/guard logic for Kafka messages.
src/server/utils/email.ts	Adds Redis-based per-second mail rate limiting before sending.
src/server/utils/dependencies/redis.ts	Introduces a Redis client wrapper (ioredis).
src/server/utils/dependencies/kafka.ts	Extracts Kafka client creation into a shared helper.
src/package.json	Adds ioredis dependency.
pnpm-lock.yaml	Locks ioredis dependency version.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

This PR aims to improve backend reliability by hardening Kafka event-stream processing (retries + dead-letter queue) and adding Redis-backed email rate limiting, while refactoring notification/upload message handling into “raw” processors.

Changes:

• Add Kafka consumer retry/backoff handling and a dead-letter queue flow for failed messages.
• Introduce Redis-based email rate limiting (per-day + per-second) and update notification emailing to use it.
• Add new utility/dependency modules (retry helper, Redis/Kafka client factories) and related runtime config/deps.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/shared/utils/constants.ts	Rename/add email rate limit constants.
src/server/utils/upload.ts	Add processRawUpload wrapper for Kafka message decoding/validation.
src/server/utils/retry.ts	Add generic retryWithBackoff helper used by Kafka processing.
src/server/utils/notification.ts	Refactor to accept Redis + new rate-limit config; add processRawNotification.
src/server/utils/email.ts	Add Redis-based per-second gating and switch to per-day limit naming.
src/server/utils/dependencies/redis.ts	Add singleton Redis client accessor.
src/server/utils/dependencies/kafka.ts	Add Kafka client factory driven by runtime config.
src/server/plugins/redis.ts	Add Nitro plugin to initialize/close Redis connection.
src/server/plugins/event-stream.ts	Refactor Kafka plugin: topic handler map, retries, DLQ producer/consumer, Redis injection.
src/package.json	Add ioredis.
src/nuxt.config.ts	Add runtime config for email rateLimit, Kafka brokers, and Redis URL.
pnpm-lock.yaml	Lockfile update for ioredis.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.