[FEATURE] Real-time notification system

[cjlapao]

Is your feature request related to a problem? Please describe.

The product currently has no standardised way to surface events — operational or informational — to users in real time. When something significant happens (a host goes offline, a disk reaches capacity, a background job completes, a service recovers), there is no mechanism to inform the user immediately, nor any way for users who were offline at the time to catch up on what they missed.
This creates three concrete pain points:

Missed critical events: Users who are not actively watching the UI when an alert fires have no way of knowing it happened. There is no persistent record they can check after the fact.
No contextual resolution path: Even when a user does see that something went wrong, there is no in-product signal pointing them to the right place to act or confirming when the situation has resolved.
No correlation between related events: An alert that a host went down and a subsequent notification that it recovered are today completely unrelated signals. There is no way to understand that two notifications are describing the lifecycle of the same underlying event.

Describe how would you expect the feature to work

Overview

We will implement a real-time, persistent, per-user notification system delivered over the existing WebSocket infrastructure. The system will allow any internal component of the service to raise a notification, route it to the correct users, persist it for later retrieval, and push it live to any user who is currently connected. Users who were offline when a notification was raised can retrieve their full notification history via a REST endpoint when they reconnect.

Notification structure

Every notification will carry the following fields:

Field	Type	Description
id	string (UUID)	Unique identifier for this notification instance
user_id	string	The user this notification belongs to
type	enum	One of: alert, warning, info, success
title	string	Short human-readable headline
message	string	Full description of the event
host_id	string (nullable)	ID of the host this notification relates to
record_id	string (nullable)	ID of the primary record this notification relates to (e.g. a host, job, or service)
correlation_id	string (nullable)	Groups related notifications together (e.g. an alert and its subsequent resolution share the same correlation_id). The value is typically the record_id of the originating entity
actions	[]Action	Zero or more actions the UI can render as buttons (see below)
meta	[]map[string]string	Arbitrary key-value pairs for additional context (tags, labels, supplementary IDs)
is_read	bool	Per-user read/unread flag. Defaults to false
created_at	timestamp	When the notification was created

Action structure

Each action in the actions array describes something the UI can offer the user as a next step:

Field	Type	Description
label	string	Button label displayed in the UI
type	enum	navigate (UI-side routing) or api_call (triggers a backend endpoint)
target	string	For navigate: a UI route. For api_call: a relative API path
method	string (nullable)	HTTP method for api_call actions (e.g. POST, DELETE)
payload	map[string]string (nullable)	Optional body payload for api_call actions

Notification types

The type field is a fixed enum that determines how the UI should present the notification:

• alert — a critical condition requiring immediate attention (e.g. host unreachable, service crashed)
• warning — a degraded or at-risk condition that does not yet require immediate action (e.g. disk at 85% capacity)
• info — a neutral informational event (e.g. a scheduled job has started)
• success — a positive outcome or resolution (e.g. a host has recovered, a job completed successfully)

Fan-out and targeting

Notifications can be raised in two modes:

• System fan-out: The caller emits a single notification event without specifying recipients. The dispatcher resolves the target user set (e.g. all users with access to the affected host) and creates a per-user notification record for each.
• Explicit targeting: The caller specifies one or more user_id values. The dispatcher creates a notification record only for those users.

Event correlation

Related notifications (e.g. an alert that a host went down, followed later by a success notification that it recovered) share the same correlation_id. This value is set by the caller at emit time — typically to the record_id of the underlying entity — and is stored verbatim. There is no enforced hierarchy; correlation is a flat grouping. The UI can use correlation_id to thread related notifications together or to visually link a resolution back to its originating alert.

Persistence and delivery

• All notifications are persisted to the database at creation time, regardless of whether the target user is currently connected.
• When a user is online, the notification is additionally pushed over the WebSocket connection immediately after being persisted.
• When a user reconnects after a period of absence, the UI polls the REST history endpoint to retrieve any notifications created since the user's last session. There is no server-side replay over WebSocket on reconnect.
• Notifications are never automatically expired or deleted by the server. Each user manages their own inbox — they can mark individual notifications as read or delete them. Deletion is per-user and does not affect other users' copies of the same notification.

Read/unread state

Each notification carries a per-user is_read flag. This enables the UI to display a badge count of unread notifications. The flag is set to false on creation and can be updated to true via the REST API (individually or in bulk — mark all as read).

REST API surface (summary)

Method	Path	Description
GET	/notifications	List notifications for the authenticated user. Supports filtering by type, is_read, correlation_id, and pagination
GET	/notifications/unread-count	Returns the count of unread notifications for badge display
PATCH	/notifications/:id/read	Mark a single notification as read
PATCH	/notifications/read-all	Mark all notifications as read
DELETE	/notifications/:id	Delete a single notification for the authenticated user

WebSocket message format

Notifications delivered over WebSocket will use the existing message envelope. The notification payload will be delivered as a message of type notification.created, containing the full notification object as defined above.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Out of scope for this feature

• UI notification centre / inbox component (to be addressed in a separate frontend feature)
• Server-side TTL or automatic expiry
• Push notifications (mobile/browser push)
• Notification preferences or per-user opt-out rules