Consolidate alert notification system into alert_notifier role

[jackaltx]

Current State

Three independent alert scripts manually deployed to monitor11:

• ispconfig-login-alert (email + Matrix) - alerts on ANY ISPConfig login
• matrix-synapse-alert (Matrix only) - alerts on Matrix Synapse errors
• fail2ban-alert (email only) - alerts on high ban rate

Deployment: Manual scp + systemd setup on monitor11

Location: Scripts in /usr/local/bin/, configs in /etc/{service}/, systemd units manually created

Desired State

Unified alert_notifier role in solti-ensemble that:

• Deploys all alert scripts via Ansible templates
• Shares common utilities (matrix-send.py, email helpers)
• Configurable per-alert via role variables
• Idempotent deployment with proper systemd timer management
• Version controlled and repeatable

Benefits

• Repeatable deployments - Deploy to new monitoring servers easily
• Easier to add new alerts - Template pattern for new alert types
• Consistent configuration - All alerts use same variable structure
• Version controlled - Track changes to alert logic over time
• Testing - Molecule scenarios for alert deployment

Design Questions to Answer

1. Shared utility strategy: Should matrix-send.py be a shared utility or per-alert?
2. Template vs. scripts: Alert templates (Jinja2) vs. separate Python scripts?
3. Dependencies: How to handle alert-specific Python dependencies (requests, yaml)?
4. State file management: Where to store state files, cleanup strategy?
5. Alert toggles: Should alerts be independently enabled/disabled via variables?
6. Credentials: How to pass Matrix tokens, SMTP passwords securely?
7. Multiple targets: Support deploying different alerts to different hosts?

Proposed Architecture

solti-ensemble/roles/alert_notifier/
├── defaults/main.yml          # Default variables for all alerts
├── templates/
│   ├── matrix-send.py.j2      # Shared Matrix notification utility
│   ├── ispconfig-login-alert.py.j2
│   ├── matrix-synapse-alert.py.j2
│   ├── fail2ban-alert.py.j2
│   └── alert-wrapper.sh.j2    # Generic wrapper for env vars
├── tasks/
│   ├── main.yml               # Orchestrates alert deployment
│   ├── matrix-send.yml        # Deploy matrix-send.py utility
│   ├── ispconfig-alert.yml    # Deploy ISPConfig alert
│   ├── matrix-synapse-alert.yml
│   └── fail2ban-alert.yml
└── molecule/
    └── default/
        └── converge.yml       # Test alert deployment

Example Usage

- name: Deploy alerts to monitor11
  hosts: monitor11
  roles:
    - role: jackaltx.solti_ensemble.alert_notifier
      vars:
        # Matrix notification config
        alert_matrix_enabled: true
        alert_matrix_homeserver: "https://matrix-web.jackaltx.com"
        alert_matrix_token: "{{ vault_matrix_token }}"
        alert_matrix_room: "#solti-verify:jackaltx.com"
        
        # ISPConfig alert
        ispconfig_alert_enabled: true
        ispconfig_alert_loki_url: "http://localhost:3100"
        ispconfig_alert_email_enabled: true
        ispconfig_alert_smtp_host: "mail.lavnet.net"
        
        # Matrix Synapse alert
        matrix_synapse_alert_enabled: true
        matrix_synapse_alert_check_interval: 10
        
        # Fail2ban alert
        fail2ban_alert_enabled: true
        fail2ban_alert_threshold: 50
        fail2ban_alert_matrix_enabled: false  # Email only

Testing Strategy

1. Molecule scenario: Deploy all alerts to test container
2. Verify systemd timers: Check timers are created and enabled
3. Mock Loki responses: Test alert logic with fake data
4. Matrix notification test: Verify matrix-send.py works
5. Idempotency: Run role twice, ensure no changes

Related

• Existing role: solti-ensemble/roles/alert_notifier (currently has fail2ban-alert only)
• Documentation: mylab/docs/matrix-notifications.md
• Manual deployment: Scripts currently on monitor11 in /usr/local/bin/
• Matrix collection: solti-matrix-mgr for Matrix API integration

Acceptance Criteria

• All three alert scripts deployed via Ansible role
• Matrix notification utility shared across alerts
• Role variables documented in README
• Molecule test passes
• Systemd timers properly enabled
• Idempotent deployment (no changes on second run)
• Migration guide from manual deployment
• Role can be deployed to clean host successfully

[jackaltx]

Architecture Refinement: Decouple Checker from Notifier

The current monolithic design embeds notification logic in each alert script. A better approach is to separate concerns:

Current Design (Monolithic)

[ispconfig-login-alert.py]
  ├─ Query Loki
  ├─ Detect violation
  ├─ Format email message
  ├─ Send SMTP
  ├─ Format Matrix message
  └─ Send Matrix notification

Problems:

• Notification logic duplicated across all alerts
• Hard to add new channels (modify every script)
• Can't change routing without code changes
• Testing requires mocking SMTP/Matrix

Proposed Design (Decoupled)

[Checker Scripts]                    [Event Queue]              [Notifier Daemon]
ispconfig-login-alert.py ──┐         /var/lib/alerts/           alert-notifier.py
matrix-synapse-alert.py ───┼──JSON──>  queue/*.json  ──read──>  ├─ Route by category
fail2ban-alert.py ─────────┘                                    ├─ Send Matrix
other-alert.py                                                  ├─ Send Email
                                                                ├─ Send SMS (future)
                                                                └─ Archive/cleanup

Components:

1. Checker Scripts (existing, simplified)

   • Query Loki for violations
   • Emit structured events to queue
   • Exit (no notification logic)

2. Event Queue (/var/lib/alerts/queue/)

   • File-based queue (JSON files)
   • One file per event
   • Processed and archived by notifier

3. Notifier Daemon (new, single responsibility)

   • Runs on timer (every 1 minute)
   • Reads queue, routes events by category/severity
   • Sends to configured channels
   • Archives processed events

Event Schema

{
  "id": "20260225-230045-ispconfig-001",
  "timestamp": "2026-02-25T23:00:45Z",
  "source": "ispconfig-login-alert",
  "category": "security.authentication",
  "severity": "warning",
  "tags": ["login", "ispconfig", "admin"],
  "data": {
    "username": "admin",
    "source_ip": "71.11.224.94",
    "action": "login_success",
    "service": "ispconfig",
    "host": "fleur.lavnet.net"
  },
  "message": "ISPConfig login: admin from 71.11.224.94",
  "loki_query": "{service_type=\"ispconfig_auth\"}"
}

Notification Routing Config

# /etc/alert-notifier/routing.yml
notification_routes:
  # Authentication events → Matrix + Email
  - match:
      category: "security.authentication"
    channels:
      - type: matrix
        room: "#solti-verify:jackaltx.com"
      - type: email
        recipients: "jack@lavnet.net"
        
  # Critical intrusions → All channels
  - match:
      category: "security.intrusion"
      severity: critical
    channels:
      - type: matrix
        room: "#solti-verify:jackaltx.com"
      - type: email
        recipients: "jack@lavnet.net"
      - type: sms  # future
        
  # Application errors → Matrix only
  - match:
      category: "application.error"
    channels:
      - type: matrix
        room: "#solti-verify:jackaltx.com"
        
  # Threshold violations → Email only
  - match:
      category: "monitoring.threshold"
    channels:
      - type: email
        recipients: "jack@lavnet.net"
        
  # Default fallback
  - match: {}
    channels:
      - type: matrix
        room: "#solti-verify:jackaltx.com"

Benefits

1. Separation of Concerns

   • Checkers: Domain logic (what to monitor)
   • Notifier: Delivery logic (how to notify)

2. Reusable Notification Logic

   • Single implementation of Matrix/Email/SMS
   • Shared formatting, retry, error handling

3. Configuration-Driven Routing

   • Change notification channels without code changes
   • Add new channels without touching checkers
   • Category-based routing rules

4. Easy Testing

   • Test checkers: Verify event generation
   • Test notifier: Mock queue, verify routing
   • No need to mock SMTP/Matrix in checkers

5. Flexible Delivery

   • Batch notifications (daily digest)
   • Throttling (max N per hour)
   • Deduplication (same event, don't spam)

6. Resilience

   • Queue survives crashes
   • Notifier can retry failed sends
   • Easy to inspect/debug queue

File-Based Queue Implementation

Why file-based:

• Simple, no daemon needed for queue
• Resilient (survives crashes)
• Easy to debug (ls /var/lib/alerts/queue/)
• Natural ordering (filename = timestamp)

Queue operations:

# Checker writes event
echo '{...}' > /var/lib/alerts/queue/20260225-230045-ispconfig-001.json

# Notifier processes queue
for event in /var/lib/alerts/queue/*.json; do
  route_and_send "$event"
  mv "$event" /var/lib/alerts/archive/
done

Directory structure:

/var/lib/alerts/
├── queue/           # Pending events (*.json)
├── archive/         # Processed events (timestamped)
├── failed/          # Failed delivery attempts
└── state/           # Notifier state (last run, etc.)

Example: ISPConfig Alert (Simplified)

Before (monolithic):

def send_alert(logins, config):
    # 50 lines of email formatting
    # 30 lines of SMTP sending
    # 40 lines of Matrix formatting
    # 20 lines of Matrix sending

After (checker only):

def emit_event(logins, config):
    for login in logins:
        event = {
            "id": f"{datetime.now().isoformat()}-ispconfig-{uuid.uuid4().hex[:6]}",
            "timestamp": datetime.utcnow().isoformat(),
            "source": "ispconfig-login-alert",
            "category": "security.authentication",
            "severity": "warning" if login['auth_result'] == 'success' else "critical",
            "data": login,
            "message": f"ISPConfig {login['auth_result']}: {login['username']} from {login['source_ip']}"
        }
        
        # Write to queue
        queue_path = Path("/var/lib/alerts/queue")
        queue_path.mkdir(parents=True, exist_ok=True)
        event_file = queue_path / f"{event['id']}.json"
        event_file.write_text(json.dumps(event, indent=2))

Notifier (handles all alerts):

def process_queue():
    queue_dir = Path("/var/lib/alerts/queue")
    
    for event_file in sorted(queue_dir.glob("*.json")):
        event = json.loads(event_file.read_text())
        
        # Match routing rules
        routes = match_routes(event, config['notification_routes'])
        
        # Send to all matched channels
        for route in routes:
            for channel in route['channels']:
                send_notification(event, channel)
        
        # Archive processed event
        archive_event(event_file)

Migration Path

1. Phase 1: Implement notifier daemon with file queue
2. Phase 2: Refactor existing alerts to emit events (backward compatible - keep old notification code)
3. Phase 3: Test dual mode (events + direct notifications)
4. Phase 4: Remove direct notification code from checkers
5. Phase 5: Add new channels (SMS, Slack, etc.) to notifier only

Systemd Units

Checker timers (unchanged):

• ispconfig-login-alert.timer - Every 5 min
• matrix-synapse-alert.timer - Every 10 min
• fail2ban-alert.timer - Every 5 min

Notifier timer (new):

• alert-notifier.timer - Every 1 min
• Processes queue, routes events, sends notifications

Open Questions

1. Event retention: How long to keep archived events?
2. Queue size limits: Max events in queue before alerting?
3. Notification throttling: Max N events per channel per hour?
4. Deduplication: Hash-based or exact match?
5. Failed delivery: Retry strategy (exponential backoff)?
6. Multiple notifiers: Support multiple monitoring hosts?

Acceptance Criteria (Updated)

• Event schema defined and documented
• File-based queue implementation
• Notifier daemon with routing engine
• Configuration-driven routing rules
• Checkers emit events to queue
• Notifier processes queue and sends notifications
• Archive/cleanup for processed events
• Failed delivery handling
• Migration guide from monolithic alerts
• Molecule tests for checker + notifier

[jackaltx]

Pattern-Based Alert Framework

After analyzing the current alerts, there are really only 3-4 core patterns that cover all monitoring scenarios. Instead of writing custom scripts for each alert, we can use a declarative, pattern-based framework.

Core Alert Patterns

1. Threshold Alert (rate-based)

count(events) > threshold [over time_window]

Examples:

• Fail2ban: >50 bans/hour
• API errors: >100 errors/5min
• Matrix Synapse: >0 errors/10min

LogQL:

sum(count_over_time({service_type="fail2ban"} | regexp `[Ban]` [1h])) > 50

2. Existence Alert (event-based)

event exists → alert

Examples:

• ISPConfig login: ANY login → alert
• Root SSH login: ANY root login → alert
• Certificate expiring: ANY cert <30 days → alert

LogQL:

{service_type="ispconfig_auth"}  # Any result = alert

3. Absence Alert (heartbeat/availability)

event NOT exists [over time_window] → alert

Examples:

• Service down: No heartbeat in 5min → alert
• Backup missing: No backup logged today → alert
• Telegraf silent: No metrics in 10min → alert

LogQL:

count_over_time({service_type="telegraf"} [10m]) == 0

4. Change Detection (state change)

current_value != previous_value → alert

Examples:

• Config file modified
• New user added
• Firewall rules changed

Implementation: Requires state tracking (hash comparison)

---

Declarative Alert Definitions

Instead of custom Python scripts, define alerts in YAML:

# /etc/alert-checker/alerts.yml
alerts:
  # Pattern: Threshold
  - name: fail2ban_high_ban_rate
    pattern: threshold
    query: 'sum(count_over_time({service_type="fail2ban"} | regexp `[Ban]` [1h]))'
    threshold: 50
    operator: ">"
    category: "monitoring.threshold"
    severity: warning
    message: "Fail2ban ban rate exceeded: {value} bans/hour (threshold: {threshold})"
    
  # Pattern: Existence
  - name: ispconfig_login
    pattern: exists
    query: '{service_type="ispconfig_auth"}'
    time_window: 5m
    category: "security.authentication"
    severity: warning
    message: "ISPConfig login: {username} from {source_ip}"
    extract_fields:
      - username
      - source_ip
    
  # Pattern: Threshold with subcategories
  - name: matrix_synapse_errors
    pattern: threshold
    query: 'count_over_time({service_type="matrix_synapse", is_error="true"} [10m])'
    threshold: 0
    operator: ">"
    category: "application.error"
    severity: warning
    message: "Matrix Synapse errors: {value} in last 10min"
    extract_fields:
      - error_category
      - module_group
    group_by: error_category  # Emit separate events per category
    
  # Pattern: Absence
  - name: telegraf_heartbeat_monitor11
    pattern: absent
    query: '{service_type="telegraf", hostname="monitor11"}'
    time_window: 10m
    category: "monitoring.availability"
    severity: critical
    message: "Telegraf heartbeat missing for monitor11 (no data in 10min)"
    
  # Pattern: Existence with filtering
  - name: ssh_root_login
    pattern: exists
    query: '{service_type="sshd"} |= "Accepted" |= "root"'
    time_window: 5m
    category: "security.intrusion"
    severity: critical
    message: "Root SSH login detected: {source_ip}"
    extract_fields:
      - source_ip
      - hostname

---

Generic Alert Checker

Single Python script that handles all patterns:

#!/usr/bin/env python3
"""
Generic Alert Checker - Pattern-based Loki monitoring
Reads alert definitions, queries Loki, emits events
"""
import json
import yaml
import requests
import time
from pathlib import Path

def check_threshold_alert(alert, loki_url):
    """Pattern: metric > threshold"""
    # Query Loki for instant metric value
    params = {
        'query': alert['query'],
        'time': int(time.time() * 1e9)
    }
    resp = requests.get(f"{loki_url}/loki/api/v1/query", params=params)
    data = resp.json()
    
    if data['status'] == 'success' and data['data']['result']:
        value = float(data['data']['result'][0]['value'][1])
        threshold = alert['threshold']
        operator = alert['operator']
        
        # Evaluate threshold condition
        if eval(f"{value} {operator} {threshold}"):
            emit_event({
                'source': alert['name'],
                'category': alert['category'],
                'severity': alert['severity'],
                'message': alert['message'].format(value=value, threshold=threshold),
                'data': {'value': value, 'threshold': threshold}
            })

def check_exists_alert(alert, loki_url):
    """Pattern: event exists"""
    # Calculate time range
    now_ns = int(time.time() * 1e9)
    window_sec = parse_duration(alert.get('time_window', '5m'))
    start_ns = now_ns - int(window_sec * 1e9)
    
    params = {
        'query': alert['query'],
        'start': start_ns,
        'end': now_ns,
        'limit': 100
    }
    resp = requests.get(f"{loki_url}/loki/api/v1/query_range", params=params)
    data = resp.json()
    
    if data['status'] == 'success' and data['data']['result']:
        for stream in data['data']['result']:
            labels = stream['stream']
            
            # Extract specified fields from labels
            extracted = {
                field: labels.get(field, 'unknown')
                for field in alert.get('extract_fields', [])
            }
            
            # Emit event for each log entry (or group by field)
            group_by = alert.get('group_by')
            if group_by:
                # Group events by field value
                grouped = {}
                for value in stream['values']:
                    key = labels.get(group_by, 'other')
                    if key not in grouped:
                        grouped[key] = []
                    grouped[key].append(value)
                
                # Emit one event per group
                for key, values in grouped.items():
                    emit_event({
                        'source': alert['name'],
                        'category': alert['category'],
                        'severity': alert['severity'],
                        'message': alert['message'].format(**extracted, **{group_by: key}),
                        'data': {**extracted, group_by: key, 'count': len(values)}
                    })
            else:
                # Emit event per stream
                emit_event({
                    'source': alert['name'],
                    'category': alert['category'],
                    'severity': alert['severity'],
                    'message': alert['message'].format(**extracted),
                    'data': extracted
                })

def check_absent_alert(alert, loki_url):
    """Pattern: event absent (no data)"""
    now_ns = int(time.time() * 1e9)
    window_sec = parse_duration(alert['time_window'])
    start_ns = now_ns - int(window_sec * 1e9)
    
    params = {
        'query': alert['query'],
        'start': start_ns,
        'end': now_ns,
        'limit': 1
    }
    resp = requests.get(f"{loki_url}/loki/api/v1/query_range", params=params)
    data = resp.json()
    
    # Alert if NO results found
    if not data['data']['result']:
        emit_event({
            'source': alert['name'],
            'category': alert['category'],
            'severity': alert['severity'],
            'message': alert['message'],
            'data': {'time_window': alert['time_window']}
        })

def emit_event(event):
    """Write event to queue"""
    import uuid
    from datetime import datetime
    
    event_id = f"{datetime.utcnow().strftime('%Y%m%d-%H%M%S')}-{event['source']}-{uuid.uuid4().hex[:6]}"
    full_event = {
        'id': event_id,
        'timestamp': datetime.utcnow().isoformat(),
        **event
    }
    
    queue_dir = Path('/var/lib/alerts/queue')
    queue_dir.mkdir(parents=True, exist_ok=True)
    
    event_file = queue_dir / f"{event_id}.json"
    event_file.write_text(json.dumps(full_event, indent=2))

def parse_duration(duration_str):
    """Parse duration like '5m', '1h', '30s' to seconds"""
    units = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}
    value = int(duration_str[:-1])
    unit = duration_str[-1]
    return value * units[unit]

def main():
    # Load config
    config = yaml.safe_load(Path('/etc/alert-checker/config.yml').read_text())
    alerts = yaml.safe_load(Path('/etc/alert-checker/alerts.yml').read_text())
    
    # Pattern handlers
    handlers = {
        'threshold': check_threshold_alert,
        'exists': check_exists_alert,
        'absent': check_absent_alert
    }
    
    # Process each alert
    for alert in alerts['alerts']:
        if not alert.get('enabled', True):
            continue
            
        try:
            handler = handlers[alert['pattern']]
            handler(alert, config['loki_url'])
        except Exception as e:
            print(f"❌ Error checking alert {alert['name']}: {e}")
    
    print(f"✅ Checked {len(alerts['alerts'])} alerts")

if __name__ == '__main__':
    main()

---

Configuration

# /etc/alert-checker/config.yml
loki_url: "http://monitor11.a0a0.org:3100"

# State tracking
state_dir: "/var/lib/alert-checker"

# Event queue
queue_dir: "/var/lib/alerts/queue"

---

Updated Architecture

┌──────────────────────────────────────────────────────────────┐
│ Generic Alert Checker (alert-checker.py)                     │
│   ├─ Load alerts.yml                                         │
│   ├─ For each alert definition:                              │
│   │   ├─ Identify pattern (threshold/exists/absent)          │
│   │   ├─ Query Loki with alert's LogQL                       │
│   │   ├─ Apply pattern logic                                 │
│   │   └─ If condition met → emit event to queue              │
│   └─ Exit                                                     │
└──────────────────────────────────────────────────────────────┘
                         ↓ Events (JSON)
┌──────────────────────────────────────────────────────────────┐
│ Event Queue (/var/lib/alerts/queue/*.json)                   │
│   {                                                           │
│     "id": "20260225-230045-ispconfig-001",                  │
│     "source": "ispconfig_login",                             │
│     "category": "security.authentication",                   │
│     "severity": "warning",                                   │
│     "message": "ISPConfig login: admin from 71.11.224.94",   │
│     "data": {"username": "admin", "source_ip": "..."}      │
│   }                                                           │
└──────────────────────────────────────────────────────────────┘
                         ↓ Read & route
┌──────────────────────────────────────────────────────────────┐
│ Alert Notifier (alert-notifier.py)                           │
│   ├─ Read queue/*.json                                       │
│   ├─ Match routing rules (category + severity)               │
│   ├─ Send to channels:                                       │
│   │   ├─ Matrix (#solti-verify:jackaltx.com)                 │
│   │   ├─ Email (jack@lavnet.net)                             │
│   │   └─ SMS (future)                                        │
│   └─ Archive processed events                                │
└──────────────────────────────────────────────────────────────┘

---

Benefits

1. No code for new alerts

   • Add alert definition to YAML
   • Checker handles pattern logic automatically

2. Consistent behavior

   • All threshold alerts work identically
   • Predictable, testable patterns

3. Self-documenting

   • Alert definitions are the documentation
   • Clear what's monitored and why

4. Easy validation

   • JSON schema for alert definitions
   • Catch config errors before deployment

5. Flexible extraction

   • Pull fields from Loki labels
   • Group by categories (e.g., error_category)

6. Testable

   • Mock Loki responses
   • Test each pattern independently
   • Verify event emission

---

Migration from Current Alerts

Current custom scripts → Pattern-based config:

# ispconfig-login-alert.py → alert definition
- name: ispconfig_login
  pattern: exists
  query: '{service_type="ispconfig_auth"}'
  time_window: 5m
  category: "security.authentication"
  severity: warning
  message: "ISPConfig login: {username} from {source_ip}"
  extract_fields: [username, source_ip]

# matrix-synapse-alert.py → alert definition
- name: matrix_synapse_errors
  pattern: threshold
  query: 'count_over_time({service_type="matrix_synapse", is_error="true"} [10m])'
  threshold: 0
  operator: ">"
  category: "application.error"
  severity: warning
  message: "Matrix Synapse: {value} errors in 10min"
  extract_fields: [error_category, module_group]
  group_by: error_category

# fail2ban-alert.py → alert definition
- name: fail2ban_high_rate
  pattern: threshold
  query: 'sum(count_over_time({service_type="fail2ban"} | regexp `[Ban]` [1h]))'
  threshold: 50
  operator: ">"
  category: "monitoring.threshold"
  severity: warning
  message: "Fail2ban: {value} bans/hour exceeds threshold {threshold}"

Result:

• 3 custom Python scripts → 1 generic checker + 3 YAML definitions
• Future alerts: Just add YAML, no code

---

Systemd Units

Single checker timer:

# /etc/systemd/system/alert-checker.timer
[Unit]
Description=Alert Checker Timer
Documentation=https://github.com/jackaltx/solti-ensemble

[Timer]
OnBootSec=2min
OnUnitActiveSec=5min  # Check all alerts every 5 min
Unit=alert-checker.service

[Install]
WantedBy=timers.target

Notifier timer:

# /etc/systemd/system/alert-notifier.timer
[Unit]
Description=Alert Notifier Timer

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min  # Process queue every 1 min
Unit=alert-notifier.service

[Install]
WantedBy=timers.target

---

Updated Acceptance Criteria

• Core alert patterns identified and documented
• Generic checker supports threshold/exists/absent patterns
• YAML schema for alert definitions
• Schema validation for alert configs
• Field extraction from Loki labels
• Group-by support for subcategories
• Event emission to queue
• All current alerts migrated to pattern-based definitions
• Single checker replaces 3 custom scripts
• Molecule tests for each pattern
• Documentation for adding new alerts (just YAML)

[jackaltx]

Security Model: Service Account & Secrets Management

Service Account (Defense in Depth)

Create dedicated service user:

useradd --system --home /var/lib/alert-service --shell /usr/sbin/nologin alert-service

Why use a dedicated user when lavender has sudo?

Even though the primary admin (lavender) has sudo access, a dedicated service user provides defense in depth:

Threat	Without Service User	With Service User
Compromised web app	Can read lavender's secrets	Cannot access alert-service files
Malicious script (as lavender)	Direct access to secrets	Requires explicit sudo (logged)
Accidental damage	rm -rf can delete state	Service files isolated
Audit trail	No log of secret access	Sudo logs access attempts
Least privilege	Service has all user perms	Service has minimal perms

Benefits:

• ✅ Protects against accidental and opportunistic attacks
• ✅ Creates audit trail (sudo logs)
• ✅ Industry best practice for system services
• ✅ Future-proof - already isolated if other services added
• ✅ Ansible-friendly - Role creates user with become: true

Limitations (be realistic):

• ❌ Does NOT protect against determined attacker with lavender's password
• ❌ Lavender can still sudo cat /etc/alert-service/secrets.env
• ✅ But requires deliberate action and creates audit log

File Ownership & Permissions

# Scripts (root-owned, world-readable)
/usr/local/bin/alert-checker.py          → root:root (755)
/usr/local/bin/alert-notifier.py         → root:root (755)
/usr/local/bin/matrix-send.py            → root:root (755)

# Configuration (root-owned, root-readable only)
/etc/alert-service/config.yml            → root:root (600)
/etc/alert-service/secrets.env           → root:root (600)
/etc/alert-service/alerts.yml            → root:root (644)  # No secrets
/etc/alert-service/routing.yml           → root:root (644)  # No secrets

# State & queue (service-owned, service-writable)
/var/lib/alert-service/                  → alert-service:alert-service (750)
/var/lib/alerts/queue/                   → alert-service:alert-service (750)
/var/lib/alerts/archive/                 → alert-service:alert-service (750)
/var/lib/alerts/failed/                  → alert-service:alert-service (750)

# Admin access (lavender in alert-service group for read-only debugging)
usermod -a -G alert-service lavender

Secrets Management Strategy

Approach: Systemd EnvironmentFile + Ansible Vault

1. Store Secrets in Ansible Vault

# group_vars/monitor11/vault.yml (encrypted with ansible-vault)
vault_matrix_token: "syt_c29sdGktbG9nZ2Vy_LzOVkVKSEyEFmDaBOSqM_0ujkoB"
vault_smtp_password: "qrz6#9A8?cHXoPbE"
vault_loki_url: "http://monitor11.a0a0.org:3100"

2. Deploy via Ansible Template

{# templates/secrets.env.j2 #}
# Alert Service Secrets
# Deployed by: solti-ensemble/roles/alert_notifier
# DO NOT EDIT MANUALLY - managed by Ansible

MATRIX_HOMESERVER_URL={{ vault_matrix_homeserver | default('https://matrix-web.jackaltx.com') }}
MATRIX_SOLTI_LOGGER_TOKEN={{ vault_matrix_token }}
MATRIX_ROOM_ID={{ vault_matrix_room | default('#solti-verify:jackaltx.com') }}

LOKI_URL={{ vault_loki_url }}

SMTP_HOST={{ vault_smtp_host | default('mail.lavnet.net') }}
SMTP_PORT={{ vault_smtp_port | default(587) }}
SMTP_USERNAME={{ vault_smtp_username }}
SMTP_PASSWORD={{ vault_smtp_password }}
SMTP_FROM={{ vault_smtp_from | default('claude_alerts@jackaltx.com') }}

# tasks/deploy-secrets.yml
- name: Deploy secrets file
  ansible.builtin.template:
    src: secrets.env.j2
    dest: /etc/alert-service/secrets.env
    owner: root
    group: root
    mode: '0600'
  become: true
  no_log: true  # Don't log secret values
  notify: Restart alert services

3. Systemd Service Configuration

# /etc/systemd/system/alert-checker.service
[Unit]
Description=Alert Checker - Pattern-based Loki monitoring
Documentation=https://github.com/jackaltx/solti-ensemble

[Service]
Type=oneshot
User=alert-service
Group=alert-service
WorkingDirectory=/var/lib/alert-service

# Load secrets (systemd injects into environment)
EnvironmentFile=/etc/alert-service/secrets.env

# Security hardening
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/etc/alert-service
ReadWritePaths=/var/lib/alert-service
ReadWritePaths=/var/lib/alerts
PrivateTmp=true
NoNewPrivileges=true
CapabilityBoundingSet=

ExecStart=/usr/local/bin/alert-checker.py

StandardOutput=journal
StandardError=journal

How it works:

1. Secrets file owned by root:root (600)
2. Service user alert-service cannot read the file directly
3. Systemd reads file at service start (systemd runs as root)
4. Systemd injects variables into service environment
5. Script accesses via os.getenv('MATRIX_SOLTI_LOGGER_TOKEN')

Access Control

Who can access what:

User	Secrets File	State Files	Service Control
root	✅ Read/write	✅ Read/write	✅ Full control
alert-service	❌ No access	✅ Read/write (own files)	❌ No direct access
lavender	⚠️ Via sudo only	✅ Read-only (group member)	✅ systemctl (via sudo)
Other users	❌ No access	❌ No access	❌ No access

Lavender's management capabilities:

# Service management (standard systemctl)
sudo systemctl status alert-checker
sudo systemctl restart alert-checker
sudo journalctl -u alert-checker -f

# Edit secrets (explicit sudo required, logged)
sudo vi /etc/alert-service/secrets.env
sudo systemctl daemon-reload
sudo systemctl restart alert-checker

# Debug queue (read-only via group membership)
ls -la /var/lib/alerts/queue/
cat /var/lib/alerts/queue/20260225-230045-ispconfig-001.json

# Deploy via Ansible (uses become: true)
ansible-playbook deploy-alerts.yml --ask-become-pass

Security Best Practices

1. Never commit secrets to git

   • Use Ansible Vault for encrypted storage
   • Keep vault password in ~/.secrets/vault_pass

2. Rotate secrets periodically

   • Matrix bot token (generate new via Admin API)
   • SMTP password
   • Update vault, redeploy

3. Audit access

   • Review sudo logs: sudo grep alert-service /var/log/auth.log
   • Monitor failed systemd starts

4. Backup encrypted

   • Vault file can be backed up (it's encrypted)
   • Never backup secrets.env in plain text

5. Use per-service tokens

   • Don't reuse Matrix token across services
   • Dedicated SMTP credentials per service

Ansible Role Implementation

# roles/alert_notifier/tasks/main.yml
- name: Create alert service user
  ansible.builtin.user:
    name: alert-service
    system: true
    home: /var/lib/alert-service
    shell: /usr/sbin/nologin
    comment: "Alert notification service account"
  become: true

- name: Add admin user to alert-service group
  ansible.builtin.user:
    name: "{{ ansible_user }}"
    groups: alert-service
    append: true
  become: true

- name: Create alert service directories
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: "{{ item.owner }}"
    group: "{{ item.group }}"
    mode: "{{ item.mode }}"
  loop:
    - { path: /etc/alert-service, owner: root, group: root, mode: '0755' }
    - { path: /var/lib/alert-service, owner: alert-service, group: alert-service, mode: '0750' }
    - { path: /var/lib/alerts/queue, owner: alert-service, group: alert-service, mode: '0750' }
    - { path: /var/lib/alerts/archive, owner: alert-service, group: alert-service, mode: '0750' }
    - { path: /var/lib/alerts/failed, owner: alert-service, group: alert-service, mode: '0750' }
  become: true

- name: Deploy secrets from vault
  ansible.builtin.template:
    src: secrets.env.j2
    dest: /etc/alert-service/secrets.env
    owner: root
    group: root
    mode: '0600'
  become: true
  no_log: true
  notify: Restart alert services

Migration from Current Setup

Current (running as lavender):

/home/lavender/.secrets/LabMatrix
User=lavender in systemd units

New (dedicated service user):

/etc/alert-service/secrets.env (from Ansible Vault)
User=alert-service in systemd units

Migration steps:

1. Deploy role with new service user
2. Test alerts work with new user
3. Remove old secrets from ~/.secrets/ (or keep for other uses)
4. Update systemd units to use alert-service user
5. Restart services

Updated Acceptance Criteria

• Dedicated service user created (alert-service)
• Service user has no login shell
• Secrets managed via Ansible Vault
• Secrets deployed with root:root 600 permissions
• Systemd EnvironmentFile pattern used
• Admin user in alert-service group (read-only debugging)
• State/queue directories owned by service user
• Systemd security hardening applied (ProtectSystem, etc.)
• Ansible role creates all directories/users
• Migration guide from current lavender-based deployment