Merge pull request #83 from Cod-e-Codes/feat/server-multi-db-durable-state

feat(server): multi-DB backends with durable reactions/read receipts/channel state

Author: Cod-e-Codes

ARCHITECTURE.md

@@ -20,9 +20,9 @@ Marchat is a self-hosted, terminal-based chat application built in Go with a cli
          ▼                       ▼                       ▼
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │ Configuration   │    │ Shared Types    │    │ Database Layer  │
-│ • Profiles      │    │ • Message Types │    │ • SQLite        │
-│ • Encryption    │    │ • Crypto Utils  │    │ • Persistence   │
-│ • Themes        │    │ • Protocols     │    │ • State Mgmt    │
+│ • Profiles      │    │ • Message Types │    │ • SQL Backends  │
+│ • Encryption    │    │ • Crypto Utils  │    │ • Postgres/MySQL│
+│ • Themes        │    │ • Protocols     │    │ • Dialect State │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
 ```
 
@@ -108,7 +108,7 @@ The server package contains the core server logic and components that are used b
 #### Core Components
 
 - **WebSocket Handlers**: Connection management and message routing
-- **Database Layer**: SQLite integration with message persistence
+- **Database Layer**: Pluggable SQL backends (SQLite/PostgreSQL/MySQL) with dialect-aware schema and query helpers
 - **Admin Interfaces**: Both TUI and web-based administrative panels
 - **Plugin Integration**: Plugin command handling and execution
 - **Health Monitoring**: System metrics and health check endpoints
@@ -201,15 +201,15 @@ The **client** stores `config.json`, `profiles.json`, keystore (unless legacy `k
 #### Key Settings (server-oriented)
 
 - **Server Configuration**: Port, TLS certificates, admin authentication
-- **Database Settings**: SQLite file path and connection parameters
+- **Database Settings**: `MARCHAT_DB_PATH` selects backend and connection details (SQLite path or Postgres/MySQL DSN)
 - **Plugin Configuration**: Registry URL and installation directories
 - **Security Settings**: Admin keys, encryption keys, and authentication
 - **File Transfer**: Size limits and allowed file types
 - **Logging**: Log levels and output destinations
 
 ### Diagnostics (`internal/doctor`)
 
-Shared package invoked by **`marchat-client`** and **`marchat-server`** when passed **`-doctor`** (human-readable report) or **`-doctor-json`** (JSON on stdout). It summarizes Go/OS, resolved config directories, known `MARCHAT_*` variables with secrets masked, role-specific checks (client: profiles, clipboard, TTY; server: `.env`, validation, DB/TLS/sqlite ping), and optionally compares the embedded version to the latest GitHub release. Set **`MARCHAT_DOCTOR_NO_NETWORK=1`** to skip the release check (e.g. air-gapped environments).
+Shared package invoked by **`marchat-client`** and **`marchat-server`** when passed **`-doctor`** (human-readable report) or **`-doctor-json`** (JSON on stdout). It summarizes Go/OS, resolved config directories, known `MARCHAT_*` variables with secrets masked, role-specific checks (client: profiles, clipboard, TTY; server: `.env`, validation, detected DB dialect, DB connection-string format validation, DB/TLS ping checks), and optionally compares the embedded version to the latest GitHub release. Set **`MARCHAT_DOCTOR_NO_NETWORK=1`** to skip the release check (e.g. air-gapped environments).
 
 ### Command Line Tools (`cmd/`)
 
@@ -235,11 +235,26 @@ Both **`marchat-client`** and **`marchat-server`** embed diagnostics via **`inte
 
 ```
 Client: Input → Encrypt → WebSocket Send
-Server: WebSocket Receive → Hub → Plugin Processing → Database
-Server: Database → Hub → WebSocket Broadcast  
+Server: WebSocket Receive → Hub → Plugin Processing → SQL Backend
+Server: SQL Backend → Hub → WebSocket Broadcast  
 Client: WebSocket Receive → Decrypt → Display
 ```
 
+### Database Backends and Durability
+
+- The server chooses a backend at runtime using `MARCHAT_DB_PATH`:
+  - SQLite path (default local setup)
+  - PostgreSQL DSN (`postgres://` / `postgresql://`)
+  - MySQL DSN (`mysql:` / `mysql://`)
+- Schema creation and upsert/insert-ignore SQL are dialect-aware.
+- Placeholder rebinding keeps shared query callsites portable across backends.
+- SQLite-specific optimizations (for example WAL mode) are applied only when the selected backend is SQLite.
+- Durable state includes:
+  - message history
+  - reactions
+  - read receipts
+  - last channel per user
+
 ### WebSocket Protocol
 
 The WebSocket communication uses JSON messages with the following structure:
@@ -304,14 +319,15 @@ CREATE TABLE ban_history (
 
 ### Key Features
 
-- **WAL Mode**: Write-Ahead Logging for better concurrency and crash recovery
-- **Database Files**: Creates three files - `marchat.db` (main), `marchat.db-wal` (write-ahead log), `marchat.db-shm` (shared memory)
+- **Backend Selection**: `MARCHAT_DB_PATH` chooses SQLite/PostgreSQL/MySQL at runtime
+- **WAL Mode (SQLite only)**: Write-Ahead Logging for better concurrency and crash recovery when SQLite is selected
+- **SQLite Database Files**: `marchat.db` (main), `marchat.db-wal` (write-ahead log), `marchat.db-shm` (shared memory)
 - **Message ID Tracking**: Sequential message IDs for user state management
 - **Encryption Support**: Binary storage for encrypted message data
 - **Performance Indexes**: Optimized queries for message retrieval and user state
 - **Message Cap**: Automatic cleanup maintaining 1000 most recent messages
 - **Ban History**: Comprehensive tracking of user moderation actions
-- **Performance Tuning**: Optimized SQLite settings for chat workloads
+- **Performance Tuning**: Backend-aware optimizations (SQLite pragmas when SQLite is selected)
 
 ## Administrative Interfaces
 
@@ -384,13 +400,13 @@ The web-based interface provides the same functionality through a browser:
 
 ### Database Optimization
 
-- **WAL Mode**: Write-Ahead Logging enabled for improved concurrency and performance
+- **WAL Mode (SQLite only)**: Write-Ahead Logging enabled for improved concurrency and performance
 - **Indexed Queries**: Performance indexes on frequently queried columns
 - **Batch Operations**: Efficient bulk message operations
 - **Connection Reuse**: Persistent database connections
 - **Query Optimization**: Prepared statements for common operations
-- **Performance Tuning**: Optimized SQLite pragmas for chat workloads
-- **Backup Considerations**: WAL mode creates additional files; backups may miss recent uncommitted data if taken while server is running
+- **Performance Tuning**: SQLite-specific pragmas are applied only on SQLite; Postgres/MySQL use driver/backend defaults
+- **Backup Considerations**: SQLite WAL mode creates additional files; backups may miss recent uncommitted data if taken while server is running
 
 ## Development Patterns
 

PROTOCOL.md

@@ -158,9 +158,10 @@ Messages initiated by the server to update client state.
 - On user connect/disconnect:
   - Broadcasts updated user list.
 - On message send:
-  - Persists eligible messages to SQLite.
+  - Persists eligible messages to the configured SQL backend selected by `MARCHAT_DB_PATH` (SQLite path, PostgreSQL DSN, or MySQL DSN).
   - Delivers to all connected clients **or** only to members of a channel when `channel` is non-empty and `sender` is not `System` (see [Channels](#channels)). Direct messages use a separate path (sender and recipient only).
-- Messages are saved to SQLite and capped at 1000 messages.
+- Reactions, read receipts, and last channel per user may be persisted server-side and replayed to reconnecting clients.
+- Message history is capped at 1000 messages.
 
 ---
 
@@ -186,7 +187,7 @@ Exceeded messages are dropped silently from the client’s perspective (the serv
 
 ## Message Retention
 
-- Messages are stored in SQLite.
+- Messages are stored in the backend configured by `MARCHAT_DB_PATH` (SQLite/PostgreSQL/MySQL).
 - The most recent 1000 messages are retained.
 - Older messages are deleted automatically.
 
@@ -206,7 +207,7 @@ If `admin` is not requested, `admin_key` is not required.
 
 ## Configuration
 
-Client settings are usually stored in **`config.json`** under the **client configuration directory** (per-user application data, or `MARCHAT_CONFIG_DIR`). That directory is separate from the **server** directory (`.env`, SQLite DB). Use **`marchat-client -doctor`** / **`marchat-server -doctor`** to print resolved paths.
+Client settings are usually stored in **`config.json`** under the **client configuration directory** (per-user application data, or `MARCHAT_CONFIG_DIR`). That directory is separate from the **server** directory (`.env` plus local DB file when SQLite is used). Use **`marchat-client -doctor`** / **`marchat-server -doctor`** to print resolved paths.
 
 Example `config.json` shape:
 

README.md

@@ -54,7 +54,7 @@ Full changelog on [GitHub releases](https://github.com/Cod-e-Codes/marchat/relea
 ## Features
 
 - **Terminal UI** - Beautiful TUI built with Bubble Tea
-- **Real-time Chat** - Fast WebSocket messaging with SQLite backend (PostgreSQL/MySQL planned)
+- **Real-time Chat** - Fast WebSocket messaging with SQLite, PostgreSQL, or MySQL backends
 - **Message Management** - Edit, delete, pin, react to, and search messages
 - **Direct Messages** - Private DM conversations between users
 - **Channels** - Multiple chat rooms with join/leave and per-channel messaging
@@ -75,7 +75,7 @@ Full changelog on [GitHub releases](https://github.com/Cod-e-Codes/marchat/relea
 
 ## Overview
 
-marchat started as a fun weekend project for father-son coding sessions and has evolved into a lightweight, self-hosted terminal chat application designed specifically for developers who love the command line. Currently runs with SQLite, with PostgreSQL and MySQL support planned for greater scalability.
+marchat started as a fun weekend project for father-son coding sessions and has evolved into a lightweight, self-hosted terminal chat application designed specifically for developers who love the command line. It supports SQLite by default and can also run against PostgreSQL or MySQL for larger deployments.
 
 **Key Benefits:**
 - **Self-hosted**: No external services required
@@ -205,7 +205,7 @@ go build -o marchat-client ./client
 | `MARCHAT_ADMIN_KEY` | Yes | - | Admin authentication key |
 | `MARCHAT_USERS` | Yes | - | Comma-separated admin usernames |
 | `MARCHAT_PORT` | No | `8080` | Server port |
-| `MARCHAT_DB_PATH` | No | `./config/marchat.db` | Database file path |
+| `MARCHAT_DB_PATH` | No | `./config/marchat.db` | Database path/DSN. Supports SQLite file path, `postgres://...`, or `mysql:...` |
 | `MARCHAT_TLS_CERT_FILE` | No | - | TLS certificate (enables wss://) |
 | `MARCHAT_TLS_KEY_FILE` | No | - | TLS private key |
 | `MARCHAT_GLOBAL_E2E_KEY` | No | - | Base64 32-byte global encryption key |
@@ -215,6 +215,28 @@ go build -o marchat-client ./client
 
 **Additional variables:** `MARCHAT_LOG_LEVEL`, `MARCHAT_CONFIG_DIR`, `MARCHAT_BAN_HISTORY_GAPS`, `MARCHAT_PLUGIN_REGISTRY_URL`
 
+### Database backend setup
+
+`MARCHAT_DB_PATH` accepts either a SQLite path or a DSN-style backend URL/prefix:
+
+```bash
+# SQLite (default)
+export MARCHAT_DB_PATH=./config/marchat.db
+
+# PostgreSQL
+export MARCHAT_DB_PATH='postgres://marchat:marchat@127.0.0.1:5432/marchat?sslmode=disable'
+
+# MySQL
+export MARCHAT_DB_PATH='mysql:marchat:marchat@tcp(127.0.0.1:3306)/marchat?parseTime=true'
+```
+
+Notes:
+- PostgreSQL requires a reachable database and credentials with schema/table create permissions.
+- MySQL DSNs should include `parseTime=true` so timestamp fields decode correctly.
+- **MariaDB** generally works with the same `mysql:` DSN shape and `parseTime=true` as MySQL.
+- The server creates **dialect-specific DDL** (for example, MySQL/MariaDB use fixed-width strings where indexes, primary keys, or unique constraints apply, because full `TEXT` keys are rejected). Long message bodies still use a large text type.
+- SQLite remains the easiest local development option.
+
 **Doctor / diagnostics:** Set `MARCHAT_DOCTOR_NO_NETWORK` to `1` to skip the GitHub latest-release check in `-doctor` / `-doctor-json`.
 
 **File Size Configuration:** Use either `MARCHAT_MAX_FILE_BYTES` (exact bytes) or `MARCHAT_MAX_FILE_MB` (megabytes). If both are set, `MARCHAT_MAX_FILE_BYTES` takes priority.
@@ -248,7 +270,7 @@ The repository’s `config/` directory holds **server** runtime files and the **
 
 ### Diagnostics (`-doctor`)
 
-Run **`./marchat-client -doctor`** or **`./marchat-server -doctor`** for a text report (paths, masked `MARCHAT_*` env, sanity checks). Use **`-doctor-json`** for machine-readable output. If both flags were passed, `-doctor-json` wins. Exits without starting the TUI or listening on a port. See [ARCHITECTURE.md](ARCHITECTURE.md) for details.
+Run **`./marchat-client -doctor`** or **`./marchat-server -doctor`** for a text report (paths, masked `MARCHAT_*` env, sanity checks). Server doctor also reports the detected DB dialect, validates the configured DB connection string format, and attempts a DB ping. Use **`-doctor-json`** for machine-readable output. If both flags were passed, `-doctor-json` wins. Exits without starting the TUI or listening on a port. See [ARCHITECTURE.md](ARCHITECTURE.md) for details.
 
 ## Admin Commands
 
@@ -681,6 +703,9 @@ Profiles stored in platform-appropriate locations:
 | Clipboard issues (Linux) | Install xclip: `sudo apt install xclip` |
 | Port in use | Change port: `export MARCHAT_PORT=8081` |
 | Database migration fails | Check file permissions, backup before source build |
+| PostgreSQL connection fails | Verify URL format: `postgres://user:pass@host:5432/db?sslmode=disable`; test with `psql` using same creds |
+| MySQL connection fails | Verify DSN prefix `mysql:` and DSN body `user:pass@tcp(host:3306)/db?parseTime=true`; test with `mysql` CLI |
+| SQL syntax error after backend switch | Ensure tables were created by the current server version and restart after changing `MARCHAT_DB_PATH` |
 | Message history missing | Expected after updates - user states reset for ban/unban improvements |
 | Ban history gaps not working | Ensure `MARCHAT_BAN_HISTORY_GAPS=true` (disabled by default) and `ban_history` table exists |
 | TLS certificate errors | Use `--skip-tls-verify` for dev with self-signed certs |

ROADMAP.md

@@ -1,55 +1,54 @@
 ## Roadmap
 
+This file tracks implementation status. Completed items use `- [x]`; future ideas are plain bullets without a checkbox.
+
 ### Recently Completed
-- CLI diagnostics (`-doctor`, `-doctor-json`) on client and server binaries for env/config/update checks
-- Message editing, deletion, and pinning
-- Message reactions (client-session scoped)
-- Direct messages between users
-- Chat channels (join/leave with per-channel messaging)
-- Typing indicators and read receipts
-- Message search (server-side)
-- E2E encryption for file transfers
-- Connection status indicator, tab completion, unread count
-- Multi-line input (Alt+Enter / Ctrl+J)
-- Chat history export
-- WebSocket rate limiting
-- Docker Compose for local development
+- [x] CLI diagnostics (`-doctor`, `-doctor-json`) on client and server binaries for env/config/update checks
+- [x] Message editing, deletion, and pinning
+- [x] Message reactions
+- [x] Direct messages between users
+- [x] Chat channels (join/leave with per-channel messaging)
+- [x] Typing indicators and read receipts
+- [x] Message search (server-side)
+- [x] E2E encryption for file transfers
+- [x] Connection status indicator, @mention tab completion, unread count
+- [x] Multi-line input (Alt+Enter / Ctrl+J)
+- [x] Chat history export
+- [x] WebSocket rate limiting
+- [x] Docker Compose for local development
 
 ### Phase 1: DB Abstraction Layer
-- Refactor database connection and initialization logic into a unified function.
-- Dynamically select DB driver and connection string at runtime.
-- Add support for PostgreSQL and MySQL in addition to SQLite.
+- [x] Refactor database connection and initialization logic into a unified function.
+- [x] Dynamically select DB driver and connection string at runtime.
+- [x] Add support for PostgreSQL and MySQL in addition to SQLite.
 
 ### Phase 2: Multi-Backend Compatibility
-- Ensure schema and queries work with:
-  - SQLite (existing)
-  - PostgreSQL
-  - MySQL
-- Adjust types where necessary (BOOLEAN, TIMESTAMP/DATETIME).
-- Consider a unified schema that works across all backends.
+- [x] Ensure schema and queries work with SQLite, PostgreSQL, and MySQL.
+- [x] Adjust types where necessary (BOOLEAN, TIMESTAMP/DATETIME).
+- [x] Maintain a unified schema that works across all backends.
 
 ### Phase 3: Schema & Query Adaptation
-- Add conditional logic for DB-specific schema tweaks.
-- Validate CREATE TABLE statements in all target backends.
-- Test queries for compatibility and performance.
+- [x] Add conditional logic for DB-specific schema tweaks.
+- [x] Validate CREATE TABLE statements in all target backends.
+- [x] Test queries for compatibility and performance.
 
 ### Phase 4: Performance Enhancements
-- ✅ Enable SQLite Write-Ahead Logging (WAL) mode for performance gains.
-- Implement batch TTL-based message deletion.
-- Add indexing for frequently queried columns.
-- Cache displayed messages in the terminal to reduce DB queries.
+- [x] Enable SQLite Write-Ahead Logging (WAL) mode for performance gains.
+- [x] Implement batch TTL-based message deletion.
+- [x] Add indexing for frequently queried columns.
+- [x] Cache displayed/recent messages in server memory to reduce repeated DB reads.
 
 ### Phase 5: Persistence & Durability
-- Persist reactions to the database (currently session-only).
-- Persist channel membership across reconnects.
-- Add read receipt state tracking per user.
+- [x] Persist reactions to the database.
+- [x] Persist last channel per user across reconnects (`user_channels`).
+- [x] Add read receipt state tracking per user.
 
 ### Phase 6: Testing & Documentation
-- Write unit/integration tests for all supported backends.
-- Document setup steps for PostgreSQL and MySQL.
-- Provide working connection string examples.
-- Include troubleshooting tips for common DB connection issues.
-- Increase test coverage for client and server packages.
+- [x] Dialect tests (DSN detection, placeholder rebinding) and SQLite-backed integration tests; exercise PostgreSQL/MySQL against live databases separately.
+- [x] Document setup steps for PostgreSQL and MySQL.
+- [x] Provide working connection string examples.
+- [x] Include troubleshooting tips for common DB connection issues.
+- [x] Increase test coverage for client and server packages.
 
 ### Phase 7: Future Improvements
 - Consider using `sqlx` or a lightweight ORM to reduce SQL dialect handling.