feat: enable WAL mode for SQLite backend to improve concurrency and performance (#116)

Author: Giovani Guizzo

.gitignore

@@ -315,7 +315,10 @@ $RECYCLE.BIN/
 # End of https://www.toptal.com/developers/gitignore/api/node,macos,osx,windows,linux,visualstudiocode,vim,emacs
 
 *.sqlite
+*.sqlite-wal
+*.sqlite-shm
+*.sqlite-journal
 
 .sidequest.config.json
 .turbo
-.vscode
+.vscode

packages/backends/sqlite/README.md

@@ -75,10 +75,38 @@ backend: {
 - **Zero Configuration** - No database server setup required, just specify a file path
 - **File-Based Storage** - Self-contained database in a single file for easy deployment
 - **ACID Transactions** - Full transaction support for data integrity and safe job claiming
+- **WAL Mode** - Write-Ahead Logging enabled by default for better concurrent access
 - **Lightweight** - Minimal resource footprint, perfect for development and small deployments
 - **Migration Support** - Automatic database schema management with Knex.js migrations
 - **Portable** - Database files can be easily backed up, moved, or shared
 
+## WAL Mode and Concurrency
+
+The SQLite backend automatically enables **WAL (Write-Ahead Logging) mode** for improved concurrent access. This provides several benefits:
+
+- **Better Concurrency**: Allows multiple readers and one writer simultaneously
+- **Reduced Lock Contention**: Minimizes `SQLITE_BUSY` errors during concurrent job processing
+- **Improved Performance**: Faster writes and better throughput for job queue operations
+- **Safer Transactions**: More reliable atomic operations for job claiming
+
+### WAL Mode Files
+
+When WAL mode is enabled, SQLite creates additional files alongside your main database:
+
+- `sidequest.sqlite` - Main database file
+- `sidequest.sqlite-wal` - Write-ahead log file
+- `sidequest.sqlite-shm` - Shared memory index file
+
+These files are managed automatically by SQLite and should not be manually edited or deleted.
+
+### Limitations
+
+While WAL mode significantly improves concurrency, SQLite is still not recommended for high-concurrency production deployments. For production use with multiple workers or distributed systems, consider using:
+
+- **PostgreSQL** (`@sidequest/postgres-backend`) - Best for production
+- **MySQL** (`@sidequest/mysql-backend`) - Good for production
+- **MongoDB** (`@sidequest/mongo-backend`) - Alternative for production
+
 ## License
 
 LGPL-3.0-or-later

packages/backends/sqlite/migrations/3_enable_wal_mode.cjs

@@ -0,0 +1,24 @@
+/**
+ * Migration to enable WAL (Write-Ahead Logging) mode for SQLite
+ * 
+ * WAL mode provides better concurrency for job processing:
+ * - Allows simultaneous readers and one writer
+ * - Reduces SQLITE_BUSY errors
+ * - Better performance for write-heavy workloads
+ * - More predictable behavior under concurrent load
+ * 
+ * Learn more: https://www.sqlite.org/wal.html
+ */
+
+exports.up = async function(knex) {
+  await knex.schema.raw('PRAGMA journal_mode = WAL;');
+  await knex.schema.raw('PRAGMA busy_timeout = 5000;');
+  await knex.schema.raw('PRAGMA cache_size = -20000;');
+  await knex.schema.raw('PRAGMA temp_store = memory;');
+};
+
+exports.down = function(knex) {
+  // Revert to default rollback journal mode
+  return knex.schema.raw('PRAGMA journal_mode = DELETE;')
+    .then(() => knex.schema.raw('PRAGMA synchronous = FULL;'));
+};