Merge #312: docs: [#310] research database backup strategies

1168780 docs: [#310] add lessons learned from backup POC implementation (Jose Celano)
74a082d docs: [#310] update issue spec progress - all tasks complete (Jose Celano)
c92c199 feat: [#310] add SQLite backup support to backup container (Jose Celano)
c7cc2c5 docs: [#310] recommend maintenance-window as primary backup solution (Jose Celano)
2d97bf1 docs: [#310] add unit tests and improve documentation clarity (Jose Celano)
1c63945 docs: [#310] add maintenance-window solution artifacts and update default backup interval (Jose Celano)
b9fd407 docs: [#310] add Torrust Demo database analysis for exclude-statistics (Jose Celano)
717b4b8 docs: [#310] add alternative backup solutions for large databases (Jose Celano)
6cff3b6 docs: [#310] reorganize backup-strategies folder structure (Jose Celano)
7b3fb8a docs: [#310] complete Phase 7 documentation and update conclusions (Jose Celano)
7ea89cf docs: [#310] add SQLite large database backup findings (Jose Celano)
d040367 docs: [#310] Phase 6 - Restore validation complete (Jose Celano)
e370b48 test: [#310] add unit tests for backup script with bats-core (Jose Celano)
e9b506c refactor: [#310] add Rust-style documentation to backup script (Jose Celano)
bf7e719 docs: [#310] implement Phase 5 - backup maintenance (packaging & retention) (Jose Celano)
4eb29fe docs: [#310] update Phase 5 plan with maintenance approach (Jose Celano)
7076964 docs: [#310] mark Phase 4 as complete in progress table (Jose Celano)
672fafc docs: [#310] reference issue #313 for Ansible ownership fix (Jose Celano)
cb36c17 docs: [#310] add production considerations and fix non-root container (Jose Celano)
745b102 docs: [#310] update Phase 4 documentation with implementation details (Jose Celano)
fd8c19a docs: [#310] implement Phase 4 - config files backup (Jose Celano)
550d382 docs: [#310] add backup integrity verification tests to Phase 3 (Jose Celano)
bfbedd9 docs: [#310] reorganize artifacts with backup-container folder (Jose Celano)
270ef7c docs: [#310] complete Phase 3 - MySQL backup with mariadb-dump (Jose Celano)
7b661aa docs: [#310] complete Phase 2 - minimal backup container (Jose Celano)
37fca8a docs: [#310] reorganize PoC into structured folder with phase docs (Jose Celano)
1406df4 docs: [#310] complete Phase 1 of sidecar backup PoC (Jose Celano)
8b8aa51 docs: [#310] add PoC plan and answer backup requirements questions (Jose Celano)
2fcf835 docs: [#310] add MySQL backup research and sidecar container solution (Jose Celano)
17b15a8 docs: [#310] add preliminary conclusions for backup research (Jose Celano)
848dbde docs: [#310] research database backup strategies (Jose Celano)

Pull request description:

  ## Summary

  Comprehensive research documentation for database backup strategies as part of Epic #309 (Add backup support).

  This PR includes **complete research** for SQLite and MySQL backup strategies, backup tools evaluation, container backup architectures, a **working proof-of-concept backup container** with **58 bats-core unit tests**, and a **recommended solution** (Maintenance Window Hybrid approach).

  ## What's Included

  ### Database Backup Strategies

  #### SQLite
  - **Backup approaches**: `.backup` command (Online Backup API), `VACUUM INTO`, file copy risks
  - **WAL mode analysis**: Checkpointing behavior, persistence, pros/cons
  - **Backup verification and restore procedures**: Integrity checks, recovery steps
  - **Torrust Live Demo analysis**: Current implementation (unsafe `cp`), proposed improvements
  - **⚠️ Critical Large Database Finding**: SQLite `.backup` stalls at 10% after 16+ hours for 17GB database (~37 MB/hour effective rate). Maintenance window backup completes in 72 seconds.

  #### MySQL
  - **Backup approaches**: `mysqldump`, physical backups, binary log backups
  - **Container-specific considerations**: Accessing MySQL in Docker containers
  - **Backup verification and restore procedures**

  ### Container Backup Architectures
  - 5 patterns documented: Host Crontab, Centralized, Sidecar, Orchestrator, External Tool
  - Comparison matrix with pros/cons
  - Decision flowchart for pattern selection

  ### Backup Tools Evaluation
  - ✅ **Restic**: Recommended - mature, encrypted, deduplicated, Docker support
  - ⚠️ **Kopia**: Alternative - newer, more features (GUI, ECC, server mode), less mature
  - ❌ **Rustic**: Discarded - beta status, not production-ready
  - Two-phase backup approach documented (DB dump → file backup)

  ### Solution Comparison (NEW)

  Four backup solutions evaluated with detailed trade-off analysis:

  | Solution | Best For | Complexity |
  |----------|----------|------------|
  | Continuous Sidecar | Hot backups, simple setup | Low |
  | **Maintenance Window** | Large DBs, complete consistency | Medium |
  | External Scheduler | Multi-service environments | High |
  | Native Database | WAL-enabled SQLite | Low |

  **Recommended Solution**: Maintenance Window Hybrid (95% container, 5% host script)

  ### Maintenance Window Backup POC (Complete - NEW)

  A working proof-of-concept with **58 bats-core unit tests** supporting both MySQL and SQLite:

  | Feature | Status |
  |---------|--------|
  | MySQL backup with mysqldump | ✅ Complete |
  | SQLite backup with sqlite3 | ✅ Complete |
  | Config file backup | ✅ Complete |
  | Retention policy (delete old backups) | ✅ Complete |
  | Single mode (run once, exit) | ✅ Complete |
  | Continuous mode (loop) | ✅ Complete |
  | Host orchestration script | ✅ Complete |
  | Crontab configuration | ✅ Complete |
  | 58 unit tests | ✅ All passing |

  **POC Artifacts**:
  - Multi-stage Dockerfile with MySQL and SQLite support
  - `backup.sh` script with modular functions
  - `maintenance-backup.sh` host orchestration script
  - Docker Compose examples for MySQL and SQLite
  - Production and test crontab configurations
  - Lessons learned document with implementation concerns

  ## Key Findings

  | Finding | Details |
  |---------|---------|
  | SQLite Safe Backup | Use `.backup` command (Online Backup API) - safe during concurrent writes |
  | SQLite Large DB Limitation | `.backup` impractical for DBs > 1GB due to locking overhead (~37 MB/hour) |
  | Maintenance Window Backup | 72 seconds for 17GB SQLite (vs ~17 days with `.backup`) |
  | Disk I/O Capacity | 445 MB/s proven - SQLite locking is bottleneck, not disk |
  | MySQL Backup | `mysqldump` works reliably for containerized deployments |
  | WAL Mode | Optional for safe backups, useful for read performance under high load |
  | Recommended Tool | Restic - battle-tested, simple, Docker-native, sufficient features |
  | Recommended Solution | Maintenance Window Hybrid - container + host crontab |
  | Sidecar Pattern | Best for single-server deployments with few services |

  ## Lessons Learned (Implementation Concerns)

  Key pain points discovered during POC that affect future implementation:

  | Pain Point | Severity | Notes |
  |------------|----------|-------|
  | Template conditionals for DB type | Medium | Docker Compose env vars differ for MySQL vs SQLite |
  | Path translation (host/container) | Medium | Multiple representations of same path |
  | SSH agent key selection | Low | Use `IdentitiesOnly=yes` |
  | Container exits in single mode | Low | Expected behavior, just surprising |
  | Log rotation missing | Low | Easy to add, often forgotten |
  | Backup verification missing | Medium | Important for production |

  ## Related Issues

  - Closes #310 (Research Database Backup Strategies)
  - Part of Epic #309
  - Created on torrust-demo repo:
    - Issue #85: Use `.backup` instead of `cp`
    - Issue #86: Evaluate WAL mode for high-traffic scenario

  ## Checklist

  ### Research Complete
  - [x] SQLite backup approaches documented
  - [x] SQLite large database findings (17GB test)
  - [x] MySQL backup approaches documented
  - [x] WAL mode analysis with checkpointing behavior
  - [x] Backup verification and restore procedures
  - [x] Torrust Live Demo analysis
  - [x] Container backup architectures (5 patterns)
  - [x] Backup tools evaluation (Restic, Kopia, Rustic)
  - [x] Solution comparison (4 approaches)
  - [x] Recommended solution documented

  ### POC Complete
  - [x] Multi-stage Dockerfile with MySQL and SQLite support
  - [x] 58 bats-core unit tests (all passing)
  - [x] MySQL backup/restore validated
  - [x] SQLite backup/restore validated
  - [x] Config file backup
  - [x] Retention policy (delete expired backups)
  - [x] Single mode (run once, exit)
  - [x] Continuous mode (loop with interval)
  - [x] Host orchestration script
  - [x] Crontab configurations (production + test)
  - [x] Docker Compose examples (MySQL + SQLite)
  - [x] Lessons learned document
  - [x] Issue spec progress updated (all tasks complete)

  ### Future Work (out of scope for this PR)
  - [ ] Implement backup command in deployer
  - [ ] Off-site transfer automation (S3, Backblaze B2)
  - [ ] Backup encryption
  - [ ] Backup verification command

  ## Documentation Structure

  ```
  docs/research/backup-strategies/
  ├── README.md                           # Overview and navigation
  ├── conclusions.md                      # Key findings and recommendations
  ├── requirements.md                     # Design preferences
  ├── architectures/
  │   └── container-patterns.md           # 5 architecture patterns
  ├── databases/
  │   ├── mysql/
  │   │   ├── README.md
  │   │   └── backup-approaches.md
  │   └── sqlite/
  │       ├── README.md
  │       ├── backup-approaches.md
  │       ├── large-database-backup.md    # Critical 17GB findings
  │       └── torrust-live-demo/
  │           ├── README.md
  │           ├── current-implementation.md
  │           └── proposed-improvements.md
  ├── tools/
  │   ├── README.md                       # Tools overview
  │   ├── restic.md                       # Detailed Restic evaluation
  │   └── restic-vs-kopia.md              # Comparison document
  └── solutions/
      ├── README.md                       # Solution comparison (NEW)
      ├── sidecar-container/              # Original sidecar POC
      └── maintenance-window/             # Recommended solution (NEW)
          ├── README.md                   # Architecture and workflow
          ├── implementation-recommendations.md  # Lessons learned
          └── artifacts/
              ├── backup-container/
              │   ├── Dockerfile
              │   ├── backup.sh
              │   └── backup_test.bats    # 58 tests
              ├── docker-compose-with-backup-mysql.yml
              ├── docker-compose-with-backup-sqlite.yml
              ├── maintenance-backup.sh
              ├── maintenance-backup.cron
              └── maintenance-backup-test.cron
  ```

ACKs for top commit:
  josecelano:
    ACK 1168780

Tree-SHA512: 90338487494b44eefe56fc6943497a2caa2715e9e459a24d36f287d5ff50938d48ce33c9d2dfb3eef08929dfb79ec80fbf73b228c570ee16b0e26d2034939c98

Author: josecelano

docs/issues/310-research-database-backup-strategies.md

@@ -12,33 +12,33 @@ Research and document backup strategies for SQLite databases, MySQL databases, a
 
 ### SQLite Backup
 
-- [ ] Learn how to backup SQLite files safely while used in production (no locks, safe copying)
-- [ ] Research tools and techniques for copying and compressing SQLite backups
-- [ ] Investigate redundancy strategies for SQLite backups (cloud volumes, S3, backup services, snapshots)
-- [ ] Document current Torrust Live Demo SQLite backup implementation
+- [x] Learn how to backup SQLite files safely while used in production (no locks, safe copying)
+- [x] Research tools and techniques for copying and compressing SQLite backups
+- [x] Investigate redundancy strategies for SQLite backups (cloud volumes, S3, backup services, snapshots)
+- [x] Document current Torrust Live Demo SQLite backup implementation
 
 ### MySQL Backup
 
-- [ ] Research MySQL backup approaches for containerized deployments
-- [ ] Learn about MySQL-specific backup tools (mysqldump, hot backup, volume snapshots)
-- [ ] Investigate compression and redundancy strategies for MySQL backups
+- [x] Research MySQL backup approaches for containerized deployments
+- [x] Learn about MySQL-specific backup tools (mysqldump, hot backup, volume snapshots)
+- [x] Investigate compression and redundancy strategies for MySQL backups
 
 ### Complete Storage Folder Backup
 
-- [ ] Research approaches for backing up the entire deployment storage folder
-- [ ] Learn about tools for full directory backups (tar, rsync, volume snapshots)
-- [ ] Understand trade-offs between full storage backup and selective approaches
+- [x] Research approaches for backing up the entire deployment storage folder
+- [x] Learn about tools for full directory backups (tar, rsync, volume snapshots)
+- [x] Understand trade-offs between full storage backup and selective approaches
 
 ### Selective Files Backup
 
-- [ ] Identify which configuration files and directories need backup
-- [ ] Research strategies for backing up specific files (docker-compose, tracker config, etc.)
-- [ ] Learn about version control and organization for selective backups
+- [x] Identify which configuration files and directories need backup
+- [x] Research strategies for backing up specific files (docker-compose, tracker config, etc.)
+- [x] Learn about version control and organization for selective backups
 
 ### General Research
 
-- [ ] Explore different backup scope strategies and their trade-offs
-- [ ] Document all findings in `docs/research/backup-strategies/` (to be created during research)
+- [x] Explore different backup scope strategies and their trade-offs
+- [x] Document all findings in `docs/research/backup-strategies/` (to be created during research)
 
 ## 🏗️ Architecture Requirements
 
@@ -221,87 +221,87 @@ Research different approaches to defining backup scope:
 
 ### Phase 1: SQLite Research (estimated 4-6 hours)
 
-- [ ] Read SQLite backup documentation
-- [ ] Research safe file copy approaches while database is in use
-- [ ] Investigate SQLite locking mechanisms and WAL mode
-- [ ] Research compression tools and techniques
-- [ ] Learn about cloud volume attachment and snapshot strategies
-- [ ] Study S3 and backup service integration options
-- [ ] Analyze Torrust Live Demo backup script implementation
-- [ ] Document all findings in `docs/research/backup-strategies/sqlite-backup-strategies.md` (create folder and file)
+- [x] Read SQLite backup documentation
+- [x] Research safe file copy approaches while database is in use
+- [x] Investigate SQLite locking mechanisms and WAL mode
+- [x] Research compression tools and techniques
+- [x] Learn about cloud volume attachment and snapshot strategies
+- [x] Study S3 and backup service integration options
+- [x] Analyze Torrust Live Demo backup script implementation
+- [x] Document all findings in `docs/research/backup-strategies/sqlite-backup-strategies.md` (create folder and file)
 
 ### Phase 2: MySQL Research (estimated 4-6 hours)
 
-- [ ] Read MySQL backup documentation
-- [ ] Research `mysqldump` usage and locking behavior
-- [ ] Investigate physical backup tools (Percona XtraBackup)
-- [ ] Learn about Docker volume backup strategies
-- [ ] Research compression techniques for MySQL dumps
-- [ ] Study cloud redundancy options for MySQL backups
-- [ ] Test basic mysqldump in Docker container (optional hands-on)
-- [ ] Document all findings in `docs/research/backup-strategies/mysql-backup-strategies.md`
+- [x] Read MySQL backup documentation
+- [x] Research `mysqldump` usage and locking behavior
+- [x] Investigate physical backup tools (Percona XtraBackup)
+- [x] Learn about Docker volume backup strategies
+- [x] Research compression techniques for MySQL dumps
+- [x] Study cloud redundancy options for MySQL backups
+- [x] Test basic mysqldump in Docker container (optional hands-on)
+- [x] Document all findings in `docs/research/backup-strategies/mysql-backup-strategies.md`
 
 ### Phase 3: Configuration Research (estimated 2-3 hours)
 
-- [ ] Identify all configuration files and directories
-- [ ] Research file copy and archive tools (`tar`, `rsync`)
-- [ ] Learn about compression options and trade-offs
-- [ ] Study configuration storage strategies
-- [ ] Research version control for config backups
-- [ ] Document all findings in `docs/research/backup-strategies/configuration-backup-strategies.md`
+- [x] Identify all configuration files and directories
+- [x] Research file copy and archive tools (`tar`, `rsync`)
+- [x] Learn about compression options and trade-offs
+- [x] Study configuration storage strategies
+- [x] Research version control for config backups
+- [x] Document all findings in `docs/research/backup-strategies/configuration-backup-strategies.md`
 
 ### Phase 4: Backup Scope Strategies (estimated 2-3 hours)
 
-- [ ] Research full storage backup approaches
-- [ ] Compare database-only backup patterns
-- [ ] Study selective backup strategies
-- [ ] Learn about layered backup approaches
-- [ ] Document trade-offs for each strategy
-- [ ] Document all findings in `docs/research/backup-strategies/backup-scope-strategies.md`
+- [x] Research full storage backup approaches
+- [x] Compare database-only backup patterns
+- [x] Study selective backup strategies
+- [x] Learn about layered backup approaches
+- [x] Document trade-offs for each strategy
+- [x] Document all findings in `docs/research/backup-strategies/backup-scope-strategies.md`
 
 ### Phase 5: Documentation Review (estimated 1 hour)
 
-- [ ] Review all research documents for completeness
-- [ ] Create README in research folder with overview
-- [ ] Ensure all research questions are addressed
-- [ ] Cross-reference with Torrust Live Demo implementation
-- [ ] Run linters and ensure documentation quality
-- [ ] Update issue with any follow-up questions or findings
+- [x] Review all research documents for completeness
+- [x] Create README in research folder with overview
+- [x] Ensure all research questions are addressed
+- [x] Cross-reference with Torrust Live Demo implementation
+- [x] Run linters and ensure documentation quality
+- [x] Update issue with any follow-up questions or findings
 
 ## Acceptance Criteria
 
 > **Note for Contributors**: These criteria define what the PR reviewer will check. Use this as your pre-review checklist before submitting the PR to minimize back-and-forth iterations.
 
 **Quality Checks**:
 
-- [ ] Pre-commit checks pass: `./scripts/pre-commit.sh`
+- [x] Pre-commit checks pass: `./scripts/pre-commit.sh`
 
 **Research Documentation**:
 
-- [ ] SQLite backup approaches documented (safe copying, compression, redundancy)
-- [ ] MySQL backup approaches documented (tools, techniques, containerization)
-- [ ] Configuration backup approaches documented
-- [ ] Backup scope strategies compared
-- [ ] Torrust Live Demo implementation analyzed and documented
-- [ ] All research questions addressed with sufficient detail
-- [ ] Cloud redundancy strategies documented (volumes, S3, snapshots)
-- [ ] Compression techniques compared
+- [x] SQLite backup approaches documented (safe copying, compression, redundancy)
+- [x] MySQL backup approaches documented (tools, techniques, containerization)
+- [x] Configuration backup approaches documented
+- [x] Backup scope strategies compared
+- [x] Torrust Live Demo implementation analyzed and documented
+- [x] All research questions addressed with sufficient detail
+- [x] Cloud redundancy strategies documented (volumes, S3, snapshots)
+- [x] Compression techniques compared
 
 **Research Completeness**:
 
-- [ ] All research questions in specifications section answered
-- [ ] Tools and techniques identified for each backup type
-- [ ] Trade-offs documented for different approaches
-- [ ] References to official documentation included
-- [ ] Findings organized in `docs/research/backup-strategies/` folder
-- [ ] README created in research folder with overview
+- [x] All research questions in specifications section answered
+- [x] Tools and techniques identified for each backup type
+- [x] Trade-offs documented for different approaches
+- [x] References to official documentation included
+- [x] Findings organized in `docs/research/backup-strategies/` folder
+- [x] README created in research folder with overview
 
 **Documentation Quality**:
 
-- [ ] Markdown linting passes (markdownlint)
-- [ ] Spell checking passes (cspell)
-- [ ] All links valid and properly formatted
-- [ ] Code examples properly formatted with syntax highlighting (if any)
+- [x] Markdown linting passes (markdownlint)
+- [x] Spell checking passes (cspell)
+- [x] All links valid and properly formatted
+- [x] Code examples properly formatted with syntax highlighting (if any)
 
 ## Related Documentation
 

docs/research/backup-strategies/README.md

@@ -0,0 +1,97 @@
+# Backup Strategies Research
+
+**Issue**: [#310 - Research database backup strategies](https://github.com/torrust/torrust-tracker-deployer/issues/310)
+**Parent Epic**: [#309 - Add backup support](https://github.com/torrust/torrust-tracker-deployer/issues/309)
+
+## Overview
+
+This folder contains research documentation for backup strategies in the context of Torrust Tracker deployments. The research covers different backup types, database-specific approaches, and general requirements.
+
+> **📋 See [Conclusions](conclusions.md) for a summary of key findings and recommendations.**
+
+## Backup Types
+
+The research is organized around three backup types:
+
+| Type                 | Description                                     | Status      |
+| -------------------- | ----------------------------------------------- | ----------- |
+| **Storage Backup**   | Complete backup of entire storage (config + DB) | Not started |
+| **Database Backup**  | Database-specific backup using safe tools       | In progress |
+| **Selective Backup** | Partial storage backup (e.g., config only)      | Not started |
+
+See [requirements.md](requirements.md) for detailed explanation of each type.
+
+## Documents
+
+### General
+
+| Document                        | Description                                                      |
+| ------------------------------- | ---------------------------------------------------------------- |
+| [Requirements](requirements.md) | Collected requirements, constraints, and backup type definitions |
+| [Conclusions](conclusions.md)   | Summary of key findings and recommendations                      |
+
+### Database-Specific
+
+| Folder                                 | Description                                                          |
+| -------------------------------------- | -------------------------------------------------------------------- |
+| [databases/sqlite/](databases/sqlite/) | SQLite backup research (approaches, current implementation analysis) |
+| [databases/mysql/](databases/mysql/)   | MySQL backup research (mysqldump, hot backups, locking behavior)     |
+
+### Architectures
+
+| Document                                                                   | Description                                  |
+| -------------------------------------------------------------------------- | -------------------------------------------- |
+| [architectures/container-patterns.md](architectures/container-patterns.md) | Container-based backup architecture patterns |
+
+### Tools
+
+| Document         | Description                        |
+| ---------------- | ---------------------------------- |
+| [tools/](tools/) | Backup tool research (restic, etc) |
+
+### Solutions
+
+| Folder                   | Description                                          |
+| ------------------------ | ---------------------------------------------------- |
+| [solutions/](solutions/) | Proposed backup solutions and architectural patterns |
+
+**⭐ Recommended**: [Maintenance Window Pattern](solutions/maintenance-window/) -
+A hybrid approach combining container-based backup with host-level orchestration.
+The backup container runs once per day (triggered by crontab), not continuously.
+This works for databases of any size and preserves container portability.
+
+**Alternative**: [Sidecar Container Pattern](solutions/sidecar-container/) -
+A dedicated backup container that runs continuously. Only practical for small
+databases (< 1GB) due to SQLite locking issues under load.
+
+## Research Status
+
+### SQLite
+
+- [x] Document backup approaches
+- [x] Analyze Torrust Live Demo implementation
+- [x] Investigate journal mode
+- [ ] Test in containerized environment
+
+### MySQL
+
+- [x] Research mysqldump approaches
+- [x] Research hot backup tools (Percona XtraBackup)
+- [x] Document locking behavior (no lock needed for InnoDB)
+
+### General
+
+- [x] Document backup types
+- [x] Collect requirements from discussions
+- [ ] Research compression strategies
+- [ ] Research retention policies
+- [ ] Research restore procedures
+
+## Key Decisions Captured
+
+From research discussions:
+
+1. **No data loss acceptable** - Safety is priority over simplicity
+2. **User provides backup path** - Deployer is storage-location agnostic
+3. **Keep complexity low** - No cloud provider APIs, focus on local backups
+4. **Database-aware backups** - Simple `cp` not acceptable for databases