Add bottomless restore interruption integration tests

Add integration tests for libsql-server bottomless replication restore
behavior when interrupted by various failure modes.

Tests verify sqld can resume and complete an interrupted restore from
S3-compatible object storage (minio) without requiring a restart.

Test cases:
- basic_restore: Sanity check that sqld restores from minio
- sqld_interrupted: sqld killed mid-restore, restarted, completes
- minio_interrupted: minio stopped mid-restore, restarted, sqld retries
- network_partition: sqld disconnected from network mid-restore, reconnected

Infrastructure:
- Docker-based fixtures with isolated networks per test
- Unique container/network names and ports via atomic counters
- Port mapping (not host networking) for isolation
- Automatic cleanup of Docker resources after each test

Files added:
- tests/bottomless/mod.rs
- tests/bottomless/fixtures.rs
- tests/bottomless/basic_restore.rs
- tests/bottomless/sqld_interrupted.rs
- tests/bottomless/minio_interrupted.rs
- tests/bottomless/network_partition.rs
- tests/bottomless/README.md

Files modified:
- tests/tests.rs: Add bottomless module
- Cargo.toml: Add reqwest dev-dependency, remove duplicate hex

Author: aredridel

Cargo.lock

@@ -99,6 +99,7 @@ indicatif = "0.17.8"
 [dev-dependencies]
 arbitrary = { version = "1.3.0", features = ["derive_arbitrary"] }
 env_logger = "0.10"
+hex = "0.4"
 hyper = { workspace = true, features = ["client"] }
 insta = { version = "1.26.0", features = ["json"] }
 libsql = { path = "../libsql/"}
@@ -114,6 +115,7 @@ s3s-fs = "0.8.1"
 ring = { version = "0.17.8", features = ["std"] }
 tonic-build = "0.11"
 prost-build = "0.12"
+reqwest = { version = "0.11", features = ["json"] }
 
 [build-dependencies]
 vergen = { version = "8", features = ["build", "git", "gitcl"] }

libsql-server/Cargo.toml

@@ -0,0 +1,47 @@
+# Bottomless Restore Interruption Tests
+
+Integration tests for libsql-server's (sqld) bottomless S3 backup/restore feature. These tests verify that sqld correctly handles interrupted restores from object storage (minio).
+
+## Test Cases
+
+### 1. `basic_restore`
+Sanity check - verifies sqld can restore a database from minio after local files are deleted.
+
+### 2. `sqld_interrupted`
+Simulates sqld crashing mid-restore (SIGKILL). After restart, sqld must detect the incomplete restore and complete it successfully.
+
+### 3. `minio_interrupted`
+Minio (S3) is stopped mid-restore, then restarted. sqld must retry and complete the restore once S3 is available again.
+
+### 4. `network_partition`
+The sqld container is disconnected from the Docker network mid-restore (simulating a network partition), then reconnected. sqld must resume and complete the restore without requiring a restart.
+
+## Running Tests
+
+```bash
+# Run all bottomless tests
+cargo test --test tests bottomless -- --nocapture
+
+# Run individual tests
+cargo test --test tests bottomless::basic_restore -- --nocapture
+cargo test --test tests bottomless::sqld_interrupted -- --nocapture
+cargo test --test tests bottomless::minio_interrupted -- --nocapture
+cargo test --test tests bottomless::network_partition -- --nocapture
+```
+
+## Architecture
+
+- **MinIO**: Runs in a Docker container with a dedicated Docker network per test
+- **sqld**: Runs in a Docker container on the same network, with port mapping for HTTP access
+- Each test gets unique container/network names and ports to avoid conflicts when running in parallel
+- Data directory is a temp directory mounted into the sqld container
+
+## Prerequisites
+
+- Docker daemon running
+- Port range 20000-30000 available on localhost
+
+## Notes
+
+- Tests never skip or ignore. If the Docker environment is unavailable, they fail loudly.
+- All temporary containers and networks are cleaned up after each test.

libsql-server/tests/bottomless/README.md

@@ -0,0 +1,61 @@
+use super::fixtures::{MinioFixture, SqldFixture, TestDatabase};
+use std::time::Duration;
+
+#[tokio::test]
+async fn test_basic_restore() {
+    let _ = tracing_subscriber::fmt::try_init();
+
+    let minio = MinioFixture::start()
+        .await
+        .expect("Failed to start minio");
+
+    let data_dir = tempfile::tempdir().expect("Failed to create temp dir");
+    let mut sqld = SqldFixture::new(&minio);
+
+    // Phase 1: Create database and replicate to minio
+    sqld.start(data_dir.path())
+        .await
+        .expect("Failed to start sqld");
+    sqld.wait_for_ready(Duration::from_secs(30))
+        .await
+        .expect("sqld did not become ready");
+
+    let endpoint = sqld.http_endpoint();
+    let db = TestDatabase::new(endpoint.clone());
+    db.create_schema().await.expect("Failed to create schema");
+    db.insert_test_data(100)
+        .await
+        .expect("Failed to insert data");
+    db.wait_for_replication()
+        .await
+        .expect("Failed to wait for replication");
+
+    sqld.stop().await.expect("Failed to stop sqld");
+
+    // Phase 2: Delete local database files to force restore
+    sqld.cleanup_data_dir(data_dir.path())
+        .await
+        .expect("Failed to cleanup dbs dir");
+
+    // Phase 3: Start sqld - should restore from minio
+    let endpoint2 = sqld.http_endpoint();
+    sqld.start(data_dir.path())
+        .await
+        .expect("Failed to restart sqld");
+    sqld.wait_for_ready(Duration::from_secs(60))
+        .await
+        .expect("sqld did not become ready after restore");
+
+    // Phase 4: Verify database is intact
+    let db2 = TestDatabase::new(endpoint2);
+    
+    tokio::time::sleep(Duration::from_secs(2)).await;
+    
+    let restored_data = db2.query_all().await.expect("Failed to query data");
+    assert_eq!(restored_data.len(), 100, "Expected 100 rows after restore");
+
+    db2.verify_integrity().await.expect("Data integrity check failed");
+
+    sqld.cleanup().await.ok();
+    minio.cleanup().await.ok();
+}