Skip to content

Commit d9cc352

Browse files
bugerclaude
andauthored
🛡️ Critical: Implement database protection system for testing (#1)
* 🛡️ Implement comprehensive database protection system for testing Critical fix to prevent development database from being wiped during testing. ## Problem - Tests with RefreshDatabase trait were wiping the development database - Running `php artisan test` without proper environment setup destroyed all data - This happened multiple times causing significant data loss ## Solution - Multi-layer Protection System ### 1. Safe Test Runner Script (scripts/safe-test.sh) - Forces DB_DATABASE=learning_app_test explicitly - Validates database name isn't 'learning_app' - Creates test database if it doesn't exist - Shows configuration before running tests - Exit immediately if wrong database detected ### 2. TestCase.php Runtime Protection - Added database validation in setUp() method - Throws exception if tests attempt to run on non-test database - Provides clear error message with recovery instructions ### 3. phpunit.xml Configuration Hardening - Added force="true" to DB_DATABASE environment variable - Ensures test database is always used regardless of .env - Added warning comments about RefreshDatabase dangers ### 4. Makefile Integration - Updated test-unit and test-unit-coverage commands - Both now use safe-test.sh instead of direct artisan test - Added clear warnings about database safety ### 5. Documentation Updates - Updated CLAUDE.md with safe testing instructions - Listed dangerous commands to avoid - Provided safe alternatives and examples - Created TESTING_SAFETY.md with comprehensive guide - Includes recovery procedures if database gets wiped ## Testing - Verified safe-test.sh prevents connection to development database - Confirmed TestCase.php throws exception on wrong database - Tested Makefile commands work with new safety wrapper ## Impact - Developers can now run tests without fear of data loss - Clear documentation prevents future incidents - Multiple failsafes ensure database protection - Recovery procedures documented if issues occur 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix CI: Allow GitHub Actions test database names The CI pipeline uses 'homeschoolai_test' and 'homeschoolai_e2e' as database names, but our protection was only allowing 'learning_app_test'. This commit updates the TestCase.php to allow all valid test database names while still protecting the development database from being wiped. * Add fail-fast configuration for CI e2e tests - Set max failures to 3 in CI (stop after 3 test failures) - Disable retries in CI for faster feedback - Add 10-minute timeout for e2e job - Reduce individual test timeouts in CI (30s test, 5s expect, 5s action) - Use BASE_URL environment variable properly - Add simpler 'line' reporter for CI output --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 146a5c5 commit d9cc352

8 files changed

Lines changed: 258 additions & 46 deletions

File tree

.github/workflows/ci.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,9 @@ jobs:
177177
- name: Run E2E tests
178178
env:
179179
BASE_URL: http://localhost:8000
180-
run: npx playwright test
180+
CI: true
181+
run: npx playwright test --max-failures=3
182+
timeout-minutes: 10
181183
continue-on-error: true
182184

183185
- name: Upload Playwright report

CLAUDE.md

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -48,24 +48,32 @@ supabase start
4848
supabase status # Should show PostgreSQL on port 54322
4949
```
5050

51-
#### **PHP Unit Tests (100% Working)**
51+
#### **🚨 CRITICAL: PHP Unit Tests - Database Protection Required**
5252

53-
⚠️ **WARNING**: PHP tests use `RefreshDatabase` trait which **WIPES THE DATABASE**!
54-
Always use `APP_ENV=testing` to protect your development data.
53+
**DANGER**: PHP tests use `RefreshDatabase` trait which **COMPLETELY WIPES THE DATABASE**!
54+
**NEVER run tests without proper database isolation or you will lose all development data!**
5555

5656
```bash
57-
# SAFE: Use the safe test runner (RECOMMENDED)
58-
./scripts/safe-test.sh # Automatically uses test database
57+
# ✅ ALWAYS USE THIS METHOD (SAFE):
58+
./scripts/safe-test.sh # Run all tests safely
5959
./scripts/safe-test.sh --filter TestName # Run specific test safely
60+
./scripts/safe-test.sh --coverage # Run with coverage
6061

61-
# OR manually specify environment (REQUIRED FORMAT)
62-
APP_ENV=testing DB_CONNECTION=pgsql SESSION_DRIVER=file CACHE_STORE=array php artisan test
62+
# Or use Makefile commands (also safe):
63+
make test-unit # Runs safe-test.sh internally
64+
make test-unit-coverage # Safe coverage testing
6365

64-
# Run specific test
65-
APP_ENV=testing DB_CONNECTION=pgsql SESSION_DRIVER=file CACHE_STORE=array php artisan test --filter TestName
66+
# ❌ NEVER RUN THESE (WILL WIPE YOUR DATABASE):
67+
php artisan test # DANGEROUS - might use wrong database
68+
artisan test # DANGEROUS
69+
vendor/bin/phpunit # DANGEROUS
6670
```
6771

68-
**Note**: Environment variables MUST be explicitly set to use the test database (`learning_app_test`) instead of your development database (`learning_app`).
72+
**Safety Features**:
73+
- `safe-test.sh` forces use of `learning_app_test` database
74+
- `TestCase.php` throws exception if wrong database detected
75+
- `phpunit.xml` has forced test database configuration
76+
- Development database (`learning_app`) is protected from accidental wipes
6977

7078
#### **E2E Tests (Infrastructure Working)**
7179
```bash

Makefile

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -95,14 +95,15 @@ test: ## Run all tests (PHPUnit + E2E)
9595
@$(MAKE) test-e2e
9696

9797
.PHONY: test-unit
98-
test-unit: ## Run PHPUnit tests
99-
@echo "$(GREEN)Running PHPUnit tests...$(NC)"
100-
APP_ENV=testing DB_CONNECTION=pgsql SESSION_DRIVER=file CACHE_STORE=array $(ARTISAN) test
98+
test-unit: ## Run PHPUnit tests (SAFE - uses test database)
99+
@echo "$(GREEN)Running PHPUnit tests on TEST database...$(NC)"
100+
@echo "$(YELLOW)⚠️ Using safe test runner to protect development database$(NC)"
101+
@./scripts/safe-test.sh
101102

102103
.PHONY: test-unit-coverage
103-
test-unit-coverage: ## Run PHPUnit tests with coverage
104-
@echo "$(GREEN)Running PHPUnit tests with coverage...$(NC)"
105-
APP_ENV=testing DB_CONNECTION=pgsql SESSION_DRIVER=file CACHE_STORE=array $(ARTISAN) test --coverage
104+
test-unit-coverage: ## Run PHPUnit tests with coverage (SAFE - uses test database)
105+
@echo "$(GREEN)Running PHPUnit tests with coverage on TEST database...$(NC)"
106+
@./scripts/safe-test.sh --coverage
106107

107108
.PHONY: test-e2e
108109
test-e2e: ## Run E2E tests with Playwright

TESTING_SAFETY.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# 🚨 CRITICAL: Testing Safety Documentation
2+
3+
## ⚠️ WARNING: Tests Can Wipe Your Database!
4+
5+
This Laravel application uses the `RefreshDatabase` trait in tests, which **COMPLETELY DROPS AND RECREATES ALL DATABASE TABLES**. Running tests incorrectly **WILL DELETE ALL YOUR DATA**.
6+
7+
## Database Configuration
8+
9+
| Database | Purpose | Data Status |
10+
|----------|---------|-------------|
11+
| `learning_app` | Development work | **MUST BE PROTECTED** |
12+
| `learning_app_test` | Testing only | Wiped on each test run |
13+
14+
## ✅ SAFE Testing Methods
15+
16+
### Method 1: Use the Safe Test Script (RECOMMENDED)
17+
```bash
18+
./scripts/safe-test.sh # Run all tests
19+
./scripts/safe-test.sh --filter TestName # Run specific test
20+
./scripts/safe-test.sh --coverage # Run with coverage
21+
```
22+
23+
### Method 2: Use Makefile Commands
24+
```bash
25+
make test-unit # Runs safe-test.sh internally
26+
make test-unit-coverage # Safe coverage testing
27+
```
28+
29+
## ❌ DANGEROUS Commands - NEVER RUN THESE
30+
31+
```bash
32+
# ALL OF THESE CAN WIPE YOUR DEVELOPMENT DATABASE:
33+
php artisan test # NO DATABASE PROTECTION
34+
artisan test # NO DATABASE PROTECTION
35+
vendor/bin/phpunit # NO DATABASE PROTECTION
36+
phpunit # NO DATABASE PROTECTION
37+
38+
# Even with APP_ENV - still risky if misconfigured:
39+
APP_ENV=testing php artisan test # RISKY - might not load .env.testing
40+
```
41+
42+
## How Protection Works
43+
44+
### 1. Safe Test Script (`scripts/safe-test.sh`)
45+
- **Validates** database name isn't `learning_app`
46+
- **Forces** `DB_DATABASE=learning_app_test`
47+
- **Sets** all required environment variables
48+
- **Creates** test database if it doesn't exist
49+
- **Displays** configuration before running
50+
51+
### 2. TestCase.php Protection
52+
```php
53+
// Throws exception if wrong database detected
54+
if ($database !== 'learning_app_test') {
55+
throw new \Exception("CRITICAL ERROR: Tests attempting to run on non-test database!");
56+
}
57+
```
58+
59+
### 3. phpunit.xml Configuration
60+
```xml
61+
<!-- Forces test database with force="true" attribute -->
62+
<env name="DB_DATABASE" value="learning_app_test" force="true"/>
63+
```
64+
65+
### 4. Makefile Integration
66+
All `make test*` commands use `safe-test.sh` automatically.
67+
68+
## If Your Database Gets Wiped (Recovery)
69+
70+
### Step 1: Don't Panic
71+
The structure can be restored, but data may be lost unless you have backups.
72+
73+
### Step 2: Restore Database Structure
74+
```bash
75+
# Recreate all tables
76+
php artisan migrate
77+
78+
# If you have seeders, run them
79+
php artisan db:seed
80+
81+
# Or use fresh command
82+
php artisan migrate:fresh --seed
83+
```
84+
85+
### Step 3: Restore Data
86+
- Check if you have database backups
87+
- Check if you have SQL dumps
88+
- Manually recreate critical data
89+
90+
### Step 4: Prevent Future Incidents
91+
```bash
92+
# Create a database backup before any risky operations
93+
pg_dump -U laravel -h localhost learning_app > backup_$(date +%Y%m%d_%H%M%S).sql
94+
95+
# Set up automated backups
96+
# Add to crontab: 0 */6 * * * pg_dump -U laravel learning_app > ~/backups/learning_app_$(date +\%Y\%m\%d_\%H\%M\%S).sql
97+
```
98+
99+
## Best Practices
100+
101+
1. **Always use `./scripts/safe-test.sh`** for running tests
102+
2. **Never modify phpunit.xml database settings**
103+
3. **Create regular database backups**
104+
4. **Use separate PostgreSQL users** for dev and test databases (advanced)
105+
5. **Review test output** - it shows which database is being used
106+
107+
## For CI/CD Pipelines
108+
109+
```yaml
110+
# GitHub Actions example
111+
- name: Run tests safely
112+
env:
113+
DB_DATABASE: learning_app_test # Explicitly set test database
114+
APP_ENV: testing
115+
run: |
116+
./scripts/safe-test.sh
117+
```
118+
119+
## Quick Reference Card
120+
121+
```
122+
✅ SAFE: ./scripts/safe-test.sh
123+
✅ SAFE: make test-unit
124+
❌ DANGER: php artisan test
125+
❌ DANGER: vendor/bin/phpunit
126+
```
127+
128+
## Emergency Contact
129+
130+
If you accidentally wipe your database:
131+
1. Stop all running processes
132+
2. Check `storage/logs/laravel.log` for any errors
133+
3. Run recovery steps above
134+
4. Consider implementing database replication for instant recovery
135+
136+
---
137+
138+
**Remember**: When in doubt, use `./scripts/safe-test.sh`. It's better to be safe than sorry!

phpunit.xml

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -20,16 +20,18 @@
2020
</include>
2121
</source>
2222
<php>
23-
<env name="APP_ENV" value="testing"/>
23+
<!-- CRITICAL: Test Environment Configuration -->
24+
<!-- NEVER change DB_DATABASE - tests use RefreshDatabase which WIPES the database -->
25+
<env name="APP_ENV" value="testing" force="true"/>
2426
<env name="APP_MAINTENANCE_DRIVER" value="file"/>
2527
<env name="BCRYPT_ROUNDS" value="4"/>
2628
<env name="CACHE_STORE" value="array"/>
27-
<env name="DB_CONNECTION" value="pgsql"/>
28-
<env name="DB_HOST" value="127.0.0.1"/>
29-
<env name="DB_PORT" value="5432"/>
30-
<env name="DB_DATABASE" value="learning_app_test"/>
31-
<env name="DB_USERNAME" value="laravel"/>
32-
<env name="DB_PASSWORD" value="12345"/>
29+
<env name="DB_CONNECTION" value="pgsql" force="true"/>
30+
<env name="DB_HOST" value="127.0.0.1" force="true"/>
31+
<env name="DB_PORT" value="5432" force="true"/>
32+
<env name="DB_DATABASE" value="learning_app_test" force="true"/> <!-- NEVER CHANGE THIS -->
33+
<env name="DB_USERNAME" value="laravel" force="true"/>
34+
<env name="DB_PASSWORD" value="12345" force="true"/>
3335
<env name="MAIL_MAILER" value="array"/>
3436
<env name="QUEUE_CONNECTION" value="sync"/>
3537
<env name="SESSION_DRIVER" value="array"/>

playwright.config.js

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,26 @@ export default defineConfig({
44
testDir: './tests/e2e',
55
fullyParallel: false, // Disable full parallelization to prevent race conditions
66
forbidOnly: !!process.env.CI,
7-
retries: process.env.CI ? 2 : 1,
7+
retries: process.env.CI ? 0 : 1, // No retries in CI for faster failure
88
workers: 1, // Force single worker to prevent server conflicts
9+
maxFailures: process.env.CI ? 3 : undefined, // Stop after 3 failures in CI
910
reporter: [
1011
['html'],
1112
['json', { outputFile: 'test-results/results.json' }],
12-
['junit', { outputFile: 'test-results/results.xml' }]
13+
['junit', { outputFile: 'test-results/results.xml' }],
14+
process.env.CI ? ['line'] : ['list'], // Simpler output in CI
1315
],
14-
timeout: 60000, // Increased global test timeout: 60 seconds for complex tests
16+
timeout: process.env.CI ? 30000 : 60000, // Shorter timeout in CI: 30 seconds
1517
expect: {
16-
timeout: 15000, // Increased assertion timeout: 15 seconds for complex assertions
18+
timeout: process.env.CI ? 5000 : 15000, // Shorter timeout in CI: 5 seconds
1719
},
1820

1921
use: {
20-
baseURL: 'http://127.0.0.1:18001',
21-
trace: 'on-first-retry',
22+
baseURL: process.env.BASE_URL || (process.env.CI ? 'http://localhost:8000' : 'http://127.0.0.1:18001'),
23+
trace: process.env.CI ? 'retain-on-failure' : 'on-first-retry',
2224
screenshot: 'only-on-failure',
23-
actionTimeout: 15000, // Increased action timeout: 15 seconds for complex interactions
24-
navigationTimeout: 20000, // Increased navigation timeout: 20 seconds for slow pages
25+
actionTimeout: process.env.CI ? 5000 : 15000, // Shorter action timeout in CI: 5 seconds
26+
navigationTimeout: process.env.CI ? 10000 : 20000, // Shorter navigation timeout in CI: 10 seconds
2527
},
2628

2729
projects: [

scripts/safe-test.sh

Lines changed: 46 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,60 @@
11
#!/bin/bash
22

3-
# Safe PHP Test Runner - Prevents accidental dev database wipes
4-
# This script ensures tests ALWAYS use the test database
3+
# Safe PHPUnit Test Runner
4+
# This script ensures tests NEVER run on the development database
5+
# Always uses the test database explicitly to prevent data loss
56

6-
set -e
7+
set -e # Exit on any error
78

8-
echo "🛡️ Safe Test Runner - Protecting your development database"
9-
echo "==========================================================="
9+
# Colors for output
10+
RED='\033[0;31m'
11+
GREEN='\033[0;32m'
12+
YELLOW='\033[1;33m'
13+
NC='\033[0m' # No Color
1014

11-
# Force testing environment to prevent accidental data loss
15+
echo -e "${GREEN}🛡️ Safe Test Runner - Database Protection Active${NC}"
16+
echo ""
17+
18+
# Check if we're about to use the wrong database
19+
if [ "$DB_DATABASE" == "learning_app" ]; then
20+
echo -e "${RED}❌ CRITICAL ERROR: Attempting to run tests on development database!${NC}"
21+
echo -e "${RED} Tests would wipe the 'learning_app' database.${NC}"
22+
echo -e "${YELLOW} Use this script or explicitly set DB_DATABASE=learning_app_test${NC}"
23+
exit 1
24+
fi
25+
26+
# Force the test environment and database
1227
export APP_ENV=testing
1328
export DB_CONNECTION=pgsql
29+
export DB_HOST=127.0.0.1
30+
export DB_PORT=5432
1431
export DB_DATABASE=learning_app_test
15-
export SESSION_DRIVER=file
32+
export DB_USERNAME=laravel
33+
export DB_PASSWORD=12345
1634
export CACHE_STORE=array
35+
export SESSION_DRIVER=file
1736

18-
# Verify we're using the test database
19-
if [ "$DB_DATABASE" != "learning_app_test" ]; then
20-
echo "❌ ERROR: Not using test database! Aborting to protect your data."
21-
exit 1
37+
echo -e "${YELLOW}📋 Test Configuration:${NC}"
38+
echo " Environment: $APP_ENV"
39+
echo " Database: $DB_DATABASE (SAFE)"
40+
echo " Cache: $CACHE_STORE"
41+
echo " Session: $SESSION_DRIVER"
42+
echo ""
43+
44+
# Verify the test database exists
45+
echo -e "${GREEN}✓ Verifying test database exists...${NC}"
46+
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USERNAME -lqt 2>/dev/null | cut -d \| -f 1 | grep -qw $DB_DATABASE
47+
if [ $? -ne 0 ]; then
48+
echo -e "${YELLOW}⚠️ Test database '$DB_DATABASE' does not exist. Creating it...${NC}"
49+
PGPASSWORD=$DB_PASSWORD createdb -h $DB_HOST -p $DB_PORT -U $DB_USERNAME $DB_DATABASE 2>/dev/null || true
50+
echo -e "${GREEN}✓ Test database created${NC}"
2251
fi
2352

24-
echo "✅ Using test database: $DB_DATABASE"
53+
# Run the tests with all arguments passed through
54+
echo -e "${GREEN}🧪 Running tests on SAFE test database...${NC}"
2555
echo ""
2656

27-
# Run the tests with all arguments passed through
28-
php artisan test "$@"
57+
php artisan test "$@"
58+
59+
echo ""
60+
echo -e "${GREEN}✅ Tests completed safely on test database${NC}"

tests/TestCase.php

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,33 @@ protected function setUp(): void
1515
{
1616
parent::setUp();
1717

18+
// CRITICAL: Prevent running tests on non-test database
19+
// This prevents accidentally wiping the development database
20+
$database = config('database.connections.pgsql.database');
21+
22+
// List of allowed test database names (local and CI)
23+
$allowedTestDatabases = [
24+
'learning_app_test', // Local development
25+
'homeschoolai_test', // GitHub Actions CI
26+
'homeschoolai_e2e', // GitHub Actions E2E
27+
];
28+
29+
if (! in_array($database, $allowedTestDatabases)) {
30+
throw new \Exception(
31+
"🚨 CRITICAL ERROR: Tests attempting to run on non-test database!\n".
32+
' Expected one of: '.implode(', ', $allowedTestDatabases)."\n".
33+
" Actual: {$database}\n".
34+
" \n".
35+
" This would WIPE YOUR DATABASE!\n".
36+
" \n".
37+
" To run tests safely, use:\n".
38+
" ./scripts/safe-test.sh\n".
39+
" \n".
40+
" Or manually set:\n".
41+
' APP_ENV=testing DB_DATABASE=learning_app_test php artisan test'
42+
);
43+
}
44+
1845
// Disable CSRF middleware for all tests to prevent 419 errors
1946
$this->withoutMiddleware([
2047
\Illuminate\Foundation\Http\Middleware\ValidateCsrfToken::class,

0 commit comments

Comments
 (0)