Skip to content

cleanup_setup.md

Latest commit

 

History

History
210 lines (158 loc) · 7.16 KB

cleanup_setup.md

File metadata and controls

210 lines (158 loc) · 7.16 KB
Raw

Periodic Cleanup Setup

This document explains how to set up periodic cleanup of temporary files generated by the Math2Visual API.

Overview

The Math2Visual API generates temporary files and database records that need periodic cleanup:

  1. Visualization Output (storage/output/): Temporary SVG files for visualization requests using unique UUIDs to prevent conflicts in parallel requests.
  2. SVG Generation (storage/temp_svgs/): Temporary AI-generated SVG icons that haven't been confirmed by the user yet.
  3. Tutor Sessions (database): Expired tutor session records stored in PostgreSQL that are no longer active.

These files and records are not cleaned up immediately to avoid blocking the API. Instead, a periodic cleanup job removes old files and expired sessions.

Files

  • app/utils/cleanup.py - Cleanup utility classes and functions
  • scripts/cleanup_temp_files.py - CLI script for manual or scheduled cleanup
  • docs/cleanup_setup.md - This documentation

Manual Cleanup

You can run the cleanup script manually:

# Clean up with defaults (output files >24h, temp SVGs >1h)
cd backend
python scripts/cleanup_temp_files.py

# Clean up output files older than 1 hour
python scripts/cleanup_temp_files.py --age-hours 1

# Clean up temp SVG files older than 30 minutes
python scripts/cleanup_temp_files.py --temp-svg-age-hours 0.5

# Dry run to see what would be cleaned (both directories)
python scripts/cleanup_temp_files.py --dry-run

# Show storage statistics (both directories)
python scripts/cleanup_temp_files.py --stats

Automated Cleanup (Cron)

To set up automated cleanup, add a cron job:

1. Edit crontab

crontab -e

2. Add cleanup job

# Recommended: Clean up every hour (output files >24h, temp SVGs >1h)
0 * * * * cd /path/to/math2visual/backend && python scripts/cleanup_temp_files.py >> /var/log/math2visual_cleanup.log 2>&1

# Alternative: Clean up every 6 hours with custom thresholds
0 */6 * * * cd /path/to/math2visual/backend && python scripts/cleanup_temp_files.py --age-hours 24 --temp-svg-age-hours 2 >> /var/log/math2visual_cleanup.log 2>&1

# More aggressive: Clean up temp SVGs every 30 minutes
*/30 * * * * cd /path/to/math2visual/backend && python scripts/cleanup_temp_files.py --temp-svg-age-hours 0.5 >> /var/log/math2visual_cleanup.log 2>&1

3. Make script executable (optional)

chmod +x backend/scripts/cleanup_temp_files.py

Configuration

File Patterns Cleaned

Visualization Output Directory (storage/output/)

  • formal_*-*-*-*-*.svg - Temporary formal visualization files (formal_{uuid}.svg)
  • intuitive_*-*-*-*-*.svg - Temporary intuitive visualization files (intuitive_{uuid}.svg)

Temp SVG Directory (storage/temp_svgs/)

  • *.svg - All temporary AI-generated SVG icon files

Age Thresholds

Visualization Output Files

  • Default: 24 hours
  • Configurable via --age-hours parameter
  • Files older than this threshold are removed

Temp SVG Files

  • Default: 1 hour
  • Configurable via --temp-svg-age-hours parameter
  • Shorter threshold since these are meant to be very temporary (user-aborted generations)

Tutor Sessions (Database)

  • Default: 2 hours of inactivity
  • Fixed expiration time (not configurable via CLI)
  • Sessions are automatically expired based on last_activity timestamp
  • Only cleaned up when database is available (gracefully skips if database is unavailable)
  • Required for proper operation with multiple Gunicorn workers (sessions are shared via database)

Archive Files

Archive files are only created when running in debug mode (FLASK_DEBUG=true):

  • formal_{timestamp}_{uuid}.svg
  • intuitive_{timestamp}_{uuid}.svg

To include archive files in cleanup, use the --include-archives flag:

python scripts/cleanup_temp_files.py --include-archives

This will also clean up:

  • formal_*_*-*-*-*-*.svg - Archive formal files (formal_{timestamp}_{uuid}.svg)
  • intuitive_*_*-*-*-*-*.svg - Archive intuitive files (intuitive_{timestamp}_{uuid}.svg)

Monitoring

Check Storage Usage

python scripts/cleanup_temp_files.py --stats

View Cleanup Logs

tail -f /var/log/math2visual_cleanup.log

Manual Verification

# Check what files exist in output directory
ls -la storage/output/

# Check what files exist in temp SVG directory
ls -la storage/temp_svgs/

# Check file ages in output directory
find storage/output/ -name "*.svg" -type f -exec ls -la {} \;

# Check file ages in temp SVG directory
find storage/temp_svgs/ -name "*.svg" -type f -exec ls -la {} \;

Troubleshooting

Permission Issues

If cleanup fails due to permissions:

# Ensure the cleanup script can write to both directories
chmod 755 storage/output/ storage/temp_svgs/
chown -R www-data:www-data storage/output/ storage/temp_svgs/  # Adjust user/group as needed

Disk Space Issues

If storage is filling up:

  1. Reduce the age thresholds: --age-hours 6 --temp-svg-age-hours 0.5
  2. Run cleanup more frequently in cron (e.g., every 30 minutes)
  3. Check for very large files: 
    find storage/output/ -size +10M
find storage/temp_svgs/ -size +10M
  4. Monitor both directories: python scripts/cleanup_temp_files.py --stats

Cleanup Not Running

  1. Check cron service: systemctl status cron
  2. Check cron logs: grep CRON /var/log/syslog
  3. Test script manually: python scripts/cleanup_temp_files.py --dry-run

Tutor Sessions Not Being Cleaned

If tutor session cleanup is not working:

  1. Check database connection: Ensure DATABASE_URL is set correctly in your environment
  2. Verify database is running: Check that PostgreSQL is accessible
  3. Check cleanup logs: Look for "Could not clean up tutor sessions" messages
  4. Manual verification: Sessions expire after 2 hours of inactivity - check tutor_sessions table in database: 
    SELECT session_id, last_activity, 
       NOW() - last_activity AS age 
FROM tutor_sessions 
ORDER BY last_activity DESC;
  5. Note: Tutor session cleanup gracefully skips if database is unavailable (won't cause errors)

Performance Impact

  • Minimal: Cleanup runs outside of API requests
  • Storage efficient: Prevents disk space issues
  • Non-blocking: API responses are not delayed by cleanup
  • Configurable: Age thresholds can be tuned based on usage patterns

Security Considerations

  • Cleanup only removes files matching specific patterns
  • Archive files with timestamps are preserved by default
  • Cleanup errors are logged but don't affect API functionality
  • Temp SVG files are cleaned more aggressively (1 hour default) to prevent accumulation from aborted generations

Use Cases

Why Two Directories?

  1. Visualization Output (storage/output/):

    • Created during normal API operations
    • Persist longer (24h) for debugging and caching
    • May be referenced by active sessions

  2. Temp SVG Directory (storage/temp_svgs/):

    • Created when users generate AI SVG icons
    • Only needed until user confirms or cancels
    • Should be cleaned quickly (1h) to handle:
      • User-aborted generations
      • Browser crashes/closes
      • Network interruptions
      • Race conditions between frontend and backend