refactor: reorganize automation directory structure (#3164)

Reorganize directory structure to better separate concerns:
- automation/ → Only workflow-specific scripts used by GitHub Actions
- scripts/ → One-time migrations and manual utilities
- core/generators/ → Reusable business logic components

Changes:
- Move one-time migration scripts to scripts/:
  - backfill_review_metadata.py
  - fix_library_versions.py
  - migrate_metadata_format.py
  - migrate_to_new_structure.py
  - upgrade_specs.py
  - upgrade_specs_ai.py

- Move reusable generators to core/generators/:
  - plot_generator.py (from automation/generators/)

- Update imports in tests and documentation
- Update repository.md with new directory structure
- Add sections explaining automation/ vs scripts/ principles

All tests pass successfully.

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: MarkusNeusinger

automation/generators/__init__.py core/generators/__init__.pyautomation/generators/__init__.py renamed to core/generators/__init__.py

@@ -76,11 +76,13 @@ pyplots/
 ├── core/                              # Shared business logic
 │   ├── __init__.py
 │   ├── config.py                      # Configuration (.env-based)
-│   └── database/                      # Database layer
-│       ├── __init__.py
-│       ├── connection.py              # Async connection management
-│       ├── models.py                  # SQLAlchemy ORM models
-│       └── repositories.py            # Repository pattern
+│   ├── database/                      # Database layer
+│   │   ├── __init__.py
+│   │   ├── connection.py              # Async connection management
+│   │   ├── models.py                  # SQLAlchemy ORM models
+│   │   └── repositories.py            # Repository pattern
+│   └── generators/                    # Reusable code generators
+│       └── plot_generator.py          # Plot code generation utilities
 │
 ├── api/                               # FastAPI backend
 │   ├── __init__.py
@@ -95,9 +97,12 @@ pyplots/
 │   ├── package.json
 │   └── Dockerfile
 │
-├── automation/                        # Automation scripts
-│   └── scripts/
-│       └── sync_to_postgres.py        # Sync plots/ to database
+├── automation/                        # Workflow automation
+│   └── scripts/                       # Workflow-specific utilities
+│       ├── sync_to_postgres.py        # Sync plots/ to database
+│       ├── workflow_utils.py          # Utilities for GitHub Actions
+│       ├── label_manager.py           # Label operations
+│       └── workflow_cli.py            # CLI for workflows
 │
 ├── tests/                             # Test suite
 │   └── unit/
@@ -121,6 +126,15 @@ pyplots/
 ├── alembic/                           # Database migrations
 │   └── versions/
 │
+├── scripts/                           # One-time and manual scripts
+│   ├── evaluate-plot.py               # Manual plot evaluation
+│   ├── regenerate-thumbnails.py       # Image processing
+│   ├── backfill_review_metadata.py    # One-time migration
+│   ├── fix_library_versions.py        # One-time fix
+│   ├── migrate_metadata_format.py     # One-time migration
+│   ├── migrate_to_new_structure.py    # One-time migration
+│   └── upgrade_specs*.py              # Spec upgrade utilities
+│
 ├── docs/                              # Documentation
 │   ├── architecture/
 │   ├── workflow.md
@@ -390,12 +404,42 @@ plt.savefig('plot.png', dpi=300)
 
 ### `core/`
 
-**Purpose**: Shared business logic used by API
+**Purpose**: Shared business logic and reusable components
 
 **Key Components**:
 - `database/connection.py` - Async database connection
 - `database/models.py` - SQLAlchemy ORM models
 - `database/repositories.py` - Repository pattern for data access
+- `generators/plot_generator.py` - Reusable plot code generation utilities
+
+---
+
+### `automation/`
+
+**Purpose**: Workflow-specific automation used by GitHub Actions
+
+**Key Components**:
+- `scripts/sync_to_postgres.py` - Database sync (used by sync-postgres.yml)
+- `scripts/workflow_utils.py` - Parsing and utilities for workflows
+- `scripts/label_manager.py` - Label operations and transitions
+- `scripts/workflow_cli.py` - CLI interface for workflow steps
+
+**Principle**: Only contains components actively used by workflows. One-time or manual scripts belong in `scripts/`.
+
+---
+
+### `scripts/`
+
+**Purpose**: One-time migrations and manually-run utilities
+
+**Key Components**:
+- `evaluate-plot.py` - Manual plot quality evaluation
+- `regenerate-thumbnails.py` - Image processing utilities
+- `backfill_review_metadata.py` - One-time backfill migration
+- `migrate_*.py` - One-time structure migrations
+- `upgrade_specs*.py` - Spec upgrade utilities
+
+**Principle**: Scripts that are not part of automated workflows. Used for maintenance, migrations, and manual operations.
 
 ---
 

automation/generators/plot_generator.py core/generators/plot_generator.pyautomation/generators/plot_generator.py renamed to core/generators/plot_generator.py

@@ -500,7 +500,7 @@ claude-skill plot-generation \
 ### Example 2: From Python
 
 ```python
-# automation/generators/claude_generator.py
+# core/generators/claude_generator.py
 from claude_skills import invoke_skill
 
 result = invoke_skill(

docs/architecture/repository.md

@@ -11,8 +11,8 @@
 always takes the LAST one (the one that led to merge).
 
 Usage:
-    python automation/scripts/backfill_review_metadata.py --dry-run
-    python automation/scripts/backfill_review_metadata.py --execute
+    python scripts/backfill_review_metadata.py --dry-run
+    python scripts/backfill_review_metadata.py --execute
 
 Requires:
     - gh CLI authenticated

docs/concepts/claude-skill-plot-generation.md

@@ -7,7 +7,7 @@
 2. metadata/{library}.yaml: Flatten current:, remove history, add created/updated/review
 3. implementations/*.py: Update docstring header to new format
 
-Run: uv run python automation/scripts/migrate_metadata_format.py
+Run: uv run python scripts/migrate_metadata_format.py
 """
 
 import re

…tion/scripts/backfill_review_metadata.py scripts/backfill_review_metadata.pyautomation/scripts/backfill_review_metadata.py renamed to scripts/backfill_review_metadata.py automation/scripts/backfill_review_metadata.py renamed to scripts/backfill_review_metadata.py

@@ -12,7 +12,7 @@
 - metadata/{library}.yaml (per-library: preview_url, current, history)
 
 Usage:
-    python automation/scripts/migrate_to_new_structure.py [--dry-run]
+    python scripts/migrate_to_new_structure.py [--dry-run]
 """
 
 import argparse