feat(workflows): unified plot-prepare workflow#535
Merged
MarkusNeusinger merged 11 commits intomainfrom Dec 9, 2025
Merged
Conversation
…info Add a new `metadata/` directory that stores structured metadata for each spec: - Tags (plot_type, domain, features, audience, data_type) at spec level - Per-library implementation metadata (generation info, quality scores, preview URLs) - History of previous attempts with feedback and improvement suggestions Changes: - Add metadata/*.yaml files for all 8 existing specs - Add metadata/schema.yaml defining the YAML structure - Update database models with new fields (structured_tags, generated_at, generated_by, etc.) - Add migration 002 for the new database columns - Update sync_to_postgres.py to read and sync metadata from YAML files - Update sync-postgres.yml workflow to trigger on metadata/* changes - Document the metadata system in CLAUDE.md This enables tracking which AI model generated each implementation, quality scores with evaluator details, and improvement suggestions for future spec updates.
Reorganize from three separate directories (specs/, metadata/, plots/)
to a unified structure where each plot has its own directory containing
all related files:
plots/{spec-id}/
├── spec.md # Description, data requirements, use cases
├── metadata.yaml # Tags, generation info, quality history
└── implementations/ # Library implementations (matplotlib.py, etc.)
Key changes:
- Migrate 8 specs with 72 implementations to new structure
- Add content field to Spec model (full markdown)
- Add code field to Implementation model (Python source)
- Remove plot_function field (no longer needed)
- Update sync script to scan new structure
- Update API to use PLOTS_DIR instead of SPECS_DIR
- Update CLAUDE.md documentation throughout
- Add migration 003 for database schema changes
This enables the app to display full spec descriptions and
implementation code directly from the database.
Update all documentation files to reflect the new structure where
everything for one plot lives in plots/{spec-id}/:
- docs/architecture/repository.md - Complete rewrite
- docs/specs-guide.md - Update file locations
- docs/workflow.md - Update path references in diagrams
- docs/plot-types-catalog.md - Update spec creation path
- docs/development.md - Update implementation paths and examples
…→ Notes - Rename "Use Cases" to "Applications" for clearer terminology - Remove Tags section (tags belong in metadata.yaml only) - Simplify Data section formatting (list without bold headers) - Add optional Notes section for implementation hints - Update all 8 existing spec files to new format - Update template and documentation to reflect changes
- Add created_at, created_by_issue, updated_at to metadata.yaml - Create templates/metadata.yaml template - Update all 8 existing metadata files with new fields - Document new tracking fields in CLAUDE.md
- Rename created_at → created, created_by_issue → issue, updated_at → updates - Add 'suggested' field to credit contributors (GitHub username) - Add 'updates' list for tracking spec changes with issue references - Set suggested: pyplots for all seed plots - Update template and CLAUDE.md documentation
Specs table:
- description (text), applications/data/notes (arrays)
- created, issue, suggested from metadata.yaml
- tags as JSONB (structured)
- Remove: content, data_requirements, optional_params, structured_tags
Implementations table:
- preview_url, preview_thumb, preview_html for multiple formats
- python_version, library_version (created with)
- tested as JSONB array [{py, lib, ok, error?}]
- Remove: variant, file_path, tested bool, workflow_run, evaluator_scores, quality_feedback, improvements_suggested
Libraries table:
- Remove: active, created_at (not needed)
- Changed spec file paths from `specs/` to `plots/` - Updated implementation file paths to reflect new directory structure - Adjusted documentation and comments to align with new paths
- Change template paths from `templates/` to `prompts/templates/` - Move `sync_to_postgres.py` to `automation/scripts/` - Update documentation to reflect new file structure
- Simplified GCS preview URL to point directly to latest.png - Added versioning, date, issue, and quality score to current and history fields in metadata - Updated repository documentation to reflect new GCS storage structure
- Merge bot-validate-request + gen-create-spec into plot-prepare.yml - Disable old workflows (renamed to .yml.disabled) - Add update request analysis (validates if update reason is sensible) - Update issue templates for clearer workflow
Contributor
⏭️ Plot Tests SkippedNo plot files changed Compatibility: Note: Only Python 3.13 is required to pass. Python 3.12 is tested for compatibility. |
MarkusNeusinger
deleted the
claude/plots-restructure-01Pi7U6YJE6erMpGqL1ea5kg
branch
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements a major restructuring from a library-centric to a plot-centric repository organization. The changes consolidate all assets for each plot type (specification, metadata, and implementations across 9 libraries) into single directories under plots/{spec-id}/. This includes:
- Migration from
specs/{spec-id}.mdandplots/{library}/{plot_type}/{spec-id}/default.pytoplots/{spec-id}/spec.mdandplots/{spec-id}/implementations/{library}.py - Introduction of structured metadata files (
metadata.yaml) alongside specs - Updated database models to track version history, quality scores, and generation metadata
- New sync script that parses the plot-centric structure and syncs to PostgreSQL
- Comprehensive documentation updates reflecting the new architecture
- Template files for AI-generated specs and metadata
Key Improvements:
- Self-contained plot directories (easier navigation)
- Structured metadata for tracking (tags, versions, quality scores)
- Simplified file naming (library.py instead of default.py in nested paths)
- Better separation between spec content and metadata
Reviewed changes
Copilot reviewed 66 out of 139 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
specs/.template.md |
Deleted - Replaced by prompts/templates/spec.md |
scripts/sync_to_postgres.py |
Deleted - Replaced by automation/scripts/sync_to_postgres.py |
prompts/templates/spec.md |
Added - New spec template with simplified structure (Applications, Data, Notes) |
prompts/templates/metadata.yaml |
Added - Comprehensive metadata template with tags and implementation tracking |
plots/{spec-id}/spec.md |
Added - New plot-centric specs for multiple plot types |
plots/{spec-id}/metadata.yaml |
Added - Metadata files for each plot |
plots/{spec-id}/implementations/*.py |
Added - Library implementations in new structure |
plots/{library}/{plot_type}/{spec-id}/default.py |
Deleted - Old library-centric implementations removed |
core/database/models.py |
Modified - Enhanced schema with history tracking, quality metadata |
automation/scripts/sync_to_postgres.py |
Added - New sync script for plot-centric structure |
api/main.py |
Modified - Updated to use PLOTS_DIR instead of SPECS_DIR |
docs/*.md |
Modified - Documentation updated to reflect new structure |
prompts/*.md |
Modified - AI prompts updated with new file paths |
alembic/versions/004_*.py |
Added - Migration for history fields |
Files not reviewed (1)
- .idea/data_source_mapping.xml: Language not supported
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bot-validate-request+gen-create-specinto singleplot-prepare.yml.yml.disabled)New Workflow Flow
Test Plan