Update

Author: nikhilwoodruff

.env.example

@@ -20,6 +20,7 @@ STORAGE_BUCKET=datasets
 # API
 API_TITLE=PolicyEngine API
 API_VERSION=0.1.0
+API_PORT=8000
 DEBUG=true
 
 # Logfire (get token from https://logfire.pydantic.dev)

Dockerfile

@@ -10,4 +10,4 @@ COPY src/ ./src/
 RUN uv pip install --system -e .
 
 # Run migrations and start server
-CMD ["uvicorn", "policyengine_api.main:app", "--host", "0.0.0.0", "--port", "80"]
+CMD uvicorn policyengine_api.main:app --host 0.0.0.0 --port ${API_PORT:-80}

README.md

@@ -1,13 +1,14 @@
 # PolicyEngine API v2
 
-A FastAPI service for running PolicyEngine microsimulations with Supabase backend, object storage, and background task processing.
+FastAPI service for PolicyEngine microsimulations with Supabase backend, object storage, and background task processing.
 
 ## Features
 
-- RESTful API for creating and managing simulations
-- Supabase for database and storage
-- Redis caching and Celery for background task processing
+- RESTful API for tax-benefit microsimulations
+- Supabase for PostgreSQL database and object storage
+- Celery workers for background simulation tasks
 - SQLModel for type-safe database models
+- Logfire observability and monitoring
 - Terraform deployment to AWS
 
 ## Quick start
@@ -16,257 +17,192 @@ A FastAPI service for running PolicyEngine microsimulations with Supabase backen
 
 - [Supabase CLI](https://supabase.com/docs/guides/cli)
 - Docker and Docker Compose
-- Python 3.11+
+- Python 3.13+ with uv
 
-### Local development with Supabase
+### Local development
 
 1. **Set up environment**
 
-Copy the example environment file:
-
 ```bash
 cp .env.example .env
 ```
 
-The `.env` file contains default values for local Supabase development. Key variables:
+The defaults in `.env.example` work with local Supabase. Key settings:
 
 ```bash
-# Supabase local instance (defaults work with `supabase start`)
 SUPABASE_URL=http://127.0.0.1:54321
-SUPABASE_KEY=<anon-key>                  # Public anon key
-SUPABASE_SERVICE_KEY=<service-role-key>  # Admin key for seeding
 SUPABASE_DB_URL=postgresql://postgres:postgres@127.0.0.1:54322/postgres
-
-# Storage
-STORAGE_BUCKET=datasets
-
-# Redis (via Docker)
 REDIS_URL=redis://localhost:6379/0
+STORAGE_BUCKET=datasets
 ```
 
-**Important:** The service key is required for database seeding operations (uploading datasets to storage). The default keys in `.env.example` work with local Supabase.
-
 2. **Start Supabase**
 
 ```bash
 supabase start
 ```
 
-This creates a local instance at `http://localhost:54321` with:
-- PostgreSQL on port 54322
-- Storage API with S3-compatible interface
-- Auth API
-- Studio dashboard at `http://localhost:54323`
+Creates a local instance with PostgreSQL (port 54322), Storage API, and Studio dashboard at `http://localhost:54323`.
 
 3. **Run integration tests**
 
-This will set up the database, seed it with UK/US models, and run tests:
+Sets up database, seeds models, and runs tests:
 
 ```bash
 make integration-test
 ```
 
 This command:
-- Starts Supabase (if not running)
 - Creates all database tables from SQLModel definitions
-- Applies RLS policies and storage bucket configuration
-- Seeds UK and US tax-benefit models with all variables, parameters, and datasets
-- Runs integration tests to verify everything works
+- Applies RLS policies and storage bucket migrations
+- Seeds UK and US tax-benefit models with variables, parameters, and datasets
+- Runs integration tests
 
 4. **Start the API**
 
 ```bash
 docker compose up
 ```
 
-The API will be available at `http://localhost:8000`. Visit `http://localhost:8000/docs` for interactive API documentation.
-
-## Observability
-
-The API uses [Logfire](https://logfire.pydantic.dev/) for observability and monitoring. Logfire automatically instruments:
-- All HTTP requests and responses
-- Database queries
-- Background tasks
-- Performance metrics
-
-To view traces and logs, visit the [Logfire dashboard](https://logfire.pydantic.dev).
-
-## Architecture
-
-The system consists of three main components:
-
-1. **API server** - FastAPI application serving REST endpoints
-2. **Database** - Supabase PostgreSQL storing models, parameters, simulations
-3. **Storage** - Supabase storage for dataset files (.h5)
-4. **Worker** - Celery workers executing simulations in background
-
-### Models
-
-The API provides eleven core resources:
-
-- **Datasets** - Microdata for simulations (stored in Supabase storage)
-- **Policies** - Parametric reforms to tax-benefit systems
-- **Simulations** - Execution of tax-benefit models on datasets
-- **Variables** - Calculated outputs (income_tax, universal_credit, etc.)
-- **Parameters** - System settings (personal_allowance, benefit_rates, etc.)
-- **ParameterValues** - Time-bound parameter values
-- **Dynamics** - Dynamic modeling configurations
-- **TaxBenefitModels** - Country models (UK, US)
-- **TaxBenefitModelVersions** - Model versions
-- **AggregateOutputs** - Aggregate statistics from simulations
-- **ChangeAggregates** - Reform impact analysis
-
-### Workflow
-
-1. Upload dataset: `POST /datasets` with file upload to Supabase storage
-2. Create policy reform: `POST /policies`
-3. Create simulation: `POST /simulations`
-4. Poll simulation status: `GET /simulations/{id}`
-5. Create aggregates: `POST /outputs/aggregate`
+API available at `http://localhost:8000`. Visit `http://localhost:8000/docs` for interactive documentation.
 
 ## API endpoints
 
-All endpoints at root level:
+Base URL: `http://localhost:8000`
 
-```bash
-GET  /health                           → {"status": "healthy"}
-GET  /datasets                         → List all datasets
+### Core resources
+
+```
+GET  /health                           → Health check
+GET  /datasets                         → List datasets
 POST /datasets                         → Create dataset
-GET  /policies                         → List all policies
+GET  /policies                         → List policies
 POST /policies                         → Create policy
-GET  /simulations                      → List all simulations
-POST /simulations                      → Create and queue simulation
-GET  /variables                        → List all variables
-GET  /parameters                       → List all parameters
-GET  /parameter-values                 → List all parameter values
-GET  /dynamics                         → List all dynamics
-GET  /tax-benefit-models               → List all models
-GET  /tax-benefit-model-versions       → List all model versions
-GET  /outputs/aggregate                → List aggregates
-POST /outputs/aggregate                → Compute aggregate
-GET  /change-aggregates                → List change aggregates
-POST /change-aggregates                → Compute reform impact
-
-POST /initialize/uk                    → Initialize UK model
-POST /initialize/us                    → Initialize US model
+GET  /simulations                      → List simulations
+POST /simulations                      → Create simulation
+GET  /simulations/{id}                 → Get simulation status
 ```
 
-## Configuration
-
-Set environment variables or create a `.env` file:
+### Model metadata
 
-```bash
-SUPABASE_URL=http://localhost:54321
-SUPABASE_KEY=your-anon-key
-SUPABASE_DB_URL=postgresql://postgres:postgres@localhost:54322/postgres
-REDIS_URL=redis://localhost:6379/0
-CELERY_BROKER_URL=redis://localhost:6379/0
-CELERY_RESULT_BACKEND=redis://localhost:6379/1
-STORAGE_BUCKET=datasets
-DEBUG=false
+```
+GET  /variables                        → List variables
+GET  /parameters                       → List parameters
+GET  /parameter-values                 → List parameter values
+GET  /tax-benefit-models               → List models
+GET  /tax-benefit-model-versions       → List model versions
 ```
 
-## Storage
+### Aggregates and analysis
 
-Datasets are stored in Supabase storage:
+```
+GET  /aggregates                       → List aggregates
+POST /aggregates                       → Create aggregate
+GET  /aggregates/{id}                  → Get aggregate
+GET  /change-aggregates                → List change aggregates
+POST /change-aggregates                → Create change aggregate
+GET  /change-aggregates/{id}           → Get change aggregate
+DELETE /change-aggregates/{id}         → Delete change aggregate
+```
 
-```python
-from policyengine_api.services import storage
+## Typical workflow
 
-# Upload dataset
-url = storage.upload_dataset("/path/to/dataset.h5", "frs_2023_24.h5")
+1. Create simulation: `POST /simulations` with dataset and policy
+2. Poll status: `GET /simulations/{id}` until status is "completed"
+3. Request aggregates: `POST /aggregates` with simulation_id and variable
+4. Compare reforms: `POST /change-aggregates` with baseline and reform simulation IDs
 
-# Download dataset
-storage.download_dataset("frs_2023_24.h5", "/local/path.h5")
+## Database management
 
-# Get public URL
-url = storage.get_dataset_url("frs_2023_24.h5")
+### Local development
 
-# List all datasets
-files = storage.list_datasets()
+```bash
+make integration-test  # Full setup: tables, migrations, seeding, tests
+make seed             # Seed UK/US models only
+make reset            # Reset Supabase database
 ```
 
-## Development
-
-### Running tests
+### Production
 
 ```bash
-pytest
+make db-reset-prod    # Reset production database (requires confirmation)
 ```
 
-### Code formatting
+**Warning:** `db-reset-prod` drops all tables, recreates schema, and reseeds data. It requires typing "yes" to confirm.
+
+## Development
+
+### Code quality
 
 ```bash
-ruff format .
-ruff check --fix .
+make format          # Format code with ruff
+make lint            # Lint and fix with ruff
+make test            # Run unit tests
 ```
 
 ### Database schema
 
-The database schema uses a hybrid approach:
+Schema is defined in two parts:
 
-**SQLModel for tables:** All table schemas are defined in Python using SQLModel (see `src/policyengine_api/models/`). This provides:
-- Single source of truth in Python code
-- Type safety and IDE autocomplete
-- Automatic relationship handling
-- Easy testing
+**SQLModel tables** (`src/policyengine_api/models/`): All table definitions use SQLModel for type safety and single source of truth.
 
-**SQL migrations for Postgres features:** SQL files in `supabase/migrations/` handle:
-- Row-level security (RLS) policies
-- Storage bucket configuration
-- Postgres-specific features SQLModel can't express
+**SQL migrations** (`supabase/migrations/`): Row-level security policies, storage buckets, and Postgres-specific features.
 
-To regenerate the schema:
+To recreate tables:
 
 ```bash
-# Creates all tables from SQLModel and applies SQL migrations
 uv run python scripts/create_tables.py
 ```
 
-**When to create SQL migrations:**
+Only create SQL migrations for RLS policies or storage configuration, not table schemas.
 
-Only create new SQL migrations for RLS policies, storage buckets, or other Postgres-specific features. Table schemas should be defined in SQLModel classes, not SQL.
+## Observability
 
-```bash
-# Create a new migration (only for RLS/storage/triggers)
-supabase migration new add_new_rls_policy
-```
+[Logfire](https://logfire.pydantic.dev/) instruments:
+- HTTP requests and responses
+- Database queries
+- Background tasks
+- Performance metrics
 
-### Supabase management
+View traces at the [Logfire dashboard](https://logfire.pydantic.dev).
 
-```bash
-# Start local Supabase
-supabase start
+## Architecture
 
-# Stop Supabase
-supabase stop
+### Components
 
-# Reset database
-supabase db reset
+- **API server**: FastAPI application (port 8000 local, 80 production)
+- **Database**: Supabase PostgreSQL
+- **Storage**: Supabase object storage for .h5 dataset files
+- **Worker**: Celery workers for background simulations
+- **Cache**: Redis for Celery broker and API caching
 
-# View logs
-supabase logs
+### Data models
 
-# Open Studio dashboard
-supabase studio
-```
+- **Datasets**: Microdata files in Supabase storage
+- **Policies**: Parameter reforms
+- **Simulations**: Tax-benefit calculations
+- **Variables**: Model outputs (income_tax, universal_credit)
+- **Parameters**: System settings (personal_allowance, benefit_rates)
+- **ParameterValues**: Time-bound parameter values
+- **Aggregates**: Statistics from simulations
+- **ChangeAggregates**: Reform impact analysis
 
-## Project structure
+### Project structure
 
 ```
 policyengine-api-v2/
-├── src/
-│   └── policyengine_api/
-│       ├── api/              # FastAPI routers
-│       ├── config/           # Settings
-│       ├── models/           # SQLModel database models
-│       ├── services/         # Database, storage, initialization
-│       ├── tasks/            # Celery tasks
-│       └── main.py           # FastAPI application
-├── supabase/                 # Supabase configuration and migrations
-├── tests/                    # Test suite
-├── docker-compose.yml        # Docker services (API, worker, Redis)
-└── pyproject.toml            # Dependencies
+├── src/policyengine_api/
+│   ├── api/              # FastAPI routers
+│   ├── config/           # Settings
+│   ├── models/           # SQLModel database models
+│   ├── services/         # Database, storage
+│   ├── tasks/            # Celery tasks
+│   └── main.py           # FastAPI app
+├── supabase/
+│   └── migrations/       # RLS policies and storage
+├── scripts/              # Database setup and seeding
+├── tests/                # Test suite
+└── docker-compose.yml    # Local services
 ```
 
 ## License