|
1 | | -Monorepo containing all the libraries, applications, terraform and github actions required to build/test/deploy/release the PolicyEngine api V2. |
2 | | - |
3 | | -# Local Development Quick Start |
4 | | -* [install poetry](https://python-poetry.org/docs/#installation) |
5 | | -* ``make build`` - install and pytest all libraries and projects. |
6 | | - |
7 | | -# Cloud Development Quick Start |
8 | | -__NOTE: MOST development should be possible locally. Deployment is slow and hard to debug. Change with caution__ |
9 | | - |
10 | | -* One time setup - this will create a new project in your GCP account you can deploy the api to. |
11 | | - * Create a gcp account _with organization_ |
12 | | - * authenticate with gcloud as a user with permission to create projects. |
13 | | - * ``gcloud auth application-default login`` |
14 | | - * Have your organization ID and billing account number handy. |
15 | | - * ``cd terraform/infra-policyengine-api && make bootstrap`` |
16 | | - * You should now have a terraform/.bootstrap_settings folder containing your project settings. |
17 | | -* build the api docker image |
18 | | - * ``cd projects/policyengine-api-full && make deploy`` |
19 | | - * There should now be a new hash under the tag "desktop" in the project artifact repository. |
20 | | -* deploy to the cloud |
21 | | - * ``cd terraform/project-policyengine-api-full && make deploy`` |
22 | | - * The cloudrun service should now be running using the latest image version of the tag "desksop" from your project artifact respository |
23 | | - |
24 | | -# Github Deploy to Cloud Quick Start |
25 | | - |
26 | | -__checkout a clean version of the repository__ you cannot bootstrap more than one project |
27 | | -in a workspace at a time. |
28 | | - |
29 | | -* bootstrap the beta project |
30 | | - * have your github repo owner id and repo (i.e. org/repo) ready |
31 | | - * have your GCP organization id and billing account number ready |
32 | | - * log into your gcp account via gcloud as a user able to create projects. |
33 | | - * ``cd terraform/project-poicyengine-api && make bootstrap_beta`` |
34 | | -* WAIT FOR AT LEAST AN HOUR (permissions configuration sometimes takes up to an hour for the github federation.) You may get errors about lack of permission (or possibly resource) for thinkgs like the deployment state bucket. |
35 | | -* configure github |
36 | | - * create a new environment in your github repo settings called "beta" and, using the ouput of the bootstrap, configure the following values |
37 | | - * REGION (generally us-central1) |
38 | | - * PROJECT_ID (in the output of the bootstrap target) |
39 | | - * _GITHUB_IDENTITY_POOL_PROVIDER_NAME (in the output of the bootstrap build target) |
40 | | - * ORG_ID |
41 | | - * BILLING_ACCOUNT |
| 1 | +# PolicyEngine API v2 |
| 2 | + |
| 3 | +Monorepo for PolicyEngine's API infrastructure, containing all services, libraries, and deployment configuration. |
| 4 | + |
| 5 | +## Quick start |
| 6 | + |
| 7 | +### Prerequisites |
| 8 | + |
| 9 | +- Docker and Docker Compose |
| 10 | +- Python 3.13+ |
| 11 | +- [uv](https://docs.astral.sh/uv/) package manager |
| 12 | +- gcloud CLI (for deployment) |
| 13 | +- Terraform 1.5+ (for deployment) |
| 14 | + |
| 15 | +### Local development |
| 16 | + |
| 17 | +Start all services: |
| 18 | +```bash |
| 19 | +make up # Start services on ports 8081-8083 |
| 20 | +make logs # View logs |
| 21 | +make down # Stop services |
| 22 | +``` |
| 23 | + |
| 24 | +Run the test suite: |
| 25 | +```bash |
| 26 | +make test # Unit tests only |
| 27 | +make test-integration-with-services # Full integration tests (manages services automatically) |
| 28 | +make test-complete # Everything: unit + integration tests |
| 29 | +``` |
| 30 | + |
| 31 | +## Architecture |
| 32 | + |
| 33 | +The repository contains three main API services: |
| 34 | + |
| 35 | +- **api-full** (port 8081): Main PolicyEngine API with household calculations |
| 36 | +- **api-simulation** (port 8082): Economic simulation engine |
| 37 | +- **api-tagger** (port 8083): Cloud Run revision management |
| 38 | + |
| 39 | +Each service generates OpenAPI specs and Python client libraries for integration testing. |
| 40 | + |
| 41 | +## Development workflow |
| 42 | + |
| 43 | +### Making changes |
| 44 | + |
| 45 | +1. Edit code locally - services hot-reload automatically when running via `make up` |
| 46 | +2. Run tests: `make test-complete` |
| 47 | +3. Commit changes to a feature branch |
| 48 | +4. Open a PR - GitHub Actions will run tests automatically |
| 49 | + |
| 50 | +### Testing |
| 51 | + |
| 52 | +Unit tests run in isolated containers: |
| 53 | +```bash |
| 54 | +make test # All services |
| 55 | +make test-service service=api-full # Single service |
| 56 | +``` |
| 57 | + |
| 58 | +Integration tests use generated client libraries: |
| 59 | +```bash |
| 60 | +make generate-clients # Generate OpenAPI clients (done automatically by test commands) |
| 61 | +make test-integration # Run integration tests (requires services running) |
| 62 | +``` |
| 63 | + |
| 64 | +### Project structure |
| 65 | + |
| 66 | +``` |
| 67 | +/ |
| 68 | +├── projects/ # Service applications |
| 69 | +│ ├── policyengine-api-full/ |
| 70 | +│ ├── policyengine-api-simulation/ |
| 71 | +│ ├── policyengine-api-tagger/ |
| 72 | +│ └── policyengine-apis-integ/ # Integration tests |
| 73 | +├── libs/ # Shared libraries |
| 74 | +│ └── policyengine-fastapi/ # Common FastAPI utilities |
| 75 | +├── deployment/ # Deployment configuration |
| 76 | +│ ├── docker-compose.yml # Local development |
| 77 | +│ ├── docker-compose.prod.yml # Production builds |
| 78 | +│ └── terraform/ # Infrastructure as code |
| 79 | +├── scripts/ # Utility scripts |
| 80 | +└── .github/workflows/ # CI/CD pipelines |
| 81 | +``` |
| 82 | + |
| 83 | +## Deployment |
| 84 | + |
| 85 | +### Setting up a new GCP project |
| 86 | + |
| 87 | +**Important**: Most development should be done locally. Cloud deployment is slow and harder to debug. |
| 88 | + |
| 89 | +1. Configure environment: |
| 90 | +```bash |
| 91 | +cp deployment/.env.example deployment/.env |
| 92 | +# Edit deployment/.env with your GCP project details |
| 93 | +``` |
| 94 | + |
| 95 | +2. Deploy infrastructure: |
| 96 | +```bash |
| 97 | +make deploy # Builds images, pushes to registry, runs terraform |
| 98 | +``` |
| 99 | + |
| 100 | +For existing GCP projects with resources: |
| 101 | +```bash |
| 102 | +make terraform-import # Import existing resources |
| 103 | +./deployment/terraform/handle-existing-workflows.sh $PROJECT_ID --delete # Handle workflows |
| 104 | +``` |
| 105 | + |
| 106 | +See [deployment guide](deployment/DEPLOYMENT_GUIDE.md) for detailed instructions. |
| 107 | + |
| 108 | +### GitHub Actions deployment |
| 109 | + |
| 110 | +The repository includes automated deployment pipelines: |
| 111 | + |
| 112 | +1. **Pull requests**: Runs tests and builds |
| 113 | +2. **Merge to main**: |
| 114 | + - Deploys to beta environment |
| 115 | + - Runs integration tests |
| 116 | + - Deploys to production |
| 117 | + |
| 118 | +Configure GitHub environments with these variables: |
| 119 | +- `PROJECT_ID`: GCP project ID |
| 120 | +- `REGION`: GCP region (usually us-central1) |
| 121 | +- `_GITHUB_IDENTITY_POOL_PROVIDER_NAME`: Workload identity provider |
| 122 | + |
| 123 | +### Important notes from experience |
| 124 | + |
| 125 | +- **Wait after bootstrap**: GCP permission propagation can take up to an hour |
| 126 | +- **Workflows can't be imported**: Use the provided script to handle existing workflows |
| 127 | +- **Always test locally first**: Cloud debugging is painful |
| 128 | +- **Check terraform state**: If deployments fail, check if resources already exist |
| 129 | + |
| 130 | +## Commands reference |
| 131 | + |
| 132 | +### Development |
| 133 | +- `make up` - Start services locally |
| 134 | +- `make down` - Stop services |
| 135 | +- `make logs` - View service logs |
| 136 | +- `make build` - Build Docker images |
| 137 | + |
| 138 | +### Testing |
| 139 | +- `make test` - Run unit tests |
| 140 | +- `make test-integration` - Run integration tests |
| 141 | +- `make test-complete` - Run all tests with service management |
| 142 | +- `make generate-clients` - Generate API client libraries |
| 143 | + |
| 144 | +### Deployment |
| 145 | +- `make deploy` - Full deployment to GCP |
| 146 | +- `make terraform-plan` - Preview infrastructure changes |
| 147 | +- `make terraform-import` - Import existing resources |
| 148 | +- `make terraform-destroy` - Remove all infrastructure |
| 149 | + |
| 150 | +## Troubleshooting |
| 151 | + |
| 152 | +### Services won't start |
| 153 | +- Check Docker is running |
| 154 | +- Ensure ports 8081-8083 are free |
| 155 | +- Run `make build` to rebuild images |
| 156 | + |
| 157 | +### Integration tests fail |
| 158 | +- Regenerate clients: `make generate-clients` |
| 159 | +- Check services are healthy: `make logs` |
| 160 | +- Verify port configuration matches docker-compose.yml |
| 161 | + |
| 162 | +### Deployment issues |
| 163 | +- Check deployment/.env configuration |
| 164 | +- Verify GCP authentication: `gcloud auth list` |
| 165 | +- For "already exists" errors: `make terraform-import` |
| 166 | +- For workflow errors: `./deployment/terraform/handle-existing-workflows.sh` |
| 167 | + |
| 168 | +## Contributing |
| 169 | + |
| 170 | +1. Create a feature branch |
| 171 | +2. Make changes and test locally |
| 172 | +3. Ensure `make test-complete` passes |
| 173 | +4. Open a PR with a clear description |
| 174 | +5. Wait for CI checks to pass |
0 commit comments