prefect

Workflow orchestration with Prefect

Getting Started

1. Install dependencies from pyproject.toml and activate the created virtualenv:

uv sync && source .venv/bin/activate

2. (Optional) Install pre-commit:

brew install pre-commit

# From root folder where `.pre-commit-config.yaml` is located, run:
pre-commit install

3. Start the Prefect Server:

Note: The Prefect Orion server is now called Prefect Server

prefect server start

4. Setup Prefect server for the flows:

prefect config set PREFECT_API_URL=http://localhost:4200/api

Prefect Flows

flows/web_csv_to_gcs.py:

Set the GOOGLE_APPLICATION_CREDENTIALS env variable:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcs_credentials.json

Then, execute with:

python flows/web_cs_to_gcs.py

flows/sqlalchemy_ingest.py:

Set the environment variables:

export DB_USERNAME=postgres
export DB_PASSWORD=postgres
export DB_HOST=localhost
export DB_PORT=5432
export DB=nyc_taxi

Now, execute it with:

python flows/sqlalchemy_ingest.py

TODO's:

• PEP-517: Packaging and dependency management with uv
• Deploy Prefect Server / Agent on Docker
• Code format/lint with Ruff
• Run Prefect flows on Docker