python-ingest

Python ingestion with polars and pandas

This cli script is set to be able to fetch the CSV datasets for NYC Yellow Trip Data, Green Trip Data, and Lookup Zones based on the endpoints in datasets.yaml.

Getting Started

1. Install dependencies from pyproject.toml and activate the created virtualenv:

uv sync && source .venv/bin/activate

2. (Optional) Install pre-commit:

brew install pre-commit

# From root folder where `.pre-commit-config.yaml` is located, run:
pre-commit install

3. Spin-up the nyc-tlc-db:

docker compose up -d nyc-tlc-db

4. Export ENV VARS to connect to DB:

export DB_HOST=localhost
export DB_NAME=nyc_tlc
export DB_USERNAME=postgres
export DB_PASSWORD=postgres

5. Run the script with the intended flags or use --help:

tlc ingest -y or --yellow:

• fetches the datasets under the key yellow_taxi_trip_data only
• persists to Postgres, on table yellow_taxi_trips

tlc ingest -g or --green:

• fetches the datasets under the key green_taxi_trip_data only,
• persists to Postgres, on table green_taxi_trips

tlc ingest -f or --fhv:

• fetches the datasets under the key fhv_trip_data
• persists to Postgres, on table: fhv_trips

tlc ingest -z or --zones:

• fetches the datasets under the key zone_lookups
• persists to Postgres, on table: zone_lookup

Additionally, you can use --use-polars for a major speed boost with Polars.

You can use any combination of options above to fetch more than dataset group at a time. For instance: tlc ingest -gz --use-polars fetches the NYC GreenTaxi Trip Data and NYC Lookup Zones while using Polars as the Dataframe library.

Containerization

1. Build the Docker Image with:

docker build -t nyc-tlc-cli:latest . --no-cache

2. Start a container with it:

docker run -d --rm \
  -e DB_HOST=host.docker.internal \
  -e DB_PORT=5432 \
  -e DB_NAME=nyc_tlc \
  -e DB_USERNAME=postgres \
  -e DB_PASSWORD=postgres \
  --name tlc-cli \
  iobruno/tlc-cli:latest

TODO's:

• PEP-517: Packaging and dependency management with uv
• Build a CLI app with Typer
• Progress Bars to keep track of the execution with rich
• Re-Implement the pipeline with Polars
• Containerize it and run app on Docker
• Fix Dockerfile to add support for Podman