github-tools

Note

This repo is actively developed. Main branch may be rewritten.

Tools for extracting data from GitHub, storing it in a local SQLite database, querying it, and visualizing trends.

Directory Layout

src/             # Rust source (single binary: github-tools)
scripts/util/    # Python: plot.py (charts), json_to_csv.py (utility)
data/            # Committed snapshots: JSON inputs and PNG charts
  {owner}_{repo}/issues/  # Issues/PRs JSON split by year (2024.json, 2025.json, ...)
  images/        # Committed PNG charts (promoted from out/images/)
out/             # Gitignored — all generated and local-only files
  historical/    # Raw JSON fetched from GitHub API
  db/            # SQLite databases
  summaries/     # Generated CSVs
  images/        # Generated PNG charts (promote to data/images/ to commit)
  purge/         # Purge audit logs (local only)

Build

cargo build --release
# Binary: target/release/github-tools

Configuration

Most commands take an --env-file pointing to a .env file:

GITHUB_TOKEN=...
REPO_OWNER=vectordotdev
REPO_NAME=vector
DOCKER_USERNAME=...   # purge commands only
DOCKER_PASSWORD=...   # purge commands only

Commands

github-tools <COMMAND>

Fetch:
  fetch-all          Fetch issues + discussions for all repos (workflow)
  fetch-issues       Fetch all issues/PRs for a repository
  fetch-discussions  Fetch all discussions for a repository
  fetch-labels       Fetch all labels for a repository

Pipeline:
  generate-all       Build DB + summaries for all repos (workflow)
  build-db           Load issues JSON into SQLite database
  generate-summaries Generate CSV summaries from SQLite database

Purge:
  purge-all          Run all purge operations (workflow)
  purge nightly      Purge old nightly images from GitHub and Docker Hub
  purge untagged     Purge untagged GitHub container images
  purge vector-dev   Purge old vector-dev images from Docker Hub

Maintenance:
  close-old-prs          Close PRs with 'meta: awaiting author' older than 6 months
  delete-stale-branches  Delete branches with no commits in 4 years
  remove-legacy-label    Remove legacy type labels from issues/PRs

Run github-tools <COMMAND> --help for full argument details.

Workflow

1. (Optional) Fetch fresh data from GitHub

github-tools fetch-all --env-file vector.env --env-file vrl.env --env-file quickwit.env

Writes to out/historical/. The fetched JSON must be split by year and promoted to data/ to commit as a snapshot. Issues/PRs are stored in data/{owner}_{repo}/issues/{year}.json.

2. Generate DB, summaries, and charts

github-tools generate-all \
  --env-file vector.env --env-file vrl.env --env-file quickwit.env \
  --exclude-labels "no-changelog,meta: awaiting author"

# Charts (still Python):
python -m scripts.util.plot --env-file vector.env --input-dir out/summaries \
  --start $(date -d "$(date +%Y-%m-01) -12 months" +%Y-%m) \
  --exclude-labels "no-changelog,meta: awaiting author"

Promote charts to commit:

cp out/images/*.png data/images/

3. (Optional) Purge stale container images

github-tools purge-all --env-file vector.env --dry-run
github-tools purge-all --env-file vector.env  # omit --dry-run to execute

Audit logs written to out/purge/ (local only).

Trends

Per-repo trend pages with all charts:

• Vector
• VRL
• Quickwit

Note

Issues and PRs with the following labels are excluded from all charts: no-changelog, meta: awaiting author.