From a25664ee905fe2aaa08c21426d3983be4b2d60f5 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Mar 2026 17:26:30 +0000 Subject: [PATCH 1/2] Initial plan From 155d80d33118bd01252c84b421a128f725cab28e Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 25 Mar 2026 17:31:09 +0000 Subject: [PATCH 2/2] Update README with comprehensive documentation sections Co-authored-by: hechth <12066490+hechth@users.noreply.github.com> Agent-Logs-Url: https://github.com/RECETOX/specdatri_reporting/sessions/51c3fa1e-a7c7-4863-a7dd-060f2ab4dc48 --- README.md | 312 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 287 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 5a75565..ae394c0 100644 --- a/README.md +++ b/README.md @@ -1,46 +1,308 @@ # specdatri_reporting -The code base includes a collection of scripts and GitHub Actions designed to gather various metrics on RECETOX's impact. -## Local development +## Overview + +**specdatri_reporting** is an automated reporting tool that collects and aggregates download and usage statistics for [RECETOX](https://www.recetox.muni.cz/) research software packages. It monitors the adoption and impact of the organisation's scientific software across multiple distribution platforms. + +### What it does + +- Fetches download and traffic data from PyPI, CRAN, Bioconda, and GitHub on a weekly schedule. +- Aggregates raw data into human-readable TSV reports grouped by month (for package downloads) or by week (for GitHub traffic). +- Commits the updated reports back to the repository automatically via GitHub Actions. + +### What kind of data is presented + +| Platform | Metric | Aggregation | +|----------|--------|-------------| +| [PyPI](https://pypi.org/) | Package downloads | Monthly | +| [CRAN](https://cran.r-project.org/) | Package downloads | Monthly | +| [Bioconda](https://bioconda.github.io/) | Package downloads | Monthly | +| [GitHub](https://github.com/) | Repository views | Weekly | +| [GitHub](https://github.com/) | Repository clones | Weekly | + +Generated reports are stored as TSV files under `reports//` and are versioned in the repository. + +### How it works (GitHub Actions) + +A scheduled GitHub Actions workflow (`.github/workflows/actions.yml`) runs every Monday at 00:00 UTC and performs the following steps: + +1. **Checkout** the repository. +2. **Install dependencies** (`pip install -r requirements.txt`). +3. **Collect statistics** by calling `./specdatri collect-stats`, which queries each configured data source and saves raw JSON responses to `tmp/runs//`. +4. **Generate reports** by calling `./specdatri generate-reports`, which reads all collected JSON files and produces aggregated TSV files in `reports//`. +5. **Commit and push** the updated report files back to the `main` branch. + +The workflow can also be triggered manually from the GitHub Actions UI via `workflow_dispatch`. + +--- + +## User Guide + +This section explains how to run the reporting tool manually. + +### Prerequisites + +- Python 3.12+ +- A GitHub personal access token with `repo` scope (`github_token`). Read access is sufficient for collecting traffic statistics manually; write access is required by the automated workflow to push report commits. +- A [PePy](https://pepy.tech/) API key for PyPI statistics (`pepy_x_api_key`). + +### Setup + +```bash +# Clone the repository +git clone https://github.com/RECETOX/specdatri_reporting.git +cd specdatri_reporting + +# Create and activate a virtual environment +python -m venv .venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate + +# Install dependencies +pip install -r requirements.txt + +# Make the entry point executable (Linux/macOS) +chmod +x specdatri + +# Provide API tokens +cp example.env .env +# Edit .env and fill in your tokens: +# github_token="your_github_token" +# pepy_x_api_key="your_pepy_api_key" +``` + +> **Note:** The tool loads `.env` automatically via `python-dotenv`. Alternatively, export the variables in your shell before running any command. + +### CLI commands + +The `specdatri` entry point exposes three subcommands. + +#### 1. Add a repository to track (`add-repo`) + +```bash +# Add a package tracked on PyPI and GitHub +./specdatri add-repo --project mypackage --pypi --github + +# Add a package with a custom repository path and all sources +./specdatri add-repo --project mypackage --repository OWNER/mypackage --pypi --bioconda --cran --github + +# Add an R package tracked on CRAN only +./specdatri add-repo --project MyRPackage --cran +``` + +Options: + +| Option | Description | Default | +|--------|-------------|---------| +| `--project TEXT` | Project name (required) | — | +| `--repository TEXT` | GitHub repo path `OWNER/REPO` | `RECETOX/` | +| `--repository-list PATH` | Path to the tracking list | `./repository_list.tsv` | +| `--pypi` | Track PyPI downloads | off | +| `--bioconda` | Track Bioconda downloads | off | +| `--cran` | Track CRAN downloads | off | +| `--github` | Track GitHub views and clones | off | + +#### 2. Collect statistics (`collect-stats`) + +```bash +# Collect stats for all configured packages +./specdatri collect-stats + +# Use custom paths +./specdatri collect-stats --repository-list custom_list.tsv --tmp-dir data/tmp +``` + +Output: JSON files in `tmp/runs//` named using the pattern +`{timestamp}___{project}___{package}___{source}___{action}.json`. + +#### 3. Generate reports (`generate-reports`) + +```bash +# Generate reports for the current year +./specdatri generate-reports + +# Generate reports for a specific year +./specdatri generate-reports --year 2025 + +# Use custom paths +./specdatri generate-reports --year 2025 --tmp-dir data/tmp --output-dir data/reports +``` + +Output: five TSV files per year in `reports//`: + +- `pypi_downloads.tsv` +- `bioconda_downloads.tsv` +- `cran_downloads.tsv` +- `github_views.tsv` +- `github_clones.tsv` + +### Full manual workflow example + +```bash +# 1. Add packages (one-time setup) +./specdatri add-repo --project matchms --pypi --bioconda --github +./specdatri add-repo --project spec2vec --pypi --bioconda --github + +# 2. Collect statistics +./specdatri collect-stats + +# 3. Generate reports +./specdatri generate-reports --year 2025 +``` + +--- + +## Developer Guide ### Project setup -It is assumed you can clone and change directories into the development repo. -Create a virtualenv or conda environment (whatever your poison). -Once in the repos directory, activate your env then run the following command to install the needed python libraries. +```bash +# Install development dependencies (includes pre-commit hooks) +pip install -r ./requirements/local.txt + +# Install pre-commit hooks +pre-commit install +``` + +### Running tests + +```bash +# Run all tests +python -m unittest discover -s tests + +# Run with coverage +coverage run -m unittest discover -s tests +coverage report -m +coverage html # generates htmlcov/index.html +``` + +### Testing GitHub Actions locally + +You need [act](https://nektosact.com/) to simulate GitHub Actions on your machine. + +```bash +# Simulate the weekly schedule trigger +act --secret-file .env schedule +``` + +> **Important notes when testing with `act`:** +> +> - Use a token **without** push permissions so that test data is never written to the production branch. +> - The push step will fail if branch protection rules are active — this is expected behaviour. +> - Files created inside the Docker container are **not** written to your local filesystem. +> - Never push directly to `main`; always work on a feature branch. + +### Adding new repositories + +Use the `add-repo` CLI command (see [User Guide](#1-add-a-repository-to-track-add-repo)) and then commit the updated `repository_list.tsv`: + +```bash +./specdatri add-repo --project newpackage --pypi --github +git add repository_list.tsv +git commit -m "Track newpackage on PyPI and GitHub" +``` + +The `repository_list.tsv` file is a tab-separated table with the columns: + +| Column | Description | +|--------|-------------| +| `repository` | GitHub repository path (`OWNER/REPO`) | +| `project` | Human-readable project identifier | +| `package` | Package name on the distribution platform | +| `source` | One of: `pypi`, `bioconda`, `CRAN`, `GitHub` | +| `action` | One of: `downloads`, `views`, `clones` | + +### Integrating a new data source + +Data sources live in `src/data_sources/` and inherit from the abstract `DataSource` base class in `src/data_sources/base.py`. + +To add a new source: + +1. **Create a new module** `src/data_sources/.py` and implement the `DataSource` interface: + + ```python + from .base import DataSource + + class MyNewDataSource(DataSource): + def fetch(self, package: str, project: str) -> dict: + # Call the external API and return raw data as a dict + ... + ``` + +2. **Register the source** in `src/cli.py` by mapping the source identifier to the new class, following the same pattern as existing sources. + +3. **Add a new source flag** to the `add-repo` command if users need to opt in to this source. + +4. **Write tests** in `tests/` following the patterns in `tests/test_data_sources.py`. + +See `src/data_sources/README.md` for a detailed description of the data source architecture. + +### Integrating a new report type + +Report generators live in `src/reports/` and inherit from the abstract `ReportGenerator` base class in `src/reports/base.py`. + +To add a new report: + +1. **Create a new module** `src/reports/.py` and implement the `ReportGenerator` interface: + + ```python + from .base import ReportGenerator + + class MyNewReportGenerator(ReportGenerator): + source = "mysource" + action = "downloads" + period = "monthly" # or "weekly" + + def aggregate(self, data: list[dict]) -> dict: + # Aggregate raw records into period → count mapping + ... + ``` + +2. **Register the generator** in `src/cli.py` so that `generate-reports` picks it up automatically. + +3. **Write tests** in `tests/` following the patterns in `tests/test_report.py`. + +See `src/reports/README.md` for a detailed description of the report generation architecture. + +--- + +## Adopting This Framework -> pip install -r ./requirements/local.txt +If you want to set up a similar automated download-statistics tracking system for your own organisation's repositories, follow these steps: -### Simulating Github Actions +### 1. Fork or copy the repository -You need [act](https://nektosact.com/) to test your code in development mode. -Install act for your chosen OS. -At your terminal, run (This simulates a GitHub action on your local device): +Fork this repository into your GitHub organisation (or copy the relevant files into a new repository). -> act --secret-file .env schedule +### 2. Configure the packages to track -### Things to note +Edit `repository_list.tsv` (or use `./specdatri add-repo`) to list your own packages and their distribution channels. -1: Do not push local development changes from `tmp` folder and `reports` folder. In fact do not edit them at all !!! +### 3. Configure GitHub Actions secrets -2: When testing with `act` do not use a token that has the permission to make push requests else your test data will mess with "production" data. +In your repository's **Settings → Secrets and variables → Actions**, add: -3: When testing with `act` know that the push may fail due to the fact that you can't directly push to main. +| Secret | Description | +|--------|-------------| +| `RECEBOT_REPORTING_TOKEN` | A GitHub personal access token (PAT) with `repo` scope. Used to read traffic data from the GitHub API and push report commits. Consider using a dedicated bot account. | +| `pepy_x_api_key` | Your PePy API key for PyPI download statistics. | -4: Always, I repeat always devlop on another branch not main and never push directly to main. +> The `github_token` secret used by the push step is the built-in `secrets.github_token` provided by GitHub Actions and does not need to be configured manually. -5: You need tokens to test the code locally, place said tokens in `example.env` and change the filename to `.env` +### 4. Adjust the workflow schedule (optional) -6: When testing with act files will be created in the docker image but never written to your file system. +Edit `.github/workflows/actions.yml` and change the `cron` expression to your preferred schedule: -### Running tests: +```yaml +schedule: + - cron: '0 0 * * 1' # Every Monday at 00:00 UTC +``` -#### Running with unittest -> python -m unittest discover -s tests +### 5. Remove or adapt RECETOX-specific content -#### Running with coverage -> coverage run -m unittest discover -s tests +- Update the repository description and any references to RECETOX in the README. +- Remove or replace entries in `repository_list.tsv` with your own packages. +- Clear the `reports/` directory (or delete historical data you do not need). -> coverage report -m +### 6. Protect the default branch (recommended) -> coverage html +Enable branch protection on `main` to prevent accidental direct pushes. The GitHub Actions workflow pushes via the `ad-m/github-push-action` action using the workflow token, which is exempt from push restrictions when `secrets.github_token` is used.