Skip to content

Commit cd035ad

Browse files
CalinLCalinL
committed
Add comprehensive documentation on low-volume private Python package management alternatives on GitHub
1 parent a300331 commit cd035ad

2 files changed

Lines changed: 627 additions & 0 deletions

File tree

‎docs/19-python-package-management-low-volume-alternatives.md‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
# Low-volume private Python packages on GitHub — alternatives to JFrog and SaaS
2+
3+
*Research compiled 2026-04-21. Backing reports in `./files/research-low-volume-python-hosting.md` (Claude Sonnet 4.6) and `./files/research-private-actions-poetry.md` (GPT-5.3-Codex).*
4+
5+
**Scenario:** 5–20 private Python packages, ~10 pushes/month, ~100 installs/month. GitHub-hosted org, Poetry + pip. You want to avoid JFrog/Cloudsmith/Gemfury costs at this scale.
6+
7+
---
8+
9+
## TL;DR — answer to both your questions
10+
11+
### Q1 — "Can we use ACR or similar instead of JFrog?"
12+
13+
**Yes, several viable alternatives exist for low volume**, ranked best-to-worst for your situation:
14+
15+
| Rank | Option | Monthly cost | pip-native? | OIDC? | Verdict |
16+
|---|---|---|---|---|---|
17+
| 🥇 **1** | **Azure Artifacts free tier (2 GiB/org)** | **$0** | ✅ | ❌ (PAT only) | **Cheapest legitimate registry. If you tolerate a PAT, this is the simplest "just works" path.** |
18+
| 🥈 **2** | **Azure Blob Storage as a static PEP 503 index** | **<$0.10** | ✅ | ✅ (push) / SAS (pull) | **Cheapest pip-native option. Real simple index, no glue code. Best if you're Azure-shop and OK with SAS token rotation.** |
19+
| 🥉 **3** | **Self-hosted pypiserver or devpi** on Azure Container Apps / App Service free tier | **$0–$15** | ✅ | ❌ (basic auth) | **Only if you want full PyPI semantics + OSS + self-host, and you're OK operating a tiny service.** |
20+
| 4 | **AWS S3 static index (`s3-pypi` / `dumb-pypi`)** | <$0.10 | ✅ | ✅ (push) | AWS equivalent of #2; auth on read is weaker (pip doesn't speak SigV4) |
21+
| 5 | **ACR + ORAS** (OCI artifacts) | ~$5 (Basic) | ❌ | ✅ | **Only pick this if you already own ACR.** `pip` can't install OCI artifacts directly — you need a glue step every install. Not recommended as a primary pattern. |
22+
| 6 | **GHCR + ORAS** | ~$0 | ❌ | ✅ | Same limitation as ACR. The "GitHub-native" appeal is real but pip can't consume it. |
23+
| 7 | **Poetry git-deps + private composite Action** | **$0** | ❌ (not a registry) | ✅ | **Great for 1–5 tiny libs. Breaks down past that.** Details in Q2. |
24+
25+
### Q2 — "Can we use a private GitHub Action with Poetry to install private Python packages?"
26+
27+
**Yes — and for your volume (few packages), this is actually a serious contender.** A private **composite Action** in a private repo can be called by every other repo in the org. It bundles `setup-python + install-poetry + configure-git-auth + poetry-install` into one reusable step. Combined with `git+https://` deps in `pyproject.toml`, this gives you a registry-free workflow that costs $0.
28+
29+
**But it is not a replacement for a package registry.** It's an orchestration/boilerplate-reduction pattern. No dependency resolution across private libs that depend on each other, no proxying of public PyPI, no retention policy, install latency grows with N.
30+
31+
**Upgrade signal:** when you exceed ~5 private packages, or private packages start depending on each other, move to a real index (Azure Artifacts free tier, Azure Blob static index, or Cloudsmith).
32+
33+
---
34+
35+
## Option 1 — Azure Artifacts free tier *(my top pick for your profile)*
36+
37+
**Why this wins:**
38+
- 2 GiB free per Azure DevOps organization — easily covers 5–20 small Python packages for years.
39+
- Real PyPI-compatible feed. pip, Poetry, and twine work without glue.
40+
- Upstream proxying of public PyPI is built in (so consumers get one index URL).
41+
- Microsoft-maintained; no ops burden.
42+
43+
**Trade-offs you must accept:**
44+
- You're creating an Azure DevOps org even if you don't use ADO otherwise. (Free; 5 min setup.)
45+
- Auth is PAT-based for pip/Poetry workflows. GitHub OIDC → Azure managed identity → Azure DevOps personal access token is doable but requires a Microsoft token-exchange step.
46+
47+
**Poetry config:**
48+
```toml
49+
[[tool.poetry.source]]
50+
name = "internal"
51+
url = "https://pkgs.dev.azure.com/<ORG>/<PROJECT>/_packaging/<FEED>/pypi/simple/"
52+
priority = "explicit" # force internal packages to come from here
53+
```
54+
55+
**Publish (GitHub Actions):**
56+
```yaml
57+
- run: poetry config http-basic.internal "" "${{ secrets.ADO_PAT }}"
58+
- run: poetry publish --repository internal --build
59+
```
60+
61+
**Verdict:** If you don't have a strong reason not to touch Azure DevOps, **start here**. It's the lowest-effort legitimate registry available at your scale.
62+
63+
---
64+
65+
## Option 2 — Azure Blob Storage as a static PEP 503 index
66+
67+
**Why this works:**
68+
- You generate a tiny HTML index (PEP 503 "simple" format) from your wheel filenames and upload it + the wheels to a private blob container.
69+
- `pip` and Poetry consume it **natively** via `--index-url` + a SAS token.
70+
- Cost at your volume: **well under $0.10/month.**
71+
- Push side uses GitHub OIDC → Azure managed identity (no stored secrets).
72+
73+
**The index-regeneration workflow:**
74+
```yaml
75+
name: publish-wheel-to-blob
76+
on:
77+
push:
78+
tags: ["v*"]
79+
80+
permissions:
81+
id-token: write
82+
contents: read
83+
84+
jobs:
85+
publish:
86+
runs-on: ubuntu-latest
87+
steps:
88+
- uses: actions/checkout@v6
89+
- uses: actions/setup-python@v6
90+
with: { python-version: "3.12" }
91+
- run: pip install poetry dumb-pypi
92+
- run: poetry build --format wheel
93+
- uses: azure/login@v2
94+
with:
95+
client-id: ${{ secrets.AZ_CLIENT_ID }}
96+
tenant-id: ${{ secrets.AZ_TENANT_ID }}
97+
subscription-id: ${{ secrets.AZ_SUBSCRIPTION_ID }}
98+
# Upload wheels
99+
- run: az storage blob upload-batch
100+
--account-name myorgpypi --destination 'wheels'
101+
--source dist --overwrite --auth-mode login
102+
# Regenerate the simple index from the container contents
103+
- run: |
104+
dumb-pypi --package-list <(az storage blob list \
105+
--account-name myorgpypi --container-name wheels \
106+
--auth-mode login --query '[].name' -o tsv) \
107+
--packages-url https://myorgpypi.blob.core.windows.net/wheels \
108+
--output-dir ./index
109+
az storage blob upload-batch \
110+
--account-name myorgpypi --destination 'simple' \
111+
--source ./index --overwrite --auth-mode login
112+
```
113+
114+
**Consumer `pyproject.toml`:**
115+
```toml
116+
[[tool.poetry.source]]
117+
name = "internal"
118+
url = "https://myorgpypi.blob.core.windows.net/simple/?<SAS_TOKEN>"
119+
priority = "explicit"
120+
```
121+
122+
**Trade-offs:**
123+
- SAS token rotation is your responsibility. Roll it via a GitHub secret update every 90 days.
124+
- No retention/cleanup policy unless you script it.
125+
- No audit log of who pulled what (Azure Monitor on blob access can provide this).
126+
127+
**Verdict:** Cheapest pip-native option. Excellent if you're Azure-shop and want near-zero cost with real PEP 503 semantics.
128+
129+
---
130+
131+
## Option 3 — Private composite GitHub Action + Poetry git-deps
132+
133+
This directly answers your second question.
134+
135+
### Architecture
136+
- Your private libraries are individual private GitHub repos with a normal `pyproject.toml`.
137+
- Consumer repos declare them as git-deps in `pyproject.toml`, pinning by commit SHA.
138+
- A single **private composite Action** in e.g. `your-org/.github-actions/poetry-install-private` encapsulates setup + auth so every consumer repo just calls one step.
139+
140+
### `action.yml` (private composite action)
141+
```yaml
142+
name: "Poetry install (private git deps)"
143+
description: "setup-python + poetry + git auth + poetry install, for repos with private GitHub git deps"
144+
145+
inputs:
146+
python-version: { default: "3.12", required: false }
147+
poetry-version: { default: "1.8.4", required: false }
148+
github-token: { required: true, description: "Token with Contents:Read on dep repos" }
149+
working-directory: { default: ".", required: false }
150+
151+
runs:
152+
using: "composite"
153+
steps:
154+
- uses: actions/setup-python@v6
155+
with: { python-version: ${{ inputs.python-version }} }
156+
157+
- shell: bash
158+
run: |
159+
python -m pip install --upgrade pip
160+
python -m pip install "poetry==${{ inputs.poetry-version }}"
161+
162+
# Rewrite git+https URLs to use the installation token transparently
163+
- shell: bash
164+
run: |
165+
git config --global \
166+
url."https://x-access-token:${{ inputs.github-token }}@github.com/".insteadOf \
167+
"https://github.com/"
168+
169+
- shell: bash
170+
working-directory: ${{ inputs.working-directory }}
171+
run: poetry install --no-interaction --no-ansi
172+
```
173+
174+
### Consumer `pyproject.toml`
175+
```toml
176+
[tool.poetry.dependencies]
177+
python = ">=3.11,<4.0"
178+
acme-lib = { git = "https://github.com/your-org/acme-lib.git", rev = "3b6c5f9a..." }
179+
acme-core = { git = "https://github.com/your-org/acme-core.git", rev = "a1b2c3d4..." }
180+
```
181+
182+
**Always pin to full commit SHAs in production.** Tags are mutable on GitHub.
183+
184+
### Caller workflow in a consumer repo
185+
```yaml
186+
name: ci
187+
on: [push, pull_request]
188+
189+
jobs:
190+
test:
191+
runs-on: ubuntu-latest
192+
steps:
193+
- uses: actions/checkout@v6
194+
# Mint a GitHub App token with Contents:Read on the dep repos
195+
- uses: actions/create-github-app-token@v3
196+
id: app-token
197+
with:
198+
client-id: ${{ vars.APP_CLIENT_ID }}
199+
private-key: ${{ secrets.APP_PRIVATE_KEY }}
200+
owner: your-org
201+
repositories: "acme-lib,acme-core"
202+
- uses: your-org/.github-actions/poetry-install-private@v1
203+
with:
204+
github-token: ${{ steps.app-token.outputs.token }}
205+
- run: poetry run pytest
206+
```
207+
208+
### Private-action access setup (one-time)
209+
- In the action's source repo: **Settings → Actions → General → Access → "Accessible from repositories owned by your-org"**.
210+
- Pin to a tag (`@v1`), not a branch, so consumers don't drift.
211+
212+
### When to use this pattern
213+
✅ Use it if: ≤5 private packages, no inter-library dependencies, infrequent releases, you want zero infrastructure.
214+
❌ Upgrade away from it when:
215+
- Private packages start depending on each other (lockfile grows linearly per git dep).
216+
- Install latency in CI exceeds ~30 seconds from git clones.
217+
- You need to proxy/cache public PyPI.
218+
- You need retention policies, RBAC on package scope, or audit logs.
219+
- Dependency-confusion-style attacks become a realistic threat.
220+
221+
**Verdict:** **Legitimate pattern for your low-volume case.** It's what many early-stage teams use before they have pain. Just know when to graduate.
222+
223+
---
224+
225+
## Why ACR/GHCR via ORAS is *not* a good primary choice
226+
227+
Both Azure Container Registry and GitHub Container Registry are OCI-compliant (v1.1) and can store wheels as OCI artifacts using ORAS. This sounds appealing ("reuse the container registry I already have!"), but:
228+
229+
- **`pip` cannot install OCI artifacts.** There is no PEP for OCI-as-package-source. Every install needs an explicit `oras pull → pip install ./dist/*.whl` shell step.
230+
- **No dependency resolution.** If package A depends on package B (both internal, both in ACR), `pip` has no way to walk that graph via OCI — you'd have to pre-pull every transitive dep.
231+
- **No proxying of public PyPI.** So you still need a separate path for all your public deps.
232+
- **Monthly cost for ACR Basic ~$5/mo** gets you nothing you couldn't get from Azure Blob (Option 2) for pennies, with a working simple index that pip actually understands.
233+
234+
ACR/GHCR + ORAS works for **container images and Helm charts**, where OCI is the natural format. For Python wheels it's a force-fit. Only consider it if you already have a strong OCI/supply-chain workflow that demands artifacts live alongside container images.
235+
236+
---
237+
238+
## Decision flow for your profile
239+
240+
```
241+
You have ≤5 private Python packages, no inter-library deps, and you want zero infra cost?
242+
└── Use Option 3 (private composite Action + Poetry git-deps). $0.
243+
244+
You want the simplest "just works" registry, and you can tolerate creating an Azure DevOps org + a PAT?
245+
└── Use Option 1 (Azure Artifacts free tier). $0, pip-native, fully managed.
246+
247+
You want the cheapest pip-native option in Azure, with OIDC on publish, and you accept SAS rotation?
248+
└── Use Option 2 (Azure Blob static PEP 503 index). <$0.10/mo.
249+
250+
You want OSS + self-host + full PyPI semantics?
251+
└── Use Option 6 (pypiserver or devpi on Azure Container Apps free tier). $0–$15/mo.
252+
253+
Your package count or inter-dependency complexity grew past ~5 packages?
254+
└── Graduate to Cloudsmith or Azure Artifacts paid tier, or JFrog if you truly need enterprise features.
255+
```
256+
257+
---
258+
259+
## My recommendation for your stated scenario
260+
261+
Given "few private Python packages, low volume, migrating from Nexus OSS, already on GitHub":
262+
263+
1. **Start with Option 3** (private composite Action + Poetry git-deps) *today*. It's $0, it takes ~1 hour to set up, and it's exactly the pattern you asked about. Use a GitHub App token (not a PAT) for CI auth.
264+
2. **Once you hit ~5 packages or packages start depending on each other, add Option 1** (Azure Artifacts free tier) as a proper internal index. Azure Artifacts can coexist with the composite Action pattern during the transition.
265+
3. **Skip ACR/GHCR+ORAS entirely** for Python unless you have a separate reason to standardize on OCI.
266+
4. **Skip JFrog/Cloudsmith/Gemfury** unless you outgrow Azure Artifacts' 2 GiB free quota and/or need SSO/SCIM/geo-replication that only the big SaaS vendors provide.
267+
268+
This gets you off Nexus OSS, onto GitHub, with genuine zero-cost infra for Python package distribution, and a clean upgrade path when you actually need one.
269+
270+
---
271+
272+
## Source reports
273+
274+
- `files/research-low-volume-python-hosting.md` (Claude Sonnet 4.6) — 8 options in depth (ACR, GHCR, Azure Blob, S3, Azure Artifacts free, pypiserver/devpi, composite action, git-deps)
275+
- `files/research-private-actions-poetry.md` (GPT-5.3-Codex) — composite action patterns, reusable workflows, wheel-via-Release distribution, full YAML snippets
276+
277+
### Key primary references
278+
279+
- Azure Artifacts free tier: <https://learn.microsoft.com/azure/devops/artifacts/start-using-azure-artifacts>
280+
- Azure Artifacts Python setup: <https://learn.microsoft.com/azure/devops/artifacts/python/project-setup-python>
281+
- Azure Blob Storage pricing: <https://azure.microsoft.com/pricing/details/storage/blobs/>
282+
- ACR OCI artifact support: <https://learn.microsoft.com/azure/container-registry/container-registry-oci-artifacts>
283+
- ORAS CLI: <https://oras.land/>
284+
- GitHub composite actions: <https://docs.github.com/en/actions/tutorials/create-actions/create-a-composite-action>
285+
- Share private actions across repos: <https://docs.github.com/en/actions/how-tos/reuse-automations/share-across-private-repositories>
286+
- `create-github-app-token` action: <https://github.com/actions/create-github-app-token>
287+
- Poetry git dependencies: <https://python-poetry.org/docs/dependency-specification/#git-dependencies>
288+
- `dumb-pypi`: <https://github.com/chriskuehl/dumb-pypi>
289+
- `s3-pypi`: <https://github.com/novemberfiveco/s3pypi>
290+
- pypiserver: <https://github.com/pypiserver/pypiserver>
291+
- devpi: <https://devpi.net/>
292+
- PEP 503 (simple index): <https://peps.python.org/pep-0503/>
293+
294+
---
295+
296+
## Validation and accuracy
297+
298+
*Validated 2026-04-23 against primary sources.*
299+
300+
All factual claims were cross-checked against live documentation and source repositories:
301+
302+
| Claim | Source | Status |
303+
|---|---|---|
304+
| Azure Artifacts free tier: 2 GiB per org | [MS Learn — Azure Artifacts](https://learn.microsoft.com/azure/devops/artifacts/start-using-azure-artifacts) | ✅ Verified |
305+
| Azure Artifacts feed URL format | [MS Learn — Python project setup](https://learn.microsoft.com/azure/devops/artifacts/python/project-setup-python) | ✅ Verified |
306+
| Poetry `priority = "explicit"` source constraint syntax | [Poetry Repositories docs](https://python-poetry.org/docs/repositories/) | ✅ Verified |
307+
| Composite action `using: "composite"` syntax | [GitHub composite actions tutorial](https://docs.github.com/en/actions/tutorials/create-actions/create-a-composite-action) | ✅ Verified |
308+
| Private action sharing via Settings → Actions → Access | [GitHub sharing private actions](https://docs.github.com/en/actions/how-tos/reuse-automations/share-across-private-repositories) | ✅ Verified |
309+
| `actions/create-github-app-token` inputs and version | [GitHub repo](https://github.com/actions/create-github-app-token) — v3.1.1 current | ✅ Verified |
310+
| `git config url.insteadOf` token auth pattern | GitHub and Git documentation | ✅ Verified |
311+
| ACR/GHCR cannot serve pip-native installs | OCI spec / pip documentation — no PEP for OCI-as-package-source | ✅ Verified |
312+
313+
**Corrections applied during validation:**
314+
315+
1. `actions/create-github-app-token@v1` updated to `@v3` (current release).
316+
2. Deprecated `app-id` input replaced with `client-id`; secret reference changed to `vars.APP_CLIENT_ID` per GitHub's recommendation.
317+
3. Shell line-continuation backslashes (`\`) added to multi-line `az` CLI commands inside YAML literal blocks.
318+
319+
*Pricing figures are directional estimates and should be confirmed with vendors before commitment.*

0 commit comments

Comments
 (0)