Skip to content

Commit d582dec

Browse files
feat: Add credential helpers for Docker and pip with custom domain support
This PR adds credential helper infrastructure for Docker and pip package managers, enabling automatic authentication to Cloudsmith registries without manual login commands or embedding credentials in URLs. This builds upon #267 (OIDC authentication) and extends the credential provider chain to Docker and pip workflows. ## Features ### Docker Credential Helper - Command: `cloudsmith credential-helper docker` - Binary: `docker-credential-cloudsmith` (for Docker CLI integration) - Supports standard domains: `docker.cloudsmith.io`, `*.docker.cloudsmith.io` - Supports custom vanity domains via API auto-discovery ### Pip Keyring Backend - Auto-discovered by pip/twine via `keyring.backends` entry point - Supports standard domains: `python.cloudsmith.io`, `dl.cloudsmith.io`, `*.cloudsmith.sh` - Supports custom vanity domains via API auto-discovery - Priority: 9.9 (runs before system keychains) ### Custom Domain Support - Set `CLOUDSMITH_ORG=my-org` to enable custom domain discovery - Fetches domains from `GET /orgs/{org}/custom-domains/` API endpoint - Caches results in `~/.cloudsmith/cache/custom_domains/` for 1 hour - Automatic, no manual configuration needed ### Shared Architecture - Both helpers use the same credential provider chain - Order: Environment Variable → Config File → Keyring → OIDC - Consistent authentication behavior across all package managers - Extensible design for future helpers (npm, cargo, maven, etc.) ## Type of Change - [x] New feature - [x] Documentation update ## Backward Compatibility - ✅ Fully backward compatible - ✅ No breaking changes - ✅ Existing authentication methods unchanged - ✅ Optional features - enable by configuration ## Testing ### Manual Testing - Docker Environment setup: ```bash $ env | grep CLOUDSMITH_ CLOUDSMITH_ORG=iduffy-demo CLOUDSMITH_SERVICE_SLUG=default-v9ty $ stat ~/.cloudsmith/config.ini stat: cannot stat '/Users/iduffy/.cloudsmith/config.ini': No such file or directory $ cloudsmith whoami --verbose Retrieving your authentication status from the API ... OK User: default (slug: default-v9ty) Authentication Method: OIDC Auto-Discovery Source: OIDC auto-discovery: AWS (org: iduffy-demo) Token Slug: 6FmYSZVQrEho Created: 2025-06-07T19:43:47.840466Z SSO Status: Not configured Keyring: Enabled (no tokens stored) ``` Before configuration: ```bash $ cat ~/.docker/config.json cat: /Users/iduffy/.docker/config.json: No such file or directory $ docker pull docker.cloudsmith.io/iduffy-demo/default/library/ubuntu:latest Error response from daemon: Head "https://docker.cloudsmith.io/v2/iduffy-demo/default/library/ubuntu/manifests/latest": unauthorized ``` After configuration: ```bash $ cat ~/.docker/config.json { "credHelpers": { "docker.cloudsmith.io": "cloudsmith" } } $ docker pull docker.cloudsmith.io/iduffy-demo/default/library/ubuntu:latest latest: Pulling from iduffy-demo/default/library/ubuntu cc43ec4c1381: Pull complete Digest: sha256:9cbed754112939e914291337b5e554b07ad7c392491dba6daf25eef1332a22e8 Status: Downloaded newer image for docker.cloudsmith.io/iduffy-demo/default/library/ubuntu:latest docker.cloudsmith.io/iduffy-demo/default/library/ubuntu:latest ``` ✅ **Success:** Docker authenticated using AWS OIDC credentials via credential helper ### Manual Testing - Pip Environment setup: ```bash $ pip config list :env:.default-timeout='100' :env:.disable-pip-version-check='1' $ python -c 'import keyring; print(keyring.backend.get_all_keyring())' [<keyring.backends.fail.Keyring object at 0x1033a4460>, <cloudsmith_cli.credential_helpers.pip.CloudsmithKeyringBackend object at 0x1033a4c80>, <keyring.backends.chainer.ChainerBackend object at 0x1033a5360>, <keyring.backends.macOS.Keyring object at 0x1033a5720>] ``` Test installation (no credentials in URL): ```bash $ pip install --no-cache-dir --index-url=https://dl.cloudsmith.io/basic/iduffy-demo/default/python/simple/ cloudsmith-python-native Looking in indexes: https://dl.cloudsmith.io/basic/iduffy-demo/default/python/simple/ Collecting cloudsmith-python-native Downloading https://dl.cloudsmith.io/basic/iduffy-demo/default/python/cloudsmith_python_native-1.0.1050047-py2.py3-none-any.whl (2.2 kB) Requirement already satisfied: toml in /Users/iduffy/projects/cloudsmith-cli/.venv/lib/python3.10/site-packages (from cloudsmith-python-native) (0.10.2) Installing collected packages: cloudsmith-python-native Successfully installed cloudsmith-python-native-1.0.1050047 ``` ✅ **Success:** Pip authenticated using AWS OIDC credentials via keyring backend ### Automated Testing - ✅ Pylint: 10.00/10 (perfect score) - ✅ Tests: 199 passed, 39 skipped - ✅ No new test failures introduced ## Documentation Added comprehensive documentation: - `CUSTOM_DOMAINS.md` - Complete guide for custom domain configuration - `PIP_KEYRING_POC.md` - Proof of concept guide for pip keyring backend - `PIP_KEYRING_SUCCESS.md` - Success report with usage examples - `pip_keyring_poc.sh` - Automated testing script ## Additional Notes ### Architecture Validation The credential helper architecture was validated by implementing two different package manager protocols (Docker binary stdin/stdout vs Python keyring API) to ensure the design is flexible and extensible. ### Custom Domain Support Custom domains are discovered automatically when `CLOUDSMITH_ORG` is set: 1. First request fetches domains from API 2. Results cached for 1 hour in `~/.cloudsmith/cache/custom_domains/` 3. Subsequent requests use cache (no repeated API calls) 4. Works seamlessly with OIDC in CI/CD environments ### Future Extensions The architecture is designed to support additional credential helpers: - npm (`.npmrc` credential helper) - cargo (Cargo credential helper) - maven (Maven settings credential helper) - gradle (Gradle properties credential helper) - nuget (NuGet credential helper) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9c3199c commit d582dec

9 files changed

Lines changed: 773 additions & 1 deletion

File tree

cloudsmith_cli/cli/commands/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
auth,
55
check,
66
copy,
7+
credential_helper,
78
delete,
89
dependencies,
910
docs,
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
"""
2+
Credential helper commands for Cloudsmith.
3+
4+
This module provides credential helper commands for various package managers
5+
(Docker, pip, npm, etc.) that follow their respective credential helper protocols.
6+
"""
7+
8+
import click
9+
10+
from ..main import main
11+
from .docker import docker as docker_cmd
12+
13+
14+
@click.group()
15+
def credential_helper():
16+
"""
17+
Credential helpers for package managers.
18+
19+
These commands provide credentials for package managers like Docker, pip, npm, etc.
20+
They are typically called by wrapper binaries (e.g., docker-credential-cloudsmith)
21+
or used directly for debugging.
22+
23+
Examples:
24+
# Test Docker credential helper
25+
$ echo "docker.cloudsmith.io" | cloudsmith credential-helper docker
26+
27+
# Called by Docker via wrapper
28+
$ echo "docker.cloudsmith.io" | docker-credential-cloudsmith get
29+
"""
30+
31+
32+
# Register subcommands
33+
credential_helper.add_command(docker_cmd, name="docker")
34+
35+
# Register with main CLI
36+
main.add_command(credential_helper, name="credential-helper")
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
"""
2+
Docker credential helper command.
3+
4+
Implements the Docker credential helper protocol for Cloudsmith registries.
5+
"""
6+
7+
import json
8+
import sys
9+
10+
import click
11+
12+
from ....credential_helpers.docker import get_credentials, is_cloudsmith_registry
13+
14+
15+
@click.command()
16+
def docker():
17+
"""
18+
Docker credential helper for Cloudsmith registries.
19+
20+
Reads a Docker registry server URL from stdin and returns credentials in JSON format.
21+
This command implements the 'get' operation of the Docker credential helper protocol.
22+
23+
Only provides credentials for Cloudsmith Docker registries (docker.cloudsmith.io).
24+
25+
Input (stdin):
26+
Server URL as plain text (e.g., "docker.cloudsmith.io")
27+
28+
Output (stdout):
29+
JSON: {"Username": "token", "Secret": "<cloudsmith-token>"}
30+
31+
Exit codes:
32+
0: Success
33+
1: Error (no credentials available, not a Cloudsmith registry, etc.)
34+
35+
Examples:
36+
# Manual testing
37+
$ echo "docker.cloudsmith.io" | cloudsmith credential-helper docker
38+
{"Username":"token","Secret":"eyJ0eXAiOiJKV1Qi..."}
39+
40+
# Called by Docker via wrapper
41+
$ echo "docker.cloudsmith.io" | docker-credential-cloudsmith get
42+
{"Username":"token","Secret":"eyJ0eXAiOiJKV1Qi..."}
43+
44+
Environment variables:
45+
CLOUDSMITH_API_KEY: API key for authentication (optional)
46+
CLOUDSMITH_ORG: Organization slug (required for OIDC)
47+
CLOUDSMITH_SERVICE_SLUG: Service account slug (required for OIDC)
48+
"""
49+
try:
50+
# Read server URL from stdin
51+
server_url = sys.stdin.read().strip()
52+
53+
if not server_url:
54+
click.echo("Error: No server URL provided on stdin", err=True)
55+
sys.exit(1)
56+
57+
# Check if this is a Cloudsmith registry
58+
if not is_cloudsmith_registry(server_url):
59+
click.echo(f"Error: Not a Cloudsmith registry: {server_url}", err=True)
60+
sys.exit(1)
61+
62+
# Get credentials using the credential provider chain
63+
credentials = get_credentials(server_url, debug=False)
64+
65+
if not credentials:
66+
click.echo(
67+
"Error: Unable to retrieve credentials. "
68+
"Make sure you have either CLOUDSMITH_API_KEY set, "
69+
"or CLOUDSMITH_ORG + CLOUDSMITH_SERVICE_SLUG for OIDC authentication.",
70+
err=True,
71+
)
72+
sys.exit(1)
73+
74+
# Output credentials in Docker credential helper JSON format
75+
click.echo(json.dumps(credentials))
76+
77+
except Exception as e: # pylint: disable=broad-exception-caught
78+
# Broad exception catch to ensure we never crash without a message
79+
click.echo(f"Error: {e}", err=True)
80+
sys.exit(1)
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""
2+
Credential helpers for various package managers.
3+
4+
This package provides credential helper implementations for Docker, pip, npm, etc.
5+
Each helper follows its respective package manager's credential helper protocol.
6+
"""
Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
"""
2+
Helper for discovering Cloudsmith custom domains.
3+
4+
This module provides functions to fetch custom domains from the Cloudsmith API
5+
for use in credential helpers. Results are cached on the filesystem.
6+
"""
7+
8+
import json
9+
import logging
10+
import os
11+
import time
12+
from pathlib import Path
13+
from typing import List, Optional
14+
15+
logger = logging.getLogger(__name__)
16+
17+
# Cache custom domains for 1 hour
18+
CACHE_TTL_SECONDS = 3600
19+
20+
21+
def get_cache_dir() -> Path:
22+
"""
23+
Get the cache directory for custom domains.
24+
25+
Returns:
26+
Path to cache directory (e.g., ~/.cloudsmith/cache/custom_domains/)
27+
"""
28+
home = Path.home()
29+
cache_dir = home / ".cloudsmith" / "cache" / "custom_domains"
30+
cache_dir.mkdir(parents=True, exist_ok=True)
31+
return cache_dir
32+
33+
34+
def get_cache_path(org: str) -> Path:
35+
"""
36+
Get the cache file path for an organization's custom domains.
37+
38+
Args:
39+
org: Organization slug
40+
41+
Returns:
42+
Path to cache file
43+
"""
44+
cache_dir = get_cache_dir()
45+
# Use org slug as filename (sanitized)
46+
safe_org = "".join(c if c.isalnum() or c in "-_" else "_" for c in org)
47+
return cache_dir / f"{safe_org}.json"
48+
49+
50+
def is_cache_valid(cache_path: Path) -> bool:
51+
"""
52+
Check if a cache file exists and is still valid.
53+
54+
Args:
55+
cache_path: Path to cache file
56+
57+
Returns:
58+
bool: True if cache exists and hasn't expired
59+
"""
60+
if not cache_path.exists():
61+
return False
62+
63+
try:
64+
mtime = cache_path.stat().st_mtime
65+
age = time.time() - mtime
66+
return age < CACHE_TTL_SECONDS
67+
except OSError:
68+
return False
69+
70+
71+
def read_cache(cache_path: Path) -> Optional[List[str]]:
72+
"""
73+
Read custom domains from cache file.
74+
75+
Args:
76+
cache_path: Path to cache file
77+
78+
Returns:
79+
List of domain strings or None if cache invalid/missing
80+
"""
81+
if not is_cache_valid(cache_path):
82+
return None
83+
84+
try:
85+
with open(cache_path, encoding="utf-8") as f:
86+
data = json.load(f)
87+
if isinstance(data, dict) and "domains" in data:
88+
domains = data["domains"]
89+
if isinstance(domains, list):
90+
logger.debug(
91+
"Read %d domains from cache: %s", len(domains), cache_path
92+
)
93+
return domains
94+
except (OSError, json.JSONDecodeError) as exc:
95+
logger.debug("Failed to read cache %s: %s", cache_path, exc)
96+
97+
return None
98+
99+
100+
def write_cache(cache_path: Path, domains: List[str]) -> None:
101+
"""
102+
Write custom domains to cache file.
103+
104+
Args:
105+
cache_path: Path to cache file
106+
domains: List of domain strings to cache
107+
"""
108+
try:
109+
data = {
110+
"domains": domains,
111+
"cached_at": time.time(),
112+
}
113+
with open(cache_path, "w", encoding="utf-8") as f:
114+
json.dump(data, f)
115+
logger.debug("Wrote %d domains to cache: %s", len(domains), cache_path)
116+
except OSError as exc:
117+
logger.debug("Failed to write cache %s: %s", cache_path, exc)
118+
119+
120+
def get_custom_domains_for_org(
121+
org: str, api_host: str = "https://api.cloudsmith.io"
122+
) -> List[str]:
123+
"""
124+
Fetch custom domains for a Cloudsmith organization.
125+
126+
Results are cached on the filesystem for 1 hour to avoid excessive API calls.
127+
128+
Args:
129+
org: Organization slug
130+
api_host: Cloudsmith API host
131+
132+
Returns:
133+
List of custom domain strings (e.g., ['docker.customer.com', 'dl.customer.com'])
134+
Empty list if API call fails or org has no custom domains
135+
136+
Example:
137+
>>> domains = get_custom_domains_for_org('my-org')
138+
>>> print(domains)
139+
['docker.acme.com', 'dl.acme.com']
140+
"""
141+
# Check cache first
142+
cache_path = get_cache_path(org)
143+
cached_domains = read_cache(cache_path)
144+
if cached_domains is not None:
145+
logger.debug("Using cached custom domains for %s", org)
146+
return cached_domains
147+
148+
# Cache miss - fetch from API
149+
logger.debug("Fetching custom domains from API for %s", org)
150+
151+
try:
152+
import requests
153+
154+
# Construct API URL
155+
url = f"{api_host}/orgs/{org}/custom-domains/"
156+
157+
# Make API request (no auth needed for custom domains endpoint)
158+
response = requests.get(url, timeout=5)
159+
160+
if response.status_code == 404:
161+
logger.debug("Organization %s not found or has no custom domains", org)
162+
# Cache empty result to avoid repeated 404s
163+
write_cache(cache_path, [])
164+
return []
165+
166+
if response.status_code != 200:
167+
logger.debug(
168+
"Failed to fetch custom domains for %s: HTTP %d",
169+
org,
170+
response.status_code,
171+
)
172+
return []
173+
174+
data = response.json()
175+
176+
# Extract domain names from response
177+
# Expected format: [{"domain": "docker.customer.com", ...}, ...]
178+
domains = []
179+
if isinstance(data, list):
180+
for item in data:
181+
if isinstance(item, dict) and "domain" in item:
182+
domains.append(item["domain"])
183+
184+
logger.debug("Fetched %d custom domains for %s", len(domains), org)
185+
186+
# Cache the result
187+
write_cache(cache_path, domains)
188+
189+
return domains
190+
191+
except ImportError:
192+
logger.debug("requests library not available, cannot fetch custom domains")
193+
return []
194+
except Exception as exc: # pylint: disable=broad-exception-caught
195+
logger.debug("Error fetching custom domains: %s", exc)
196+
return []
197+
198+
199+
def get_custom_domains_from_env() -> List[str]:
200+
"""
201+
Get custom domains by fetching from API if CLOUDSMITH_ORG is set.
202+
203+
Returns:
204+
List of custom domain strings
205+
Empty list if CLOUDSMITH_ORG not set or API call fails
206+
207+
Example:
208+
>>> os.environ['CLOUDSMITH_ORG'] = 'my-org'
209+
>>> domains = get_custom_domains_from_env()
210+
>>> print(domains)
211+
['docker.acme.com', 'dl.acme.com']
212+
"""
213+
# Check for org and fetch custom domains from API
214+
org = os.environ.get("CLOUDSMITH_ORG", "").strip()
215+
if not org:
216+
return []
217+
218+
api_host = os.environ.get("CLOUDSMITH_API_HOST", "https://api.cloudsmith.io")
219+
return get_custom_domains_for_org(org, api_host)
220+
221+
222+
def is_custom_cloudsmith_domain(server_url: str, org: Optional[str] = None) -> bool:
223+
"""
224+
Check if a server URL matches a custom Cloudsmith domain.
225+
226+
Args:
227+
server_url: The server URL to check
228+
org: Optional organization slug (if not provided, uses CLOUDSMITH_ORG from env)
229+
230+
Returns:
231+
bool: True if this is a custom Cloudsmith domain, False otherwise
232+
233+
Example:
234+
>>> os.environ['CLOUDSMITH_ORG'] = 'my-org'
235+
>>> is_custom_cloudsmith_domain('docker.acme.com')
236+
True
237+
>>> is_custom_cloudsmith_domain('docker.hub.io')
238+
False
239+
"""
240+
if not server_url:
241+
return False
242+
243+
# Normalize URL
244+
normalized = server_url.lower()
245+
if "://" in normalized:
246+
normalized = normalized.split("://", 1)[1]
247+
normalized = normalized.split(":")[0] # Remove port
248+
249+
# Get custom domains
250+
custom_domains = get_custom_domains_from_env()
251+
252+
# Check if normalized URL matches any custom domain
253+
return normalized in custom_domains

0 commit comments

Comments
 (0)