Skip to content

Commit 76fd933

Browse files
meeechrosieyohannan
meeech
and
rosieyohannan
authored
Add PyPI OIDC trusted publishing guide (#10566)
add pypi guide for upcoming trusted publishing on that platform. --------- Co-authored-by: Rosie Yohannan <rosie@circleci.com>
1 parent cfd45da commit 76fd933

4 files changed

Lines changed: 281 additions & 1 deletion

File tree

‎antora-playbook.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ antora:
120120
- rst
121121
- require: ./extensions/unlisted-pages-extension.js
122122
allowedUnlistedPages:
123+
- 'deploy:deploy-to-pypi-registry.adoc'
123124
- require: '@sntke/antora-mermaid-extension'
124125
mermaid_initialize_options:
125126
start_on_load: true

‎docs/guides/modules/deploy/pages/deploy-to-pypi-registry.adoc‎

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
= Publish to PyPI using OIDC trusted publishing
2+
:page-badge: Preview
3+
:page-platform: Cloud
4+
:page-description: Publish packages to PyPI from CircleCI using OIDC trusted publishing, the recommended tokenless approach.
5+
:experimental:
6+
7+
NOTE: Publishing to PyPI with trusted publishing is currently in preview.
8+
9+
Follow this how-to guide to configure CircleCI to publish to the Python Package Index (PyPI) using trusted publishing. Trusted publishing uses OpenID Connect (OIDC) to exchange a short-lived CircleCI token for a PyPI API token at publish time. No long-lived `PYPI_API_TOKEN` exists to store, rotate, or leak.
10+
11+
For a functional example you can clone and adapt, see the link:https://github.com/CircleCI-Public/sign-and-publish-examples/tree/main/pypi[CircleCI PyPI trusted publishing example].
12+
13+
[#introduction]
14+
== Introduction
15+
16+
Trusted publishing is the recommended way to publish to PyPI from CircleCI. Compared to using a long-lived API token stored in an environment variable, trusted publishing offers the following benefits:
17+
18+
* Removes the need to create, store, and rotate a publish token.
19+
* Issues a short-lived API token that is valid only for the duration of the publishing job.
20+
* Lets you scope publishing to a specific organization, project, pipeline definition, and (optionally) context.
21+
22+
NOTE: PyPI also supports a _pending publisher_ for projects that do not exist yet. Unlike some registries, you do not need to publish a first release manually before configuring trusted publishing. See <<set-up-the-trusted-publisher-on-pypi,Step 2>>.
23+
24+
[#prerequisites]
25+
== Prerequisites
26+
27+
* A CircleCI Cloud account integrated with a supported VCS. See xref:getting-started:first-steps.adoc[Sign up and Try CircleCI] for more information.
28+
* A project on CircleCI that builds your Python package.
29+
* A PyPI account with two-factor authentication enabled.
30+
* Owner access to the PyPI project, or permission to create a pending publisher for a new project name.
31+
* link:https://pypi.org/project/twine/[Twine] version 6.1.0 or later, and Python 3.9 or later, in the Docker image (or executor) used by your publishing job. From version 6.1.0, Twine has built-in support for trusted publishing and detects CircleCI OIDC credentials automatically.
32+
33+
[#gather-the-data-you-need]
34+
== 1. Gather the data you need
35+
36+
You need CircleCI identifiers to configure your project as a trusted publisher. These are:
37+
38+
* Organization ID
39+
* Project ID
40+
* Pipeline definition ID
41+
* Context ID (optional)
42+
* VCS origin (optional)
43+
* VCS ref (optional)
44+
45+
Collect these before you visit PyPI.
46+
47+
=== 1.1 Organization ID (required)
48+
You need your CircleCI organization ID, in UUID format. You can find your org ID in menu:Org Settings[Overview], or follow these steps:
49+
50+
include::guides:ROOT:partial$app-navigation/steps-to-org-id.adoc[]
51+
52+
=== 1.2 Project ID (required)
53+
You need your CircleCI project ID, in UUID format. You can find your project ID in menu:Project Settings[Overview], or follow these steps:
54+
55+
include::guides:ROOT:partial$app-navigation/steps-to-project-id.adoc[]
56+
57+
=== 1.3 Pipeline definition ID (required)
58+
You need your pipeline definition ID, in UUID format. You can find your pipeline definition ID in menu:Project Settings[Project Setup], where you will find details of all pipelines that have been set up for your project. Follow these steps to find your pipeline definition IDs:
59+
60+
include::guides:ROOT:partial$app-navigation/steps-to-pipeline-definition-id.adoc[]
61+
62+
=== 1.4 Context ID (optional)
63+
If you intend to restrict publishing to a CircleCI context (recommended, see <<lock-it-down,Step 3>>), get the UUID for the context now. To find a context ID, follow these steps:
64+
65+
include::guides:ROOT:partial$app-navigation/steps-to-org-settings.adoc[]
66+
. Select *Contexts* from the menu.
67+
. Select the context you want to find the ID for.
68+
. Select the btn:[Copy] icon next to the context ID to copy it to your clipboard.
69+
70+
Gather context ID the context you want to restrict publishing to.
71+
72+
=== 1.5 VCS origin and VCS ref (optional)
73+
74+
Using VCS origin and VCS ref, you can create trusted publishers that can only publish from specific branches. This can be useful if you have branch protections that control who can merge and therefore trigger pipelines. One example is to have a trusted publisher bound to a main branch for production releases, and to a staging branch for generating pre-releases.
75+
76+
VCS origin:: You need the version control system origin URL for your project, for example `github.com/myorg/myrepo` or `gitlab.com/mylab/myproject`.
77+
VCS ref:: The reference to the change that triggered the pipeline, for example, `refs/heads/main`.
78+
79+
NOTE: VCS origin and VCS ref info is not available for pipelines triggered via xref:orchestrate:triggers-overview.adoc#trigger-a-pipeline-from-a-custom-webhook[Custom Webhooks].
80+
81+
[#set-up-the-trusted-publisher-on-pypi]
82+
== 2. Set up the trusted publisher on PyPI
83+
84+
You can configure a trusted publisher for an existing project, or create a _pending publisher_ for a project name that has not been published yet.
85+
86+
For an existing project:
87+
88+
. Sign in to https://pypi.org[PyPI] and navigate to *Your projects*.
89+
. Select *Manage* for the project you want to add Trusted Publishing to.
90+
. Select *Publishing* to begin the process.
91+
92+
. Fill in the fields with the values you collected in the previous step:
93+
+
94+
** Organization ID
95+
** Project ID
96+
** Pipeline Definition ID
97+
** Context ID (optional)
98+
** VCS origin (optional)
99+
** VCS ref (optional)
100+
101+
. Save the publisher.
102+
103+
To publish a project name that does not exist yet, create a pending publisher instead from the *Publishing* page, then publish your first release through the workflow.
104+
105+
[#lock-it-down]
106+
== 3. Lock it down with restricted contexts
107+
108+
Using restricted contexts allows you to control who can run pipelines that publish. You can also add information about your source repo and branch.
109+
110+
This step is optional but recommended. If you do not want context-based restrictions, skip to <<configure-your-circleci-pipeline,Step 4>>.
111+
112+
You can further restrict who and what can run the publishing job by attaching the job to a xref:security:contexts.adoc[CircleCI Context] and applying restrictions to that context. If you do this, remember to record the context's UUID under *Context ID* when you configure the trusted publisher on PyPI (see <<set-up-the-trusted-publisher-on-pypi,Step 2>>). Three types of restrictions are available, as described in the following sections.
113+
114+
Security group restrictions:: xref:security:contexts.adoc#security-group-restrictions[Security Group Restrictions] limit job execution to members of a specific group. Anyone outside the group who attempts to run the job fails. Use a security group restriction when you want only a small set of people to be able to publish.
115+
116+
Project restrictions:: xref:security:contexts.adoc#project-restrictions[Project Restrictions] limit a context to one or more specific projects. With a project restriction in place, a job in any other project that tries to use the context will fail. Use a project restriction to ensure the publishing context can only be used by the project that owns your PyPI package.
117+
118+
Expression restrictions::
119+
120+
xref:security:contexts.adoc#expression-restrictions[Expression Restrictions] are more powerful. The most common use is restricting a context to a specific branch. For example:
121+
+
122+
```
123+
pipeline.git.branch == "main"
124+
```
125+
+
126+
This causes any job on a different branch that tries to use the context to fail.
127+
+
128+
*SSH rerun restrictions*
129+
+
130+
PyPI rejects any OIDC token whose `ssh_rerun` claim is `true`. As defense in depth, you can also add the following expression restriction to your publishing context so that the job cannot be re-run with SSH at all:
131+
+
132+
```
133+
not job.ssh.enabled
134+
```
135+
+
136+
You can combine expressions to restrict the context to a specific branch and disallow SSH reruns in a single rule:
137+
+
138+
```
139+
pipeline.git.branch == "main" and not job.ssh.enabled
140+
```
141+
142+
[#configure-your-circleci-pipeline]
143+
== 4. Configure your CircleCI pipeline
144+
145+
Add a publishing job to your `.circleci/config.yml`. From version 6.1.0, Twine handles the following steps:
146+
147+
* Detects it is running on CircleCI.
148+
* Retrieves an OIDC token for the `pypi` audience.
149+
* Exchanges it with PyPI for a short-lived API token.
150+
* Uploads your distributions.
151+
152+
In the examples below, the job is attached to the context from <<lock-it-down,Step 3>> (here named `trusted-publishing-guard`) and the workflow filters to the `main` branch only. If you skipped Step 3, remove the `context:` block from the workflow.
153+
154+
[tabs]
155+
====
156+
Automatic (recommended)::
157+
+
158+
--
159+
With Twine 6.1.0 or later, run `twine upload` and let Twine handle the OIDC exchange for you.
160+
161+
[source,yaml]
162+
----
163+
version: 2.1
164+
165+
jobs:
166+
publish:
167+
docker:
168+
- image: cimg/python:3.12
169+
steps:
170+
- checkout
171+
- run:
172+
name: Build distributions
173+
command: |
174+
python -m pip install --upgrade build twine
175+
python -m build
176+
- run:
177+
name: Publish to PyPI
178+
command: |
179+
# twine detects the CircleCI OIDC credentials and performs the
180+
# trusted publishing exchange automatically. No token is needed.
181+
twine upload dist/*
182+
183+
workflows:
184+
publish:
185+
jobs:
186+
- publish:
187+
context:
188+
- trusted-publishing-guard
189+
filters:
190+
branches:
191+
only:
192+
- main
193+
----
194+
--
195+
Manual exchange (fallback)::
196+
+
197+
--
198+
If you cannot use Twine 6.1.0 or later, retrieve the OIDC token and exchange it for a short-lived API token yourself.
199+
200+
[source,yaml]
201+
----
202+
version: 2.1
203+
204+
jobs:
205+
publish:
206+
docker:
207+
- image: cimg/python:3.12
208+
steps:
209+
- checkout
210+
- run:
211+
name: Build distributions
212+
command: |
213+
python -m pip install --upgrade build twine
214+
python -m build
215+
- run:
216+
name: Publish to PyPI
217+
command: |
218+
# Retrieve a CircleCI OIDC token for the PyPI audience
219+
PYPI_ID_TOKEN=$(circleci run oidc get --claims '{"aud": "pypi"}')
220+
# Exchange the OIDC token for a short-lived PyPI API token
221+
API_TOKEN=$(curl -s -X POST https://pypi.org/_/oidc/mint-token \
222+
-d "{\"token\": \"${PYPI_ID_TOKEN}\"}" | jq -r '.token')
223+
# Upload using the minted token
224+
twine upload --non-interactive \
225+
--username __token__ --password "${API_TOKEN}" dist/*
226+
227+
workflows:
228+
publish:
229+
jobs:
230+
- publish:
231+
context:
232+
- trusted-publishing-guard
233+
filters:
234+
branches:
235+
only:
236+
- main
237+
----
238+
239+
--
240+
====
241+
242+
For background on OIDC tokens in CircleCI, see xref:permissions-authentication:openid-connect-tokens.adoc[Use OpenID Connect Tokens in Jobs].
243+
244+
[#publish-a-new-version]
245+
== 5. Publish a new version
246+
247+
With the workflow above, every merge to `main` publishes a new version. Bump the `version` in your `pyproject.toml` (or `setup.py`) on a feature branch, open a pull request, and merge to `main`. CircleCI runs the publishing workflow on the merge commit, and `twine` exchanges the OIDC token for a short-lived API token to upload the package.
248+
249+
NOTE: PyPI does not allow re-uploading a version that already exists. Make sure each merge to `main` carries a new version number.
250+
251+
[#self-hosted-runners]
252+
== Self-hosted runners
253+
254+
Trusted publishing from CircleCI self-hosted runners works.
255+
256+
[#notes]
257+
== Notes
258+
259+
* You can test the full flow against link:https://test.pypi.org[TestPyPI] before publishing to production PyPI. Configure a separate trusted publisher on TestPyPI and point `twine` at it with `twine upload --repository testpypi dist/*`. If you use the manual exchange, use the `testpypi` audience and the `https://test.pypi.org/_/oidc/mint-token` endpoint instead.
260+
* Pull requests from forks do not have access to CircleCI secrets, including OIDC tokens. A fork cannot trigger a publish.
261+
262+
[#track-deployments-with-deploy-markers]
263+
== Track your deployments with deploy markers
264+
265+
Deploy markers provide a way to track and manage your PyPI package publications in the CircleCI web app. When you add deploy markers to your deployment job, you can view a timeline of all deployments, track their status, and enable rollback and deploy pipelines.
266+
267+
You have two options for setting up deploy markers:
268+
269+
* *In-app setup*: Use the guided setup in the CircleCI web app when configuring a xref:set-up-rollbacks.adoc[Rollback Pipeline] or xref:set-up-deploys.adoc[Deploy Pipeline]. The setup walks you through adding deploy markers to your configuration. If you are using GitHub and have the CircleCI GitHub App installed, you can use AI to generate the deploy marker configuration automatically.
270+
271+
* *Manual setup*: Add deploy marker commands directly to your `.circleci/config.yml` file by following the xref:configure-deploy-markers.adoc[Configure Deploy Markers] guide.
272+
273+
Both approaches enable you to track deployment history and manage rollbacks directly from the CircleCI web app.
274+
275+
[#questions-and-feedback]
276+
== Feedback
277+
278+
If you have questions or feedback, drop by the https://discuss.circleci.com/[CircleCI Discuss forum] or the https://discord.gg/circleci[CircleCI Discord community].

‎ui/src/css/doc.css‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1552,6 +1552,7 @@
15521552
.platform-badge {
15531553
position: relative;
15541554
cursor: default;
1555+
flex-shrink: 0;
15551556
}
15561557

15571558
/* Light mode tooltip - dark background with white text */

‎ui/src/partials/article.hbs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<h1 class="page flex items-center gap-2">
44
{{{this}}}
55
{{#if @root.page.attributes.badge}}
6-
<span class="text-[10px] rounded-full py-1 px-1.5 {{#if @root.page.attributes.badge-classes}}{{@root.page.attributes.badge-classes}}{{else}}text-terminal-black border{{/if}}" {{#if @root.page.attributes.badge-bg}}style="font-family: 'Roboto', sans-serif; font-weight: 700; background-color: {{@root.page.attributes.badge-bg}}; {{#if @root.page.attributes.badge-border}}border: 1px solid {{@root.page.attributes.badge-border}};{{/if}}"{{else}}style="font-family: 'Roboto', sans-serif; font-weight: 700;"{{/if}}>{{@root.page.attributes.badge}}</span>
6+
<span class="text-[10px] rounded-full py-1 px-1.5 shrink-0 whitespace-nowrap {{#if @root.page.attributes.badge-classes}}{{@root.page.attributes.badge-classes}}{{else}}text-terminal-black border{{/if}}" {{#if @root.page.attributes.badge-bg}}style="font-family: 'Roboto', sans-serif; font-weight: 700; background-color: {{@root.page.attributes.badge-bg}}; {{#if @root.page.attributes.badge-border}}border: 1px solid {{@root.page.attributes.badge-border}};{{/if}}"{{else}}style="font-family: 'Roboto', sans-serif; font-weight: 700;"{{/if}}>{{@root.page.attributes.badge}}</span>
77
{{/if}}
88
</h1>
99
{{/with}}

0 commit comments

Comments
 (0)