Add Podman quadlet .image file support by ehelms · Pull Request #501 · theforeman/foremanctl

ehelms · 2026-05-08T14:00:57Z

Why are you introducing these changes? (Problem description, related links)

Container image references were embedded directly in each service role as variable defaults, making it difficult to override images for product builds, disconnected installs, or developer testing without modifying foremanctl-managed files.

Resolves: #277

What are the changes introduced in this pull request?

Introduce a shared images role with deploy_image.yaml that generates a .image quadlet unit file and .image.d/ drop-in directory for each container image using containers.podman.podman_image (state: quadlet)
Each service role gains an image.yaml task that calls the images role with its image name and tag, removing per-role image variable defaults
Container roles now reference Image=<name>.image instead of full registry URLs, binding image pull to the quadlet unit lifecycle managed by systemd
pull-images playbook updated to deploy image units and run daemon_reload rather than pulling images directly; actual pulls happen on first service start
Drop-in hierarchy (/usr/share/containers/systemd/<name>.image.d/10-product.conf, 20-archive.conf, /etc/containers/systemd/<name>.image.d/90-user.conf) allows RPM vendors, ISO extraction, and users to override images without modifying foremanctl-generated files
images_registry_auth_file defaults to /etc/foreman/registry-auth.json; users with authenticated registries run podman login <registry> --authfile=/etc/foreman/registry-auth.json before deploying
Add tests/images_test.py for core image files, drop-in directories, and systemd service existence; IOP image tests in tests/iop/images_test.py

How to test this pull request

Steps to reproduce:

Deploy with ./foremanctl deploy
Verify .image files exist under /etc/containers/systemd/ for each service
Verify <name>-image systemd services exist
Run ./forge test and confirm images_test.py passes
Place /etc/containers/systemd/foreman.image.d/90-user.conf with an alternate image tag, run systemctl daemon-reload, and confirm the override takes precedence

Checklist

Tests added/updated (if applicable)
Documentation updated (if applicable)

ekohl · 2026-05-08T14:35:25Z

Do you intend to resolve #277 with this?

ekohl · 2026-05-08T14:42:23Z

Add a Podman quadlet .image unit system with three-tier precedence: admin overrides (/etc/foremanctl/images.d/) > vendor/RPM overrides (/usr/share/foremanctl/images.d/) > generated defaults from Ansible variables

I'm wondering if we can leverage the built in /usr/share/containers/systemd/ somehow. It's a major change, but I've been wondering if we should ship the container definitions in an RPM and deploy them to /usr/share/containers/systemd, letting foremanctl only add overrides in /etc/containers/systemd.

Also not sure if you can use /etc/containers/systemd/foreman.image.d/override.conf to replace parts of it. If so, perhaps an RPM can ship /usr/share/containers/systemd/foreman.image.d/disconnected.conf to provide disconnected installation instructions.

ehelms · 2026-05-08T20:58:41Z

I'm wondering if we can leverage the built in /usr/share/containers/systemd/ somehow. It's a major change, but I've been wondering if we should ship the container definitions in an RPM and deploy them to /usr/share/containers/systemd, letting foremanctl only add overrides in /etc/containers/systemd.

As we stated before, tieing ourselves to an RPM laying down some base container will slow us down as any time it requires a change we have to wait for the RPM to cycle and it would split the base from any overrides. I think we would end up just deploying everything via overrides. Just feel like friction right now.

I did think about this and here is an alternative using this concept:

Updated Proposal

Place .image files alongside .container files in /etc/containers/systemd/, using quadlet's native drop-in mechanism for overrides.

Layout:

/etc/containers/systemd/
foreman.image # base, templated by foremanctl
foreman.image.d/
50-disconnected.conf # optional: disconnected registry override
90-user.conf # optional: user override

Precedence (last wins):

foreman.image -- foremanctl default
50-disconnected.conf -- disconnected layer
90-user.conf -- user layer

Numbered prefixes enforce ordering with gaps for future layers.

What this eliminates:

Custom /etc/foremanctl/images.d/ and /usr/share/foremanctl/images.d/ directories
Manual stat/copy/conditional precedence logic in deploy_image.yaml
The entire images role

What stays the same:

Container roles reference Image=foreman.image (no change)
foremanctl templates the base .image file (just changes the destination)
daemon-reload after deployment (already happening)

ekohl · 2026-05-12T08:42:05Z

Updated Proposal

This is what I had in mind.

Custom /etc/foremanctl/images.d/ and /usr/share/foremanctl/images.d/ directories

This is IMHO the big benefit.

evgeni · 2026-05-13T13:29:32Z

I like the idea!

There is an alternative approach from @bochi in #503, which is much simpler in the diff size, but requires custom code vs using native systemd/podman override options. Mostly mentioning it here as I wonder if this PR would also solve their usecase.

bochi · 2026-05-13T18:57:07Z

i like this approach too and will change my PR to only have the feature addition but no changes to image locations. looking forward to this getting merged 🙂

ianballou · 2026-05-13T21:01:11Z

Being able to override the container images easily would be really beneficial for development. If we eventually have containers built from PRs much like packit does with RPMs today, you could just drop those in. Or, when we need to test new versions of Pulp, we could simply drop that new Pulp container in (theforeman/pulp-oci-images#13).

ekohl · 2026-05-13T21:36:21Z

@@ -1,6 +1,5 @@
 ---
 iop_advisor_frontend_container_image: "quay.io/iop/advisor-frontend"
 iop_advisor_frontend_container_tag: "foreman-3.18"


Unrelated, but shouldn't this follow nightly?

ekohl · 2026-05-15T23:39:32Z

+Image=quay.io/foreman/foreman:nightly
+```
+
+**User's own registry:** The user creates a `90-user.conf` drop-in to point at their own registry. This overrides the base layer:


This particular use case may be easier with containers-registries.conf. From reading the docs I think adding the following to /etc/containers/registries.conf should work:

[[registry]] prefix = "quay.io/theforeman" location = "katello.example.com/Default_Organization"

arvind4501 · 2026-05-19T13:28:13Z

  post_tasks:
-    - name: Pull an image
-      containers.podman.podman_image:
+    - name: Deploy core image units


From what i understand is this only writes .image files to /etc/containers/systemd/ and does a daemon_reload but does that actually pulls a image? i mean does it download the images that can be used without re-pulling in deploy?

https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html#usage-summary says enerating a systemd .service that runs podman image pull. is confusing, does it pull by default or on start of .service

I think it generates a .service (like foreman.image.service) file but it requires someone/something to start it. Whether that's a user running systemctl or another service depending on it (I'd expect foreman.service to).

In this case I'd expect foremanctl pull-images to ensure the service is started, but some questions that pop up:

What if the service (like foreman.image.service) was already started? Is it a noop or does it run again?

What happens if an updated image is pulled? Does it restart dependent services?

Looking at the docs Policy influences this.

The quadlet generator makes each .image unit a Type=oneshot service. When a container starts, systemd activates its image service dependency, which runs podman image pull (or no-ops if the image already exists), then exits. The service transitions active → inactive when the oneshot finishes.

This strategy has lots of benefits and one small downside, on systemctl start foreman.target:

Each container Requires its image service

Since the image services are inactive (they completed as oneshots), systemd re-runs them

Each one checks whether the image exists, which is fast if it's cached — but with 20 image services being re-checked concurrently on a busy system, it adds latency

Because of this I had to increase the foreman.target retries when we test start/stop/restart of it.

The docs mention Policy=always and that implies it always connects over the network. Perhaps we should use Policy=missing so it's only a local operation after the first installation?

Then we do need an explicit process to update images, but we already have an update and upgrade guide. Perhaps that's good anyway? Otherwise you can unexpectedly pull in z-streams after a reboot.

When a container starts, systemd activates its image service dependency, which runs podman image pull

so technically it happens when containers start not when we run foremanctl pull-images? thats seems odd with what we expect from pull-images, as a user i can pull-images and then deploy which reduces the deployment time as i already have latest images.
or i am interpreting wrong?

evgeni · 2026-05-19T14:17:37Z

+    dest: "{{ images_quadlet_dir }}/{{ images_definition.name }}.image"
+    mode: "0644"
+
+- name: Create drop-in directory for {{ images_definition.name }}


we don't deploy anything here, right? it's just "so it's there so someone else can do it"?

Yes. So, arguably, I can drop this here and just let whatever needs to do this create them (e.g. downstreams with different images or upstream release RPMs). Thoughts?

I thought packages would not touch it here anyway but use /usr/lib?

/usr/share but yes you are correct. This override directory would be used for these two scenarios:

development / testing image overrides (e.g. developers being able to do Pulp upgrade container for development pulp-oci-images#13 and then drop it in and test it)

"hotfix" or modified containers with a different name:tag

One argument is these are specialized and thus the directory should be created at the time of use. This adds a step the user or developer has to remember. Whereas, having the directory allows quick and easy drop in with less overhead remembering two steps.

What do you think?

One argument is these are specialized and thus the directory should be created at the time of use. This adds a step the user or developer has to remember. Whereas, having the directory allows quick and easy drop in with less overhead remembering two steps.

IMHO you can't count on things existing just because Ansible created it so I'd prefer the tools to be robust by having a mkdir -p in there. So my suggestion is to remove the drop-in directory from Ansible.

ekohl · 2026-05-19T16:19:17Z

@@ -0,0 +1,15 @@
+---
+- name: Generate image file for {{ images_definition.name }}
+  containers.podman.podman_image:


Following up on #501 (comment): I think we should set Policy=missing so it doesn't automatically update on every reboot/restart.

Then we can use podman auto-update to force updates.

Introduce .image quadlet units to decouple image sourcing from container definitions. Container roles now reference Image=<name>.image instead of full registry URLs, and a new images role handles deployment with a three-tier precedence model: admin overrides (/etc/foremanctl/images.d/) > vendor RPMs (/usr/share/foremanctl/images.d/) > generated defaults. Resolves: theforeman#277 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ekohl · 2026-05-19T22:57:53Z

+    quadlet_file_mode: "0644"
+    quadlet_options:
+      - "Policy=missing"
+      - "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"


Per containers/ansible-podman-collections#1034 (comment) you can use authfile:

Suggested change

- "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"

authfile: "{{ images_registry_auth_file }}"

Nope. This will set AuthFile and this will enforce the existence of this file. What we want is to set the environment variable like we do in puppet-iop.

ehelms · 2026-05-20T01:50:37Z

I'm not yet sure what change in the design is not leading to the failure. I will need to follow up with deeper investigation.

arvind4501 · 2026-05-20T10:26:41Z

+    quadlet_file_mode: "0644"
+    quadlet_options:
+      - "Policy=missing"
+      - "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"


Suggested change

- "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"

- "AUTH_FILE={{ images_registry_auth_file }}"

we can't use Environment or REGISTRY_AUTH_FILE directly in [Image] section as thats not one of the valid options(https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html#options)
we could use REGISTRY_AUTH_FILE but not directly in .image file https://docs.podman.io/en/latest/markdown/podman-image.unit.5.html#authfile-path

and mostly CI is also failing because of same reason

systemd[1]: Reloading. May 20 09:47:23 quadlet.example.com quadlet-generator[34868]: converting "postgresql.image": unsupported key 'Environment' in group 'Image' in /etc/containers/systemd/postgresql.image May 20 09:47:23 quadlet.example.com quadlet-generator[34868]: processing encountered some errors May 20 09:47:23 quadlet.example.com systemd-rc-local-generator[34879]: /etc/rc.d/rc.local is not marked executable, skipping. May 20 09:47:23 quadlet.example.com systemd[34865]: /usr/lib/systemd/system-generators/podman-system-generator failed with exit status 1.

pr-processor Bot added the Not yet reviewed label May 8, 2026

ehelms mentioned this pull request May 8, 2026

Pulp upgrade container for development theforeman/pulp-oci-images#13

Open

1 task

evgeni mentioned this pull request May 13, 2026

allow downstream product overrides for features #503

Open

2 tasks

ekohl reviewed May 13, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label May 13, 2026

ehelms force-pushed the image-quadlet-support branch 2 times, most recently from 9b0d041 to 2f9968f Compare May 15, 2026 20:24

ekohl approved these changes May 15, 2026

View reviewed changes

arvind4501 reviewed May 19, 2026

View reviewed changes

Comment thread docs/developer/deployment.md Outdated

arvind4501 reviewed May 19, 2026

View reviewed changes

ehelms force-pushed the image-quadlet-support branch from 2f9968f to e392b79 Compare May 19, 2026 14:11

evgeni reviewed May 19, 2026

View reviewed changes

Comment thread src/roles/foreman/tasks/main.yaml

evgeni reviewed May 19, 2026

View reviewed changes

Comment thread src/roles/images/tasks/deploy_image.yaml Outdated

evgeni reviewed May 19, 2026

View reviewed changes

Comment thread src/vars/images.yml Outdated

evgeni reviewed May 19, 2026

View reviewed changes

Comment thread docs/developer/deployment.md Outdated

ehelms force-pushed the image-quadlet-support branch from e392b79 to f0e2546 Compare May 19, 2026 16:01

ekohl requested changes May 19, 2026

View reviewed changes

pr-processor Bot added the Waiting on contributor label May 19, 2026

ekohl mentioned this pull request May 19, 2026

[BUG]: podman_image with state: quadlet ignores auth_file parameter containers/ansible-podman-collections#1034

Closed

ehelms force-pushed the image-quadlet-support branch from f0e2546 to 6057acc Compare May 19, 2026 19:12

pr-processor Bot removed the Waiting on contributor label May 19, 2026

pr-processor Bot added the Needs re-review label May 19, 2026

ekohl reviewed May 19, 2026

View reviewed changes

pr-processor Bot removed the Needs re-review label May 19, 2026

arvind4501 reviewed May 20, 2026

View reviewed changes

bochi added a commit to bochi/foremanctl that referenced this pull request May 22, 2026

Merge PR theforeman#501: Podman quadlet .image file support

bec957f

	- "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"
	authfile: "{{ images_registry_auth_file }}"

	- "Environment=REGISTRY_AUTH_FILE={{ images_registry_auth_file }}"
	- "AUTH_FILE={{ images_registry_auth_file }}"

Conversation

ehelms commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

How to test this pull request

Checklist

Uh oh!

ekohl commented May 8, 2026

Uh oh!

ekohl commented May 8, 2026

Uh oh!

ehelms commented May 8, 2026

Updated Proposal

Uh oh!

ekohl commented May 12, 2026

Uh oh!

evgeni commented May 13, 2026

Uh oh!

bochi commented May 13, 2026

Uh oh!

ianballou commented May 13, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arvind4501 May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehelms commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arvind4501 May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ehelms commented May 8, 2026 •

edited

Loading

arvind4501 May 19, 2026 •

edited

Loading

arvind4501 May 20, 2026 •

edited

Loading