Containers won't start when enabling the postfix-exporter in versions >= 5.0.0

[mohazza00]

Starting from version 5.0.0, when enabling the postfix-exporter the containers won't start, and I'll get this error:

★★★★★ POSTFIX STARTING UP (debian) ★★★★★
‣ NOTE  System accounts: postfix=100:102, opendkim=101:104. Careful when switching distros.
‣ INFO  Not setting any timezone for the container
‣ DEBUG /tmp writable.
‣ INFO  Using plain log format for rsyslog.
‣ INFO  Removing /var/log/mail.log from /etc/logrotate.d/logrotate.conf
sed: cannot rename /etc/logrotate.d/sedYk0RXB: Device or resource busy

Do you have any idea what could be wrong?

[CoolSpot]

Can confirm.
The root cause is that the startup script (I believe this one) does a sed -i rewrite of /etc/logrotate.d/logrotate.conf.
On Kubernetes, if that file is a mounted file (subPath or other injection), sed does atomic rename and fails with Device or resource busy.

The /etc/logrotate.d/logrotate.conf file comes from this configmap-metrics.yaml, so it is immutable from within a container.
Even though the script checks if file is writable:

if [ -w "$default_logrotate_conf_path" ]

This check won't indicate if underlying file is immutable because it is a k8s configmap mount.
One way to fix this (but maybe defeating logrotate logic, idk) is to mount the file as read-only so it is marked so on the fs level:

volumeMounts:
  - name: metrics-config
    mountPath: /etc/logrotate.d/logrotate.conf
    subPath: logrotate.conf
    readOnly: true # mark as read-only on fs level, so logrotate_remove_duplicate_mail_log() can properly detect that

[bokysan]

Huh, this is a conundrum. Not sure what would be the best way to solve it.

Possibly copying over logrotate.conf into another location on startup? Any ideas?

[CoolSpot]

@bokysan , If we have full control over logrotate.conf contents via helm templates, why there could be any "duplicates" to be removed at all?
I don't know the exact logic of all this, but if the /var/log/mail.log entry needs to be removed from the logrotate.conf file, then it can be removed in the Helm template (conditionally depending on whatever is the logic), so the startup bash script doesn't have to do that.
Mounting logrotate.conf with readOnly: true allows startup script to remain unchanged - it will skip attempt to modify logrotate.conf if it is read-only on the fs level.

[bokysan]

It's not about removing mail.log. It's about adding it.

The image by default logs to the standard out. This is the gold standard of containers so you can inspect the logs and pipe them to your favorite log service.

Postfix exporter, however, needs to have a log file to read from to work. That's where /var/log/mail.log comes in. So, if you enable postfix-exporter, logs need to be written to the log file additionaly to the standard out. And you want to keep the log file relatively small (aggressive rotation) to not let it overflow the disk.

As to why this is done in bash - because everybody's use case is different and there are actually quite some people running this image directly in Docker with postfix-exporterenabled. The idea was to have one common solution to the problem.

[CoolSpot]

It's not about removing mail.log. It's about adding it.

The part that is failing clearly tries to remove not add:

logrotate_remove_duplicate_mail_log() {
	# /etc/logrotate.d/logrotate.conf does not exist on a default install of Alpine
	local default_logrotate_conf_path="/etc/logrotate.d/logrotate.conf"
	local rsyslog_logrotate_conf_path="/etc/logrotate.d/rsyslog"

	if [[ -f "$default_logrotate_conf_path" ]]; then
		if [ -w "$default_logrotate_conf_path" ] && egrep -q '^/var/log/mail.log' "$default_logrotate_conf_path"; then
			info "Removing /var/log/mail.log from $default_logrotate_conf_path"
			sed -i -E '/^\/var\/log\/mail.log/d' "$default_logrotate_conf_path"
		fi
	elif [[ -f "$rsyslog_logrotate_conf_path" ]]; then
		if [ -w "$rsyslog_logrotate_conf_path" ] && egrep -q '^/var/log/mail.log' "$rsyslog_logrotate_conf_path"; then
			info "Removing /var/log/mail.log from $rsyslog_logrotate_conf_path"
			sed -i -E '/^\/var\/log\/mail.log/d' "$rsyslog_logrotate_conf_path"
		fi
	fi
}

If it just skips removing (e.g. by checking if the file is read-only), everything should be good.

[mohazza00]

How was it working in versions before 5.0.0?

[dancollins]

I tried editing the stateful set to apply logrotate config as readonly, but get the same error message from sed. I tried disabling logrotate, but then I get critical errors from postfix with file permission issues.

FWIW I was using an NFS storage class, so perhaps that was messing with it?