TLS private keys leaked in controller logs on runtime cert update failure

[kkozlow]

Note: This issue was analyzed and written with the help of AI — I apologize for any inaccuracies in the description. The underlying problem and the fix have been verified manually.

Summary

When a runtime certificate update fails (e.g., due to malformed PEM in a Kubernetes Secret), the full PEM payload including the TLS private key is logged at INFO level by the ingress controller. Anyone with access to controller logs can extract private keys for any TLS certificate managed by the controller.

Related Work

I noticed that PR #784 by @Lappihuan addresses the same root cause trigger (multiple consecutive newlines in PEM data causing NO_START_LINE). That fix prevents this specific failure scenario by normalizing newlines before sending the payload to HAProxy. However, the PEM-in-logs exposure remains for any other runtime cert error (cert/key mismatch, socket timeout, transaction conflict, etc.) — the private key leak I'm describing here is a separate, broader security concern that exists independently of the newline trigger.

Root Cause

From my analysis, the issue spans two repositories (client-native and kubernetes-ingress):

1. client-native — runtime/runtime_single_client.go, ExecuteWithResponse():

When the HAProxy runtime API returns an error, ExecuteWithResponse() embeds the full command in the error message:

return "", fmt.Errorf("[%c] %s [%s]", rawdata[1], rawdata[4:], command)

For certificate operations, command contains the full set ssl cert heredoc with the entire PEM payload:

// runtime/certs.go – SetCertificate()
cmd := fmt.Sprintf("set ssl cert %s <<\n%s\n", storageName, payload)

So the resulting error string looks like:

[3]  unable to load certificate from file '/path/cert.pem': NO_START_LINE. [...] [set ssl cert /path/cert.pem <<
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASC...
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDazCCAlOgAwIBAgIUY3K3MkYVBFPq...
-----END CERTIFICATE-----
]

2. kubernetes-ingress — pkg/haproxy/certs/main.go, writeCert():

This error is passed directly to instance.Reload() without any sanitization:

updated, err := c.updateRuntime(filename, content, isCa)
if err != nil {
    instance.Reload("Runtime update of cert file '%s' failed : %s", filename, err.Error())
}

instance.Reload() logs the message at INFO level, making the private key visible in controller logs. The same pattern exists for deleteRuntime errors a few lines above.

Impact

• TLS private keys are exposed in plain text in controller logs
• Any user, system, or log aggregation pipeline with read access to controller logs can extract private keys
• The leak happens for every runtime cert update/delete failure, not just a specific error type
• Common triggers: malformed PEM in Secret (e.g., trailing empty line causing heredoc termination), cert/key mismatch, HAProxy socket issues

Steps to Reproduce

1. Deploy the HAProxy Kubernetes Ingress Controller
2. Create a TLS Secret with a trailing empty line in tls.key:

   # Add an extra newline at the end of the key file:
   printf '\n' >> server.key
   kubectl create secret tls test-cert --cert=server.crt --key=server.key

3. Create an Ingress resource referencing this secret
4. Observe controller logs — the Runtime update of cert file ... failed message contains the full private key

Expected Behavior

Error messages passed to instance.Reload() (and any other log call) should never contain PEM-encoded blocks. The diagnostic information (file path, HAProxy error description, OpenSSL error code) should be preserved, but cryptographic material must be stripped.

Suggested Fix

Option A (defense-in-depth in kubernetes-ingress):

Sanitize errors before logging in writeCert() and refreshCerts():

var pemBlockRegexp = regexp.MustCompile(`(?s)-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----`)

func certErrorForLog(err error) string {
    if err == nil {
        return ""
    }
    return pemBlockRegexp.ReplaceAllString(err.Error(), "[REDACTED]")
}

Then replace err.Error() with certErrorForLog(err) in the two instance.Reload() calls.

Option B (root cause fix in client-native):

Avoid embedding the full command in error messages from ExecuteWithResponse() when the command contains sensitive data. For example, truncate or omit the command for set ssl cert operations.

Ideally both fixes should be applied — together with #784 for complete coverage.

I already have option A implemented and tested in my deployment and opened a PR with this fix: #786.

Additional Context

I discovered this issue when investigating NO_START_LINE errors caused by Kubernetes Secrets containing PEM files with trailing empty lines. The trailing \n\n in tls.key prematurely terminates the HAProxy stat socket heredoc (set ssl cert ... <<), causing HAProxy to receive only the private key without the certificate. The resulting error then exposes the full PEM in logs via the mechanism described above.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.