Add max_retries and retry_interval to [(api_)database] conf

[melwitt]

Nova's default database connection retry settings (max_retries=10, retry_interval=10) mean nova-api can spend up to 100 seconds retrying a connection to an unavailable database. This exceeds the Kubernetes startup probe window (60 seconds), so the pod gets killed before nova finishes retrying, leading to a CrashLoopBackOff when a cell database is down.

In RHOSO, Kubernetes provides its own higher-level retry mechanism by killing and recreating pods that fail to start. This is preferable to Nova retrying internally because it reports the situation clearly via CR status fields and events, and allows Kubernetes to reschedule the pod to another worker if needed.

Set max_retries to 30 and retry_interval to 1 second for both [database] and [api_database] so that Nova gives up on an unreachable database more quickly and lets Kubernetes handle the recovery.

Resolves: OSPRH-30130

[openshift-ci]

Hi @melwitt. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[gibizer]

Nova's default database connection retry settings (max_retries=-1, retry_interval=10) cause nova-api pods to hang indefinitely when a cell database is unavailable. This prevents the pods from becoming ready within the Kubernetes startup probe window (60 seconds), leading to a CrashLoopBackOff.

I tis not -1 it is 10 by default:
https://docs.openstack.org/nova/latest/configuration/config.html#api_database.max_retries
https://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries

Still enough to slow down startup and trigger the failure of the startup k8s probe that will trigger the kill of the pod.

[gibizer]

Please fix the commit message, otherwise looks good to me.

[gibizer]

looks good. Thank for fixing

[centosinfra-prod-github-app]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/34010c119d374f5ca33c859e2917e1be

❌ openstack-meta-content-provider FAILURE in 4m 23s
⚠️ nova-operator-kuttl SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-meta-content-provider
⚠️ nova-operator-tempest-multinode-ceph SKIPPED Skipped due to failed job openstack-meta-content-provider

[gibizer]

/retest

    - '2026-06-16 12:03:36,198 - repo-setup-get-hash - ERROR - Invalid response received
      from the delorean server. Queried URL: https://trunk.rdoproject.org/centos9-antelope/current-podified/delorean.repo.md5.
      Response code: -1. Response text: The read operation timed out. Failed to create
      HashInfo object.'

[gibizer]

recheck

[openshift-ci]

@melwitt: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/functional	b8071eb	link	true	/test functional

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[gibizer]

The functional test failure is relevant. You can reproduce it via make test locally. We have some tests that are too specific about the content of the generated content. e.g.

nova-operator/test/functional/nova/multicell_test.go

Line 232 in 7ba93ca

"[database]\nconnection = mysql+pymysql://%s:%s@hostname-for-%s.%s.svc/nova_cell0?read_default_file=/etc/my.cnf",

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: melwitt
Once this PR has been reviewed and has the lgtm label, please ask for approval from gibizer. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SeanMooney]

https://docs.openstack.org/nova/latest/configuration/config.html#api_database.max_retries
https://docs.openstack.org/nova/latest/configuration/config.html#api_database.retry_interval

so im not actully sure that relying on k8s retry is actully preferable

the defualt is 10
im not sure turing this to 3 relly helps with stablity in gerneal in ci.

i woudl suggest we increase max_reties to say 30
but lower retry_interval to 1 as your proposing

that keeps the time less the the k8s interval but should be more stable over all.

with that said gibi is right that we will need ot update

nova-operator/test/functional/nova/multicell_test.go

Lines 230 to 246 in 7ba93ca

	ContainSubstring(
	fmt.Sprintf(
	"[database]\nconnection = mysql+pymysql://%s:%s@hostname-for-%s.%s.svc/nova_cell0?read_default_file=/etc/my.cnf",
	cell0Account.Spec.UserName, cell0Secret.Data[mariadbv1.DatabasePasswordSelector],
	cell0.MariaDBDatabaseName.Name, novaNames.Namespace)),
	)
	apiAccount := mariadb.GetMariaDBAccount(novaNames.APIMariaDBDatabaseAccount)
	apiSecret := th.GetSecret(types.NamespacedName{Name: apiAccount.Spec.Secret, Namespace: novaNames.APIMariaDBDatabaseAccount.Namespace})
	Expect(configData).To(
	ContainSubstring(
	fmt.Sprintf(
	"[api_database]\nconnection = mysql+pymysql://%s:%s@hostname-for-%s.%s.svc/nova_api?read_default_file=/etc/my.cnf",
	apiAccount.Spec.UserName, apiSecret.Data[mariadbv1.DatabasePasswordSelector],
	novaNames.APIMariaDBDatabaseName.Name, novaNames.Namespace)),
	)

the simple solution woul db to put this after the existing assert but we shoudl jsut add the new content to them as well so that we dont reguress this overried

[melwitt]

Fair point ... otherwise it would give up after only 3 seconds. I have updated to max_retries = 30.