Skip to content

Commit aa3d9e1

Browse files
committed
fix: add retry logic for database migration during startup recovery
--bug=1067545@tapd-62980211 --user=刘瑞斌 【启动】使用window桌面docker重启容器时,偶发数据库连接失败的问题 https://www.tapd.cn/62980211/s/1891342
1 parent a625975 commit aa3d9e1

File tree

1 file changed

+25
-5
lines changed

1 file changed

+25
-5
lines changed

main.py

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,34 @@ def collect_static():
3232
def perform_db_migrate():
3333
"""
3434
初始化数据库表
35+
wait-for-it 仅检测 TCP 端口,PostgreSQL 崩溃恢复期间端口已开放但拒绝查询,
36+
因此在此处增加重试逻辑,等待数据库真正就绪后再执行 migrate。
3537
"""
3638
logging.info("Check database structure change ...")
3739
logging.info("Migrate model change to database ...")
38-
try:
39-
management.call_command('migrate')
40-
except Exception as e:
41-
logging.error('Perform migrate failed, exit', exc_info=True)
42-
sys.exit(11)
40+
max_retries = 10
41+
retry_interval = 5 # seconds
42+
for attempt in range(1, max_retries + 1):
43+
try:
44+
management.call_command('migrate')
45+
return
46+
except Exception as e:
47+
err_msg = str(e)
48+
# 判断是否为数据库仍在启动中(崩溃恢复场景)
49+
is_db_starting = (
50+
'the database system is starting up' in err_msg
51+
or 'starting up' in err_msg
52+
or 'Connection refused' in err_msg
53+
)
54+
if is_db_starting and attempt < max_retries:
55+
logging.warning(
56+
f'Database is not ready yet (attempt {attempt}/{max_retries}), '
57+
f'retrying in {retry_interval}s... Error: {err_msg}'
58+
)
59+
time.sleep(retry_interval)
60+
else:
61+
logging.error('Perform migrate failed, exit', exc_info=True)
62+
sys.exit(11)
4363

4464

4565
def start_services():

0 commit comments

Comments
 (0)