Admin API user renew updates panel DB but sing-box protocols (Hy2/TUIC/Mieru/Naive) stay stale until manual apply-users

[mktwix]

Summary

When creating or renewing users via the Admin API (PATCH /api/v2/admin/user/{uuid}/), the panel database is updated correctly, but sing-box-based protocols (Hysteria2, TUIC, Mieru, Naive) may continue serving an old user list until apply-users / apply_configs is run manually on the server.

VLESS (Xray) works after API update because Xray can hot-reload via API. Hy2/TUIC/Mieru/Naive fail because they depend on regenerated static sing-box config files + service reload.

This makes external automation (billing panels, renewal scripts) appear successful while users cannot connect on several protocols.

Environment

• Hiddify Manager: 12.3.3
• OS: Ubuntu (multiple VPS nodes)
• Invocation: External automation via Admin API (not panel UI)
• Affected nodes: 3 independent edge servers (same version, same behavior)

Steps to reproduce

1. Have an existing user with Hy2/TUIC/Mieru/Naive enabled.
2. Let the user expire (or disable), then renew via Admin API:
   • PATCH /api/v2/admin/user/{uuid}/ with updated package_days, start_date, enable: true, etc.
3. Confirm API returns success and GET /api/v2/admin/user/{uuid}/ shows the renewed user as enabled.
4. Try connecting with:
   • VLESS → works
   • Hysteria2 / TUIC / Mieru / Naive → auth fails / user not recognized
5. On server, inspect sing-box configs:

   grep -rl '<uuid>' /opt/hiddify-manager/singbox/configs/*.json

   UUID may be missing from Hy2/TUIC/Mieru/Naive inbound files, or files updated but service not reloaded.
6. Check sing-box service age:

   systemctl show hiddify-singbox -p ActiveEnterTimestamp

   In our case it was still running a config from 2 days earlier, despite successful API renew.

Actual result

• Panel/API user record is correct (enable=true, correct package_days, limits, dates).
• VLESS subscriptions work immediately.
• Hy2/TUIC/Mieru/Naive subscriptions fail until manual apply.
• Automatic background apply often does not complete.

What we observed on the server

1) apply_configs can hang indefinitely (non-interactive)

apply_configs.sh → install.sh apply_configs → python -m cli_progress ... runs but never exits.

ps -forest -o pid,etime,cmd
  bash apply_configs.sh
    install.sh apply_configs
      python -m cli_progress ... ./install.sh apply_configs --no-gui --no-log   # R state, 17+ minutes

• /opt/hiddify-manager/log/system/0-install.log stayed 0 bytes
• No 0-install.lock present
• hiddify-singbox remained active with an old ActiveEnterTimestamp
• Likely related to Apply Configuration hangs forever when run without a TTY (cli_progress urwid loop never exits) #5479 (cli_progress urwid loop without TTY)

2) apply-users often fails with exit 12 (install lock)

Another installation is running.... Please wait until it finishes or execute 'rm $LOCK_FILE'
exit 12

Happens when a concurrent/stuck apply_configs is running or left a stale lock.

3) Manual apply-users fixes it

After killing the stuck cli_progress process and running:

/opt/hiddify-manager/common/commander.py apply-users
systemctl restart hiddify-singbox

• Hy2/TUIC/Mieru/Naive configs regenerated with the UUID present
• Protocols started working again

Expected result

After Admin API user create/renew/update:

1. Panel DB is updated (already works)
2. apply-users runs reliably in background
3. sing-box configs are regenerated
4. hiddify-singbox is reloaded/restarted
5. All protocols (including Hy2/TUIC/Mieru/Naive) reflect the change without manual SSH intervention

Related issues

This appears to be a combination of several open bugs/limitations:

• Apply Configuration hangs forever when run without a TTY (cli_progress urwid loop never exits) #5479 — apply_configs / cli_progress hangs without TTY (we hit this directly)
• Background commander fails to apply user changes due to missing thread argument #5498 — background commander(...) thread missing cmd argument, so panel-triggered apply-users silently fails
• Hysteria and Tuic new Users #4926 — new users not added to Hy2/TUIC sing-box JSON files

Suggested fixes

1. Fix cli_progress non-TTY hang (Apply Configuration hangs forever when run without a TTY (cli_progress urwid loop never exits) #5479) — bypass urwid when !sys.stdout.isatty()
2. Fix background commander thread (Background commander fails to apply user changes due to missing thread argument #5498) — pass args=(base_cmd,) to cmd_in_back
3. After Admin API user mutations, ensure apply-users is queued and verified (and surface failures to API caller / logs)
4. Consider documenting that API-only updates are not sufficient for sing-box protocols without a successful apply-users
5. Add health check: compare panel user UUID set vs sing-box inbound UUID set; alert on drift

Workaround (what we implemented externally)

For anyone automating renewals via API:

# after API user update
/opt/hiddify-manager/common/commander.py apply-users
systemctl restart hiddify-singbox   # if reload did not happen

With retries when exit 12 (lock held).

---

Happy to provide additional logs or test a patched build. This is affecting production renewals on multiple nodes.

[mktwix]

Opened PRs for the apply hang fix:

• fix: bypass cli_progress wrapper when stdout is not a TTY #5506 (bypass cli_progress in utils.sh when no TTY)
• fix: bypass urwid UI when stdout is not a TTY cli_progress#8 (same fix in cli_progress main.py)

For the background commander / apply-users silent failure (#5498), there is already an open fix: hiddify/hiddifypanel#32