[Bug] Install failure using deploy script in Debian LXC - database encryption related

[tgmct]

Hey there! 👋 Thanks for taking the time to report this bug — we appreciate it!

Please keep in mind: PegaProx is developed and maintained entirely by volunteers in our free time. We do our best, but we can't work magic 🪄 — please be patient, we'll get to your issue as soon as we can.

Want to help keep PegaProx alive?

• ⭐ Star the project — it's free and helps a lot!
• 💖 Become a Sponsor — helps us dedicate more time to development
• 🤝 Contributing code or docs is always welcome too!

---

Describe the bug

The result of performing an installation using the curl deploy script in a Debian LXC container is that the service FAILS to start.

Steps to Reproduce

1. curl -sSL https://raw.githubusercontent.com/PegaProx/project-pegaprox/refs/heads/main/deploy.sh | sudo bash
2. select option 2 - 443 access

Expected behavior

start process at end of installation

Environment

• PegaProx Version: unknown - does NOT start
• Installation Method: see steps to reproduce
• OS: Debian LXC (13)
• Browser: n/a
• Behind Reverse Proxy? n/a
• UI? n/a

Logs

journalctl -u pegaprox -f
May 14 17:02:09 pegaprox5 pegaprox[3319]: ~~~~~~~~~^^
May 14 17:02:09 pegaprox5 pegaprox[3319]: File "/usr/lib/python3.13/pathlib/_abc.py", line 482, in is_file
May 14 17:02:09 pegaprox5 pegaprox[3319]: return S_ISREG(self.stat(follow_symlinks=follow_symlinks).st_mode)
May 14 17:02:09 pegaprox5 pegaprox[3319]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:09 pegaprox5 pegaprox[3319]: File "/usr/lib/python3.13/pathlib/_local.py", line 517, in stat
May 14 17:02:09 pegaprox5 pegaprox[3319]: return os.stat(self, follow_symlinks=follow_symlinks)
May 14 17:02:09 pegaprox5 pegaprox[3319]: ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:09 pegaprox5 pegaprox[3319]: PermissionError: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:09 pegaprox5 systemd[1]: pegaprox.service: Main process exited, code=exited, status=1/FAILURE
May 14 17:02:09 pegaprox5 systemd[1]: pegaprox.service: Failed with result 'exit-code'.
May 14 17:02:14 pegaprox5 systemd[1]: pegaprox.service: Scheduled restart job, restart counter is at 214.
May 14 17:02:14 pegaprox5 systemd[1]: Started pegaprox.service - PegaProx - Proxmox Cluster Management.
May 14 17:02:14 pegaprox5 pegaprox[3324]: [DBCRYPTO] plain DB detected at /opt/PegaProx/config/pegaprox.db — auto-encrypting before first connection. This is a one-time operation.
May 14 17:02:14 pegaprox5 pegaprox[3324]: [DBCRYPTO] auto-encrypt check failed: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:14 pegaprox5 pegaprox[3324]: Traceback (most recent call last):
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/app.py", line 702, in main
May 14 17:02:14 pegaprox5 pegaprox[3324]: _r = _dbcrypto.ensure_db_encrypted(_db_path)
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/dbcrypto.py", line 200, in ensure_db_encrypted
May 14 17:02:14 pegaprox5 pegaprox[3324]: result = _run_inline_migration(db_path)
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/dbcrypto.py", line 243, in _run_inline_migration
May 14 17:02:14 pegaprox5 pegaprox[3324]: mk = load_master_key()
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/keystore.py", line 85, in load_master_key
May 14 17:02:14 pegaprox5 pegaprox[3324]: _CACHED = _resolve()
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/keystore.py", line 146, in _resolve
May 14 17:02:14 pegaprox5 pegaprox[3324]: if p.is_file():
May 14 17:02:14 pegaprox5 pegaprox[3324]: ~~~~~~~~~^^
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/usr/lib/python3.13/pathlib/_abc.py", line 482, in is_file
May 14 17:02:14 pegaprox5 pegaprox[3324]: return S_ISREG(self.stat(follow_symlinks=follow_symlinks).st_mode)
May 14 17:02:14 pegaprox5 pegaprox[3324]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:14 pegaprox5 pegaprox[3324]: File "/usr/lib/python3.13/pathlib/_local.py", line 517, in stat
May 14 17:02:14 pegaprox5 pegaprox[3324]: return os.stat(self, follow_symlinks=follow_symlinks)
May 14 17:02:14 pegaprox5 pegaprox[3324]: ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:14 pegaprox5 pegaprox[3324]: PermissionError: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:14 pegaprox5 pegaprox[3324]: Error loading server settings from database: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:15 pegaprox5 pegaprox[3324]: [push] inbox table ensure failed: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:15 pegaprox5 pegaprox[3324]: Gevent monkey-patching applied
May 14 17:02:15 pegaprox5 pegaprox[3324]: [ws-patch] simple-websocket PerMessageDeflate disabled
May 14 17:02:15 pegaprox5 pegaprox[3324]: Checking optional libraries...
May 14 17:02:15 pegaprox5 pegaprox[3324]: ✓ websockets (VNC/SSH console)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ✓ paramiko (SSH features)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ✓ gevent (high performance)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ✓ argon2-cffi (secure password hashing)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ✓ XenAPI (XCP-ng integration)
May 14 17:02:15 pegaprox5 pegaprox[3324]: Traceback (most recent call last):
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox_multi_cluster.py", line 228, in
May 14 17:02:15 pegaprox5 pegaprox[3324]: main(debug_mode=debug_mode)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/app.py", line 717, in main
May 14 17:02:15 pegaprox5 pegaprox[3324]: app = create_app()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/app.py", line 340, in create_app
May 14 17:02:15 pegaprox5 pegaprox[3324]: load_enabled_plugins(app)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~~~~~~~~~~~~^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/api/plugins.py", line 225, in load_enabled_plugins
May 14 17:02:15 pegaprox5 pegaprox[3324]: states = _get_plugin_states()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/api/plugins.py", line 120, in _get_plugin_states
May 14 17:02:15 pegaprox5 pegaprox[3324]: db = get_db()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/db.py", line 4111, in get_db
May 14 17:02:15 pegaprox5 pegaprox[3324]: _db = PegaProxDB()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/db.py", line 88, in init
May 14 17:02:15 pegaprox5 pegaprox[3324]: self._init_db()
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~~~~~^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/db.py", line 196, in _init_db
May 14 17:02:15 pegaprox5 pegaprox[3324]: conn = self.conn
May 14 17:02:15 pegaprox5 pegaprox[3324]: ^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/db.py", line 188, in conn
May 14 17:02:15 pegaprox5 pegaprox[3324]: return self._get_connection()
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~~~~~~~~~~~~^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/db.py", line 172, in _get_connection
May 14 17:02:15 pegaprox5 pegaprox[3324]: self._local.conn = dbcrypto.connect(
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~~~~~~~~^
May 14 17:02:15 pegaprox5 pegaprox[3324]: self.db_path,
May 14 17:02:15 pegaprox5 pegaprox[3324]: ^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: check_same_thread=False, # We handle thread safety ourselves
May 14 17:02:15 pegaprox5 pegaprox[3324]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: timeout=30.0
May 14 17:02:15 pegaprox5 pegaprox[3324]: ^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: )
May 14 17:02:15 pegaprox5 pegaprox[3324]: ^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/dbcrypto.py", line 98, in connect
May 14 17:02:15 pegaprox5 pegaprox[3324]: _apply_sqlcipher_pragmas(conn, db_path)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/dbcrypto.py", line 112, in _apply_sqlcipher_pragmas
May 14 17:02:15 pegaprox5 pegaprox[3324]: mk = load_master_key()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/keystore.py", line 85, in load_master_key
May 14 17:02:15 pegaprox5 pegaprox[3324]: _CACHED = _resolve()
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/opt/PegaProx/pegaprox/core/keystore.py", line 146, in _resolve
May 14 17:02:15 pegaprox5 pegaprox[3324]: if p.is_file():
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/usr/lib/python3.13/pathlib/_abc.py", line 482, in is_file
May 14 17:02:15 pegaprox5 pegaprox[3324]: return S_ISREG(self.stat(follow_symlinks=follow_symlinks).st_mode)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: File "/usr/lib/python3.13/pathlib/_local.py", line 517, in stat
May 14 17:02:15 pegaprox5 pegaprox[3324]: return os.stat(self, follow_symlinks=follow_symlinks)
May 14 17:02:15 pegaprox5 pegaprox[3324]: ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 14 17:02:15 pegaprox5 pegaprox[3324]: PermissionError: [Errno 13] Permission denied: '/etc/pegaprox/secret.key'
May 14 17:02:15 pegaprox5 systemd[1]: pegaprox.service: Main process exited, code=exited, status=1/FAILURE
May 14 17:02:15 pegaprox5 systemd[1]: pegaprox.service: Failed with result 'exit-code'.

Screenshots

Checklist

• I have searched existing issues to make sure this is not a duplicate
• I am using the latest version of PegaProx

[tgmct]

Seems to be associated with encrypting the sqlite database. The /etc/pegaprox/secret.key is accessible to root only ( -rw------- 1 root pegaprox 44 May 14 16:41 secret.key ). Install process was run from root.

[mkellermann97]

Hey tgmct — confirmed, your diagnosis is spot on. The systemd unit runs as the pegaprox user; 0600 root:pegaprox only grants read to the owner (root), so the service can't actually open its own key. Hence the boot loop.

Patch landed on the Testing branch: deploy.sh now writes the key as 0640 root:pegaprox for fresh installs, and existing installs sitting at 0600 get auto-bumped on the next update.sh run. Workaround until it rolls into a release:

sudo chmod 640 /etc/pegaprox/secret.key
sudo systemctl restart pegaprox

Nico cuts the releases and he's on holiday through May 18, so this picks up sometime next week — no fixed ETA from me. The chmod above should unblock you in the meantime.

Bug reports with the actual ls -la attached save us a debug session, so cheers for that. If you ever feel like keeping the lights on for the volunteer side of things, the tip jar's at https://opencollective.com/pegaprox.

— Marcus

[elektronen]

Hi. I installed via curl from Testing to see if this fix worked, and I'm afraid it does not...

May 15 18:22:49 xxx-XXXXXXX systemd[1]: pegaprox.service: Scheduled restart job, restart counter is at 11.
May 15 18:22:49 xxx-XXXXXXX systemd[1]: Started pegaprox.service - PegaProx - Proxmox Cluster Management.
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: [DBCRYPTO] plain DB detected at /opt/PegaProx/config/pegaprox.db — auto-encrypting before first connection. This is a one-time operation.
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: Gevent monkey-patching applied
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: [ws-patch] simple-websocket PerMessageDeflate disabled
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: Checking optional libraries...
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: ✓ websockets (VNC/SSH console)
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: ✓ paramiko (SSH features)
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: ✓ gevent (high performance)
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: ✓ argon2-cffi (secure password hashing)
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: ✓ XenAPI (XCP-ng integration)
May 15 18:22:50 xxx-XXXXXXX pegaprox[6393]: [FATAL] [KEYSTORE] key at /etc/pegaprox/secret.key has loose permissions (640) — chmod 0600 required.
May 15 18:22:50 xxx-XXXXXXX systemd[1]: pegaprox.service: Main process exited, code=exited, status=1/FAILURE
May 15 18:22:50 xxx-XXXXXXX systemd[1]: pegaprox.service: Failed with result 'exit-code'.

and it just keeps looping.

[mkellermann97]

@elektronen — ugh, sorry. Root-caused this morning.

What happened: you pulled deploy.sh from Testing, which is correct on its end (it now writes the key at 0640). But deploy.sh then runs git clone --depth 1 ... without a --branch flag, so it always pulled main. Main still has the OLD keystore.py that rejects 0640 with the chmod 0600 required text — which is exactly the error you're seeing. So you ended up with the Testing chmod + main's strict keystore. Mismatch → boot loop.

Immediate unblock for your existing install — don't reinstall, just realign ownership and mode:

sudo chown pegaprox:pegaprox /etc/pegaprox/secret.key
sudo chmod 0600 /etc/pegaprox/secret.key
sudo systemctl restart pegaprox

That makes the key owned by the service user at 0600, which main's keystore.py is happy with. Service should come up cleanly.

Proper fix for the installer landed on Testing as commit 93ff31c. Both deploy.sh and update.sh now honour a PEGAPROX_BRANCH env var. Once that's available, a from-Testing install will be:

PEGAPROX_BRANCH=Testing curl -sSL https://raw.githubusercontent.com/PegaProx/project-pegaprox/Testing/deploy.sh | sudo -E bash

(The -E on sudo is important — it passes the env var through.)

Sandbox-verified: with PEGAPROX_BRANCH unset → clones main (no behaviour change for everyone else). With PEGAPROX_BRANCH=Testing → clones Testing and the keystore.py with the 0640-permissive _enforce_perms actually lands on disk.

Apologies for the dance — the Testing branch is rolling pre-release and there's some chicken-and-egg with installer scripts pulling their own host repo. Once Nico merges Testing → main on his return (May 18), the original chmod 640 fix will be the default and none of this scaffolding will be needed.

Tip jar's at https://opencollective.com/pegaprox if testing the rough edges of pre-release branches earns its keep.

— Marcus

[mkellermann97]

Hey @tgmct + @elektronen — v0.9.10.3 just shipped: https://github.com/PegaProx/project-pegaprox/releases/tag/v0.9.10.3

The keystore-permission fix is in. Fresh installs write the master key at 0640 root:pegaprox so the service user can read it; existing installs at 0600/0400 auto-bump on the next update.sh. Plus PEGAPROX_BRANCH env-var on deploy.sh / update.sh so installing from a branch other than main actually clones that branch end-to-end.

Closing as completed. If anything still feels off after the upgrade, file a fresh ticket — the audit trail stays cleaner that way than reopening this one.

Donations at https://opencollective.com/pegaprox if PegaProx-finally-installing earns its keep.

— Marcus

[tfoks]

Hi @mkellermann97

I think there's even more going wrong during an update.

I wanted to upgrade coming from version 0.9.9.3. After running the update.sh script it should upgrade to the latest version 0.9.10.3 but failed:

[DBCRYPTO] plain DB detected at /opt/PegaProx/config/pegaprox.db — auto-encrypting before first connection. This is a one-time operation.
[DBCRYPTO] auto-encrypt check failed: [Errno 13] Permission denied: '/home/pegaprox/.config/pegaprox/secret.key'

At first it seems to be a permission problem but in fact there's no folder "/home/pegaprox" nor any descendants. Creating the path doesn't change anything.

[mkellermann97]

Hey @tfoks — thanks for flagging, that's a real bug in the v0.9.10 keystore resolver, not a configuration error on your side.

Root cause: the new multi-tier key resolver walks tier 1 → 6 (env vars → /etc/pegaprox/secret.key → ~/.config/pegaprox/secret.key → legacy config/.pegaprox.key). Tier 5 (~/.config/pegaprox/secret.key) uses Path.is_file() to check existence. That returns False on missing-file BUT raises PermissionError when the kernel returns EACCES on a path lookup — which happens when:

• the systemd unit has ProtectHome=true (most LXC-appliance / packaged installs do), so the process sees /home as empty but the home itself returns EACCES
• /home/pegaprox exists but is mode 700 owned by a different user than the service runs as
• mandatory-access ACL blocks lookup

In all three cases tier 5 raised, propagated out of _resolve(), and tier 6 — where your perfectly valid v0.9.9.3 legacy key actually lives — was never checked. So you got the EACCES error and an unbootable service despite having a working key on disk.

Fix landed on Testing as 3bcddde — _safe_is_file() helper wraps Path.is_file() and returns False on any OSError. All six tier checks now use it. Tier 5 raising on EACCES now falls through to tier 6 cleanly, which finds your legacy config/.pegaprox.key and the upgrade completes normally.

Immediate workaround to unblock you right now (pick whichever fits):

Option (a) — move the legacy key to the tier-4 system-service location:

sudo cp /opt/PegaProx/config/.pegaprox.key /etc/pegaprox/secret.key
sudo chown root:pegaprox /etc/pegaprox/secret.key
sudo chmod 0640 /etc/pegaprox/secret.key
sudo systemctl restart pegaprox

Option (b) — point PEGAPROX_KEY_FILE env-var at the legacy location via the service unit:

[Service]
Environment="PEGAPROX_KEY_FILE=/opt/PegaProx/config/.pegaprox.key"

then sudo systemctl daemon-reload && sudo systemctl restart pegaprox.

Both bypass the misbehaving tier 5 check.

Apologies for the bug — this should have been caught in the v0.9.10 release-cut. Will roll into the next release.

Donations at https://opencollective.com/pegaprox if root-causing-on-customer-comments earns its keep.

— Marcus