[Bug]: Minion does not restart after crash

[prensing]

What happened?

Occasionally, a minion on Linux will crash. Obviously not good, but it should then attempt to restart so that it is not completely offline.

I think the systemd service file should be augmented to include restart capability. This would include lines for "Restart" and probably some limits like with "StartLimitIntervalSec" and "StartLimitBurst".

Type of salt install

Official deb

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output

Salt Version:
          Salt: 3008.0rc4
 
Python Version:
        Python: 3.14.5 (main, May 12 2026, 03:56:57) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 48.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.49
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.11
         smmap: 5.0.3
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: ubuntu 22.04.5 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-143-generic
        system: Linux
       version: Ubuntu 22.04.5 jammy

[campbellmc]

In 3007.14 on Rocky 9 too - unhandled exception results in service failed.

[OrangeDog]

When configuring this, you should also disable Salt's own attempt at it:

[Service]
ExecStart=
ExecStart=/usr/bin/salt-minion --disable-keepalive
KillMode=mixed
Restart=on-failure
RestartSec=15
StartLimitBurst=3
StartLimitInterval=300

[prensing]

When configuring this, you should also disable Salt's own attempt at it:
....

I am not sure that this is relevant. Based on my situation, the minion is crashing, so its keep-alive code can't do anything. My suggestion is that you would want the internal keep-alive working, but the systemd restart is for a "last ditch" fix.

I do agree with the restart entries in the Service file, apart from disabling keep-alive.

[OrangeDog]

If you're having systemd do it, there's no point having Salt do it too. It's just a waste of resources having another process running just for restarting the first one.

See also #68544, #68406.

[prensing]

OK, maybe. I don't know the code, but I assume that an internal keep-alive does more than just catch exceptions, and maybe is quicker at restarting from scratch.

Either way, right now, the minion can crash and is then dead until there is outside intervention.

[campbellmc]

This is the crux of the matter. Unhandled exceptions crashing the minion.