[Bug]: salt.exceptions.AuthenticationError: message authentication failed from minions

[byoungstrom]

What happened?

After upgrading from 3007.1 to 3007.13, we began seeing minions fail in several ways. One consistent error is "message authentication failed". The minions begin to report the error after about 24 hours after restarting the master nodes. Anther common error is in getting pillar values. I can file a separate bug for this if they have different causes. We upgraded to 3007.14 yesterday, error persists.

We have 3 master nodes with shared gluster storage in a Master Cluster arrangement. No haproxy. All minions are configured to talk with all three master nodes.

Type of salt install

Official rpm

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

rockylinux-9

salt --versions-report output

Salt Version:
          Salt: 3007.14

Python Version:
        Python: 3.10.19 (main, Feb 17 2026, 23:32:03) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.46
        Jinja2: 3.1.6
       libgit2: 1.9.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.18.2
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.4
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: rocky 9.5 Blue Onyx
        locale: utf-8
       machine: x86_64
       release: 5.14.0-503.33.1.el9_5.x86_64
        system: Linux
       version: Rocky Linux 9.5 Blue Onyx

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here's some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt's Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• GitHub Discussions

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We're glad you've joined our community and look forward to doing awesome things with you!

[byoungstrom]

The full backtrace is

[WARNING] Error: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 310, in _crypted_transfer
    ret = yield _do_transfer()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 783, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 802, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 300, in _do_transfer
    data = self.auth.session_crypticle.loads(data, raw, nonce=nonce)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1746, in loads
    data = self.decrypt(data)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1718, in decrypt
    raise AuthenticationError("message authentication failed")
salt.exceptions.AuthenticationError: message authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 4888, in call_highstate
    top = self.get_top()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 4277, in get_top
    tops = self.get_tops(context=context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3953, in get_tops
    contents = self.client.cache_file(self.opts["state_top"], saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 187, in cache_file
    return self.get_url(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 516, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1186, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1436, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1429, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1419, in __hash_and_stat_file
    return self._channel_send(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1150, in _channel_send
    return self.channel.send(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 139, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 147, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 547, in run_sync
    return future_cell["future"].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 796, in run
    yielded = self.gen.throw(exc)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 353, in send
    ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 783, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 796, in run
    yielded = self.gen.throw(exc)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 314, in _crypted_transfer
    ret = yield _do_transfer()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 783, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/gen.py", line 802, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 300, in _do_transfer
    data = self.auth.session_crypticle.loads(data, raw, nonce=nonce)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1746, in loads
    data = self.decrypt(data)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 1718, in decrypt
    raise AuthenticationError("message authentication failed")
salt.exceptions.AuthenticationError: message authentication failed