[BUG] binlog files are not deleted any more

[oliver]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The "binary logs" (binlog files) under data/log/mysql/ are not automatically deleted any more, leading to high disk usage. Usually these files should be automatically deleted after the configured expiration time (expire_logs_days = 10). But I have a Docker container which is based on the "ghcr.io/linuxserver/mariadb" Docker image, and it has accumulated 140 GB of binlog files over the past few months. The oldest files were from March.

This appears to be the issue from https://jira.mariadb.org/browse/MDEV-34504 (also mentioned at https://stackoverflow.com/a/79088897 and https://www.reddit.com/r/archlinux/comments/1e1sa1b/after_the_latest_update_mariadb_11421_does_not/). This issue was introduced in MariaDB 11.4.1, apparently.

I have fixed this for me now by editing the defaults/custom.cnf file and adding this line in the [mysqld] section:

slave_connections_needed_for_purge=0

After restarting the container, the old binlog files were automatically deleted.

Since my database does not use any replication, I think this fix is OK. And I guess this would also be a good fix for the custom.cnf file from linuxserver.io.

Btw. thank you for providing the linuxserver.io Docker images! They have saved quite some time for me.

Expected Behavior

The binlog files should automatically be deleted after 10 days.

Steps To Reproduce

I guess it will take more than 10 days to see that the old binlog files are not deleted; so I cannot easily provide instructions how to reproduce this issue. My guess would be something like this:

• start a Docker container using the ghcr.io/linuxserver/mariadb image.
• do some queries, maybe even write data (I would guess that this is necessary so that actually some binlog files are created, but I haven't tried)
• observe that binlog files are being created as data/log/mysql/mariadb-bin.000001
• wait for 11 days
• check the binlog files in data/log/mysql/ directory:
  • desired: the oldest binlog file has been deleted
  • bad: the mariadb-bin.000001 file still exists; and there is a log message in data/log/mysql/mariadb-error.log like this: [Note] Binary log 'mariadb-bin.000001' is not purged because less than 'slave_connections_needed_for_purge' slaves have processed it

I guess the time to reproduce this issue could be reduced by setting the expire_logs_days config option to 1; or even setting binlog_expire_logs_seconds to a small value, to reproduce the issue after a few minutes?

Environment

- OS: Debian Bookworm
- How docker service was installed: distro's packagemanager

CPU architecture

x86-64

Docker creation

I did not directly run the ghcr.io/linuxserver/mariadb image; instead I used a Docker image built from the https://github.com/mediathekview/plugin.video.mediathekview/blob/master/Dockerfile file, which is based on ghcr.io/linuxserver/mariadb .

I then started the container using this `docker-compose.yml`:


---
services:
    mediathekview_db:
        image: mediathekview-kodi-db:latest
        container_name: mediathekview_db
        environment:
            - PUID=1000
            - PGID=1000
            - TZ=Europe/Berlin
            - MYSQL_DATABASE=mediathekview
            - MYSQL_USER=mediathekview
            - CRON_TIMESPEC=0 4-22/1 * * *
            - RUN_ON_STARTUP=yes
        env_file: /opt/docker-compose/mediathekview_db/docker_env_vars
        volumes:
            - /mnt/data/opt/docker-compose/mediathekview_db/data/:/config
        ports:
            - 43306:3306
        restart: unless-stopped

Container logs

[migrations] started
[migrations] no migrations found
[mediathekview_db] Export environment for use in crontab
[mediathekview_db] Install crontab:  0 4-22/1 * * * python3 /plugin.video.mediathekview/mvupdate3 mysql -v -d mediathekview -u mediathekview -p 1809362084
[mediathekview_db] Create custom stdout and stderr named pipes
[mediathekview_db] Schedule run on startup in 10s
[mediathekview_db] Start cron
usermod: no changes
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────
Linuxserver.io version: 11.4.5-r0-ls177
Build-date: 2025-03-11T07:42:29+00:00
───────────────────────────────────────
    
[custom-init] No custom files found, skipping...
250604 08:45:49 mysqld_safe Logging to '/config/log/mysql/mariadb-error.log'.
250604 08:45:49 mysqld_safe Starting mariadbd daemon with databases from /config/databases
Connection to localhost (127.0.0.1) 3306 port [tcp/mysql] succeeded!
Logrotate is enabled
[ls.io-init] done.
[mediathekview_db] Run on startup: python3 /plugin.video.mediathekview/mvupdate3 mysql -v -d mediathekview -u mediathekview -p 1809362084
2025-06-04 08:45:59.335978 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 08:46:10.169534 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 510 records from ./Filmliste-diff
2025-06-04 08:46:10.512314 INFO [mvupdate3-1.1.3:UpdateFileImport]: 403 records processed in 0 sec. Updated: 338 Inserted: 65 deleted: 0
2025-06-04 09:00:00.546315 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 09:00:00.335433 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 10:00:00.650035 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 10:00:10.045317 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 688 records from ./Filmliste-diff
2025-06-04 10:00:10.459304 INFO [mvupdate3-1.1.3:UpdateFileImport]: 562 records processed in 0 sec. Updated: 388 Inserted: 174 deleted: 0
2025-06-04 10:00:00.356772 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 10:00:10.054829 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 688 records from ./Filmliste-diff
2025-06-04 10:00:10.649198 INFO [mvupdate3-1.1.3:UpdateFileImport]: 562 records processed in 0 sec. Updated: 562 Inserted: 0 deleted: 0
2025-06-04 11:00:00.398652 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 11:00:00.571015 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 12:00:00.383627 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 12:00:10.397103 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 762 records from ./Filmliste-diff
2025-06-04 12:00:10.910560 INFO [mvupdate3-1.1.3:UpdateFileImport]: 624 records processed in 0 sec. Updated: 551 Inserted: 73 deleted: 0
2025-06-04 12:00:00.577421 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 12:00:11.174862 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 762 records from ./Filmliste-diff
2025-06-04 12:00:11.406779 INFO [mvupdate3-1.1.3:UpdateFileImport]: 624 records processed in 0 sec. Updated: 624 Inserted: 0 deleted: 0
2025-06-04 13:00:00.630116 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 13:00:00.433816 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 14:00:00.426419 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 14:00:10.431377 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 807 records from ./Filmliste-diff
2025-06-04 14:00:10.912519 INFO [mvupdate3-1.1.3:UpdateFileImport]: 664 records processed in 0 sec. Updated: 611 Inserted: 53 deleted: 0
2025-06-04 14:00:00.680761 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 14:00:10.429935 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 807 records from ./Filmliste-diff
2025-06-04 14:00:11.146590 INFO [mvupdate3-1.1.3:UpdateFileImport]: 664 records processed in 0 sec. Updated: 664 Inserted: 0 deleted: 0
2025-06-04 15:00:00.461207 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 15:00:00.701319 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 16:00:00.717270 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 16:00:10.684312 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 897 records from ./Filmliste-diff
2025-06-04 16:00:11.274820 INFO [mvupdate3-1.1.3:UpdateFileImport]: 743 records processed in 0 sec. Updated: 666 Inserted: 77 deleted: 0
2025-06-04 16:00:00.480737 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 16:00:10.699707 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 897 records from ./Filmliste-diff
2025-06-04 16:00:11.537540 INFO [mvupdate3-1.1.3:UpdateFileImport]: 743 records processed in 0 sec. Updated: 743 Inserted: 0 deleted: 0
2025-06-04 17:00:00.480499 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 17:00:00.694284 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 18:00:10.544106 ERROR [mvupdate3-1.1.3:UpdateFileDownload]: bz2 decompression failed: Invalid data stream
2025-06-04 18:00:10.544185 ERROR [mvupdate3-1.1.3:UpdateFileDownload]: Failure decompress Invalid data stream
2025-06-04 18:00:00.705498 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
Traceback (most recent call last):
  File "/plugin.video.mediathekview/mvupdate3", line 19, in <module>
    APP.run()
  File "/plugin.video.mediathekview/resources/lib/mvupdate.py", line 206, in run
    updateop = self.updater.doUpdate()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/plugin.video.mediathekview/resources/lib/updater.py", line 163, in doUpdate
    ufd.downloadIncrementalUpdateFile()
  File "/plugin.video.mediathekview/resources/lib/updateFileDownload.py", line 88, in downloadIncrementalUpdateFile
    check = self._download(downloadUrl, self._compressedFilename, self._filename)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/plugin.video.mediathekview/resources/lib/updateFileDownload.py", line 190, in _download
    retval = self._decompress_bz2(compressedFilename, targetFilename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/plugin.video.mediathekview/resources/lib/updateFileDownload.py", line 214, in _decompress_bz2
    dstfile.write(decompressor.decompress(data))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Invalid data stream
2025-06-04 18:00:00.497696 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 18:00:10.663347 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 970 records from ./Filmliste-diff
2025-06-04 18:00:11.359640 INFO [mvupdate3-1.1.3:UpdateFileImport]: 807 records processed in 0 sec. Updated: 732 Inserted: 75 deleted: 0
2025-06-04 19:00:00.758935 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 19:00:00.495438 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 20:00:00.535053 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 20:00:12.078719 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 1118 records from ./Filmliste-diff
2025-06-04 20:00:12.831220 INFO [mvupdate3-1.1.3:UpdateFileImport]: 941 records processed in 0 sec. Updated: 809 Inserted: 132 deleted: 0
2025-06-04 20:00:00.824991 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 20:00:14.029860 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 1118 records from ./Filmliste-diff
2025-06-04 20:00:14.834324 INFO [mvupdate3-1.1.3:UpdateFileImport]: 941 records processed in 0 sec. Updated: 941 Inserted: 0 deleted: 0
2025-06-04 21:00:00.746277 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 21:00:00.558550 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
Caught SIGTERM signal!
[migrations] started
[migrations] no migrations found
[mediathekview_db] Export environment for use in crontab
[mediathekview_db] Install crontab:  0 4-22/1 * * * python3 /plugin.video.mediathekview/mvupdate3 mysql -v -d mediathekview -u mediathekview -p 1809362084
[mediathekview_db] Create custom stdout and stderr named pipes
[mediathekview_db] Schedule run on startup in 10s
[mediathekview_db] Start cron
usermod: no changes
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────
Linuxserver.io version: 11.4.5-r0-ls177
Build-date: 2025-03-11T07:42:29+00:00
───────────────────────────────────────
    
[custom-init] No custom files found, skipping...
250604 21:29:26 mysqld_safe Logging to '/config/log/mysql/mariadb-error.log'.
250604 21:29:26 mysqld_safe Starting mariadbd daemon with databases from /config/databases
Connection to localhost (127.0.0.1) 3306 port [tcp/mysql] succeeded!
Logrotate is enabled
[ls.io-init] done.
[mediathekview_db] Run on startup: python3 /plugin.video.mediathekview/mvupdate3 mysql -v -d mediathekview -u mediathekview -p 1809362084
2025-06-04 21:29:36.684272 INFO [mvupdate3-1.1.3:mvupdate]: Python Version3.12.9 (main, Feb  7 2025, 17:39:29) [GCC 14.2.0]
2025-06-04 21:29:48.841946 INFO [mvupdate3-1.1.3:UpdateFileImport]: Starting import of approximately 1170 records from ./Filmliste-diff
2025-06-04 21:29:49.581518 INFO [mvupdate3-1.1.3:UpdateFileImport]: 985 records processed in 0 sec. Updated: 912 Inserted: 73 deleted: 0

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[robotfishe]

Thanks for this @oliver! I was wondering why my logs were 7GB and found my way here, and your solution worked for me. Maybe slave_connections_needed_for_purge=0 should be included in the default config file in the future?

[LinuxServer-CI]

This issue is locked due to inactivity