Support guidelines
I've found a bug and checked that ...
Description
When running fail2ban as a docker container on the same host as my nginx proxy manager docker container the system slows down over a 24 hour period
Expected behaviour
For the system to not slow down even if left alone for a week or more
Actual behaviour
I have two servers that are functionally identical - two Raspberry Pi 4B 2GB boards with 120GB SSD via USB3 connectors. They run edge services using docker compose files.
edge1: Linux edge1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
edge2: Linux edge2 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
edge1 containers
```BASH
$ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1)
NAMES SIZE NETWORKS STATUS CREATED STATE
acme-sh-1 2B (virtual 40.3MB) edge1 Up 3 minutes 4 days ago running
diun-1 0B (virtual 54.6MB) edge1 Up 3 minutes 4 days ago running
fail2ban 3.12kB (virtual 90.9MB) host Up 3 minutes (healthy) 4 days ago running
netdata-1 707kB (virtual 395MB) edge1 Up 3 minutes (healthy) 23 hours ago running
npm 42.4MB (virtual 858MB) edge1 Up 3 minutes (healthy) 4 days ago running
pihole-1 74.2MB (virtual 368MB) host Up 3 minutes (healthy) 23 hours ago running
promtail-1 0B (virtual 185MB) edge1 Up 3 minutes 3 days ago running
scrutiny-collector-1 9.37kB (virtual 189MB) edge1 Up 3 minutes 4 days ago running
wireguard-1 23kB (virtual 167MB) edge1 Up 3 minutes 4 days ago running
```
edge2 containers
```BASH
$ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1)
NAMES SIZE NETWORKS STATUS CREATED STATE
acme-sh-2 2B (virtual 40.3MB) edge2 Up 23 hours 4 days ago running
diun-2 0B (virtual 54.6MB) edge2 Up 23 hours 4 days ago running
duckdns 21.7kB (virtual 40.8MB) edge2 Up 22 hours 22 hours ago running
netdata-2 26.6kB (virtual 394MB) edge2 Up About a minute (healthy) About a minute ago running
pihole-2 128MB (virtual 422MB) host Up 23 hours (healthy) 23 hours ago running
promtail-2 0B (virtual 185MB) edge2 Up 23 hours 3 days ago running
scrutiny-collector-2 9.37kB (virtual 189MB) edge2 Up 23 hours 4 days ago running
wireguard-2 23kB (virtual 167MB) edge2 Up 23 hours 4 days ago running
```
Generally, both servers need no daily maintenance and the Diun jobs run on Sundays to let me know what containers need updating (thanks for that, btw!).
I wanted to implement fail2ban in a docker container and used this version based on the good work done with Diun. I found a tutorial and followed it, amending the configuration to suit my needs. It works, and I sometimes get ban notifications.
However, after a day or so of usage the following will occur:
- System load increases
- Disk I/O increases
- Zombie processes spike
- Threads increase
According to my netdata metrics, this is due to fail2ban. I'm just not sure how to see what it was doing at the time.
Steps to reproduce
- Start server
- Check all docker services are running
- Wait 24+ hours
- Monitor system performance
Docker info
Client: Docker Engine - Community
Version: 24.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.19.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 9
Server Version: 24.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.21-v8+
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.855GiB
Name: id-edge1
ID: 7HKR:XS7T:KP55:WXII:JDYT:SCR5:EF3N:PHZA:XOAK:EWEP:3J7W:NWFN
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Docker Compose config
name: fail2ban
services:
fail2ban:
cap_add:
- NET_ADMIN
- NET_RAW
container_name: fail2ban
dns:
- 192.168.1.1
environment:
F2B_DB_PURGE_AGE: 14d
F2B_LOG_LEVEL: WARNING
F2B_LOG_TARGET: STDOUT
TZ: America/Edmonton
hostname: edge1
image: ghcr.io/crazy-max/fail2ban:latest
network_mode: host
restart: unless-stopped
volumes:
- type: bind
source: /srv/fail2ban/data
target: /data
bind:
create_host_path: true
- type: bind
source: /srv/nginx-proxy-manager/data/logs
target: /var/log/npm
read_only: true
bind:
create_host_path: true
- type: bind
source: /etc/timezone
target: /etc/timezone
read_only: true
bind:
create_host_path: true
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
bind:
create_host_path: true
Logs
fail2ban | Setting timezone to America/Edmonton...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-07 15:49:10,951 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-07 15:49:10,956 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-07 15:49:10,958 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-07 15:49:10,959 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-07 15:49:10,959 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-07 15:49:10,967 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-07 15:49:10,969 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-07 15:49:10,998 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-07 15:49:11,002 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-07 15:49:11,008 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-07 15:49:11,011 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-07 15:49:11,014 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-07 15:49:11,016 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-07 15:49:11,043 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-07 15:49:11,045 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-07 15:49:11,047 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-07 15:49:11,051 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-07 15:49:11,053 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-07 15:49:11,055 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | Setting timezone to America/Edmonton...
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-08 23:00:12,148 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,158 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-08 23:00:12,161 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-08 23:00:12,162 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-08 23:00:12,162 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-08 23:00:12,197 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,201 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-08 23:00:12,254 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-08 23:00:12,269 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-08 23:00:12,283 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-08 23:00:12,285 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-08 23:00:12,289 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-08 23:00:12,295 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-08 23:00:12,356 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,358 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-08 23:00:12,365 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-08 23:00:12,379 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-08 23:00:12,381 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-08 23:00:12,397 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | 2023-07-08 23:00:18,583 fail2ban.utils [1]: ERROR 7f9a787270 -- exec: bash /data/action.d/telegram_notif.sh -a start
fail2ban | 2023-07-08 23:00:18,583 fail2ban.utils [1]: ERROR 7f9a787270 -- returned 6
fail2ban | 2023-07-08 23:00:18,584 fail2ban.actions [1]: ERROR Failed to start jail 'npm-general-f-b' action 'action-ban-docker-f-b': Error starting action Jail('npm-general-f-b')/action-ban-docker-f-b: 'Script error'
fail2ban | 2023-07-09 11:50:38,474 fail2ban.actions [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned
[repeated every 5 minutes until]
fail2ban | 2023-07-09 12:58:15,383 fail2ban.actions [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned
fail2ban | Setting timezone to America/Edmonton...
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | ln: /etc/localtime: File exists
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-10 08:24:32,151 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,157 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-10 08:24:32,160 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-10 08:24:32,161 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-10 08:24:32,162 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-10 08:24:32,176 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,178 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-10 08:24:32,223 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-10 08:24:32,227 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-10 08:24:32,235 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-10 08:24:32,237 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-10 08:24:32,243 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-10 08:24:32,245 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-10 08:24:32,304 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,306 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-10 08:24:32,309 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-10 08:24:32,312 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-10 08:24:32,325 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-10 08:24:32,329 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
fail2ban | 2023-07-11 09:46:53,185 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.8.92 already banned
fail2ban | 2023-07-11 17:05:32,996 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned
[repeated every minute until]
fail2ban | 2023-07-11 17:10:14,200 fail2ban.actions [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned
fail2ban | 2023-07-12 13:24:47,594 fail2ban.observer [1]: ERROR can't start new thread
fail2ban | Traceback (most recent call last):
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 221, in run
fail2ban | meth(*ev[1:])
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 361, in db_purge
fail2ban | self.add_named_timer('DB_PURGE', self.__db_purge_interval, 'db_purge')
fail2ban | File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 119, in add_named_timer
fail2ban | t.start()
fail2ban | File "/usr/lib/python3.10/threading.py", line 935, in start
fail2ban | _start_new_thread(self._bootstrap, ())
fail2ban | RuntimeError: can't start new thread
fail2ban | Setting timezone to America/Edmonton...
fail2ban | ln: /etc/localtime: File exists
fail2ban | Setting SSMTP configuration...
fail2ban | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban | Initializing files and folders...
fail2ban | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban | Setting Fail2ban configuration...
fail2ban | Checking for custom actions in /data/action.d...
fail2ban | WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban | Add custom action action-ban-docker-f-b.conf...
fail2ban | WARNING: telegram_notif.sh already exists and will be overriden
fail2ban | Add custom action telegram_notif.sh...
fail2ban | Checking for custom filters in /data/filter.d...
fail2ban | WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban | Add custom filter npm-general-f-b.conf...
fail2ban | 2023-07-12 14:10:04,526 fail2ban.configreader [1]: INFO Loading configs for fail2ban under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,531 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-12 14:10:04,535 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban | 2023-07-12 14:10:04,536 fail2ban [1]: INFO Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban | 2023-07-12 14:10:04,536 fail2ban [1]: INFO Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban | 2023-07-12 14:10:04,558 fail2ban.configreader [1]: INFO Loading configs for jail under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,560 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.conf']
fail2ban | 2023-07-12 14:10:04,593 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban | 2023-07-12 14:10:04,602 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban | 2023-07-12 14:10:04,606 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban | 2023-07-12 14:10:04,608 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-12 14:10:04,615 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban | 2023-07-12 14:10:04,617 fail2ban.configreader [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban | 2023-07-12 14:10:04,655 fail2ban.configreader [1]: INFO Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,657 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-12 14:10:04,664 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban | 2023-07-12 14:10:04,671 fail2ban.configreader [1]: INFO Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban | 2023-07-12 14:10:04,673 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | 2023-07-12 14:10:04,677 fail2ban.configparserinc[1]: INFO Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban | Server ready
Additional info
The latest issue occurred just before 2023-07-12 01:00 MDT. The only entry in the docker logs for this day was "ERROR can't start new thread" at 13:24 MDT, likely when I ssh'd into the server to check why it was so slow.
The fail2ban container can see the NPM logs:
$ docker exec fail2ban ls -lha /var/log/npm/error.log
-rwxrwxr-x 1 1000 1000 25.6K May 26 12:11 /var/log/npm/error.log
The database appears to be valid:
$ ls -lha data/db/
total 212K
drwxrwxr-x 2 idsvc docker 4.0K Jun 27 14:28 .
drwxrwxr-x 6 idsvc docker 4.0K Jul 11 14:48 ..
-rwxrwxr-x 1 idsvc docker 204K Jul 12 14:34 fail2ban.sqlite3
The issue appears to be iowait related:

Just before I rebooted the server, the number of fail2ban processes was at 1,655:

What am I missing here?
Support guidelines
I've found a bug and checked that ...
Description
When running fail2ban as a docker container on the same host as my nginx proxy manager docker container the system slows down over a 24 hour period
Expected behaviour
For the system to not slow down even if left alone for a week or more
Actual behaviour
I have two servers that are functionally identical - two Raspberry Pi 4B 2GB boards with 120GB SSD via USB3 connectors. They run edge services using docker compose files.
edge1:
Linux edge1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linuxedge2:
Linux edge2 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linuxedge1 containers
```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-1 2B (virtual 40.3MB) edge1 Up 3 minutes 4 days ago running diun-1 0B (virtual 54.6MB) edge1 Up 3 minutes 4 days ago running fail2ban 3.12kB (virtual 90.9MB) host Up 3 minutes (healthy) 4 days ago running netdata-1 707kB (virtual 395MB) edge1 Up 3 minutes (healthy) 23 hours ago running npm 42.4MB (virtual 858MB) edge1 Up 3 minutes (healthy) 4 days ago running pihole-1 74.2MB (virtual 368MB) host Up 3 minutes (healthy) 23 hours ago running promtail-1 0B (virtual 185MB) edge1 Up 3 minutes 3 days ago running scrutiny-collector-1 9.37kB (virtual 189MB) edge1 Up 3 minutes 4 days ago running wireguard-1 23kB (virtual 167MB) edge1 Up 3 minutes 4 days ago running ```edge2 containers
```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-2 2B (virtual 40.3MB) edge2 Up 23 hours 4 days ago running diun-2 0B (virtual 54.6MB) edge2 Up 23 hours 4 days ago running duckdns 21.7kB (virtual 40.8MB) edge2 Up 22 hours 22 hours ago running netdata-2 26.6kB (virtual 394MB) edge2 Up About a minute (healthy) About a minute ago running pihole-2 128MB (virtual 422MB) host Up 23 hours (healthy) 23 hours ago running promtail-2 0B (virtual 185MB) edge2 Up 23 hours 3 days ago running scrutiny-collector-2 9.37kB (virtual 189MB) edge2 Up 23 hours 4 days ago running wireguard-2 23kB (virtual 167MB) edge2 Up 23 hours 4 days ago running ```Generally, both servers need no daily maintenance and the Diun jobs run on Sundays to let me know what containers need updating (thanks for that, btw!).
I wanted to implement fail2ban in a docker container and used this version based on the good work done with Diun. I found a tutorial and followed it, amending the configuration to suit my needs. It works, and I sometimes get ban notifications.
However, after a day or so of usage the following will occur:
According to my netdata metrics, this is due to fail2ban. I'm just not sure how to see what it was doing at the time.
Steps to reproduce
Docker info
Docker Compose config
Logs
Additional info
The latest issue occurred just before 2023-07-12 01:00 MDT. The only entry in the docker logs for this day was "ERROR can't start new thread" at 13:24 MDT, likely when I ssh'd into the server to check why it was so slow.
The fail2ban container can see the NPM logs:
$ docker exec fail2ban ls -lha /var/log/npm/error.log -rwxrwxr-x 1 1000 1000 25.6K May 26 12:11 /var/log/npm/error.logThe database appears to be valid:
$ ls -lha data/db/ total 212K drwxrwxr-x 2 idsvc docker 4.0K Jun 27 14:28 . drwxrwxr-x 6 idsvc docker 4.0K Jul 11 14:48 .. -rwxrwxr-x 1 idsvc docker 204K Jul 12 14:34 fail2ban.sqlite3The issue appears to be iowait related:

Just before I rebooted the server, the number of fail2ban processes was at 1,655:

What am I missing here?