This is a short one that I do not want to forget about.
I had problems with fail2ban, it suddenly stopped working. It was running as usual and when I checked with:
fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
it did give me results, but when I was tailing
/var/log/fail2ban.log nothing happened.
Another server had a very similar setup, but it was working (and logging in its fail2ban.log), so I started checking config etc, but everything was the same… until it wasn’t. I suddenly saw that the timestamp of the logs was off from the system time.
A while ago I had changed the timezone (the machine was relatively fresh) and rsyslogd was still running in the old timezone, logging everything with an offset. This caused fail2ban to consider everything “too old” (by a couple of hours), so nothing was ever banned. Silly me! After restart of rsyslogd, it immediately started banning IP’s (a good thing, since that machine is constantly being scanned).
Takeaway: if you change the timezone of a machine, at least restart the syslog daemon.