fail2ban on Postfix not banning an IP address

What does your fail2ban log say?

@Vance:

What does your fail2ban log say?

Thanks so much for looking at this. I really appreciate it. Here is some further information:

I had the good fortune of being able to see this happen live (many times actually; the same guy had multiple IP addresses to play with). This is what I found:
* As soon as fail2ban triggers and sends me the email, it does indeed right away apply the iptables rule: I checked "sudo iptables -L" right after getting the email, and the IP address was indeed listed there as banned.

• However, the IP address can still connect to my Postfix for 1-2 minutes after that. So it appears that my iptables rules, although applied, are not taking effect immediately.
  So the question is, why aren't my iptables rules taking effect immediately?

To answer your question, the /var/log/fail2ban.log contains the following. Notice, again, how the ban takes effect, but the guy can still try connecting for 1-2 minutes after that.

2013-10-28 05:31:50,995 fail2ban.actions: WARNING [postfix] Ban 123.56.789.123
2013-10-28 05:32:17,642 fail2ban.actions: WARNING [postfix] 123.56.789.123 already banned
2013-10-28 05:32:44,173 fail2ban.actions: WARNING [postfix] 123.56.789.123 already banned
2013-10-28 05:33:11,207 fail2ban.actions: WARNING [postfix] 123.56.789.123 already banned
2013-10-28 05:33:36,239 fail2ban.actions: WARNING [postfix] 123.56.789.123 already banned
2013-10-28 05:34:04,274 fail2ban.actions: WARNING [postfix] 123.56.789.123 already banned

@dee4:

Oct 27 06:55:56 plato postfix/smtpd[15297]: warning: Connection concurrency limit exceeded: 51 from unknown[12.345.678.912] for service smtp

It looks to me like they had already made 51 connections to the smtp port before your ban was triggered. My guess is that you probably have your iptables rule applying to new connections only so the connections that were already made can continue to do their thing until they get disconnected.

@Stever:

@dee4:

Oct 27 06:55:56 plato postfix/smtpd[15297]: warning: Connection concurrency limit exceeded: 51 from unknown[12.345.678.912] for service smtp

It looks to me like they had already made 51 connections to the smtp port before your ban was triggered. My guess is that you probably have your iptables rule applying to new connections only so the connections that were already made can continue to do their thing until they get disconnected.
This "limit exceeded" thing appeared the first time, but it hasn't appeared since. Some guy from Asia has been trying to attack me whole day now using multiple IP addresses and he's still at it. After one IP address gets banned, he switches to another one maybe 30-60 mins later. Again, what bothers me is that the IP doesn't get blocked immediately, it seems.

@dee4:

So the question is, why aren't my iptables rules taking effect immediately?

That's an excellent question, and I don't seem to be able to find an answer. On one system I deal with, it's set to block an address after four failed ssh login attempts - sometimes attackers can get five or six tries in, but it takes seconds or less to stop traffic; nowhere near a full minute.

If there seems to be consistently a delay of a minute, I would guess that there's some sort of batching going on in the kernel where it waits to apply new rules. Otherwise, perhaps the load on your system prevents it from being applied immediately, or it could even be a kernel bug.

This does not directly address the problem, but you could set up firewall rules to rate-limit connections to the ports in question. This would at least reduce the speed of attempts. Be careful in your settings - you don't want to keep out legitimate connections. (Note that Postfix has its own rate-limiting which might be easier to use than iptables - this is what the "Connection concurrency limit" log message was about. I don't have first-hand experience setting it up, but it looks as simple as setting smtpdclientconnectionratelimit in main.cf and restarting Postfix.)

On a side note, it can be easy to take attacks personally when you're starting out. In truth, there isn't "some guy from Asia" sitting at a keyboard on the other end of the connection. It's a random compromised Windows box or web server that is just one of many that is part of a botnet running an automated script. Like a Terminator, it can't be reasoned with, bargained with, intimidated, or even annoyed. It'll just try until it sees it's not succeeding, then move on. Meanwhile another bot will try.

@Stever:

It looks to me like they had already made 51 connections to the smtp port before your ban was triggered. My guess is that you probably have your iptables rule applying to new connections only so the connections that were already made can continue to do their thing until they get disconnected.

If this is the case (and maybe dee4 can confirm by pasting the rules that f2b concocts), then you can force postfix to disconnect the user after a number of errors (which could include auth failures). The config parameters you'd be interested in are smtpdsofterrorlimit (connection is slowed after this many errors), smtpderrorsleeptime (connection is slowed by this much after smtpdsofterrorlimit errors are reached) and smtpdharderrorlimit (connection is severed after this many errors). See what they are currently (postconf -d | grep error) and adjust as you see fit.

Thanks for the replies. I still have Ubuntu 10.04 LTS. I'll let you know how things go after I upgrade to Ubuntu 12.04 LTS, but this could be after a few months so don't wait around for me.

It seems to me that the default Postfix parameters will work just fine for now (in particular the smtpderrorsleeptime, smtpdsofterrorlimit, and smtpdharderror_limit).

The question is, why would anyone engage in this attack? Seems to me that I'd have to have a very weak password for their attacks to be successful, even if I turn off Fail2Ban. I suppose they just keep looking until they find a very weak server, don't they?

@dee4:

The question is, why would anyone engage in this attack?
That's been answered already in this thread.

There is NOBODY doing this - it's all scripted botnets (compromised workstations or servers) sifting thru the net looking for low hanging fruit. If you have a public IP, then you're a target.

Best solution - don't use passwords on server services (like ssh) and enforce encryption and strong passwords on your email accounts - and then ignore the chaff, it will NEVER go away.

@vonskippy:

@dee4:

enforce encryption and strong passwords on your email accounts - and then ignore the chaff, it will NEVER go away.
Thanks, that makes sense, but it makes me more curious.

Suppose I follow this guide to setup email on my node: https://library.linode.com/email/postfi … 0.19-mysql">https://library.linode.com/email/postfix/postfix2.9.6-dovecot2.0.19-mysql

Suppose I then get rid of all connection/rate/login limits etc. from Postfix. I also don't install Fail2Ban and don't have any rate limits through iptables.

Under this setup, how easy would it be for a cracker to brute force crack a strong password?

What I'm asking, under the Postfix/Dovecot setup in that guide, is each password trial slow enough such that a reasonably strong password (e.g. 20 characters of numbers, letters and symbols) would take centuries to crack?

In cryptography, this is often done with lots of PBKDF2 iterations to slow down each attempt. Do Postfix/Dovecot have that after following the guide?

Based on the $6$ in the Library section on adding users, Dovecot appears to be using SHA-512 hashes, with just a single round of hashing.

It's my understanding that this a hash that is not intended to have the same work-factor properties as bcrypt or PBKDF2. It looks like you can make Dovecot use bcrypt, but only if your libc supports it (see man crypt on your system).

Edit: some of this is incorrect; the SHA-512 hash scheme uses multiple rounds and is at least partly designed to increase the work factor. See posts below for details.

@Vance:

Based on the $6$ in the Library section on adding users, Dovecot appears to be using SHA-512 hashes, with just a single round of hashing.

It's my understanding that this a hash that is not intended to have the same work-factor properties as bcrypt or PBKDF2. It looks like you can make Dovecot use bcrypt, but only if your libc supports it (see man crypt on your system).

Thanks for pointing me in the right direction.

If I look at this link: http://wiki2.dovecot.org/Tools/Doveadm/Pw

it seems to me that they use 5000 rounds by default for SHA512-CRYPT. The reason I say this is that currently (after following Linode's guide), the form of the email user passwords in my MySQL database is like this:

$6$grF.zp/BK.HuJ/vv$MLhrz7o/kmyx1ykNaz4lzL3nBpI7YqcJfaO.7OHu/NNFzajISgi.Hoj1XMF2ECJhzhhMH8vGahwcfNBu62d.v0

Indeed, if I type in

sudo doveadm pw -p "hello" -s SHA512-CRYPT

I get a similar output:

{SHA512-CRYPT}$6$yXhlPOzKNYhBViN/$SDKbHqjex/CHE2spoI4tBLh86kSFUOy5k8o5lIopCYeELaWtkVe5Y8RwuyxvDIXzF2cN0838x1RE.JGrcFrR31

However, if I specify the rounds the rounds show up in the result, so if I type

sudo doveadm pw -p "hello" -r 100000 -s SHA512-CRYPT

I get:

{SHA512-CRYPT}$6$rounds=100000$3yLxwVr5ryV8YT0I$PpzQVIq.qOg0b.eIiRPYu3z4/01hguuzX0bcRR2Tmjav3Upqj8r7sd4MwTXGRbMOm5QabX2icHIuU2STeAaA0/

Meaning that since there were no rounds stored in MySQL, it means it's using the default of 5000 rounds.

Maybe I'm missing something. By all means correct me if I'm wrong!

Good catch. I made a mistake by assuming it was a single round for SHA-512. In fact you are correct; 5000 rounds is the default.

According to that document, you can specify up to 999999999 rounds and (Linux) libc should handle it. So I think things should work if you choose 100000 rounds (although I would time it to make sure it doesn't take too long).

@Vance:

Good catch. I made a mistake by assuming it was a single round for SHA-512. In fact you are correct; 5000 rounds is the default.

According to that document, you can specify up to 999999999 rounds and (Linux) libc should handle it. So I think things should work if you choose 100000 rounds (although I would time it to make sure it doesn't take too long).
Very good, thanks, it works!

To summarize, here is how people boost the security of their default email setup after following Linode's guide:
4. Use some password generator to generate a strong > 20 character password with letters, numbers and symbols. (Or some other strong password generator such as Diceware)

7. Change the type of the password column in your virtual_users table in MySQL to TEXT (the default is VARCHAR(106) and the new password won't fit there).

8. Use sudo doveadm pw -p "theStrongPassword" -r 50000 -s SHA512-CRYPT to generate the new encrypted password. Note that we are now using doveadm to generate the password instead of using MySQL functions (as Linode do in the guide)

9. Put your new encrypted password into the database (everything including and following the $6$)

10. Open ~/.bash_history and remove the lines showing your passwords.
    Hope that helps someone, or myself when I need to do this again later 8)

@dee4:

.. /etc/fail2ban/jail.local file:

..

           iptables[name=postfix, port=smtp, protocol=tcp]
           iptables[name=postfix, port=ssmtp, protocol=tcp]

It could be a) this is creating only one of the iptables rules, or b) you're running iptables on port 587 as well and its this one being attacked. There is iptables-multiport and iptables-ipset-proto6 for dealing with blocking multiple ports. There was a problem using the same action more than once in the same jail that was only fixed one or two fail2ban versions ago.

@dee4:

..

This is my /etc/fail2ban/filter.d/postfix.local:

[Definition]

failregex = (?i): warning: [-._\w]+\[<host>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed
            reject: RCPT from (.*)\[<host>\]: 554
ignoreregex =</host></host> 

Note: this probably applies to you too now http://www.kb.cert.org/vuls/id/686662

Grab an official replacement https://github.com/fail2ban/fail2ban/bl … stfix.conf">https://github.com/fail2ban/fail2ban/blob/master/config/filter.d/postfix.conf and/or look up why un-anchored regexs are dangerous ( https://github.com/fail2ban/fail2ban/bl … er/FILTERS">https://github.com/fail2ban/fail2ban/blob/master/FILTERS ).

This thread is quite old however I thought somebody might find this piece of advice helpful enough.

On my postfix setup I make sure every failed login attempt is followed by a delay (i.e. 1 second) and then every n-th failed login is followed by longer delay (i.e. 10 seconds). I'm still probed however attacks take less cpu and attacker realises soon enough there is no possible way to crack anything when each attempt is delayed.