Repeatedly experiencing Outbound traffic DOS
> We have detected an outbound denial of service attack originating from your Linode. It appears that a process internal to your Linode is sending large amounts of malicious traffic towards other servers. We ask that you investigate this matter as soon as possible to determine why this activity is originating from your Linode.
Linode also sent list of possible things to do but somehow I am not able to target whats going on, possibly I might be lacking some sys admin skills (am not one after all)
> - "/var/log/auth.log": You may have fallen victim to a SSH brute force attack.
"lastlog": You can cross reference recent account logins with the brute force attempts in "/var/log/auth.log”.
/tmp: This directory is often used by attackers to store their files in.
Web server logs: You may have installed a vulnerable script or web application.
"ps aux": Check for foreign processes.
Next time around, we completely redeployed, moved from Ubuntu 12.04 to 14.04, disabled password access and moved to RSA key based SSH access, disabled root access as well and still the very next day, the same thing happened - another instance of outbound DOS generated and my network activity rising to crazy levels.
I really need to get the app live asap and no idea what to do about this. Can anyone help me figure out?
Apparently this has started happening in the last one year and I see a lot of loyal customers move out of Linode to Digital ocean / Amazon.
Help help help!
6 Replies
Since it's not a SSH based attack, what other things are running?
Moving to another host won't help remove whatever attack vector is being used.
Are you running IPTABLES - what are the rules?
Better Details = Better Answers
So we are running a web app with apache, myqsl, passenger and ruby on rails.
I checked the /var/log/auth.log, these are some of the fishy lines that seems like a brute force attempt but apart from the one successful attempt (that is mine), I dont see anyone else getting into the system, so the questions is how is someone getting in and initiating the outbound network heavy traffic?
Nov 23 06:45:28 roadmojo sshd[17009]: Invalid user admin from 122.225.109.117
Nov 23 06:45:28 roadmojo sshd[17009]: inputuserauthrequest: invalid user admin [preauth]
Nov 23 06:45:28 roadmojo sshd[17009]: Connection closed by 122.225.109.117 [preauth]
Nov 23 06:45:40 roadmojo sshd[17014]: fatal: Read from socket failed: Connection reset by peer [preauth]
Nov 23 06:45:58 roadmojo sshd[17008]: Connection closed by 122.225.109.117 [preauth]
..
Nov 23 06:52:36 roadmojo sshd[17059]: Invalid user admin from 175.45.24.109
Nov 23 06:52:36 roadmojo sshd[17059]: inputuserauthrequest: invalid user admin [preauth]
Nov 23 06:52:36 roadmojo sshd[17059]: Connection closed by 175.45.24.109 [preauth]
..
Nov 23 09:36:57 roadmojo sshd[20400]: Did not receive identification string from 202.120.38.28
..
Nov 23 10:17:49 roadmojo sshd[21386]: Invalid user root # from 202.120.38.28
Nov 23 10:17:49 roadmojo sshd[21386]: inputuserauthrequest: invalid user root # [preauth]
Nov 23 10:17:50 roadmojo sshd[21386]: Received disconnect from 202.120.38.28: 11: Bye Bye [preauth]
Nov 23 10:18:19 roadmojo sshd[21400]: Invalid user from 202.120.38.28
Nov 23 10:18:19 roadmojo sshd[21400]: inputuserauthrequest: invalid user [preauth]
Nov 23 10:18:20 roadmojo sshd[21400]: Received disconnect from 202.120.38.28: 11: Bye Bye [preauth]
Nov 23 10:18:50 roadmojo sshd[21414]: Invalid user from 202.120.38.28
Nov 23 10:18:50 roadmojo sshd[21414]: inputuserauthrequest: invalid user [preauth]
Nov 23 10:18:50 roadmojo sshd[21414]: Received disconnect from 202.120.38.28: 11: Bye Bye [preauth]
Nov 23 10:19:20 roadmojo sshd[21428]: Invalid user from 202.120.38.28
Nov 23 10:19:20 roadmojo sshd[21428]: inputuserauthrequest: invalid user [preauth]
Nov 23 10:19:20 roadmojo sshd[21428]: Received disconnect from 202.120.38.28: 11: Bye Bye [preauth]
Nov 23 10:19:50 roadmojo sshd[21442]: Invalid user from 202.120.38.28
..
Nov 23 12:20:04 roadmojo sshd[24409]: Did not receive identification string from 112.216.92.44
Nov 23 12:25:01 roadmojo CRON[24530]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 23 12:25:01 roadmojo CRON[24530]: pam_unix(cron:session): session closed for user root
Nov 23 12:26:00 roadmojo sshd[24545]: Invalid user oracle from 112.216.92.44
Nov 23 12:26:00 roadmojo sshd[24545]: inputuserauthrequest: invalid user oracle [preauth]
Nov 23 12:26:00 roadmojo sshd[24545]: Received disconnect from 112.216.92.44: 11: Bye Bye [preauth]
..
Nov 23 12:59:51 roadmojo sshd[25409]: invalid public DH value: <= 1 [preauth]
Nov 23 12:59:51 roadmojo sshd[25409]: Disconnecting: bad client public DH value [preauth]
..
Hope this helps. Is there anything specific that I can provide?
Checking more.
> Unless there is a good reason to have elastic search open to the world (which it's unlikely there is) firewall it off, problem solved.
We are disabling remote access -
this will only give access to localhost.