View All Products
Compute
Storage
Networking
Databases
Services
Developer Tools
Industries
Pricing
Community
Engage With Us
Community ▶ Questions ▶ server stops every few days
Log in to Ask a Question

server stops every few days

networking

I hired a contractor, who was highly recommended, build a LAMP stack on my Linode VPS (Linode also highly recommended). Every few days the server becomes un-ping-able, and often after a few hours comes up again.

My contractor says he has done nothing unusual with the setup and that something is on on the Linode itself. Linode Support has implied that the fault is with the server build, and there is nothing more they can, or need to, do since the Linode itself is running. I have not loaded my app or basically even touched the thing except for logging in via Lish to try to read logs.

Any help leading to a real solution would be greatly appreciated. I am willing to make you a full-rights user in my Linode account if it would help, since there is nothing sensitive loaded at this point.

Thanks.

16 Replies

Are you getting any reboot alerts or other alerts from Linode?

What OS are you running?

What's the contents of /var/log/syslog or /var/log/messages around the time you're having problems?

@obs:

Are you getting any reboot alerts or other alerts from Linode?

What OS are you running?

What's the contents of /var/log/syslog or /var/log/messages around the time you're having problems?

Thanks for the quick response.

Getting no Linode alerts (unless I force a reboot, so alerts are working)

OS: CentOS 7

My contractor has checked the logs and says nothing unusual appears. I will begin checking them out myself. I am now running a script (on a different server) that pings my Linode every minute and reports if the status has changed from the previous report.

I've had that happen when a server runs out of memory and begins swapping. Run "dmesg" and see if the out-of-memory killer has ever been invoked. What are you using to monitor memory usage (if anything)? Also, are you able to access the server via Lish while it's unpingable?

@masonm:

I've had that happen when a server runs out of memory and begins swapping. Run "dmesg" and see if the out-of-memory killer has ever been invoked. What are you using to monitor memory usage (if anything)? Also, are you able to access the server via Lish while it's unpingable?

Thanks for the response.

I can access via Lish while it is unpingable.

When I run "dmesg" I get a long list of firewall statements about blocking various TCP and UDP hits, but nothing else.

My Linode Dashboard shows large output spike at (what I think is) the moment of crash.

After a crash early this morning (~1 am EST) it would not come back even after three reboots. Somehow it came back by itself since then. A few minutes ago, I entered some false creds in the everheldwebgroup.com/phpmyadmin login box (not an injection attempt, mind you, just a wrong name), and the whole thing went down. Could be a coincidence, but the thought that anyone on the planet can crash my server this easily is unsettling.

If you can access it via Lish but not externally then there's clearly a networking issue, most likely involving iptables. Run "ps aux" and "iptables -L -n" as root and post the output of both commands here.

OK, Took me a while to figure out how to collect all the input, but here it is:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.2 48464 5780 ? Ss 06:13 0:05 /sbin/init nosep nodevfs

root 2 0.0 0.0 0 0 ? S 06:13 0:00 [kthreadd]

root 3 0.0 0.0 0 0 ? S 06:13 0:00 [ksoftirqd/0]

root 5 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/0:0H]

root 6 0.0 0.0 0 0 ? S 06:13 0:00 [kworker/u4:0]

root 7 0.0 0.0 0 0 ? S 06:13 0:01 [rcu_sched]

root 8 0.0 0.0 0 0 ? S 06:13 0:00 [rcu_bh]

root 9 0.0 0.0 0 0 ? S 06:13 0:00 [migration/0]

root 10 0.0 0.0 0 0 ? S 06:13 0:00 [migration/1]

root 11 0.0 0.0 0 0 ? S 06:13 0:00 [ksoftirqd/1]

root 13 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/1:0H]

root 14 0.0 0.0 0 0 ? S< 06:13 0:00 [khelper]

root 15 0.0 0.0 0 0 ? S 06:13 0:00 [kdevtmpfs]

root 16 0.0 0.0 0 0 ? S< 06:13 0:00 [netns]

root 20 0.0 0.0 0 0 ? S 06:13 0:00 [xenwatch]

root 21 0.0 0.0 0 0 ? S 06:13 0:00 [xenbus]

root 211 0.0 0.0 0 0 ? S< 06:13 0:00 [writeback]

root 214 0.0 0.0 0 0 ? S< 06:13 0:00 [crypto]

root 215 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]

root 216 0.0 0.0 0 0 ? S< 06:13 0:00 [kblockd]

root 227 0.0 0.0 0 0 ? S< 06:13 0:00 [md]

root 318 0.0 0.0 0 0 ? S< 06:13 0:00 [rpciod]

root 440 0.0 0.0 0 0 ? S 06:13 0:00 [kswapd0]

root 510 0.0 0.0 0 0 ? S 06:13 0:00 [fsnotify_mark]

root 516 0.0 0.0 0 0 ? S 06:13 0:00 [ecryptfs-kthrea]

root 518 0.0 0.0 0 0 ? S< 06:13 0:00 [nfsiod]

root 520 0.0 0.0 0 0 ? S< 06:13 0:00 [cifsiod]

root 526 0.0 0.0 0 0 ? S 06:13 0:00 [jfsIO]

root 527 0.0 0.0 0 0 ? S 06:13 0:00 [jfsCommit]

root 529 0.0 0.0 0 0 ? S 06:13 0:00 [jfsCommit]

root 531 0.0 0.0 0 0 ? S 06:13 0:00 [jfsSync]

root 533 0.0 0.0 0 0 ? S< 06:13 0:00 [xfsalloc]

root 535 0.0 0.0 0 0 ? S< 06:13 0:00 [xfsmrucache]

root 537 0.0 0.0 0 0 ? S< 06:13 0:00 [xfslogd]

root 543 0.0 0.0 0 0 ? S< 06:13 0:00 [glock_workqueue]

root 545 0.0 0.0 0 0 ? S< 06:13 0:00 [delete_workqueu]

root 555 0.0 0.0 0 0 ? S< 06:13 0:00 [gfs_recovery]

root 1150 0.0 0.0 0 0 ? S 06:13 0:00 [khvcd]

root 1247 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]

root 1248 0.0 0.0 0 0 ? S< 06:13 0:00 [drbd-reissue]

root 1264 0.0 0.0 0 0 ? S< 06:13 0:00 [kpsmoused]

root 1267 0.0 0.0 0 0 ? S< 06:13 0:00 [raid5wq]

root 1272 0.0 0.0 0 0 ? S< 06:13 0:00 [dmbufiocache]

root 1297 0.0 0.0 0 0 ? S< 06:13 0:00 [ipv6_addrconf]

root 1317 0.0 0.0 0 0 ? S< 06:13 0:00 [bioset]

root 1345 0.0 0.0 0 0 ? S< 06:13 0:00 [deferwq]

root 1348 0.0 0.0 0 0 ? S< 06:13 0:00 [reiserfs/xvda]

root 1349 0.0 0.0 0 0 ? S< 06:13 0:00 [kworker/0:1H]

root 1350 0.0 0.0 0 0 ? S 06:13 0:00 [jbd2/xvda-8]

root 1351 0.0 0.0 0 0 ? S< 06:13 0:00 [ext4-rsv-conver]

root 1373 0.0 0.4 42996 8212 ? Ss 06:13 0:02 /usr/lib/systemd/systemd-journald

root 1374 0.0 0.0 0 0 ? S 06:13 0:00 [kauditd]

root 1477 0.0 0.1 40720 3116 ? Ss 06:13 0:00 /usr/lib/systemd/systemd-udevd

root 1769 0.0 0.1 116676 3200 ? S avahi 2066 0.0 0.1 28080 2472 ? Ss 06:13 0:00 avahi-daemon: running [server.local]

root 2070 0.0 0.7 535972 14612 ? Ssl 06:13 0:05 /usr/sbin/NetworkManager –no-daemon

root 2074 0.0 0.4 207992 8316 ? Ssl 06:13 0:00 /usr/sbin/rsyslogd -n

root 2076 0.0 0.9 549980 20184 ? Ssl 06:13 0:06 /usr/bin/python -Es /usr/sbin/tuned -l -P

avahi 2079 0.0 0.0 27948 220 ? S 06:13 0:00 avahi-daemon: chroot helper

dbus 2084 0.0 0.1 26700 3020 ? Ss 06:13 0:04 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

root 2085 0.0 0.1 34688 2980 ? Ss 06:13 0:02 /usr/lib/systemd/systemd-logind

chrony 2087 0.0 0.1 24728 2524 ? S 06:13 0:00 /usr/sbin/chronyd -u chrony

root 2090 0.0 0.1 126304 3032 ? Ss 06:13 0:00 /usr/sbin/crond -n

root 2106 0.0 0.0 110008 1828 tty1 Ss+ 06:13 0:00 /sbin/agetty --noclear tty1

root 2118 0.0 0.0 6488 124 ? Ss 06:13 0:01 /sbin/iprupdate --daemon

root 2125 0.0 0.0 6488 124 ? Ss 06:13 0:01 /sbin/iprinit --daemon

polkitd 2359 0.0 0.5 513848 11512 ? Ssl 06:13 0:01 /usr/lib/polkit-1/polkitd --no-debug

root 2360 0.0 0.0 39128 92 ? Ss 06:13 0:00 /sbin/iprdump --daemon

root 2418 0.0 0.9 107248 19060 ? S 06:13 0:00 /sbin/dhclient -d -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fbad332-e2ed-4c3d-bd30-44e3507a717c-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0

mysql 2605 0.0 0.1 115348 3168 ? Ss 06:13 0:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr

root 2611 0.0 0.9 260384 18440 ? Ss 06:13 0:03 php-fpm: master process (/usr/local/php56/etc/php-fpm.conf)

root 2618 0.0 0.0 19764 1904 ? Ss 06:13 0:01 /usr/local/directadmin/da-popb4smtp

nobody 2619 0.0 0.2 64692 5168 ? Ss 06:13 0:00 /usr/local/directadmin/directadmin d

root 2628 0.0 0.1 152840 3904 ? Ss 06:13 0:00 pure-ftpd (SERVER)

root 2637 0.0 0.2 82796 6100 ? Ss 06:13 0:00 /usr/sbin/sshd -D

named 2658 0.0 1.0 240448 21352 ? Ssl 06:13 0:00 /usr/sbin/named -u named

nobody 2690 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2691 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2692 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2693 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2694 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2695 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2696 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2698 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2699 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

nobody 2700 0.0 0.0 64692 1012 ? S 06:13 0:00 /usr/local/directadmin/directadmin d

mysql 2743 0.0 4.4 697320 91084 ? Sl 06:13 0:36 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/server.everheldwebgroup.com.err --pid-file=server.everheldwebgroup.com.pid

root 2748 0.0 0.9 165696 18704 ? Ss 06:13 0:29 lfd - sleeping

root 2808 0.0 0.3 67620 7580 ? Ss 06:13 0:02 /usr/sbin/httpd -k start

apache 2879 0.0 0.5 1332260 11568 ? Sl 06:13 0:24 /usr/sbin/httpd -k start

apache 2880 0.0 0.6 1331644 12856 ? Sl 06:13 0:24 /usr/sbin/httpd -k start

root 2955 0.0 0.1 18664 2500 ? Ss 06:13 0:00 /usr/sbin/dovecot -F

dovecot 2959 0.0 0.2 48172 5592 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2960 0.0 0.2 48176 5508 ? S 06:13 0:00 dovecot/imap-login

dovecot 2961 0.0 0.1 12364 2232 ? S 06:13 0:00 dovecot/anvil [33 connections]

root 2962 0.0 0.1 12496 2364 ? S 06:13 0:00 dovecot/log

dovecot 2964 0.0 0.2 48172 5516 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2965 0.0 0.2 48172 5532 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2966 0.0 0.2 48172 5556 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2967 0.0 0.2 48172 5532 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2968 0.0 0.2 48172 5552 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2969 0.0 0.2 48172 5536 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2970 0.0 0.2 48172 5552 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2971 0.0 0.2 48172 5576 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2972 0.0 0.2 48172 5548 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2973 0.0 0.2 48172 5544 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2974 0.0 0.2 48176 5588 ? S 06:13 0:00 dovecot/imap-login

dovecot 2975 0.0 0.2 48176 5552 ? S 06:13 0:00 dovecot/imap-login

dovecot 2976 0.0 0.2 48176 5540 ? S 06:13 0:00 dovecot/imap-login

dovecot 2977 0.0 0.2 48176 5556 ? S 06:13 0:00 dovecot/imap-login

dovecot 2978 0.0 0.2 48176 5524 ? S 06:13 0:00 dovecot/imap-login

dovecot 2979 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login

dovecot 2980 0.0 0.2 48176 5564 ? S 06:13 0:00 dovecot/imap-login

dovecot 2981 0.0 0.2 48176 5692 ? S 06:13 0:00 dovecot/imap-login

dovecot 2982 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login

dovecot 2983 0.0 0.2 48176 5696 ? S 06:13 0:00 dovecot/imap-login

root 2984 0.0 0.1 15460 3216 ? S 06:13 0:00 dovecot/config

dovecot 2985 0.0 0.2 48172 5540 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2986 0.0 0.2 48172 5516 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2987 0.0 0.2 48172 5560 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2988 0.0 0.2 48172 5540 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2989 0.0 0.2 48172 5560 ? S 06:13 0:00 dovecot/pop3-login

dovecot 2990 0.0 0.2 48176 5696 ? S 06:13 0:00 dovecot/imap-login

dovecot 2991 0.0 0.2 48176 5560 ? S 06:13 0:00 dovecot/imap-login

dovecot 2992 0.0 0.2 48176 5536 ? S 06:13 0:00 dovecot/imap-login

dovecot 2993 0.0 0.2 48176 5532 ? S 06:13 0:00 dovecot/imap-login

dovecot 2994 0.0 0.2 48176 5544 ? S 06:13 0:00 dovecot/imap-login

root 2995 0.0 0.1 17836 3392 ? S 06:13 0:00 dovecot/auth [0 wait, 0 passdb, 0 userdb]

mail 3010 0.0 0.2 62944 5792 ? Ss 06:14 0:00 /usr/sbin/exim -bd -q1h

root 5257 0.0 0.0 0 0 ? S 09:17 0:00 [kworker/u4:2]

root 7233 0.0 0.0 0 0 ? S< 12:05 0:00 [kworker/1:1H]

root 12080 0.0 0.0 0 0 ? S 18:01 0:00 [kworker/1:3]

root 12499 0.0 0.0 0 0 ? S 18:19 0:00 [kworker/0:2]

root 12758 0.0 0.0 0 0 ? S 18:35 0:00 [kworker/1:0]

root 12889 0.0 0.2 188816 4376 ? Ss 18:40 0:00 login – admin

root 12897 0.0 0.0 0 0 ? S 18:41 0:00 [kworker/0:0]

root 12912 0.0 0.0 0 0 ? S 18:42 0:00 [kworker/1:1]

admin 12923 0.0 0.1 115352 3340 hvc0 Ss 18:42 0:00 -bash

root 12947 0.0 0.2 180544 4116 hvc0 S 18:42 0:00 su

root 12948 0.0 0.1 115352 3408 hvc0 S 18:42 0:00 bash

root 13081 0.0 0.1 123360 2544 hvc0 R+ 18:48 0:00 ps aux

…and from iptables:

Chain INPUT (policy DROP)

target prot opt source destination

ACCEPT tcp -- 207.192.69.5 0.0.0.0/0 tcp dpt:53

ACCEPT udp -- 207.192.69.5 0.0.0.0/0 udp dpt:53

ACCEPT tcp -- 207.192.69.5 0.0.0.0/0 tcp spt:53

ACCEPT udp -- 207.192.69.5 0.0.0.0/0 udp spt:53

ACCEPT tcp -- 207.192.69.4 0.0.0.0/0 tcp dpt:53

ACCEPT udp -- 207.192.69.4 0.0.0.0/0 udp dpt:53

ACCEPT tcp -- 207.192.69.4 0.0.0.0/0 tcp spt:53

ACCEPT udp -- 207.192.69.4 0.0.0.0/0 udp spt:53

ACCEPT tcp -- 97.107.133.4 0.0.0.0/0 tcp dpt:53

ACCEPT udp -- 97.107.133.4 0.0.0.0/0 udp dpt:53

ACCEPT tcp -- 97.107.133.4 0.0.0.0/0 tcp spt:53

ACCEPT udp -- 97.107.133.4 0.0.0.0/0 udp spt:53

LOCALINPUT all -- 0.0.0.0/0 0.0.0.0/0

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

INVALID tcp -- 0.0.0.0/0 0.0.0.0/0

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:20

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:21

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:25

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:53

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:67

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:80

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:110

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:143

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:465

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:587

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:993

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:995

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:2222

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1857

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:20

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:21

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:53

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 0 limit: avg 1/sec burst 5

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 11

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 3

LOGDROPIN all -- 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy DROP)

target prot opt source destination

Chain OUTPUT (policy DROP)

target prot opt source destination

ACCEPT tcp -- 0.0.0.0/0 207.192.69.5 tcp dpt:53

ACCEPT udp -- 0.0.0.0/0 207.192.69.5 udp dpt:53

ACCEPT tcp -- 0.0.0.0/0 207.192.69.5 tcp spt:53

ACCEPT udp -- 0.0.0.0/0 207.192.69.5 udp spt:53

ACCEPT tcp -- 0.0.0.0/0 207.192.69.4 tcp dpt:53

ACCEPT udp -- 0.0.0.0/0 207.192.69.4 udp dpt:53

ACCEPT tcp -- 0.0.0.0/0 207.192.69.4 tcp spt:53

ACCEPT udp -- 0.0.0.0/0 207.192.69.4 udp spt:53

ACCEPT tcp -- 0.0.0.0/0 97.107.133.4 tcp dpt:53

ACCEPT udp -- 0.0.0.0/0 97.107.133.4 udp dpt:53

ACCEPT tcp -- 0.0.0.0/0 97.107.133.4 tcp spt:53

ACCEPT udp -- 0.0.0.0/0 97.107.133.4 udp spt:53

LOCALOUTPUT all -- 0.0.0.0/0 0.0.0.0/0

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp spt:53

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:53

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

INVALID tcp -- 0.0.0.0/0 0.0.0.0/0

ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:20

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:21

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:25

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:53

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:67

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:80

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:110

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:113

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:2222

ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1857

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:20

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:21

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:53

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:113

ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 ctstate NEW udp dpt:123

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 0

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 8

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 11

ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmptype 3

LOGDROPOUT all -- 0.0.0.0/0 0.0.0.0/0

Chain ALLOWIN (1 references)

target prot opt source destination

ACCEPT all -- 71.174.180.0/24 0.0.0.0/0

ACCEPT all -- 71.174.180.183 0.0.0.0/0

ACCEPT all -- 134.42.112.2 0.0.0.0/0

Chain ALLOWOUT (1 references)

target prot opt source destination

ACCEPT all -- 0.0.0.0/0 71.174.180.0/24

ACCEPT all -- 0.0.0.0/0 71.174.180.183

ACCEPT all -- 0.0.0.0/0 134.42.112.2

Chain DENYIN (1 references)

target prot opt source destination

DROP all -- 76.109.51.121 0.0.0.0/0

DROP all -- 79.188.233.110 0.0.0.0/0

DROP all -- 37.233.38.46 0.0.0.0/0

DROP all -- 61.19.253.26 0.0.0.0/0

DROP all -- 77.241.99.130 0.0.0.0/0

DROP all -- 185.61.136.111 0.0.0.0/0

DROP all -- 88.211.134.50 0.0.0.0/0

DROP all -- 94.102.52.186 0.0.0.0/0

DROP all -- 85.238.127.45 0.0.0.0/0

DROP all -- 173.166.245.5 0.0.0.0/0

DROP all -- 62.33.192.25 0.0.0.0/0

DROP all -- 72.93.39.7 0.0.0.0/0

DROP all -- 50.176.69.14 0.0.0.0/0

Chain DENYOUT (1 references)

target prot opt source destination

LOGDROPOUT all -- 0.0.0.0/0 76.109.51.121

LOGDROPOUT all -- 0.0.0.0/0 79.188.233.110

LOGDROPOUT all -- 0.0.0.0/0 37.233.38.46

LOGDROPOUT all -- 0.0.0.0/0 61.19.253.26

LOGDROPOUT all -- 0.0.0.0/0 77.241.99.130

LOGDROPOUT all -- 0.0.0.0/0 185.61.136.111

LOGDROPOUT all -- 0.0.0.0/0 88.211.134.50

LOGDROPOUT all -- 0.0.0.0/0 94.102.52.186

LOGDROPOUT all -- 0.0.0.0/0 85.238.127.45

LOGDROPOUT all -- 0.0.0.0/0 173.166.245.5

LOGDROPOUT all -- 0.0.0.0/0 62.33.192.25

LOGDROPOUT all -- 0.0.0.0/0 72.93.39.7

LOGDROPOUT all -- 0.0.0.0/0 50.176.69.14

Chain INVALID (2 references)

target prot opt source destination

INVDROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x03/0x03

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x06

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x05/0x05

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x18/0x08

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x30/0x20

INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 ctstate NEW

Chain INVDROP (10 references)

target prot opt source destination

DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain LOCALINPUT (1 references)

target prot opt source destination

ALLOWIN all -- 0.0.0.0/0 0.0.0.0/0

DENYIN all -- 0.0.0.0/0 0.0.0.0/0

Chain LOCALOUTPUT (1 references)

target prot opt source destination

ALLOWOUT all -- 0.0.0.0/0 0.0.0.0/0

DENYOUT all -- 0.0.0.0/0 0.0.0.0/0

Chain LOGDROPIN (1 references)

target prot opt source destination

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:68

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:111

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:111

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:113

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:113

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:135:139

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:135:139

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:445

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:445

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:500

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:500

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:513

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:513

DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:520

DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:520

LOG tcp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: TCP_IN Blocked "

LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: UDP_IN Blocked "

LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix "Firewall: ICMP_IN Blocked "

DROP all -- 0.0.0.0/0 0.0.0.0/0

Chain LOGDROPOUT (14 references)

target prot opt source destination

LOG tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: TCP_OUT Blocked "

LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: UDP_OUT Blocked "

LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 8 level 4 prefix "Firewall: ICMP_OUT Blocked "

DROP all -- 0.0.0.0/0 0.0.0.0/0

This iptables rule is likely why your box comes unpingable:

ACCEPT icmp – 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5

What that's saying is you can only ping 5 times before it'll start limiting you to 1 a second, blocking anything more than that. That just affects pings though; everything else should work fine. Next time you try pinging, limit it to once every two seconds to make sure you don't get blocked. In Linux, this can be done with "ping -i 2".

I noticed you have login failure daemon ("lfd") running, which is probably what blocked you when you failed to login. Talk to your contractor about tweaking it if you think it's too aggressive.

@masonm:

This iptables rule is likely why your box comes unpingable:

ACCEPT icmp – 0.0.0.0/0 0.0.0.0/0 icmptype 8 limit: avg 1/sec burst 5

What that's saying is you can only ping 5 times before it'll start limiting you to 1 a second, blocking anything more than that. That just affects pings though; everything else should work fine. Next time you try pinging, limit it to once every two seconds to make sure you don't get blocked. In Linux, this can be done with "ping -i 2".

I noticed you have login failure daemon ("lfd") running, which is probably what blocked you when you failed to login. Talk to your contractor about tweaking it if you think it's too aggressive.

Thanks for the response.

Regarding ping, I have a cron-script running from a different site that pings this box only oncer per minute, so I don't think I am hitting it too hard.

As for the blocked login, I am not sure what is meant by "blocked you". Ping and phpmyadmin-log-in aside, if you try going to http://everheldwebgroup.com/ nothing is found. If ping and login attempts can result in taking the server offline like this then I am sunk. Right?

Again, thanks for all the help.

@johnrh:

Regarding ping, I have a cron-script running from a different site that pings this box only oncer per minute, so I don't think I am hitting it too hard.

Could be pings from someone else. That rule doesn't discrimate based on IP, it applies to all incoming packets.

> As for the blocked login, I am not sure what is meant by "blocked you".

From http://configserver.com/cp/csf.html:

"To complement the ConfigServer Firewall (csf), we have developed a Login Failure Daemon (lfd) process that runs all the time and periodically (every X seconds) scans the latest log file entries for login attempts against your server that continually fail within a short period of time. Such attempts are often called "Brute-force attacks" and the daemon process responds very quickly to such patterns and blocks offending IP's quickly. "

It's almost certainly using iptables for doing the actual blocking, so the next time you get blocked you should be able to login via Lish and unblock yourself by deleting the iptables rule. Talk to your contractor for more information.

> If ping and login attempts can result in taking the server offline like this then I am sunk. Right?

No. ICMP echo packets can usually be blocked completely without adversly affecting a server. I'm not saying that's a good idea (since you lose the ability to troubleshoot with ping), but just because a server can't be pinged doesn't mean it's offline.

Also, the lfd blocks are per-IP if I'm reading the docs right, so it only makes the server inaccessible to you, not anyone else.

I do not pretend to understand completely your previous reply, so I will continue to digest (thanks particularly for the link). Seeing phrases like the following:

"…next time you get blocked" and "…it only makes the server inaccessible to you…"

I am starting to think that I lead this discussion in a wrong direction with my mention of my own log-ins.

First, I have been taking care to make sure I am not looking at the site as one user. At my place of work I have access to two entirely separate Internet connections, for employees and visitors, with separate security measures, etc., and absolutely separate Internet-facing IPs. At home, I have accest to two separate DSL connections, different physical connections, with a different carrier than the one at work.

Second, I want to emphasize that my domain becomes unavailable (to all), or goes back online at times when I have not even been looking at it. It all started soon after the Linode went online and well before I put put any auto-ping script in place.

I am not disputing the assertion that it is a networking issue, but I want to make sure I have not mislead.

Here, for what it is worth, is a screenshot of one of my dashboard graphs from this morning (uploaded to my demo site on a VPS other than the Linode in question):

http://everheldwebgroupdemo.com/aa_lino … shot_1.png">http://everheldwebgroupdemo.com/aalinodescreenshot_1.png

Thanks

If you're still having problems, I'd try two things:

One, load MONIT so you can monitor what processes are doing what and when - then correlate that info with the time(s) your external ping monitor is saying you're offline.

Two, at least for testing, simplify (a lot) your IPTABLES ruleset, which seems to me, to be a right mess.

Example:

Chain ALLOWIN (1 references)
target prot opt source destination
ACCEPT all -- 71.174.180.0/24 0.0.0.0/0
ACCEPT all -- 71.174.180.183 0.0.0.0/0
ACCEPT all -- 134.42.112.2 0.0.0.0/0

Line 2 is completely superfluous, since in Line 1 you've already allowed that entire /24 subnet.

In several places, you allow several single IP's only to ALLOW ALL a few lines later.

Not sure what generated that IPTABLES ruleset, but I'd try again.

OK! So after reading the words "firewall" and "iptables" often enough from you all, I tried just shutting off the firewall, and, bam!, immediately I can get to the site! (Took me a while just to find the command. Seems that CentOS 7 is something of a different animal from previous versions). So I guess I need to get my contractor (or someone) review the settings.

It is wonderful to at least have a direction. Thanks again to all three of you.

So I looked at little more at the iptables I posted above, and, hey, 50.176.69.14 is one of the IPs from which I come to the server! I am sure my contractor did not just write that "deny" line in explicitly. I now think I was wrong to say the server is going done. Instead, it seems to block my IP. If so, it has happened to both my home and place of work, then goes back to accepting me for a while after I reboot the VPS.

What is going on here?! Its like the thing is stalking me.

It is likely that these firewall rules are added by the ConfigServer Firewall/Login Failure Daemon that masonm described. Somewhere in the logs for that application it should say why your IP was blocked. Often this is because of failed login attempts, or even the number of attempts (successful or not) in a given time period. It all depends on what triggers are set in the application.

@Vance:

It is likely that these firewall rules are added by the ConfigServer Firewall/Login Failure Daemon that masonm described. Somewhere in the logs for that application it should say why your IP was blocked. Often this is because of failed login attempts, or even the number of attempts (successful or not) in a given time period. It all depends on what triggers are set in the application.

Thanks, Vance. Very useful for me to know that even successful attempts can trigger a block.

If you, or someone, would kindly educate me a bit further, can the firewall be tuned to block only, say, ssh attempts while leaving httpd alone? Clearly I do not want to block an entire group users from getting to my website just because they are from the same IP as one user who is trying to log into my server.

I am not familiar with this application. You can check out its documentation or its user forum for more details.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Log in to Reply
Community Code of Conduct