Port 443 status randomly changes to 'filtered'. Restarting nginx fixes it for a while.
Server OS: Ubuntu 18.04.5
Nginx randomly stops receiving https requests.
Any https request gets "Connection timeout" response.
Restarting helps but only for a few days.
What I've checked:
systemctl status nginx
the service is active
there are no errors in nginx logs
nmap <server-ip></server-ip>
PORT STATE SERVICE
80/tcp open http
443/tcp **filtered** https
journalctl
no errors
dmesg
no errors
ufw status
inactive
iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
How can I find what's causing the problem?
UPDATE 11.07.21
The problem happened again and I've run some additional commands:
ss -plnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 16 128 127.0.0.1:7141 0.0.0.0:*
LISTEN 38 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 129 128 0.0.0.0:443 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 100 [::1]:8761 [::]:*
LISTEN 0 50 [::1]:8762 [::]:*
LISTEN 75 128 [::]:443 [::]:*
ss -atn sport == 443
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 129 128 0.0.0.0:443 0.0.0.0:*
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.14:50752
CLOSE-WAIT 62 0 <my-server-ip>:443 190.88.157.209:53692
CLOSE-WAIT 174 0 <my-server-ip>:443 188.120.116.128:9068
CLOSE-WAIT 414 0 <my-server-ip>:443 51.89.155.27:20400
CLOSE-WAIT 370 0 <my-server-ip>:443 63.143.42.242:35772
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63490
CLOSE-WAIT 518 0 <my-server-ip>:443 103.76.44.243:58496
CLOSE-WAIT 414 0 <my-server-ip>:443 51.89.155.27:21866
CLOSE-WAIT 518 0 <my-server-ip>:443 140.238.83.181:44782
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.14:64912
CLOSE-WAIT 304 0 <my-server-ip>:443 185.191.171.18:11146
CLOSE-WAIT 279 0 <my-server-ip>:443 68.4.27.246:63404
CLOSE-WAIT 518 0 <my-server-ip>:443 152.169.229.115:57877
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63481
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:27109
CLOSE-WAIT 218 0 <my-server-ip>:443 207.38.88.75:12518
CLOSE-WAIT 518 0 <my-server-ip>:443 124.240.214.92:32336
CLOSE-WAIT 373 0 <my-server-ip>:443 128.199.195.156:38372
CLOSE-WAIT 304 0 <my-server-ip>:443 185.191.171.26:23620
CLOSE-WAIT 518 0 <my-server-ip>:443 103.76.44.243:58500
CLOSE-WAIT 518 0 <my-server-ip>:443 174.2.25.50:48538
CLOSE-WAIT 377 0 <my-server-ip>:443 63.143.42.246:54248
CLOSE-WAIT 295 0 <my-server-ip>:443 54.156.8.33:20089
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:14355
CLOSE-WAIT 518 0 <my-server-ip>:443 124.240.214.92:32337
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.41:6010
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:55117
CLOSE-WAIT 518 0 <my-server-ip>:443 50.24.7.102:45514
CLOSE-WAIT 518 0 <my-server-ip>:443 66.102.8.216:35146
CLOSE-WAIT 518 0 <my-server-ip>:443 174.2.25.50:48552
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63510
CLOSE-WAIT 518 0 <my-server-ip>:443 103.76.44.243:58498
CLOSE-WAIT 518 0 <my-server-ip>:443 105.245.106.245:57395
CLOSE-WAIT 295 0 <my-server-ip>:443 54.156.8.33:2177
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63500
CLOSE-WAIT 373 0 <my-server-ip>:443 63.143.42.242:37358
CLOSE-WAIT 174 0 <my-server-ip>:443 188.120.116.128:9097
CLOSE-WAIT 518 0 <my-server-ip>:443 66.102.8.216:58832
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.41:18880
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63482
CLOSE-WAIT 518 0 <my-server-ip>:443 66.249.70.51:38665
CLOSE-WAIT 149 0 <my-server-ip>:443 190.88.157.209:53691
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:29591
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.41:40028
CLOSE-WAIT 1 0 <my-server-ip>:443 66.249.64.22:58748
CLOSE-WAIT 304 0 <my-server-ip>:443 185.191.171.18:43156
CLOSE-WAIT 211 0 <my-server-ip>:443 207.38.88.75:12734
CLOSE-WAIT 518 0 <my-server-ip>:443 151.82.77.142:14303
CLOSE-WAIT 1 0 <my-server-ip>:443 5.255.231.155:55850
CLOSE-WAIT 1 0 <my-server-ip>:443 66.249.64.24:51073
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63512
CLOSE-WAIT 373 0 <my-server-ip>:443 63.143.42.251:37356
CLOSE-WAIT 518 0 <my-server-ip>:443 50.24.7.102:45516
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.34:50266
CLOSE-WAIT 518 0 <my-server-ip>:443 105.245.106.245:57397
ESTAB 406 0 <my-server-ip>:443 95.91.75.32:3099
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.14:26910
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:31917
CLOSE-WAIT 518 0 <my-server-ip>:443 36.90.163.47:53848
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:16593
CLOSE-WAIT 414 0 <my-server-ip>:443 51.89.155.27:22292
CLOSE-WAIT 316 0 <my-server-ip>:443 5.255.231.155:47018
CLOSE-WAIT 518 0 <my-server-ip>:443 174.2.25.50:48546
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.14:2850
CLOSE-WAIT 304 0 <my-server-ip>:443 185.191.171.18:38278
CLOSE-WAIT 217 0 <my-server-ip>:443 93.158.90.56:49885
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63508
CLOSE-WAIT 518 0 <my-server-ip>:443 66.249.70.49:50315
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63483
ESTAB 0 0 <my-server-ip>:443 78.10.205.246:47113
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63509
CLOSE-WAIT 1 0 <my-server-ip>:443 85.52.243.124:46299
CLOSE-WAIT 525 0 <my-server-ip>:443 68.4.27.246:63485
CLOSE-WAIT 518 0 <my-server-ip>:443 66.102.8.216:45559
CLOSE-WAIT 294 0 <my-server-ip>:443 3.238.138.173:57324
CLOSE-WAIT 307 0 <my-server-ip>:443 185.191.171.41:4624
CLOSE-WAIT 304 0 <my-server-ip>:443 185.191.171.26:60256
CLOSE-WAIT 373 0 <my-server-ip>:443 63.143.42.242:57958
ESTAB 184 0 <my-server-ip>:443 95.91.75.32:34140
CLOSE-WAIT 1 0 <my-server-ip>:443 186.179.166.137:51668
...
ss -p | grep nginx | grep -i estab | wc -l
0
Note: Normally ss -plnt output looks like this:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:80 0.0.0.0:*
LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 0.0.0.0:443 0.0.0.0:*
LISTEN 0 128 127.0.0.1:7141 0.0.0.0:*
LISTEN 0 128 [::]:80 [::]:*
LISTEN 0 100 [::1]:8761 [::]:*
LISTEN 0 50 [::1]:8762 [::]:*
LISTEN 0 128 [::]:443 [::]:*
And ss -atn sport == 443 output normally looks like this:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:443 0.0.0.0:*
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:49070
ESTAB 0 0 <my-server-ip>:443 66.249.70.53:54557
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.41:30578
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:38618
TIME-WAIT 0 0 <my-server-ip>:443 66.249.64.22:38639
ESTAB 0 0 <my-server-ip>:443 17.121.113.26:64778
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:42886
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.67:47202
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:15380
ESTAB 0 0 <my-server-ip>:443 223.236.100.52:39478
ESTAB 0 0 <my-server-ip>:443 41.186.78.109:26127
TIME-WAIT 0 0 <my-server-ip>:443 3.235.40.235:33774
ESTAB 0 0 <my-server-ip>:443 66.249.64.23:55669
ESTAB 0 0 <my-server-ip>:443 37.210.126.179:48524
TIME-WAIT 0 0 <my-server-ip>:443 52.71.251.5:55276
ESTAB 0 0 <my-server-ip>:443 201.230.235.12:51712
ESTAB 0 0 <my-server-ip>:443 197.158.235.224:44578
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:26606
TIME-WAIT 0 0 <my-server-ip>:443 54.36.148.79:60100
...
4 Replies
jdutton
Linode Staff
It looks like someone on Stack Exchange may have already mentioned this, but it's possible that the number of requests could be a factor. I wasn't finding too many search results for similar situations, but it sounds like if you have any rate-limiting set up for NGINX, it could affect how NGINX receives requests past a certain threshold: Rate Limiting with NGINX and NGINX Plus
Since iptables is already set to accept all connections, you could also try disabling it entirely to see if that yields a better result. Likewise, enabling Network Helper and rebooting your server to help configure your internal network settings may help isolate the issue.
We also have a guide with some NGINX HTTPS configuration tips that might be a good point of reference for your own config: Getting Started with NGINX - Part 3: Enable TLS for HTTPS Connections
If you end up figuring out what the issue is, please post it here in case anyone else runs into a similar scenario. I'm also dying to know!
You can also do this by rate limiting the number of incoming connections on the port. Unfortunately, you can't do this with ufw…you have to modify the iptables(8) rules ufw generates. This is eason #4593 to understand iptables(8) and not use ufw!
See:
http://blog.lavoie.sl/2012/09/protect-webserver-against-dos-attacks.html
-- sw
@jdutton, @stevewi Thanks for your answers.
I know it looks like something related with the number of requests but I've checked nginx access logs and there were no anomalies or bursts of requests.
I couldn't find any other clues so I wrote a simple bot that monitors port statuses and will automatically run a few more commands when the problem happens again. Hopefully that will get me more information to analyze. I'll update this thread when that happens.