POST requests fail with Gunicorn
I have a Flask app that I deployed to Linode that works great for about an hour and then after that the pages keep loading but any database interactions fail for user logins or other post requests (like editing profiles), while get requests work fine (like queries that return data for pages). Is there anything that would impact post requests like that in terms of configuration? I don't have any problems like this when I'm running locally. When I run sudo reboot on the server, it works again, but only for another hour or so.
When checking the error logs, Gunicorn does seem to be running into problems, but as far as I can tell from reading documentation, "signal 15" is pretty vague and makes diagnosis a challenge. I've included a snapshot of the logs below.
I'm running supervisor with the following config (note I'm just testing the single core, hence the three workers for gunicorn):
[program:sims-portal]
directory=/home/jonathan/flask_app
command=/home/jonathan/flask_app/venv/bin/gunicorn -w 3 run:app
user=jonathan
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
stderr_logfile=/var/log/flaskapp/flaskapp.err.log
stdout_logfile=/var/log/flaskapp/flaskapp.out.log
Any help would be appreciated! If it helps to see the code itself, it's stored here: https://github.com/JonathanGarro/SIMS-Portal-Neo
[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 737 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 738 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 16:39:38 +0000] [733] [INFO] Shutting down: Master
[2022-07-19 16:41:37 +0000] [734] [INFO] Starting gunicorn 20.1.0
[2022-07-19 16:41:37 +0000] [734] [INFO] Listening at: http://127.0.0.1:8000 (734)
[2022-07-19 16:41:37 +0000] [734] [INFO] Using worker: sync
[2022-07-19 16:41:37 +0000] [735] [INFO] Booting worker with pid: 735
[2022-07-19 16:41:37 +0000] [736] [INFO] Booting worker with pid: 736
[2022-07-19 16:41:38 +0000] [737] [INFO] Booting worker with pid: 737
[2022-07-19 21:50:12 +0000] [737] [INFO] Worker exiting (pid: 737)
[2022-07-19 21:50:12 +0000] [735] [INFO] Worker exiting (pid: 735)
[2022-07-19 21:50:12 +0000] [736] [INFO] Worker exiting (pid: 736)
[2022-07-19 21:50:12 +0000] [734] [INFO] Handling signal: term
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 735 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 737 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [INFO] Shutting down: Master
[2022-07-19 21:51:51 +0000] [833] [INFO] Starting gunicorn 20.1.0
[2022-07-19 21:51:51 +0000] [833] [INFO] Listening at: http://127.0.0.1:8000 (833)
[2022-07-19 21:51:51 +0000] [833] [INFO] Using worker: sync
[2022-07-19 21:51:51 +0000] [834] [INFO] Booting worker with pid: 834
[2022-07-19 21:51:51 +0000] [835] [INFO] Booting worker with pid: 835
[2022-07-19 21:51:51 +0000] [836] [INFO] Booting worker with pid: 836
[2022-07-20 20:20:43 +0000] [833] [INFO] Handling signal: term
[2022-07-20 20:20:43 +0000] [834] [INFO] Worker exiting (pid: 834)
[2022-07-20 20:20:43 +0000] [836] [INFO] Worker exiting (pid: 836)
[2022-07-20 20:20:43 +0000] [835] [INFO] Worker exiting (pid: 835)
[2022-07-20 20:20:43 +0000] [833] [WARNING] Worker with pid 835 was terminated due to signal 15
[2022-07-20 20:20:45 +0000] [833] [INFO] Shutting down: Master
[2022-07-20 20:20:47 +0000] [4706] [INFO] Starting gunicorn 20.1.0
[2022-07-20 20:20:47 +0000] [4706] [INFO] Listening at: http://127.0.0.1:8000 (4706)
[2022-07-20 20:20:47 +0000] [4706] [INFO] Using worker: sync
[2022-07-20 20:20:47 +0000] [4707] [INFO] Booting worker with pid: 4707
[2022-07-20 20:20:47 +0000] [4708] [INFO] Booting worker with pid: 4708
[2022-07-20 20:20:47 +0000] [4709] [INFO] Booting worker with pid: 4709
4 Replies
isackey
Linode Staff
There's a high chance your Linode might be OOMing looking at the logs you provided. Have you considered resizing your Linode to a bigger plan?
Run dmesg to check for any OOMing errors on your Linode that may be killing your processes. A user from this post reported similar issues with Flask and it does have some good troubleshooting information that should be helpful.
Thanks for the tips. I ran dmesg and got a pretty long return, but didn't find any alerts like in post your link to (nothing about being OOM). I did just resize the Linode but the problem persists.
I don't know anything about python, Flask or Gunicorn. I do know about signal-handling (generally) though…
When checking the error logs, Gunicorn does seem to be running into problems, but as far as I can tell from reading documentation, "signal 15" is pretty vague and makes diagnosis a challenge. I've included a snapshot of the logs below.
[2022-07-19 21:50:12 +0000] [734] [INFO] Handling signal: term
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 735 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 736 was terminated due to signal 15
[2022-07-19 21:50:12 +0000] [734] [WARNING] Worker with pid 737 was terminated due to signal 15
The Gunicorn doc at https://docs.gunicorn.org/en/stable/signals.html describes SIGTERM processing (for both master & workers) as "graceful shutdown."
This would be what I would expect if you send SIGTERM (signal 15) to a master process with a bunch of children.
-- sw
same issue using flask,nginx and gunicorn. after some time my app will stop responding especially POST forms(login,contact). other html pages keep rendering fine. I constantly reboot my linode or reload supervisor for everything to run again. I checked the configuration file, has no errors. Is maybe because I am on a shared linode?
[2024-04-09 17:59:20 +0000] [12827] [INFO] Worker exiting (pid: 12827)
[2024-04-09 17:59:20 +0000] [12826] [INFO] Worker exiting (pid: 12826)
[2024-04-09 17:59:20 +0000] [12825] [INFO] Worker exiting (pid: 12825)
[2024-04-09 17:59:20 +0000] [12822] [INFO] Handling signal: term
[2024-04-09 17:59:20 +0000] [12822] [ERROR] Worker (pid:12827) was sent SIGTERM!
[2024-04-09 17:59:20 +0000] [12822] [ERROR] Worker (pid:12826) was sent SIGTERM!
[2024-04-09 17:59:20 +0000] [12822] [ERROR] Worker (pid:12825) was sent SIGTERM!
[2024-04-09 17:59:21 +0000] [12822] [INFO] Shutting down: Master
[2024-04-09 17:59:57 +0000] [939] [INFO] Starting gunicorn 21.2.0
[2024-04-09 17:59:57 +0000] [939] [INFO] Listening at: http://127.0.0.1:8000 (9$
[2024-04-09 17:59:57 +0000] [939] [INFO] Using worker: sync
[2024-04-09 17:59:57 +0000] [963] [INFO] Booting worker with pid: 963
[2024-04-09 17:59:57 +0000] [964] [INFO] Booting worker with pid: 964
[2024-04-09 17:59:57 +0000] [965] [INFO] Booting worker with pid: 965