Postgres seems to keep getting shutdown after a while
I restart postgresql and then browse around in my app but then after a while it seems to shutdown and I get this message:
exception: "Illuminate\Database\QueryException" file: "/root/maketube/api/vendor/laravel/framework/src/Illuminate/Database/Connection.php" line: 664 message: "SQLSTATE[08006] [7] could not connect to server: Connection refused↵ Is the server running on host "localhost" (::1) and accepting↵ TCP/IP connections on port 5432?↵could not connect to server: Connection refused↵ Is the server running on host "localhost" (127.0.0.1) and accepting↵ TCP/IP connections on port 5432? (SQL: select * from "tags")"
I have it installed on my production server: psql (PostgreSQL) 11.2 (Ubuntu 11.2-1.pgdg18.04+1)
My setup is: Laravel as the backend and node as the frontend. With nginx used to route.
My linode instance is: Ubuntu 18.04 LTS, Nanode 1GB: 1 CPU, 25GB Storage, 1GB RAM
My postgres log does not say much more:
2019-04-12 22:20:55.678 UTC [4311] LOG: database system is ready to accept connections
2019-04-13 10:11:27.727 UTC [10603] LOG: listening on IPv6 address "::1", port 5433
2019-04-13 10:11:27.729 UTC [10603] LOG: listening on IPv4 address "127.0.0.1", port 5433
2019-04-13 10:11:27.732 UTC [10603] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5433"
2019-04-13 10:11:27.788 UTC [10605] LOG: database system was interrupted; last known up at 2019-04-12 22:20:55 UTC
2019-04-13 10:11:28.747 UTC [10605] LOG: database system was not properly shut down; automatic recovery in progress
2019-04-13 10:11:28.784 UTC [10605] LOG: redo starts at 0/165B6C0
2019-04-13 10:11:28.784 UTC [10605] LOG: invalid record length at 0/165B6F8: wanted 24, got 0
2019-04-13 10:11:28.784 UTC [10605] LOG: redo done at 0/165B6C0
2019-04-13 10:11:28.843 UTC [10603] LOG: database system is ready to accept connections
2019-04-13 10:11:29.391 UTC [10624] [unknown]@[unknown] LOG: incomplete startup packet
Anyone know whats happening?
9 Replies
The log you posted is for postgre starting up… and it complains that it wasn't previously shut down properly…
… but that doesn't tell us why it shut down (or was killed)?
Can you look for that in the logs?
Maybe something like this?
https://www.postgresql.org/message-id/3997.192.168.1.243.1182192844.squirrel@mail.wexwarez.com
LOG: server process (PID 13904) was terminated by signal 9
LOG: terminating any other active server processes
in postgresql-11-main.log? There is literally nothing more. I delete that file and the restarted postgres and it shows the same messages and I posted initially. Is there any other log I dont know about?
I have a read stuff about OOM Kill but not sure how to check this
Maybe try systemd logs?
journalctl -f -u postgresql to watch output as it happens
or
journalctl --no-pager -u postgresql for history
As far as OOM goes - you can use e.g. htop to watch memory usage (htop is not installed by default, apt-get install htop).
And if you do find a memory leak - it could be your web app leaking connections (a fairly common type of bug).
To check that you can use SELECT usesysid, usename FROM pg_stat_activity and keep an eye on it over time if you can.
( pg command taken from here: https://stackoverflow.com/a/464641/2342806 )
To check for OOM kills that already happened - and I suppose there should be some, if that theory is right - you can do this:
grep -i killed /var/log/messages
Hmm so upgrading my linode instance to 2GB ram seems to work. lets see after some usage if the same issue still happens
btw running the command produces:
grep: /var/log/messages: No such file or directory
Can you try
dmesg | grep -i killed
( my servers are Debian, close but not identical to Ubuntu )
That returned nothing, but I guess this is now after I upgraded my linode, so maybe it was a memory issue