Out of memory exception, eheap_alloc: Cannot allocate x bytes of memory (of type "heap").
Hi,
I'm having problems with one of my servers. I have simple app that reads files from ftp and stores in database.
After maintenance restart application is crashing after opening FTP connection.CPU rises til process is killed by oom.
Slogan: eheap_alloc: Cannot allocate 255489152 bytes of memory (of type "heap").
System version: Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]
Compiled: Tue Aug 4 13:49:01 2020
Taints: crypto
Atoms: 29346
Calling Thread: scheduler:0
=scheduler:1
Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING
Scheduler Sleep Info Aux Work: THR_PRGR_LATER_OP
Current Port:
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK
Current Process:
=scheduler:2
Scheduler Sleep Info Flags: SLEEPING | POLL_SLEEPING | WAITING
Scheduler Sleep Info Aux Work:
Current Port:
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | INACTIVE
Current Process:
=dirty_cpu_scheduler:3
Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING
Scheduler Sleep Info Aux Work:
Current Process:
=dirty_cpu_scheduler:4
Scheduler Sleep Info Flags:
Scheduler Sleep Info Aux Work:
Current Process: <0.1121.0>
Current Process State: Garbing
Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | GC | DIRTY_ACTIVE_SYS | DIRTY_RUNNING_SYS
Current Process Program counter: 0x00007f5941057438 ('Elixir.Ecto.Adapters.Postgres.TypeModule':'Elixir.Ecto.Adapters.Postgres.Timestamp'/6 + 240)
Current Process Limited Stack Trace:
0x00007f594144f290:SReturn addr 0x436E35F8 ('Elixir.Postgrex.Protocol':rows_recv/4 + 136)
0x00007f594144f2a8:SReturn addr 0x436D8740 ('Elixir.Postgrex.Protocol':execute_recv/5 + 232)
0x00007f594144f2d8:SReturn addr 0x43688A80 ('Elixir.DBConnection':handle/4 + 400)
0x00007f594144f320:SReturn addr 0x43687928 ('Elixir.DBConnection':describe_run/5 + 360)
0x00007f594144f360:SReturn addr 0x4368DA80 ('Elixir.DBConnection':'-run_meter/5-fun-0-'/4 + 136)
0x00007f594144f388:SReturn addr 0x4368B888 ('Elixir.DBConnection':run_begin/3 + 112)
0x00007f594144f3b8:SReturn addr 0x4368A628 ('Elixir.DBConnection':prepare_execute/4 + 144)
0x00007f594144f3d8:SReturn addr 0x4367E5B0 ('Elixir.Ecto.Adapters.Postgres.Connection':prepare_execute/5 + 192)
0x00007f594144f3e0:SReturn addr 0x80320E18 ('Elixir.Ecto.Adapters.SQL':sql_call/6 + 840)
0x00007f594144f420:SReturn addr 0x8031ED38 ('Elixir.Ecto.Adapters.SQL':execute_and_cache/7 + 408)
0x00007f594144f460:SReturn addr 0x410E3FC8 ('Elixir.Ecto.Repo.Queryable':execute/5 + 864)
0x00007f594144f4c8:SReturn addr 0x410E37E0 ('Elixir.Ecto.Repo.Queryable':all/4 + 240)
0x00007f594144f4d0:SReturn addr 0x803009E0 ('Elixir.Neptun.Fetchers.FTPFetcher':filter_listing_for_downloaded_data/1 + 336)
0x00007f594144f4e8:SReturn addr 0x803002F0 ('Elixir.Neptun.Fetchers.FTPFetcher':do_work/0 + 248)
0x00007f594144f508:SReturn addr 0x803012E0 ('Elixir.Neptun.Fetchers.FTPFetcher':handle_info/2 + 504)
0x00007f594144f520:SReturn addr 0x84612198 (gen_server:try_dispatch/4 + 152)
0x00007f594144f568:SReturn addr 0x84613038 (gen_server:handle_msg/6 + 1696)
0x00007f594144f5a0:SReturn addr 0x8461B840 (proc_lib:init_p_do_apply/3 + 72)
0x00007f594144f5c0:SReturn addr 0x6A011F08 (<terminate normally="" process="">)
=dirty_cpu_run_queue
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | EXEC
=dirty_io_scheduler:5
Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING
Scheduler Sleep Info Aux Work:
Current Process:
=dirty_io_scheduler:6</terminate>
elixir version: 1.7.4-otp-21
erlang/otp: 21.3
postgresql: 9.6
I tried compiling with newer versions with no result.
A appreciate your feedback. Thank you.
3 Replies
jtoscani
Linode Staff
If your server is OOMing, you've run out of RAM and will need to give your processes more breathing room. There are a few steps to take:
1. Upgrade to a larger server
If you're currently using all of your RAM to it's full potential, you'll want to increase your server size to capture more space. We have a guide that walks you through this process: Resize Your Linode. This will allow you to regain access to your server.
2. Optimize your processes to run more efficiently with the available resources.
Once you're able to log into your server, you can explore the running processes and try to identify any excessive use. To that end, we have an excellent Community post which can provide further guidance: My Linode is OOMing, what should I do?
3. Investigate Crash Dumps
- In reviewing this documentation from Erlang.org, see Section 8.11.2- Memory profiling (page 244/369)
eheap_alloc: Cannot allocate 1234567890 bytes of memory (of type "heap").
The above slogan is one of the more common reasons for Erlang to terminate. For unknown reasons the Erlang RunTime System failed to allocate memory to use. When this happens a crash dump is generated that contains information about the state of the system as it ran out of memory. Use the crashdump_viewer to get a view of the memory is being used.
- In light of this, here is another resource from Erlang.org that would prove useful: How to Interpret the Erlang Crash Dumps
To address why this may have happened after maintenance: it's possible that certain settings or configuration changes were made prior to your server being shut down for maintenance, but were not yet put into effect. When Linodes are restarted, these "dormant" changes then become active, which is why upon reboot some customers run into issues such as this.
Thank you for your reply.
Thing that confuses me is that server was up over a year and everything was working just fine.
I also have two more servers with same RAM and CPU and they have peak when opening FTP but not more than 100% on CPU and it gets right back to normal.