While working in our RabbitMQ environment this week, we noticed that there was a large, unexplained amount of memory in use by RabbitMQ that we couldn’t account for by normal queue and message use. One of the first tools we use when poking around Erlang and RabbitMQ is to do a memory dump.
$ sudo rabbitmqctl eval
'lists:sublist(lists:reverse(
lists:sort([{process_info(Pid, memory), Pid,
process_info(Pid)} || Pid <- processes()])), 1).'
This command is evaluated directly into the running erlang instance within our RabbitMQ and dumps the top memory users from largest to smallest. In the above case, we're telling erlang to show us the single top user of memory. You can change the 1 to any positive number you like to get more and more erlang processes returned to evaluate the memory usage of.
This produces something like the following:
[{{memory,12344224},
<4865.28344.41>,
[{current_function,{gen_server2,process_next_msg,1}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<4865.28338.41>,<4865.28353.41>,#Port<4865.433154>]},
{dictionary,[{{credit_to,<4865.5033.42>},28},
{{#Ref<4865.0.1377.83012>,fhc_handle},
{handle,{file_descriptor,prim_file,{#Port<4865.433154>,20}},
335872,false,0,infinity,[],true,
"/var/lib/rabbitmq/mnesia/rabbit@rabbitmqhost/queues/14AI6IEJDH4TZNL8R69HT5Y6K/journal.jif",
[write,binary,raw,read],
[{write_buffer,infinity}],
true,true,
{1351,12777,400557}}},
{{ch,<4865.28440.41>},
{cr,<4865.28440.41>,#Ref<4865.0.1377.84010>,
{set,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},
1,
{[],[]},
{token,<4865.28439.41>,true},
false,19}},
{{credit_to,<4865.4869.42>},15},
{'$ancestors',[rabbit_amqqueue_sup,rabbit_sup,
<4865.28225.41>]},
{{credit_to,<4865.4228.42>},49},
{fhc_age_tree,{1,
{{1351,12777,400557},
#Ref<4865.0.1377.83012>,nil,nil}}},
{{credit_to,<4865.28474.41>},35},
{{credit_to,<4865.5066.42>},11},
{credit_blocked,[]},
{{credit_to,<4865.28507.41>},3},
{{credit_to,<4865.28490.41>},33},
{{"/var/lib/rabbitmq/mnesia/rabbit@rabbitmqhost/queues/14AI6IEJDH4TZNL8R69HT5Y6K/journal.jif",
fhc_file},
{file,1,true}},
{{credit_to,<4865.28498.41>},14},
{{credit_to,<4865.28482.41>},31},
{guid,{{4270534438,2738637279,2258006136,1881461985},0}},
{'$initial_call',{gen,init_it,6}}]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<4865.28224.41>},
{total_heap_size,1542687},
{heap_size,196418},
{stack_size,7},
{reductions,332233374},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,20706}]},
{suspending,[]}]}]
...done.
I don't know what all of the bits mean yet, but we can point out a few useful things to look at. A good reference on what is being returned can be found in the Erlang documentation for process_info/2
- memory - the current amount of memory in use by this erlang process. This directly effects the size of the RabbitMQ process size in the OS.
<4865.28344.41>- the erlang pid inside the erlang kernel- current_function - the function call currently running in the process
- registered_name - (Note: note seen above). this is the name of the process associated with this memory. If the process isn't named, this won't show up in the output
- status - What the process is currently doing. It can be something like exiting, garbage_collecting, waiting, running, runnable, or suspended.
- messages - messages associated with this process.
- dictionary -
There's more than this, so you should definitely look at the erlang docs to figure out what's going on here, but this should get you started in understanding what memory usage is for your RabbitMQ environment.
