Cull Lingering Erlang Heart Processes

February 4, 2009. Filed under erlang 20

Working with Erlang a bit more recently, and also with the very handy heart command line parameter. (I strongly recommend this tutorial as an introduction to heart.) The heart parameter allows you to specify a command to be run if your Erlang process crashes:

export HEART_COMMAND="erl -heart -detached -run myscript start 8000"
erl -heart -detached -run myscript start 8000

If myscript crashes, it will automatically be restarted, which is undeniably handy. However, it is possible to end up with heart processes which are monitoring a non-existant Erlang process. When the monitored Erlang process dies, then a new Erlang process is spawned, along with a new heart process. Sometimes the old heart process isn't correctly disposed of if your application is crashing frequently (for example you try to run invalid code with heart monitoring it), you can end up with a number of unwanted heart processes lingering around aimlessly.

This short Perl script eliminates all heart processes which are monitoring a non-existant process. It is safe to run with multiple heart processes1, and can be either run manually or added as a periodic cronjob.

If you ran this script at the exact moment a monitored process crashed, it seems possible it would inappropriately terminate the new heart process (a simply remedy would be to avoid terminating any heart processes that are less than 5 seconds old).

use strict;

my $output = `ps aux`;
my @lines = split("\n", $output);
my %pids = ();

foreach my $line (@lines) {
  if ($line =~ m/\w+[ ]+(\d+).*$/) {
    $pids{$1} = 1;
  }
}

foreach my $line (@lines) {
  if ($line =~ m/\w+[ ]+(\d+).* heart -pid (\d+)$/) {
    if ($pids{$2} != 1) {
      exec("kill $1");
    }
  } 
}

Reading that code, I can't decide if I hate Perl or love Perl. It's an issue.


  1. Unlike the first version I scraped together, which eliminated all heart processes except that with the highest PID. Sometimes worse is really genuinely and absolutely worse.