Cull Lingering Erlang Heart Processes
Working with Erlang a bit more recently, and also
with the very handy heart
command line parameter.
(I strongly recommend this tutorial as an introduction to heart
.)
The heart
parameter allows you to specify a command
to be run if your Erlang process crashes:
export HEART_COMMAND="erl -heart -detached -run myscript start 8000"
erl -heart -detached -run myscript start 8000
If myscript
crashes, it will automatically be restarted,
which is undeniably handy. However, it is possible to end up
with heart processes which are monitoring a non-existant Erlang
process. When the monitored Erlang process dies, then a new Erlang
process is spawned, along with a new heart process. Sometimes
the old heart process isn't correctly disposed of if
your application is crashing frequently (for example you try to
run invalid code with heart monitoring it), you can end up with a number
of unwanted heart processes lingering around aimlessly.
This short Perl script eliminates all heart
processes which
are monitoring a non-existant process. It is safe to run with
multiple heart processes1, and can be either run manually or
added as a periodic cronjob.
If you ran this script at the exact moment a monitored process crashed, it seems possible it would inappropriately terminate the new heart process (a simply remedy would be to avoid terminating any heart processes that are less than 5 seconds old).
use strict;
my $output = ps aux
;
my @lines = split("\n", $output);
my %pids = ();
foreach my $line (@lines) {
if ($line =~ m/\w+[ ]+(\d+).*$/) {
$pids{$1} = 1;
}
}
foreach my $line (@lines) {
if ($line =~ m/\w+[ ]+(\d+).* heart -pid (\d+)$/) {
if ($pids{$2} != 1) {
exec("kill $1");
}
}
}
Reading that code, I can't decide if I hate Perl or love Perl. It's an issue.
Unlike the first version I scraped together, which eliminated all heart processes except that with the highest PID. Sometimes worse is really genuinely and absolutely worse.↩