making munin-graph take advantage of multiple cpus/cores

I do a lot of things for Automattic, and many of the things I do are quite esoteric (for a php developer anyways.) Perl is not my language of choice, but I’ve never balked at a challenge…. just… did it have to be perl? Anyways. We have more than a thousand machines that we track with munin… which means a TON of graphs. munin-update is efficient, taking advantage of all cpus and getting done in the fastest time possible, but munin-graph started taking so long as to be useless (and munin-cgi-graph takes almost a minute to fully render the servers day/week summary page which is completely unacceptable when we’re trying to troubleshoot a sudden, urgent, problem.) So I got to dive in and make it faster…

Step 1: add in this function (which i borrowed from somewhere else)

sub afork (\@$&) {
  my ($data, $max, $code) = @_;
  my $c = 0;
  foreach my $data (@$data) {
    wait unless ++ $c < = $max;
    die "Fork failed: $!
" unless defined (my $pid = fork);
    exit $code -> ($data) unless $pid;
  }
  1 until -1 == wait;
}

Step 2: replace this

for my $service (@$work_array) {
    process_service ($service);
}

with this

afork(@$work_array, 16, \&process_service);

I also have munin-html and munin-graph running side-by-side

( [ -x /usr/local/munin/lib/munin-graph  ] &&
    nice /usr/local/munin/lib/munin-graph --cron $@ 2>&1 |
    fgrep -v "*** attempt to put segment in horiz list twice" )& $waitgraph=$!
( [ -x /usr/local/munin/lib/munin-html   ] && nice /usr/local/munin/lib/munin-html $@; )& $waithtml=$!
wait $waitgraph
wait $waithtml

I did several other, more complicated hacks as well. Such as not generating month and year graphs via cron, letting those render on-demand with munin-cgi-graph

All said we’re doing in under 2.5 minutes what was taking 7 or 8 minutes previously

4 thoughts on “making munin-graph take advantage of multiple cpus/cores”

  1. Tim Vaillancourt says:

    Hey there,

    I am unable to find the lines you speak of for Step 2 in the munin-graph perl script for versions 1.0, 1.2 and even 1.3. I also cannot find "work_array" at all in munin-graph. A similiar "for my $service" exists but appears to be very different:

    "sub process_node {

    my ($domain,$name,$node) = @_;

    # See if we should skip it because of command-line arguments

    return if (@limit_hosts and not grep (/^$name$/, @limit_hosts));

    # Make my graphs

    logger ("Processing $name") if $DEBUG;

    for my $service (keys %{$node->{client}}) {

    my $service_time= Time::HiRes::time;

    my $lastupdate = 0;

    my $now = time;

    my $fnum = 0;

    my @rrd;

    my @added = ();

    …

    "

    I am using 1.2.6, the current stable version. What version did you get this working on? Thanks a lot for the article, it seems I am in the same boat with too many graphs!

    Cheers,

    Tim

    Reply

Leave a Reply