making munin-graph take advantage of multiple cpus/cores

I do a lot of things for Automattic, and many of the things I do are quite esoteric (for a php developer anyways.) Perl is not my language of choice, but I’ve never balked at a challenge…. just… did it have to be perl? Anyways. We have more than a thousand machines that we track with munin… which means a TON of graphs. munin-update is efficient, taking advantage of all cpus and getting done in the fastest time possible, but munin-graph started taking so long as to be useless (and munin-cgi-graph takes almost a minute to fully render the servers day/week summary page which is completely unacceptable when we’re trying to troubleshoot a sudden, urgent, problem.) So I got to dive in and make it faster…

Step 1: add in this function (which i borrowed from somewhere else)

sub afork (\@$&) {
  my ($data, $max, $code) = @_;
  my $c = 0;
  foreach my $data (@$data) {
    wait unless ++ $c < = $max;
    die "Fork failed: $!
" unless defined (my $pid = fork);
    exit $code -> ($data) unless $pid;
  }
  1 until -1 == wait;
}

sub afork (\@$&) {

my ($data, $max, $code) = @_;

my $c = 0;

foreach my $data (@$data) {

wait unless ++ $c < = $max;

die "Fork failed: $!

" unless defined (my $pid = fork);

exit $code -> ($data) unless $pid;

}

1 until -1 == wait;

}

Step 2: replace this

for my $service (@$work_array) {
    process_service ($service);
}

for my $service (@$work_array) {

process_service ($service);

}

with this

afork(@$work_array, 16, \&process_service);

1	afork(@$work_array, 16, \&process_service);

I also have munin-html and munin-graph running side-by-side

( [ -x /usr/local/munin/lib/munin-graph  ] &&
    nice /usr/local/munin/lib/munin-graph --cron $@ 2>&1 |
    fgrep -v "*** attempt to put segment in horiz list twice" )& $waitgraph=$!
( [ -x /usr/local/munin/lib/munin-html   ] && nice /usr/local/munin/lib/munin-html $@; )& $waithtml=$!
wait $waitgraph
wait $waithtml

( [ -x /usr/local/munin/lib/munin-graph ] &&

nice /usr/local/munin/lib/munin-graph --cron $@ 2>&1 |

fgrep -v "*** attempt to put segment in horiz list twice" )& $waitgraph=$!

( [ -x /usr/local/munin/lib/munin-html ] && nice /usr/local/munin/lib/munin-html $@; )& $waithtml=$!

wait $waitgraph

wait $waithtml

I did several other, more complicated hacks as well. Such as not generating month and year graphs via cron, letting those render on-demand with munin-cgi-graph

All said we’re doing in under 2.5 minutes what was taking 7 or 8 minutes previously

4 thoughts on “making munin-graph take advantage of multiple cpus/cores”

Tim Vaillancourt 2009/05/02 at 4:49 AM

Hey there,

I am unable to find the lines you speak of for Step 2 in the munin-graph perl script for versions 1.0, 1.2 and even 1.3. I also cannot find "work_array" at all in munin-graph. A similiar "for my $service" exists but appears to be very different:

"sub process_node {

my ($domain,$name,$node) = @_;

# See if we should skip it because of command-line arguments

return if (@limit_hosts and not grep (/^$name$/, @limit_hosts));

# Make my graphs

logger ("Processing $name") if $DEBUG;

for my $service (keys %{$node->{client}}) {

my $service_time= Time::HiRes::time;

my $lastupdate = 0;

my $now = time;

my $fnum = 0;

my @rrd;

my @added = ();

…

"

I am using 1.2.6, the current stable version. What version did you get this working on? Thanks a lot for the article, it seems I am in the same boat with too many graphs!

Cheers,

Tim

Reply ↓
kOoLiNuS 2009/07/21 at 5:51 PM

Nice, but way too complicated for my actual skills !

Reply ↓
vinc 2009/09/27 at 2:39 AM

Hi,

indeed, could you specify the munin version your are using ?

Thanks,

V

Reply ↓
apokalyptik Post author2009/09/27 at 3:00 AM

We were using munin_1.3.4

Reply ↓