I do a lot of things for Automattic, and many of the things I do are quite esoteric (for a php developer anyways.) Perl is not my language of choice, but I’ve never balked at a challenge…. just… did it have to be perl? Anyways. We have more than a thousand machines that we track with munin… which means a TON of graphs. munin-update is efficient, taking advantage of all cpus and getting done in the fastest time possible, but munin-graph started taking so long as to be useless (and munin-cgi-graph takes almost a minute to fully render the servers day/week summary page which is completely unacceptable when we’re trying to troubleshoot a sudden, urgent, problem.) So I got to dive in and make it faster…
Step 1: add in this function (which i borrowed from somewhere else)
sub afork (\@$&) { my ($data, $max, $code) = @_; my $c = 0; foreach my $data (@$data) { wait unless ++ $c < = $max; die "Fork failed: $!\n" unless defined (my $pid = fork); exit $code -> ($data) unless $pid; } 1 until -1 == wait; }
Step 2: replace this
for my $service (@$work_array) { process_service ($service); }
with this
afork(@$work_array, 16, \&process_service);
I also have munin-html and munin-graph running side-by-side
( [ -x /usr/local/munin/lib/munin-graph ] && nice /usr/local/munin/lib/munin-graph --cron $@ 2>&1 | fgrep -v "*** attempt to put segment in horiz list twice" )& $waitgraph=$! ( [ -x /usr/local/munin/lib/munin-html ] && nice /usr/local/munin/lib/munin-html $@; )& $waithtml=$! wait $waitgraph wait $waithtml
I did several other, more complicated hacks as well. Such as not generating month and year graphs via cron, letting those render on-demand with munin-cgi-graph
All said we’re doing in under 2.5 minutes what was taking 7 or 8 minutes previously