Archive for January, 2009

making munin-graph take advantage of multiple cpus/cores

23 Jan

Posted in Business, CLI, Linux, Software Development | 4 Comments »

I do a lot of things for Automattic, and many of the things I do are quite esoteric (for a php developer anyways.) Perl is not my language of choice, but I’ve never balked at a challenge…. just… did it have to be perl? Anyways. We have more than a thousand machines that we track with munin… which means a TON of graphs. munin-update is efficient, taking advantage of all cpus and getting done in the fastest time possible, but munin-graph started taking so long as to be useless (and munin-cgi-graph takes almost a minute to fully render the servers day/week summary page which is completely unacceptable when we’re trying to troubleshoot a sudden, urgent, problem.) So I got to dive in and make it faster…

Step 1: add in this function (which i borrowed from somewhere else)

sub afork (\@$&) {
  my ($data, $max, $code) = @_;
  my $c = 0;
  foreach my $data (@$data) {
    wait unless ++ $c < = $max;
    die "Fork failed: $!\n" unless defined (my $pid = fork);
    exit $code -> ($data) unless $pid;
  }
  1 until -1 == wait;
}

Step 2: replace this

for my $service (@$work_array) {
    process_service ($service);
}

with this

afork(@$work_array, 16, \&process_service);

I also have munin-html and munin-graph running side-by-side

( [ -x /usr/local/munin/lib/munin-graph  ] &&
    nice /usr/local/munin/lib/munin-graph --cron $@ 2>&1 |
    fgrep -v "*** attempt to put segment in horiz list twice" )& $waitgraph=$!
( [ -x /usr/local/munin/lib/munin-html   ] && nice /usr/local/munin/lib/munin-html $@; )& $waithtml=$!
wait $waitgraph
wait $waithtml

I did several other, more complicated hacks as well. Such as not generating month and year graphs via cron, letting those render on-demand with munin-cgi-graph

All said we’re doing in under 2.5 minutes what was taking 7 or 8 minutes previously

Google stops development on 6 services

18 Jan

Posted in Random Thoughts | 2 Comments »

[edit: link]

I already see the Stallmanites rallying for their battle cries. Never using anything you didn’t write yourself is an asinine concept, in my opinion… This coming from someone who can write web services himself. The truth is that using services “in the cloud,” “on the web,” or anywhere else is just like using local software in one very important sense. (I particularly like one comment I heard on this once which went something like: “Would you be able to validate the source code to the ls binary on your own?”)

If your data is not in two completely separate locations, then it’s not safe.

My wife, who’s in need of some extra storage space (shes starting to get into photography some) got two external 120gb hard drives (which were on clearance.) Before I let her use them I sat her down and gave her some important advice: “If you store your data on one of these drives… it is NOT backed up… it’s just stored on that disk. And if something happens to that disk there is NOTHING that I can do to save your photos. Period. I bought you two because every so often you need to copy whats important to the second disk. That was if one disk dies, you don’t loose your stuff”

I’m not sure why people feel that if they’re using apps in the cloud that this doesn’t apply to them. A service shutting down is basically equivalent to loosing a hard disk. Be prepared. Back your data up if it’s that important!

As an aside. All WordPress installations allow you to export your data — even WordPress.com. I suggest people take advantage of that on occasion!

A Pure Memcached Queue

16 Jan

Posted in Random Thoughts | 2 Comments »

This is pretty clever… Might have to code this up in PHP… memcachequeue-a-pure-memcached-queue

php image functions failing on uploaded images

15 Jan

Posted in Random Thoughts | 1 Comment »

if you’re dealing with user uploaded images in any non-passthrough way (such as resizing, converting, etc) you may be familiar with this particular error message for corrupt images: “Corrupt JPEG data: xxx extraneous bytes before marker 0xd9″ Regardless of who, how, or why this happens the error is usually non fatal as far as the visual image data itself is concerned. And you just want php to make a gd resource from the image already… right? Well I cant make it do that, but if you have imagemagick installed you can simly execute the following:

/path/to/mogrify /path/to/image -strip

this will strip out the image metadata inside the image (with a WARNING (which is all this message is intended to be… thanks php/gd))

I have a feeling this will help a number of frustrated developers AND users who are left wondering why their image wont work…

Palms new phone

08 Jan

Posted in Random Thoughts | No Comments »

It looks nice… but will Palm be able to pull out of their nose-dive? I guess, to my mind, if Palm were a scene from a movie It’s be the end of the subway scene from the first Matrix movie. Palm is neo on the edge of the platform trying not to get sucked into the train (except that Palm didn’t just finish kicking anyones ass.) The question i guess we’re all wondering is whether they can keep from face-planting into the side of the train as it rushes past?

That was probably a really stupid analogy… I really want to have faith in Palm… I had more than one Treo… I just cant bring myself to be anything but cautious…