Google stops development on 6 services

[edit: link]

I already see the Stallmanites rallying for their battle cries. Never using anything you didn’t write yourself is an asinine concept, in my opinion… This coming from someone who can write web services himself. The truth is that using services “in the cloud,” “on the web,” or anywhere else is just like using local software in one very important sense. (I particularly like one comment I heard on this once which went something like: “Would you be able to validate the source code to the ls binary on your own?”)

If your data is not in two completely separate locations, then it’s not safe.

My wife, who’s in need of some extra storage space (shes starting to get into photography some) got two external 120gb hard drives (which were on clearance.) Before I let her use them I sat her down and gave her some important advice: “If you store your data on one of these drives… it is NOT backed up… it’s just stored on that disk. And if something happens to that disk there is NOTHING that I can do to save your photos. Period. I bought you two because every so often you need to copy whats important to the second disk. That was if one disk dies, you don’t loose your stuff”

I’m not sure why people feel that if they’re using apps in the cloud that this doesn’t apply to them. A service shutting down is basically equivalent to loosing a hard disk. Be prepared. Back your data up if it’s that important!

As an aside. All WordPress installations allow you to export your data — even WordPress.com. I suggest people take advantage of that on occasion!

php image functions failing on uploaded images

if you’re dealing with user uploaded images in any non-passthrough way (such as resizing, converting, etc) you may be familiar with this particular error message for corrupt images: “Corrupt JPEG data: xxx extraneous bytes before marker 0xd9” Regardless of who, how, or why this happens the error is usually non fatal as far as the visual image data itself is concerned. And you just want php to make a gd resource from the image already… right? Well I cant make it do that, but if you have imagemagick installed you can simly execute the following:

/path/to/mogrify /path/to/image -strip

this will strip out the image metadata inside the image (with a WARNING (which is all this message is intended to be… thanks php/gd))

I have a feeling this will help a number of frustrated developers AND users who are left wondering why their image wont work…

Palms new phone

It looks nice… but will Palm be able to pull out of their nose-dive? I guess, to my mind, if Palm were a scene from a movie It’s be the end of the subway scene from the first Matrix movie. Palm is neo on the edge of the platform trying not to get sucked into the train (except that Palm didn’t just finish kicking anyones ass.) The question i guess we’re all wondering is whether they can keep from face-planting into the side of the train as it rushes past?

That was probably a really stupid analogy… I really want to have faith in Palm… I had more than one Treo… I just cant bring myself to be anything but cautious…

A tale of two gamers

Gamer #1 — The youngster. The youngster grew up playing games… The NES isn’t so much nostalgic as archaic. They don’t have much money working on getting through high-school, college, or fresh out. But what they lack in cash flow they make up for in time. These are the purists. If this person spends $60 on a game they want it to be CHALLENGING. They want as much time and value out of a game as they can possibly get. These are the WoW players who think spending 30 hours a week is a good investment of their time.

Gamer #2 — The spouse. The spouse used to be a youngster. But they’ve since acquired this marvelous, and strange, new thing — a life. Now they have a career — not a job — a wife, and possibly a kid or two. When this person forks out $60 for a game he wants a good respite from real life without consuming his real life. It’s challenging for this person to put in 5 hours a week into a game and 30 is completely unrealistic (even if they’d love to be able to.) This gamer wants to be completely engrossed and entertained for a while but also needs to be able to put the game down and pick it up again in 2 weeks without loosing much.

I get reminded of this from time to time on forum discussions and the like. Especially when you have these two factions arguing over things like gold farmers, and cheats, and glitches. I happen to fall into the Gamer #2 category, and I can tell you that when I get 3 hours to sit down and play a game I do NOT want to spend that hard earned time grinding. __I__JUST__DON’T__. The real PITA about that is that I would LOVE to play games like WoW.

I hope that game developers start looking at my demographic (unlike kids (who in their defense can’t) I’m willing to pay for my games…) seriously. give me a WoW server with the exp and gold tweaked so that I can get past the crappy ‘kill 50 fluffy bunnies’ grinding quests, and get to some fun gaming before I turn 60.

The youngsters will, of course, argue things like “taking away from the game,” “why even bother playing,” and things like “it’s not that hard and doesnt take that long, you just suck, n00b.”

To which I say bite me. Until game makers start understanding that some gamers want a time sink, and some gamers cant afford one. I’ll be the low level guy who just payed for 60 hours of some Chinese gold farmers time…. Because one of these days I’d like to get to do something interesting… I’ll be the guy who uses the game glitch to avoid spending 15 hours forging swords or chopping wood.

I would like to not have to resort to these measures… and If the game manufacturers would just throw us a bone, I bet we wouldnt…

A counting bloom filter

This is just me screwing around, really…. This implements a counting bloom filter in native php. I read about bloom filters this morning, and wrote this tonight in between playing Command and Conquer 3 campaign missions. The hashing function is configurable (and easily extensible,) rehashing the key multiple times to reduce the chance of collisions is also configurable (though I’m not entirely sure how needed this is.) Frankly this all might not even be (or work) properly… seems to… we’ll see…

< ?php

class bloom {

	/**
	 *
	 * A counting bloom filter
	 *
	 *	$f = new bloom(10, 'md5');
	 *
	 *	$f->add('foo');
	 *	$f->add('bar');
	 *	$f->add('foo');
	 *
	 *	$f->exists('foo');        // true
	 *	$f->exists('bar');        // true
	 *	$f->exists('baz');        // false
	 *	$f->exists('foo', false); // 2
	 *	$f->exists('bar', false); // 1
	 *	$f->exists('baz', false); // false
	 *
	 **/

	var $number_of_hashing_functions;
	var $map = array();
	var $hashlen = 32;
	var $hashfunc = 'md5';

	function __construct( $functions=1, $hashtype='md5' ) {
		$this->bloom($functions, $hashtype);
	}

	function bloom( $functions=1, $hashtype='md5' ) {
		$this->number_of_hashing_functions = (int)$functions;
		if ( !$this->number_of_hashing_functions || $this->number_of_hashing_functions < 1 )
			$this->number_of_hashing_functions = 1;
		$this->set_hashing_function($hashtype);
		$this->initialize_bitmat($this->number_of_hashing_functions);
	}

	function set_hashing_function($method) {
		switch ( $method ) {
			default:
			case 'md5':
				return false;
				break;
			case 'sha1':
				$this->hashlen = 40;
				$this->hashfunc = 'sha1';
				break;
		}
	}

	function hash($key, $n=0) {
		return call_user_func( $this->hashfunc, $n.$key );
	}

	function add($key) {
		for ( $i=0; $i< $this->number_of_hashing_functions; $i++ ) {
			$k = $this->hash($key, $i);
			for ( $n=0; $n< $this->hashlen; $n++ ) {
				$this->map[$i][$n][$k{$n}]++;
			}
		}
		return true;
	}

	function remove($key) {
		for ( $i=0; $i< $this->number_of_hashing_functions; $i++ ) {
			$k = $this->hash($key, $i);
			for ( $n=0; $n< $this->hashlen; $n++ ) {
				$this->map[$i][$n][$k{$n}]--;
			}
		}
		return true;
	}

	function exists($key, $bool=true) {
		$max = 0;
		for ( $i=0; $i< $this->number_of_hashing_functions; $i++ ) {
			$k = $this->hash($key, $i);
			for ( $n=0; $n< $this->hashlen; $n++ ) {
				if ( !$v = $this->map[$i][$n][$k{$n}] )
					return false;
				else
					$max = max($v, $max);
			}
		}
		if ( $bool )
			return true;
		else
			return $max;
	}

	function initialize_bitmat($n) {
		$empty_bitmap_line = array(
			'0' => 0, '1' => 0, '2' => 0, '3' => 0, '4' => 0, '5' => 0, '6' => 0, '7' => 0,
			'8' => 0, '9' => 0, 'a' => 0, 'b' => 0, 'c' => 0, 'd' => 0, 'e' => 0, 'f' => 0  );
		$this->map=array();
		for ( $i=0; $i< $n; $i++ ) {
			$this->map[$i]=array();
			for ( $l=0; $l< $this->hashlen; $l++ ) {
				$this->map[$i][$l] = $empty_bitmap_line;
			}
		}
	}

}

?>

Using wait, $!, and () for threading in bash

This is a simplistic use of the pattern that I wrote about in my last post to wait on multiple commands in bash. In essence I have a script which runs a command (like uptime or restarting a daemon) on a whole bunch of servers (think pssh). Anyways… this is how I modified the script to run the command on multiple hosts in parallel. This is a bit simplistic as it runs, say, 10 parallel ssh commands and then waits for all 10 to complete. I’m very confident that someone could easily adapt this to run at a constant concurrency level of $threads… but I didn’t need it just then so I didn’t go that far… As a side note, this is possibly the first time I’ve ever *needed* an array in a bash script… hah…

# $1 is the commandto run on the remote hosts
# $2 is used for something not important for this script
# $3 is the (optional) number of concurrent connections to use

if [ ! "$3" == "" ]
then
    threads=$3
else
    threads=1
fi

cthreads=0;
stack=()
for s in $servers
  do
    if [ $cthreads -eq $threads ]; then
        for job in ${stack[@]}; do
              wait $job
        done
        stack=()
        cthreads=0
    fi
    (
        for i in $(ssh root@$s "$1" )
            do
                echo -e "$s:\t$i"
        done
    )& stack[$cthreads]=$!
    let cthreads=$cthreads+1
done
for job in ${stack[@]}; do
    wait $job
done

bash – collecting the return value of backgrounded processes

You know that you can run something in the background in a bash script with ( command )&, but a coworker recently wanted to run multiple commands, wait for all of them to complete, collect and decide what to do based on their return values… this proved much trickier. Luckily there is an answer

#!/bin/bash

(sleep 3; exit 1)& p1=$!
(sleep 2; exit 2)& p2=$!
(sleep 1; exit 3)& p3=$!

wait "$p1"; r1=$?
wait "$p2"; r2=$?
wait "$p3"; r3=$?

echo "$p1:$r1 $p2:$r2 $p3:$r3"

a dumbed down version of wpdb for sqlite

I’ve been working, gradually, on a project using an sqlite3 database (for its convenience) and found myself missing the clean elegance of wpdb… so I implemented it. It was actually really easy to do, and I figured I would throw it up here for anyone else wishing to use it. The functionality that I build this around is obtainable here: http://php-sqlite3.sourceforge.net/pmwiki/pmwiki.php (don’t freak… its in apt…)

With this I can focus on the sql, which is different enough, and not fumble over function names and such… $db = new sqlite_wpdb($dbfile, 3); var_dump($db->get_results(“SELECT * FROM `mytable` LIMIT 5”));

the code is below… and hopefully not too mangled…

< ?php

class sqlite_wpdb {

        var $version = null;
        var $db = null;
        var $result = null;
        var $error = null;

        function sqwpdb($file, $version=3) { 
                return $this->__construct($file, $version); 
        }

        function __construct($file, $version=3) {
                $function = "sqlite{$version}_open";
                if ( !function_exists($function) )
                        return false;
                if ( !file_exists($file) )
                        return false;
                if ( !$this->db = @$function($file) )
                        return false;
                $this->version = $version;
                $this->fquery = "sqlite{$this->version}_query";
                $this->ferror = "sqlite{$this->version}_error";
                $this->farray = "sqlite{$this->version}_fetch_array";
                return $this;
        }

        function escape($string) {
                return str_replace("'", "''", $string);
        }

        function query($query) {
                if ( $this->result = call_user_func($this->fquery, $this->db, $query) )
                        return $this->result;
                $this->error = call_user_func($this->ferror, $this->db);
                return false;
        }

        function array_to_object($array) {
                if ( ! is_array($array) )
                        return $array;

                $object = new stdClass();
                foreach ( $array as $idx => $val ) {
                        $object->$idx = $val;
                }
                return $object;
        }

        function get_results($query) {
                if ( !$this->query($query) )
                        return false;
                $rval = array();
                while ( $row = $this->array_to_object(call_user_func($this->farray, $this->result)) ) {
                        $rval[] = $row;
                }
                return $rval;
        }

        function get_row($query) {
                if ( ! $results = $this->get_results($query) )
                        return false;
                return array_shift($results);
        }

        function get_var($query) {
                return $this->get_val($query);
        }

        function get_val($query) {
                if ( !$row = $this->get_row($query) )
                        return false;
                $row = get_object_vars($row);
                if ( !count($row) )
                        return false;
                return array_shift($row);
        }

        function get_col($query) {
                if ( !$results = $this->get_results($query) )
                        return false;
                $column = array();
                foreach ( $results as $row ) {
                        $row = get_object_vars($row);
                        if ( !count($row) )
                                continue;
                        $column[] = array_shift($row);
                }
                return $column;
        }

}

?>