Archive for the ‘Web Stuff’ Category:
Just because you build it, doesnt mean they will come
Here’s a small bit of advice for all you would-be “cloud storage providers.” Just because you have a buttload of disks doesn’t mean people will be falling over themselves to use your software. If I have to spend *any* of my time worrying about your load, storage, or other internal algorithms (or unnecessary limitations for that matter) then YOU . HAVE . FAILED.
If I have to take the time to shard my data into 4096 different containers because you couldn’t be bothered to think “hey what if a service with a lot of users that create a lot of stuff decides to use us as a store?” Then you’re obviously not in it to win it (so to speak.)
Give us ABSTRACTED storage. Non abstracted storage we can do on our own thank you.
Subscribe to the comments for this postJust what you need to know to write a CouchDB reduce function
Lets say you have the CouchDB classes (located here) all compiled together and included into your test.php script. Lets also say that you have created a database with the built-in web ui called “testing”. Finally let us say that your test.php has the following code in it, which would add a record to the db every time it is run. (i know that the data in the document serves no useful purpose… but really I just want to figure out this map/reduce thing so that I can make awesome views… so this suffices sufficiently.)
require_once dirname( dirname( __FILE__ ) ) . '/includes/couchdb.php'; $couchdb = new CouchDB('testing', 'localhost', 5984); $key = microtime(); $result = $couchdb->send( '/'.md5($key), 'put', json_encode( array( "_id" => md5($key), "time" => $key, 'md5' => md5($key), 'sha1' => sha1($key), 'crc' => crc32($key) ) ) ); print_r($result->getBody(true));
After running the code a bunch of times you would end up with a bunch of documents which look more or less like this:
(click for full size)Now lets say you want to write a view that told you what the first characters of the _id were and how many documents share that first letter. This is analogous to the following in MySQL
SELECT LEFT(md5, 1) AS `lchar`, count(md5) FROM `md5table` GROUP BY `lchar`
Your map function is easy, because you dont have any selection criteria, so we process all rows
function(doc){ emit(doc._id,doc); }
The reduce function is where the actual programming comes in… And it seems there aren’t many well explained examples of exactly how to do this (I just brute forced it by trial and error)
function(key, values, rereduce) { var output = {}; if ( rereduce ) { // key is null, and values are values returned by previous calls // // see http://wiki.apache.org/couchdb/Introduction_to_CouchDB_views // // essentially we are taking the previously reduced view, and the // reduced view for new records, and we are reducing those two things // together. Summarizing two summaries, essentially for ( var i in values ) { // here we have multiple prebuilt output objects and we're simply combining them // just like below we have an array with a numeric id and an output object // // retrieve a summary var vals = values[i]; for ( var key in vals ) { // debugging // log(key); // // store in or increment our new output object if ( output[key] == undefined ) output[key] = vals[key]; else output[key] = output[key] + vals[key]; } } } else { // key is an array, which we dont care about, and values are the // values returned by the map // // see http://wiki.apache.org/couchdb/Introduction_to_CouchDB_views // // we are taking each document and processing that, reducing it down // to a summary object (output) for each of the rows passed for ( var i in values ) { // we have an array, values, with numeric ids and a document objects // // retrieve a document var doc = values[i]; // get what we want from it, the first char of the md5 var key = doc._id.substr(0, 1); // debugging // log( key + " :: " + doc._id ); // // store or increment the output object if ( output[key] !== undefined ) output[key] = output[key] + 1; else output[key] = 1; } } // done return output; }
and in code, using a temporary view, ( if you used this view all the time you would want to make it permanent… but this is about how to lay out a reduce function, nothing more ) so request code that looks like this
$view = array( 'map' => 'function(doc){ emit(doc._id,doc); }', 'reduce' => ' function(key, values, rereduce) { var output = {}; if ( rereduce ) { // key is null, and values are values returned by previous calls for ( var i in values ) { var vals = values[i]; for ( var key in vals ) { // log(key); if ( output[key] == undefined ) output[key] = vals[key]; else output[key] = output[key] + vals[key]; } } } else { // key is an array, which we dont care about, and values are the values returneb by the map for ( var i in values ) { var doc = values[i]; var key = doc._id.substr(0, 1); // log( key + " :: " + doc._id ); if ( output[key] !== undefined ) output[key] = output[key] + 1; else output[key] = 1; } } return output; } ' ); $result = $couchdb->send('/_temp_view', 'POST', json_encode($view) ); print_r($result->getBody(true));
would give you output that looks like this:
stdClass Object ( [rows] => Array ( [0] => stdClass Object ( [key] => [value] => stdClass Object ( [0] => 15 [1] => 17 [2] => 16 [3] => 13 [4] => 27 [5] => 18 [6] => 26 [7] => 15 [8] => 18 [9] => 21 [a] => 12 [b] => 23 [c] => 20 [d] => 27 [e] => 28 [f] => 26 ) ) ) )
I hope this helps somebody out.
Using wait, $!, and () for threading in bash
This is a simplistic use of the pattern that I wrote about in my last post to wait on multiple commands in bash. In essence I have a script which runs a command (like uptime or restarting a daemon) on a whole bunch of servers (think pssh). Anyways… this is how I modified the script to run the command on multiple hosts in parallel. This is a bit simplistic as it runs, say, 10 parallel ssh commands and then waits for all 10 to complete. I’m very confident that someone could easily adapt this to run at a constant concurrency level of $threads… but I didn’t need it just then so I didn’t go that far… As a side note, this is possibly the first time I’ve ever *needed* an array in a bash script… hah…
# $1 is the commandto run on the remote hosts # $2 is used for something not important for this script # $3 is the (optional) number of concurrent connections to use if [ ! "$3" == "" ] then threads=$3 else threads=1 fi cthreads=0; stack=() for s in $servers do if [ $cthreads -eq $threads ]; then for job in ${stack[@]}; do wait $job done stack=() cthreads=0 fi ( for i in $(ssh root@$s "$1" ) do echo -e "$s:\t$i" done )& stack[$cthreads]=$! let cthreads=$cthreads+1 done for job in ${stack[@]}; do wait $job done
a dumbed down version of wpdb for sqlite
I’ve been working, gradually, on a project using an sqlite3 database (for its convenience) and found myself missing the clean elegance of wpdb… so I implemented it. It was actually really easy to do, and I figured I would throw it up here for anyone else wishing to use it. The functionality that I build this around is obtainable here: http://php-sqlite3.sourceforge.net/pmwiki/pmwiki.php (don’t freak… its in apt…)
With this I can focus on the sql, which is different enough, and not fumble over function names and such… $db = new sqlite_wpdb($dbfile, 3); var_dump($db->get_results(”SELECT * FROM `mytable` LIMIT 5″));
the code is below… and hopefully not too mangled…
< ?php class sqlite_wpdb { var $version = null; var $db = null; var $result = null; var $error = null; function sqwpdb($file, $version=3) { return $this->__construct($file, $version); } function __construct($file, $version=3) { $function = "sqlite{$version}_open"; if ( !function_exists($function) ) return false; if ( !file_exists($file) ) return false; if ( !$this->db = @$function($file) ) return false; $this->version = $version; $this->fquery = "sqlite{$this->version}_query"; $this->ferror = "sqlite{$this->version}_error"; $this->farray = "sqlite{$this->version}_fetch_array"; return $this; } function escape($string) { return str_replace("'", "''", $string); } function query($query) { if ( $this->result = call_user_func($this->fquery, $this->db, $query) ) return $this->result; $this->error = call_user_func($this->ferror, $this->db); return false; } function array_to_object($array) { if ( ! is_array($array) ) return $array; $object = new stdClass(); foreach ( $array as $idx => $val ) { $object->$idx = $val; } return $object; } function get_results($query) { if ( !$this->query($query) ) return false; $rval = array(); while ( $row = $this->array_to_object(call_user_func($this->farray, $this->result)) ) { $rval[] = $row; } return $rval; } function get_row($query) { if ( ! $results = $this->get_results($query) ) return false; return array_shift($results); } function get_var($query) { return $this->get_val($query); } function get_val($query) { if ( !$row = $this->get_row($query) ) return false; $row = get_object_vars($row); if ( !count($row) ) return false; return array_shift($row); } function get_col($query) { if ( !$results = $this->get_results($query) ) return false; $column = array(); foreach ( $results as $row ) { $row = get_object_vars($row); if ( !count($row) ) continue; $column[] = array_shift($row); } return $column; } } ?>
Postfix, DKIMproxy, Spamc
If you’re running any moderately busy mail server you’re probably using spamassassins spamc/spamd to check for spam because its tons more efficient than piping the mail through the spamassassin cli. Assuming that you do, and that you plan on adding DKIM proxy to the mix to verify and sign emails you need to put things in the right order, to save you some headache here’s what I did:
- smtp|smtps => -o smtpd_proxy_filter=127.0.0.1:10035 # outgoing dkim verify port
- 127.0.0.1:10036 => -o content_filter=spamassassin
- spamassassin => pipe user=nobody argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient} # this delivers to the “pickup” service
- pickup => -o content_filter=dksign:127.0.0.1:10037 # outgoing dkim signing port
- 127.0.0.1:10038 => -o content_filter= # the buck stops here
If you arent careful with these (which I wasnt) you’ll end up causing an infinite loop between your filters (which I did). Thus concludes our public service announcement.
Writing your own shell in php
I’ve always wanted to write my own simple shell in php. Call me a glutin for punishment, but it seems like something that a lot of people could use to be able to do… If your web app had a command line interface for various things… like looking up stats, or users, or suspending naughty accounts, or whatever…. wouldnt that be cool and useful? Talk about geek porn. Anyways this this morning I got around to tinkering with the idea, and here is what i came up with… It’s rough, and empty, but its REALLY easy to extend and plug into any php application.
apokalyptik:~/phpshell$ ./shell.php /home/apokalyptik/phpshell > hello hi there /home/apokalyptik/phpshell > hello world hi there world /home/apokalyptik/phpshell > cd .. /home/apokalyptik/ > cd phpshell /home/apokalyptik/phpshell > ls shell.php /home/apokalyptik//phpshell > exit apokalyptik:~/phpshell$ ./shell.php
See the source here: shell.phps
Internally Caching Longer Than Externally Caching
We use varnish for a lot of our file caching needs, and recently we figured out how to do something rather important through a combination of technologies. Imagine you have backend servers generating dynamic content based on user input. So your users do something that fits the following categories:
- is expensive to generate dynamically, and should be served from cache
- many requests come in for the same objects, bandwidth should be conserved
- doesnt change very often
- once changed needs to take effect quickly
Now wish varnish we’ve been using the Expires header for a long time with great success, but for this we were having no luck. If we set the expires header to 3 weeks, then clients also cache the content for 3 weeks (violating requirement #3.) We can kill the Expires header in varnish at vcl_deliver, but then clients don’t cache at all (#2.) We can add Content-Control, overwrite the Age (otherwise reported Age: will be greater than max-age), and kill the Expires headers in the same place, but this isn’t pretty, and seems like a cheap hack. Ideally we could rewrite the Expires header in varnish, but that doesn’t seem doable.
So what we ended up doing, was header rewriting at the load balancer (nginx.) inside our location tag we added the following:
proxy_hide_header Age; proxy_hide_header Expires; proxy_hide_header Cache-Control; add_header Source-Age $upstream_http_Age; expires 300s;
Now nginx setsa proper Cache-Control: and Expires: headers for us, disregarding what varnish serves out. Web clients dont check back for 5 minutes (reusing the old object) and varnish can cache until judgment dat because we get wild card invalidation
Isn’t technology fun?!
This deserves some link love
Andy bogged a piece of advice that I have him which I got from Barry… and if you want to know how to get the true absolute path to the real location of the current script is from inside of it (like phps realpath and __FILE__) I suggest you check it out
I keep marking this as unread in google reader…
I keep marking this as unread in google reader so that Its there when I need it… which probably means I should just blog it… automating firefox via telnet
Logging post data
Lets say you have a relatively complex php web application, like wordpress. You have it running under apache (which is common.) You have good control of your site via .htaccess (which is common — permalinks and all.) And something happens to your blog (e.g. someone is exploiting some unknown vulnerability to compromise your content), which you want to track down. So you want to log, for instance, HTTP POST data. Your first instinct might be to add some logging code to index.php (mine was) But there are a lot of possible places which might be directly accessed, especially in the admin. So The trick I use (and this is probably the only time I’ve ever condoned this) is to use PHPs auto_prepend_file functionality.
Make a /home/user/postlog/ directory, then a /home/user/postlog/logs/ directory (and chmod a+rw that one.) Next make a simple /home/user/postlog/postlog.php file with the following contents:
<?php if ( isset($_POST) && is_array($_POST) && count($_POST) > 0 ) { $log_dir = dirname( __FILE__ ) . '/logs/'; $log_name = "posts-" . $_SERVER['REMOTE_ADDR'] . "-" . date("Y-m-d-H") . ".log"; $log_entry = gmdate('r') . "\t" . $_SERVER['REQUEST_URI'] . "\r\n" . serialize($_POST) . "\r\n\r\n"; $fp=fopen( $log_dir . $log_name, 'a' ); fputs($fp, $log_entry); fclose($fp); } ?>
Finally add this line to the top of your .htaccess file:
php_value auto_prepend_file /home/user/postlog/postlog.php
If all went well this should start logging any request to any php file with any post data into the /home/user/postlog/logs/ direcory (with a unique log per ip per day)