Archive for the ‘web stuff’ Category:
This deserves some link love
Subscribe to the comments for this postI keep marking this as unread in google reader…
Logging post data
Make a /home/user/postlog/ directory, then a /home/user/postlog/logs/ directory (and chmod a+rw that one.) Next make a simple /home/user/postlog/postlog.php file with the following contents:
<?php
if ( isset($_POST) && is_array($_POST) && count($_POST) > 0 ) {
$log_dir = dirname( __FILE__ ) . '/logs/';
$log_name = "posts-" . $_SERVER['REMOTE_ADDR'] . “-” . date(”Y-m-d-H”) . “.log”;
$log_entry = gmdate(’r') . “\t” . $_SERVER['REQUEST_URI'] . “\r\n” . serialize($_POST) . “\r\n\r\n”;
$fp=fopen( $log_dir . $log_name, ‘a’ );
fputs($fp, $log_entry);
fclose($fp); }
?>
Finally add this line to the top of your .htaccess file:
php_value auto_prepend_file /home/user/postlog/postlog.php
If all went well this should start logging any request to any php file with any post data into the /home/user/postlog/logs/ direcory (with a unique log per ip per day)
It’s good for the server. It’s good for the soul.
so-you-wanna-see-an-image
A request comes into pound for a file. Pound hashes the hostname (via a custom patch which we have not, but may, release) , to determine which of several backend servers the request should hit. Pound forwards the request to that server. This, of course, means that a given blog always serves from the same backend server. The only exception to the afore-mentioned rule is if that server is, for some reason, unavailable in which case it picks another server to serve that hostname from temporarily.
The request then comes into varnishd on the backend servers. The varnishd daemon checks its 300Gb worth of files cache and (for the sake of this example) finds nothing (hey, new images are uploaded all the time!) Varnishd then checks with the web server (running on the same machine, just bound to a different IP/Port#) and that request is handled by a custom script.
So, a http daemon on the same backend server runs the file request. The custom script checks the DB to gather information on the file (specifically which DC’s it is in, size, mod time, and whether its deleted or not) all this info is saved in memcached for 5 minutes. The script increments and checks the “hawtness” (term courtesy of Barry) of the file in memcached (if the file has been accessed over a certain # of times it is then deemed “hawt”, and a special header is sent with the response telling varnishd to put the file into its cache. When that happens the request would be served directly by varnishd in the previous paragraph and never hit the httpd or this script again (or at least not until the cache entry expires.)) At this point, assuming the file should exist (deleted = 0 in the files db) we fetch the file from a backend source.
Which backend source depends on where it is available. The order of preference is as follows: Always fetch from Amazon S3 if the file lives there (no matter what, the following preferences only ever occur if, for some reason, s3 = 0 in the files db), and if that fails fetch from the one files server we still have (which has larger slower disks, and is used for archiving purposes and fault tolerance only)
After fetching the file from the back end… the custom script hands the data and programatically generated headers to the http daemon, which hands the data to varnishd, varnishd hands the data to pound, pound hands the data to the requesting client, and the image appears in the web browser.
And there was much rejoicing (yay.)
For the visual people among us who like visuals and stuff… (I like visuals…) here goes…
Useful bash oneliner: List Server IP Addresses
Autumn Leaves Leaf #3: Commander
require 'net/http'
require 'net/https'
class Commander < AutumnLeaf
before_filter :authenticate, :only => [ :reload, :sync, :quit, :deploy ]
$authenticated = []
def authenticate_filter(sender, channel, command, msg, options)
return true if $authenticated.include?(sender)
return false
end
def did_receive_private_message(sender, msg)
# assumes there is a file at
# http://my.svnserver.com/svn/access
# whose contents are “granted”
Net::HTTP.start(’my.svnserver.com’) {|http|
req = Net::HTTP::Get.new(’/svn/access’);
req.basic_auth(sender, msg)
response = http.request(req)
$authenticated < < sender if response.body == "granted"
}
end
def someone_did_quit(sender, msg)
$authenticated.delete(sender)
end
def someone_did_leave_channel(sender, channel)
$authenticated.delete(sender)
end
def someone_did_join_channel(sender, channel)
$authenticated.delete(sender)
end
def deploy_command(sender, channel, text)
message "deploying..."
system("sudo /usr/local/bin/deploy.sh 1>/dev/null 2>/dev/null”)
end
end
Autumn Leaves Leaf #1: Announcer
require 'socket'
require 'thread'
class Announcer < AutumnLeaf
def handle_incoming(sock)
Thread.new {
line = sock.gets
message line
sock.close
}
end
def did_start_up
Thread.new {
listener = TCPServer.new('',22122)
while (new_socket = listener.accept)
handle_incoming(new_socket)
end
}
end
end
Autumn Leaves (a ruby framework for IRC bots)
Link: http://www.shutupgeorge.com/al-docs/
Throughput sensitive fputs/fwrite (in php)