elockd [more] publicly available

I’d call this an 0.5 release. It’s now in the code repo that we at Automattic use to put out little open source tools.

I’ve fixed several bugs with the code since the first time that I posted about it. It can handle at least 2k active connections, and at least 100k shared and exclusive locks (split 50/50) and can handle every single connection orphaning its locks at the same time for both shared and exclusive locks.

It’s a pretty good bit of code… not bad for under a week in my spare time.

I suppose that I should explain what it is and does. And why I care.

The idea behind lockd was to build a network oriented daemon which would handle arbitrary locking of… things. The idea was to mix the ease of Memcached with the atomic chewy goodness of MySQls named locks. It’s really handy to have something like this that can be depended upon when you have a very large distributed environment to keep things from stepping on its own toes.

Only one client can have an exclusive lock, and only the lock owner can release it. any client can inspect that exclusive lock to see if it’s held or not. If the owner disconnects from the server then all of the owned locks are released.

Any number of clients can acquire a shared lock, and the result of the locking action includes the number of other owners sharing that lock. The response for the first lock request would be 1 lock acquired, while the second lock request for the lock would be 2 lock acquired (i.e. two people have this lock.) Likewise releasing the lock decrements it, and inspecting the lock shows the number of open handles to that lock. All of an owners shared locks are also released on disconnect.

Oh, did I mention that it also keeps stats for you to use in your munin graphs? Yea. That too.

So… some obvious questions I’m sure you’re wondering:

1: why not just use Memcached? Well Memcached has no knowledge of the state of your connection. I want a lock that goes away if the parent process disconnects for any reason. You could do this with timed keys in Memcached but you run two risks: the first being that you might not get around to updating the lock and resetting its ttl before it expires leaving another process able to lock erroneously, and the second being that given enough data flowing through the cache your lock might simply be pushed off the back end of the cache by newer data — something that I don’t want. Also shared locks would be difficult here.

2. Why not just use MySQL named locks? You can only hold one named lock per connection, and there is no concept of shared named locks.

3. Why not use filesystem locks? Those tend to be difficult for people to code properly, depend on various standards implementations, cant do counted shared locks, and most importantly aren’t network enabled.

4. Whats the big deal about shared locks? They’re super powerful — great for rate limiting, etc.

5. Wasn’t there already something out there for this? I’m not going to say “no,” but I will say “not that I saw when I looked”.

6. Why did you rewrite it in erlang, was the PHP version bad? Yes, sort of, the PHP version played some interesting tricks to achieve a thread-like operational state, but I believe that there are timing issues because of those tricks that cause it to crash in as-of-yet unknown circumstances at high load. The PHP version is also slow when there are a very high number of clients or locks. Erlang was, essentially, born for this particular purpose since it sports great concurrency models immutable variables, and the way that the gen_* things work out I get atomicity built in even with all these concurrent clients grabbing at stuff.

7. Whats the API look like? It looks nice and clean…

“g $key
” — get exclusive $key
“r $key
” — release exclusive $key
“i $key
” — inspect exclusive $key
“sg $key
” — get shared $key
“sr $key
” — release shared $key
“si $key
” — inspect shared $key
“q
” — show stats

Using PHP and OpenSSH with username/password auth

It turns out that this is actually a tricky problem. It’s super easy to use the OpenSSH command line stuff via PHP when you have key based authentication set up, but it’s not at all easy to use when you want to go the user/pass route. This is for a couple of reasons:

First you cannot specify the password on the command line. Second you cannot use the php process controls directly to give the password (well this isn’t 100% true, if you want to recompile your PHP binary with pty support then you probably could bypass everything I’m about to say and just use proc_open straight). And there’s a third reason that I’ll get to in a bit.

OpenSSH supports getting a password from an executable program via the SSH_ASKPASS environment variable — with two notable gotchas. First this only works if you also specify a DISPLAY environment variable, and second it does NOT work if the controlling process has a tty or pty.

The code below works… but if you just paste it into a script file and run it directly with php ./myscript.php it fails. Why?

function ssh_user_pass_port_forward( $hostname, $username, $password, $localport, $remotehost, $remoteport ) {
	$descriptorspec = array(
	        0 => array("pipe", "r"),  // stdin is a pipe that the child will read from
	        1 => array("pipe", "w"),  // stdout is a pipe that the child will write to
	        2 => array("pipe", "w")   // stderr is a file to write to
	);
	$script = tempnam( '/tmp/', 'askpass-');
	file_put_contents( 
		$script, 
		"#!/bin/bash
echo -n ".escapeshellarg( $password ) 
	);
	chmod( $script, 0755 );
	$env = array( 
		'DISPLAY' => '0', 
		'SSH_ASKPASS' => $script 
	);
	$forward = sprintf(
		"%d:%s:%d",
		$localport,
		$remotehost,
		$remoteport
	);
	$command = sprintf(
		"/usr/bin/ssh -n -o StrictHostKeyChecking=no -l %s -L %s -N %s",
		escapeshellarg( $username ),
		escapeshellarg( $forward ),
		escapeshellarg( $hostname )
	);
	return proc_open( $command, $descriptorspec, $pipes, getcwd(), $env);
}

The answer is that you are running it, and you’re running it from a shell which has a tty or pty attached, and the script inherits those and OpenSSH sees this and then ignores the SSH_ASKPASS variable completely. The trick to testing/using this code is to execute it without running it from a terminal. If you execute the command below (roughly) it looks like it hangs. but if you open another terminal you should be able to use the port forward just created.

ssh myserver.com “php /path/to/myscript.php”

This is because since you’re sshing for the sole purpose of running that command and not using a shell the system doesn’t allocate you a pty like it would normally. I’m guessing that executing it via an http request would do something similar.

Since I previously posted some hard-won advice for working with ssh2_* in php I thought I would share this equally tricky bit that I’d also figured out in the process.

Remember that there is ALWAYS more than one way to skin a problem, and every problem can be skinned with enough effort.

Again, this is itch-scratch-ware, YMMV, this is meant as a starting point on a journey to a solution and not a drop-in-works-everywhere bit of code. It’s just the hard stuff. The useful stuff is still up to you ;)

PHP SSH2 code

I’ve had a need to use the PHP SSH2 PECL recently (working on making a product, at work, more efficient) And thought I would share some of the preliminary code. You can find it here: vpssh.phps

The most interesting thing is not vpssh_core or it’s exec (though it’s good code) the really interesting thing is the vpssh_tunnel class and the accompanying examples at the top of the file. This really shows some advanced usage of ssh2_tunnel that you can’t really find anywhere else.

It’s just the beginnings of some useful code, but it’s probably a huge jumping off point for anyone seriously looking into the ssh2 pecl functionality. Oh, it also works with both password and key based authentication.

This code is less than 12 hours old, and it works for me so far, YMMV. Feedback welcome … or not… Whatever. Hope it helps someone out. And I hope it helps me out later when I need this kind of thing again.

Temporary files

Here’s a little tip. If you’re writing PHP code for a linux only environment and you need a temp file to write something to.

You can unlink the file after you use fopen() and use it to your hearts content. Linux will keep the “file” in existence for you and you can use ftruncate(), rewind(), fflush() and the like to churn it to your hearts content. When you close the filepointer linux will reclaim the space you used. With this method of operation you can be sure that your code will not accidentally leave these files laying around if, for example, the process is terminated.

Just recently I used this trick on two tempfiles in parallel, moving data back and forth between them filtering things in and out on each pass. Handy trick…