Software Development

Debian, ProFTPD, FTPS, TLS, SSL, and SSL23_GET_SERVER_HELLO:unknown protocol

Recently I needed to test against an FTPS server. No big deal, I thought to myself, I’ll just set one up real quick. Boy did I end up having a hard time with that. Not because the task was actually hard but because there’s a bit of a general haziness about the whole idea of what FTPS is. More on that later.

The first thing I did was setup my Debian ProFTPD server via the included /etc/proftpd/tls.conf. Restarted ProFTPD, and then tried curl -v -v -k ‘ftps://localhost’ which immediately resulted in the following error

* About to connect() to localhost port 990 (#0)
*   Trying 127.0.0.1... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

Oh, right, It’s listening on port 21 not port 990… curl -v -v -k ftps://localhost:21/ which gave me this error

* About to connect() to localhost port 21 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 21 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection #0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Believe it, or not, I got stuck here for more than an entire day. Which is kind of embarrassing. I googled the hell out of this issue, and got lots of advice which centered about generating appropriate certs, and using “openssl s_client -connect 127.0.0.1:21” to test (which resulted in, essentially, the same error: “14996:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:607:” )

With the help of a friend from work we found what I had been overlooking. You see FTPS can mean one of two very different things.

FTPS can mean FTP with explicit SSL. This is where you connect to FTP, then give a command to encrypt the session after the initial plaintext connection has been established.

FTPS can also mean FTP with implicit SSL. This is where you connect to the ftp server and the connection is encrypted before any commands are sent (this is like having HTTP on port 80 and HTTPS on port 443, except using 21 and 990 for FTP.)

The two types of FTPS are not compatible with one another. Apparently FTPS/Implicit is no longer a part of the standard, but still “around” and “supported” by “things”. And curl thinks you mean this when you give it a url of ftps://something. FTPS/Implicit is also the kind of stream that “openssl s_client -connect 127.0.0.1:21” would test. FTPS/Implicit is not the configuration setup by /etc/proftpd/tls.conf. Which is why my testing failed, frustratingly, for so long.

Since ProFTPD uses FTPS/Explicit by default… how do you test? With very similar commands to the ones I used previously (lending to the confusion…)

openssl s_client -connect 127.0.0.1:21 -starttls ftp

curl -v -v -k --ftp-ssl ftp://localhost:21/

Ok. Now I’m able to setup and test an FTP/E server. What if I also need to setup and test an FTP/I server too? Thats pretty simple. in ProFTPD 1.3.3rc2, the mod_tls module was enhanced to support implicit FTPS via the UseImplicitSSL TLSOption. So by adding “TLSOption UseImplicitSSL” on an appropriately new version of ProFTPD and mod_tls you can have a server that works with “curl -v -v -k ftps://localhost:21/” and “openssl s_client -connect 127.0.0.1:21”

I hope that this saves someone else the headaches that going through all of this gave me. Had I read through the ProFTPD TLS howto carefully, instead of just searching for what I thought I needed, I would have solved this all much more quickly.

Longest Common Prefix Between Two Strings

While working on a “for fun” side project I needed to get the longest common prefix of two arbitrary strings. Since I didn’t find what I was looking for in the PHP string functions I made my own. And for some perverse reason i decided to see what the fewest number of lines I could do it in was without displaying warnings…

I got down to a one line function, and managed to avoid using an @ to silence anything…


function str_common_prefix( $s1, $s2, $i=0 ) {
return ( !empty($s1{$i}) && !empty($s2{$i}) && $s1{$i} == $s2{$i} ) ? str_common_prefix( $s1, $s2, ++$i ) : $i;
}

I’d be interested to see what others come up with.

Adding a second authentication factor to WordPress

UPDATE: I’ve added the plugin to the WordPress.org repository. If it gathers interest/attention then I may develop it further and add more stuff like SMS gateway support, configuration, etc… See: http://wordpress.org/extend/plugins/second-factor/

I really don’t know why, but the idea of adding a second authentication factor to WordPress blogs took hold of my brain tonight and needed an outlet. So I made this little proof of concept plugin: Second Factor. What it does is pretty simple:

  1. When you log in it goes through a series of cryptographic routines and generates some info which is stored in the database as a user option.
  2. A key is generated for you, and an email is sent to your listed email address.
  3. When you attempt to access a page while logged in it blocks you, asking for the key that was emailed to you
  4. Finally after entering this second authentication token you are allowed access to the site

I could see this being extended to Instant Messaging, SMS, IRC, or even integrated with a text-to-phone service to make an actual phone call which reads off the numbers to you.

What I don’t know is if anyone actually wants this… If this is even worthwhile. For me it was mainly a thought experiment. Would you want to have this kind of added security on your WP Installation?

Todays lesson: Check your assumptions

It’s easy to make assumptions about what we think is the problem with the speed of our code. Lets say you have a loop like so:

foreach( $big_list_of_things as $thing ) {
	$res1 = somefunc1( $thing );
	$res2 = somefunc2( $thing );
	$res3 = somefunc3( $thing );
	$res4 = somefunc4( $thing );
}

Since its your code you’re pretty sure most all of the time me being spent inside somefunc2 and everything seems pretty fast except somefunc2 which is noticeably slow. Maybe you did something like this to get a kind of anecdotal feel for how long things are taking (I do this kind of thing a lot…)

foreach( $big_list_of_things as $thing ) {
	$res1 = somefunc1( $thing ); echo "1";
	$res2 = somefunc2( $thing ); echo "2";
	$res3 = somefunc3( $thing ); echo "3";
	$res4 = somefunc4( $thing ); echo "4 ";
}

So you even see some slowness on func2 — Solved! well… maybe… Just because func2 is obviously a problem doesn’t mean that there aren’t other problems going on in your code that you weren’t expecting. And it’s not a huge amount of work just to make sure, so we might as well, here I’ll show you how.

$big_list_of_things = range( 1, 100 );

function somefunc1() { usleep( mt_rand(0, 100) ); }
function somefunc2() { usleep( mt_rand(0, 30000) ); }
function somefunc3() { usleep( mt_rand(0, 14000) ); }
function somefunc4() { usleep( mt_rand(0, 150) ); }

$t1 = $t2 = $t3 = $t4 = 0;
foreach( $big_list_of_things as $thing ) {
        $s = microtime(1);  
        $res1 = somefunc1( $thing );
        $t1 += ( microtime(1) - $s ); 
        $s = microtime(1);
        $res2 = somefunc2( $thing );
        $t2 += ( microtime(1) - $s ); 
        $s = microtime(1);
        $res3 = somefunc3( $thing );
        $t3 += ( microtime(1) - $s ); 
        $s = microtime(1);
        $res4 = somefunc4( $thing );
        $t4 += ( microtime(1) - $s );
}
echo "1: $t1, 2: $t2, 3: $t3, 4: $t4
";

Gives me the following output… and oh look… 2 is a problem but maybe 3 needs to be looked into as well. Would I have known about how much time 3 was taking? probably not since it “seemed” so fast…

php ./test.php
1: 0.0077567100524902, 2: 1.4149680137634, 3: 0.70513272285461, 4: 0.013875484466553
php ./test.php
1: 0.0079605579376221, 2: 1.5631670951843, 3: 0.62554883956909, 4: 0.013876676559448
php ./test.php
1: 0.0080087184906006, 2: 1.4883117675781, 3: 0.66886830329895, 4: 0.013206481933594
php ./test.php
1: 0.0074846744537354, 2: 1.5641448497772, 3: 0.64763903617859, 4: 0.012674331665039

Just last night I found and fixed a bug that must have been there for ages that I had no idea about by exactly this method. And in case you were wondering whether or not it’s worth it… Trust me… Sometimes it is…

elockd [more] publicly available

I’d call this an 0.5 release. It’s now in the code repo that we at Automattic use to put out little open source tools.

I’ve fixed several bugs with the code since the first time that I posted about it. It can handle at least 2k active connections, and at least 100k shared and exclusive locks (split 50/50) and can handle every single connection orphaning its locks at the same time for both shared and exclusive locks.

It’s a pretty good bit of code… not bad for under a week in my spare time.

I suppose that I should explain what it is and does. And why I care.

The idea behind lockd was to build a network oriented daemon which would handle arbitrary locking of… things. The idea was to mix the ease of Memcached with the atomic chewy goodness of MySQls named locks. It’s really handy to have something like this that can be depended upon when you have a very large distributed environment to keep things from stepping on its own toes.

Only one client can have an exclusive lock, and only the lock owner can release it. any client can inspect that exclusive lock to see if it’s held or not. If the owner disconnects from the server then all of the owned locks are released.

Any number of clients can acquire a shared lock, and the result of the locking action includes the number of other owners sharing that lock. The response for the first lock request would be 1 lock acquired, while the second lock request for the lock would be 2 lock acquired (i.e. two people have this lock.) Likewise releasing the lock decrements it, and inspecting the lock shows the number of open handles to that lock. All of an owners shared locks are also released on disconnect.

Oh, did I mention that it also keeps stats for you to use in your munin graphs? Yea. That too.

So… some obvious questions I’m sure you’re wondering:

1: why not just use Memcached? Well Memcached has no knowledge of the state of your connection. I want a lock that goes away if the parent process disconnects for any reason. You could do this with timed keys in Memcached but you run two risks: the first being that you might not get around to updating the lock and resetting its ttl before it expires leaving another process able to lock erroneously, and the second being that given enough data flowing through the cache your lock might simply be pushed off the back end of the cache by newer data — something that I don’t want. Also shared locks would be difficult here.

2. Why not just use MySQL named locks? You can only hold one named lock per connection, and there is no concept of shared named locks.

3. Why not use filesystem locks? Those tend to be difficult for people to code properly, depend on various standards implementations, cant do counted shared locks, and most importantly aren’t network enabled.

4. Whats the big deal about shared locks? They’re super powerful — great for rate limiting, etc.

5. Wasn’t there already something out there for this? I’m not going to say “no,” but I will say “not that I saw when I looked”.

6. Why did you rewrite it in erlang, was the PHP version bad? Yes, sort of, the PHP version played some interesting tricks to achieve a thread-like operational state, but I believe that there are timing issues because of those tricks that cause it to crash in as-of-yet unknown circumstances at high load. The PHP version is also slow when there are a very high number of clients or locks. Erlang was, essentially, born for this particular purpose since it sports great concurrency models immutable variables, and the way that the gen_* things work out I get atomicity built in even with all these concurrent clients grabbing at stuff.

7. Whats the API look like? It looks nice and clean…

“g $key
” — get exclusive $key
“r $key
” — release exclusive $key
“i $key
” — inspect exclusive $key
“sg $key
” — get shared $key
“sr $key
” — release shared $key
“si $key
” — inspect shared $key
“q
” — show stats

elockd

I’ve been working on an erlang port of my php locking daemon in erlang (which is a more appropriate language for this kind of thing.) And I have it all tricked out (ok partially tricked out but hey it’s my first real erlang project and i’ve only spent 2 afternoons on it.)

The api is completely the same between the two (read: simple), and it works great (in my tests.) It supports both exclusive and shared locks, orphaning on disconnect works great for both, stats are working, it’s all running under a supervisor to restart anything that stops, I *think* i’ve done the code well enough that hot code swapping should work as expected. I know there’s a lot of “how an erlang application is packaged” stuff that I don’t know yet.

If i had to describe in a one-liner what this does i would say that lockd is MySQL named locks meets Memcached.

I’m kind of annoyed, however, that “erl -s esuper” doesn’t run the stupid thing, I have to actually run esuper:start(). to get it going. I’ll have to figure that out. You would think that running some precompiled beams/modules/functions/args would be super easy from the outside a-la-init-script, and it probably is but I’m missing something.

Comments on the code are welcome. It’s a pretty cool little thing — my [lack of] mad erlang skills aside.

When I’m ready I’ll be testing it in production, and putting it in a public repo.

Using PHP and OpenSSH with username/password auth

It turns out that this is actually a tricky problem. It’s super easy to use the OpenSSH command line stuff via PHP when you have key based authentication set up, but it’s not at all easy to use when you want to go the user/pass route. This is for a couple of reasons:

First you cannot specify the password on the command line. Second you cannot use the php process controls directly to give the password (well this isn’t 100% true, if you want to recompile your PHP binary with pty support then you probably could bypass everything I’m about to say and just use proc_open straight). And there’s a third reason that I’ll get to in a bit.

OpenSSH supports getting a password from an executable program via the SSH_ASKPASS environment variable — with two notable gotchas. First this only works if you also specify a DISPLAY environment variable, and second it does NOT work if the controlling process has a tty or pty.

The code below works… but if you just paste it into a script file and run it directly with php ./myscript.php it fails. Why?

function ssh_user_pass_port_forward( $hostname, $username, $password, $localport, $remotehost, $remoteport ) {
	$descriptorspec = array(
	        0 => array("pipe", "r"),  // stdin is a pipe that the child will read from
	        1 => array("pipe", "w"),  // stdout is a pipe that the child will write to
	        2 => array("pipe", "w")   // stderr is a file to write to
	);
	$script = tempnam( '/tmp/', 'askpass-');
	file_put_contents( 
		$script, 
		"#!/bin/bash
echo -n ".escapeshellarg( $password ) 
	);
	chmod( $script, 0755 );
	$env = array( 
		'DISPLAY' => '0', 
		'SSH_ASKPASS' => $script 
	);
	$forward = sprintf(
		"%d:%s:%d",
		$localport,
		$remotehost,
		$remoteport
	);
	$command = sprintf(
		"/usr/bin/ssh -n -o StrictHostKeyChecking=no -l %s -L %s -N %s",
		escapeshellarg( $username ),
		escapeshellarg( $forward ),
		escapeshellarg( $hostname )
	);
	return proc_open( $command, $descriptorspec, $pipes, getcwd(), $env);
}

The answer is that you are running it, and you’re running it from a shell which has a tty or pty attached, and the script inherits those and OpenSSH sees this and then ignores the SSH_ASKPASS variable completely. The trick to testing/using this code is to execute it without running it from a terminal. If you execute the command below (roughly) it looks like it hangs. but if you open another terminal you should be able to use the port forward just created.

ssh myserver.com “php /path/to/myscript.php”

This is because since you’re sshing for the sole purpose of running that command and not using a shell the system doesn’t allocate you a pty like it would normally. I’m guessing that executing it via an http request would do something similar.

Since I previously posted some hard-won advice for working with ssh2_* in php I thought I would share this equally tricky bit that I’d also figured out in the process.

Remember that there is ALWAYS more than one way to skin a problem, and every problem can be skinned with enough effort.

Again, this is itch-scratch-ware, YMMV, this is meant as a starting point on a journey to a solution and not a drop-in-works-everywhere bit of code. It’s just the hard stuff. The useful stuff is still up to you 😉