Servers

Slow SSH login

tl;dr – my SSH client was attempting an IPV6 connection; my internet connection does not support IPV6

So. For a couple of weeks now, on and off, I’ve been trying to track down the source of my personal server being slow to log in via SSH.  IT was slow… say… 98% of the time, and 2% of the time it was fast.

I thought maybe it was compromised somehow (and I did find some xmlrpc pingback reflection attacks being done using a site of mine… which I fixed) but it didn’t appear to be so.

Then I thought maybe it was some sort of network exhaustion. But my bandwidth, and my tcp connections as reported by netstat all looked very low.

I had some swap used, so I thought maybe there was something important in swap slowing the system down.

I checked the usual suspects (dns resolvers, etc) and nothing. I checked my 2 factor authentication setup. Again nothing. I checked my sshrc script. Nada.

Turning to google and the various .*exchange(like)? site answers I tried setting usedns to no, and GSSAPIAuthentication to no. I cleared out some hosts.deny|allow entries, I even disabled some dynamic things in PAM for SSH, and finally turned PAM authentication off completely.  disabled and configured the server not to use avahi

Nothing worked. Everything looked good… sshd loglevel was set to verbose and not giving me any signs… using ssh -v to connect didn’t show me anything out of the ordinary

Well that’s not true. It showed the problem but it wasn’t an “error”, so to speak, and so I didn’t immediately notice it.

It turns out that, since my server has an IPV6 address my ssh client was attempting to connect to it via IPV6 first.  This is problematic because my internet connection has no such support.

Setting “AddressFamily inet” for “Host *” in .ssh/config fixed things right up.

I want to say this started happening right around the time I upgraded to OSX Yosemite (GM candidate), but I couldn’t swear to that.

Debian, ProFTPD, FTPS, TLS, SSL, and SSL23_GET_SERVER_HELLO:unknown protocol

Recently I needed to test against an FTPS server. No big deal, I thought to myself, I’ll just set one up real quick. Boy did I end up having a hard time with that. Not because the task was actually hard but because there’s a bit of a general haziness about the whole idea of what FTPS is. More on that later.

The first thing I did was setup my Debian ProFTPD server via the included /etc/proftpd/tls.conf. Restarted ProFTPD, and then tried curl -v -v -k ‘ftps://localhost’ which immediately resulted in the following error

* About to connect() to localhost port 990 (#0)
*   Trying 127.0.0.1... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

Oh, right, It’s listening on port 21 not port 990… curl -v -v -k ftps://localhost:21/ which gave me this error

* About to connect() to localhost port 21 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 21 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection #0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Believe it, or not, I got stuck here for more than an entire day. Which is kind of embarrassing. I googled the hell out of this issue, and got lots of advice which centered about generating appropriate certs, and using “openssl s_client -connect 127.0.0.1:21” to test (which resulted in, essentially, the same error: “14996:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:607:” )

With the help of a friend from work we found what I had been overlooking. You see FTPS can mean one of two very different things.

FTPS can mean FTP with explicit SSL. This is where you connect to FTP, then give a command to encrypt the session after the initial plaintext connection has been established.

FTPS can also mean FTP with implicit SSL. This is where you connect to the ftp server and the connection is encrypted before any commands are sent (this is like having HTTP on port 80 and HTTPS on port 443, except using 21 and 990 for FTP.)

The two types of FTPS are not compatible with one another. Apparently FTPS/Implicit is no longer a part of the standard, but still “around” and “supported” by “things”. And curl thinks you mean this when you give it a url of ftps://something. FTPS/Implicit is also the kind of stream that “openssl s_client -connect 127.0.0.1:21” would test. FTPS/Implicit is not the configuration setup by /etc/proftpd/tls.conf. Which is why my testing failed, frustratingly, for so long.

Since ProFTPD uses FTPS/Explicit by default… how do you test? With very similar commands to the ones I used previously (lending to the confusion…)

openssl s_client -connect 127.0.0.1:21 -starttls ftp

curl -v -v -k --ftp-ssl ftp://localhost:21/

Ok. Now I’m able to setup and test an FTP/E server. What if I also need to setup and test an FTP/I server too? Thats pretty simple. in ProFTPD 1.3.3rc2, the mod_tls module was enhanced to support implicit FTPS via the UseImplicitSSL TLSOption. So by adding “TLSOption UseImplicitSSL” on an appropriately new version of ProFTPD and mod_tls you can have a server that works with “curl -v -v -k ftps://localhost:21/” and “openssl s_client -connect 127.0.0.1:21”

I hope that this saves someone else the headaches that going through all of this gave me. Had I read through the ProFTPD TLS howto carefully, instead of just searching for what I thought I needed, I would have solved this all much more quickly.