tl;dr – my SSH client was attempting an IPV6 connection; my internet connection does not support IPV6

So. For a couple of weeks now, on and off, I’ve been trying to track down the source of my personal server being slow to log in via SSH.  IT was slow… say… 98% of the time, and 2% of the time it was fast.

I thought maybe it was compromised somehow (and I did find some xmlrpc pingback reflection attacks being done using a site of mine… which I fixed) but it didn’t appear to be so.

Then I thought maybe it was some sort of network exhaustion. But my bandwidth, and my tcp connections as reported by netstat all looked very low.

I had some swap used, so I thought maybe there was something important in swap slowing the system down.

I checked the usual suspects (dns resolvers, etc) and nothing. I checked my 2 factor authentication setup. Again nothing. I checked my sshrc script. Nada.

Turning to google and the various .*exchange(like)? site answers I tried setting usedns to no, and GSSAPIAuthentication to no. I cleared out some hosts.deny|allow entries, I even disabled some dynamic things in PAM for SSH, and finally turned PAM authentication off completely.  disabled and configured the server not to use avahi

Nothing worked. Everything looked good… sshd loglevel was set to verbose and not giving me any signs… using ssh -v to connect didn’t show me anything out of the ordinary

Well that’s not true. It showed the problem but it wasn’t an “error”, so to speak, and so I didn’t immediately notice it.

It turns out that, since my server has an IPV6 address my ssh client was attempting to connect to it via IPV6 first.  This is problematic because my internet connection has no such support.

Setting “AddressFamily inet” for “Host *” in .ssh/config fixed things right up.

I want to say this started happening right around the time I upgraded to OSX Yosemite (GM candidate), but I couldn’t swear to that.