Comments on: HA EC2 Part #2: Load Balancing the Load Balancer http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/ The random things that spew forth from my brain... Mon, 16 Jan 2012 18:43:29 +0000 hourly 1 http://wordpress.org/?v=3.4-alpha-19620 By: David Masover http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-2/#comment-7636 David Masover Tue, 29 Jan 2008 11:50:07 +0000 http://blog.apokalyptik.com/?p=154#comment-7636 Here's an idea I'm toying with: ANY dns service (dyndns.com, zoneedit.com, xname.org, etc) which can run in slave mode. Then, use some EC2 instance -- probably your load balancers -- as a powerdns master. I chose powerdns because, after looking at just about every DNS server and library I could find, it's the closest to a pluggable/scriptable server. Mainly, it has a 'pipe' API, in which it communicates via stdin/stdout with a simple, tab-delimited format. It's not entirely documented, but I was able to get the SOA field working by using spaces to delimit the actual SOA data. This will not perform incredibly well, but it doesn't have to. All it really has to do is implement AXFR, and maybe do some local resolution for your other EC2 instances. Of course, you have to tell them (somehow) if the master changes, and it may take a bit longer for those changes to propagate. But it means that you can do pretty much anything with your DNS. It also means that rather than learn a whole new API for your entire DNS implementation, you only have to learn it for the one operation of updating the master IP address -- could be as simple as a single POST. Here's an idea I'm toying with:

ANY dns service (dyndns.com, zoneedit.com, xname.org, etc) which can run in slave mode.

Then, use some EC2 instance — probably your load balancers — as a powerdns master. I chose powerdns because, after looking at just about every DNS server and library I could find, it's the closest to a pluggable/scriptable server.

Mainly, it has a 'pipe' API, in which it communicates via stdin/stdout with a simple, tab-delimited format. It's not entirely documented, but I was able to get the SOA field working by using spaces to delimit the actual SOA data.

This will not perform incredibly well, but it doesn't have to. All it really has to do is implement AXFR, and maybe do some local resolution for your other EC2 instances.

Of course, you have to tell them (somehow) if the master changes, and it may take a bit longer for those changes to propagate. But it means that you can do pretty much anything with your DNS. It also means that rather than learn a whole new API for your entire DNS implementation, you only have to learn it for the one operation of updating the master IP address — could be as simple as a single POST.

]]> By: Manuel Simoni http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-2/#comment-5792 Manuel Simoni Sun, 25 Feb 2007 06:10:17 +0000 http://blog.apokalyptik.com/?p=154#comment-5792 OK, thanks. BTW, great site! OK, thanks. BTW, great site!

]]> By: apokalyptik http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-2/#comment-5790 apokalyptik Sat, 24 Feb 2007 07:07:14 +0000 http://blog.apokalyptik.com/?p=154#comment-5790 yes, you can definately do that. lb1 and lb2 names are mainly there for ease of administration. It's a lot handier to just "ssh lb1.foo.com" than listing your instances and finding the one which is a load balancer. yes, you can definately do that. lb1 and lb2 names are mainly there for ease of administration. It's a lot handier to just "ssh lb1.foo.com" than listing your instances and finding the one which is a load balancer.

]]> By: Manuel Simoni http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-5788 Manuel Simoni Sat, 24 Feb 2007 00:08:13 +0000 http://blog.apokalyptik.com/?p=154#comment-5788 Instead of the extra addresses lb1.example.com and lb2.example.com, wouldn't it be possible that each load balancer just regularly checks whether lb.example.com works, and when it doesn't, register its own IP via the DNS API? Instead of the extra addresses lb1.example.com and lb2.example.com, wouldn't it be possible that each load balancer just regularly checks whether lb.example.com works, and when it doesn't, register its own IP via the DNS API?

]]> By: apokalyptik http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-4064 apokalyptik Sun, 07 Jan 2007 12:58:43 +0000 http://blog.apokalyptik.com/?p=154#comment-4064 Yes, theres a reason that DNS has been used for so long: It's robust, and it works. This is the kind of deviation from standards which make life difficult in large paradigm shifts like this. I'm sure that whoever implemented this thought to themselves "a DNS entry worth hard coding is not likely to ever change anyway..." However now that we're moving into the compute-as-a-service era the above statement no longer holds anywhere near true. I expect that this will eventually be addressed as services like EC2 become more widespread in use especially now that JAVA is going OSS. (This is exactly the kind of "bug" fixing that OSS is exceptionally well suited to deal with) Yes, theres a reason that DNS has been used for so long: It's robust, and it works.

This is the kind of deviation from standards which make life difficult in large paradigm shifts like this. I'm sure that whoever implemented this thought to themselves "a DNS entry worth hard coding is not likely to ever change anyway…" However now that we're moving into the compute-as-a-service era the above statement no longer holds anywhere near true.

I expect that this will eventually be addressed as services like EC2 become more widespread in use especially now that JAVA is going OSS. (This is exactly the kind of "bug" fixing that OSS is exceptionally well suited to deal with)

]]> By: Steve Loughran http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-4055 Steve Loughran Sun, 07 Jan 2007 12:38:18 +0000 http://blog.apokalyptik.com/?p=154#comment-4055 Its documented on Sun's site: <a href="http://java.sun.com/j2se/1.5.0/docs/guide/net/properties.html" rel="nofollow">http://java.sun.com/j2se/1.5.0/docs/guide/net/pro...</a> networkaddress.cache.ttl (default: -1) networkaddress.cache.negative.ttl (default: 10) The infinite cache was added to stop long lived applets getting access to hosts behind a firewall through DNS abuse, but it stays in apps too. really old jvms (1.2 and maybe 1.3) saved negative dns entries forever. this is bad news as a temporary outage of a DNS server would never be recovered from. TTLs -they are there for a reason, and it is the duty of client side apps to remember them. I wonder if proxies have similar behaviour? Interesting thought... Its documented on Sun's site:
http://java.sun.com/j2se/1.5.0/docs/guide/net/pro…

networkaddress.cache.ttl (default: -1)

networkaddress.cache.negative.ttl (default: 10)

The infinite cache was added to stop long lived applets getting access to hosts behind a firewall through DNS abuse, but it stays in apps too. really old jvms (1.2 and maybe 1.3) saved negative dns entries forever. this is bad news as a temporary outage of a DNS server would never be recovered from.

TTLs -they are there for a reason, and it is the duty of client side apps to remember them.

I wonder if proxies have similar behaviour? Interesting thought…

]]> By: apokalyptik http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-4014 apokalyptik Sat, 06 Jan 2007 10:47:56 +0000 http://blog.apokalyptik.com/?p=154#comment-4014 oooh interesting caveat. (I'm not a java guy personally...) oooh interesting caveat. (I'm not a java guy personally…)

]]> By: Steve Loughran http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-4012 Steve Loughran Sat, 06 Jan 2007 10:41:52 +0000 http://blog.apokalyptik.com/?p=154#comment-4012 Except that the default TTL of nslookups in a JVM is *forever*, so unless your clients start their JVMs with java.net.ttl=1000 (or whatever) you're in trouble. As long as your clients arent JVMs or Java applets (whose hostname caches cannot be changed for security reasons) this is an non issue. Except that the default TTL of nslookups in a JVM is *forever*, so unless your clients start their JVMs with java.net.ttl=1000 (or whatever) you're in trouble. As long as your clients arent JVMs or Java applets (whose hostname caches cannot be changed for security reasons) this is an non issue.

]]> By: Tom Kerswill http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-3982 Tom Kerswill Fri, 05 Jan 2007 18:02:12 +0000 http://blog.apokalyptik.com/?p=154#comment-3982 Certainly will! Am in the process of setting all the DNS stuff up at the mo. I love the idea of being able to keep firing up instances as a backup if the main server fails. Actually I'm quite optimistic about using the DNS failover in this way. What I'm now less sure about is using S3 as the backup, to store home directories, etc. Loading in all the volatile data (eg. html files, php, etc) from S3 to the EC2 seems to take hours --- far too long to make it viable as a switch-on-and-go backup :-( Though each time I test it, it seems to get faster... which is at least encouraging! Certainly will! Am in the process of setting all the DNS stuff up at the mo. I love the idea of being able to keep firing up instances as a backup if the main server fails.

Actually I'm quite optimistic about using the DNS failover in this way. What I'm now less sure about is using S3 as the backup, to store home directories, etc. Loading in all the volatile data (eg. html files, php, etc) from S3 to the EC2 seems to take hours — far too long to make it viable as a switch-on-and-go backup :-( Though each time I test it, it seems to get faster… which is at least encouraging!

]]> By: apokalyptik http://blog.apokalyptik.com/2006/11/13/ha-ec2-part-2-load-balancing-the-load-balancer/comment-page-1/#comment-3968 apokalyptik Fri, 05 Jan 2007 12:27:37 +0000 http://blog.apokalyptik.com/?p=154#comment-3968 Very interesting! I hope you'll send some of your findings on the DNS service my way? Very interesting! I hope you'll send some of your findings on the DNS service my way?

]]>