Author: apokalyptik

apokalyptikhttp://blog.apokalyptik.com/

mod_fcgid and the cfgi gem for rails with apache2 on fedora core 6

Because its not readily apparent when you’re concentrating on learning rails… I’m posting this here. While trying to get rails running on Fedora Core 6 (FC6) I was running into not being able to compile the ruby fcgi gem.

What gives, right? mod_fcgid (what FC6 comes with) is supposed to be binary compatible with mod_fastcgi. And there *IS* no mod_fastcgi or fastcgi in yum! Well it turns out that

mod_fcgid and mod_fastcgi both connect to fastcgi. That bears repeating. mod_fcgid is roughly equivalent to mod_fastcgi but niether are equivalent to fastcgi. (which in retrospect seems obvious as so many things do when you’re searching for answers)

so… on FC6, install mod_fcgid. then download and install fastcgi (not mod_fastcgi) from http://www.fastcgi.com/dist/ and then 

gem install fcgi --source \
  http://rubyforge.planetargon.com/gems.rubyforge.org/ \
  -- \
  --with-fcgi-include=/usr/local/include \
  --with-fcgi-lib=/usr/local/lib

will work and everything will be peachy-keen

since a lot of people will be searching about this in a specific scope for Fedora Core 6, and since nobody seems to directly state this issue in plain terms, here it is. Google, eat me up! 😉

(PHP code) Gracefully handling the failure of TCP resources

function check_tcp_active($host, $port) {
    $socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
    socket_set_option($socket,
      SOL_SOCKET,
      SO_RCVTIMEO,
       array(
       "sec"=>0,
       "usec"=>500
       )
    );
    $result = @socket_connect($socket, $host, $port);
    if ( $result ) {
      socket_close($socket);
      return(TRUE);
    } else {
      return(FALSE);
    }
  }

  function find_active_server($array) {
    // Format: $array['127.0.0.1']=3306
    if ( is_array($array) ) {
      foreach ( $array as $host => $port ) {
        if ( $this->check_tcp_active($host, $port) ) {
          $rval['host']=$host;
          $rval['port']=$port;
          return($rval);
        }
      }
    }
    return(FALSE);
  }

  $mysqlServers=array(
    '127.0.0.1'    => 3306,
    '192.168.0.10' => 3306,
    '192.168.0.11' => 3306,
    '192.168.0.12' => 3306,
    '192.168.0.13' => 3306,
    '192.168.0.14' => 3306,
  );

  $goodMysqlHost=find_active_server($mysqlServers);

with only a very small amount of work a pseudo random load distribution would be possible. Hope this helps someone 🙂

EC2 Economics, People just don’t seem to get it

Should you or shouldn’t you use Amazons EC2 service? If you believe everything you read without bothering to send it through the hype filter the answer is invariably YES! But in reality (where most of the rest of us live) the question actually depends on a lot of different factors.

The, theoretically, most straight forward of those factors is raw cash. Lets say you were considering purchasing a low end server from, say, Servpath. You’re looking at $1999 per year for a server with a better CPU, but less RAM and HDD space. Versus (365 * 24 * 0.10 = ) $876 with an EC2 instance. Easy, right?

AHH but the devil is in the details. you get 0 (zero, none, nada, zilch, less than any) bandwidth with your Ec2 instance. Thats extra. Lets assume you use all 1500Gb per month you get with your low end server, the price at servpath is still $1999 for the year. The price at EC2 goes up by (1500 * 12 * 0.20 = ) $3600. Even if you pay monthly instead of yearly at servpath its only ($199 * 12 = ) $2388.

Of course if you use significantly less bandwidth… say… 300GB per month…

Servpath: $1999,
Amazon: $876 + (300 * 12 * 0.20 = ) $720 = $1596

But the real complexity comes when you worry about scaling, or handling peak versus off times. And further complexity when you start talking high availability, and load balancing. And yet further if you host static and dynamic content in different places, etc.

The bottom line is that all of these blog posts I see about the choice between traditional versus new-age hosting is not as simple as it is made out to be. Before you jump on the buzzword bandwagon you should really make sure that you’re investing in what makes *sense* for your business and not in what made sense for someone else.

Right about now you’re scratching your head and wondering whether I’m telling you TO use EC2 or NOT TO use EC2. And you’re reading this whole thing wrong (if thats what you’re wondering.) What I am telling you is that EC2 is an extremely flexible and versatile tool which has a huge possible number of advantageous scenarios into which it fills a void previously delegated to the realm of “there’s no solution other than to spend more money or not. period.” You use the sed utility when it makes sense, right? You wouldn’t, for example, attempt to use it to accept http uploads? Of course not that doesn’t even make sense! Well not all situations make sense for EC2 either.

EC2 is a lot like the OSS movement. The up side is that It gives people the power of flexibility and choice. But much like the “Is linux desktop ready?” debate thats been raging on for years you have to deal with the downside which is that it gives people the power of flexibility and choice. Double edged blade indeed. But well worth the risks… If you can learn to wield it properly!

Thoroughly confused yet? That isn’t even the half of it! 🙂

So my little brother got fired…

So my little brother had a job working for a storage company. He had only held the job for about 2 weeks, and last Wed he was fired (no reason given). They failed to have his check ready that day; furthermore they failed to have gotten it to him today. They heartily promise to have it ready for him tomorrow morning. My wife and I are going to go down their with him to pick it up. Seems the California labor code was broken in several respects.

Apparently, according to sections 201 and 227.3 of the labor code, they were legally obligated to have his last check on hand and give it to him before he walked out the door. Which they failed to do.

Furthermore they’ve not given him his required last paycheck for a full 7 days (Nov 15 – 21.) So we looked into the labor code and found this little gem: “An employer who willfully fails to pay any wages due a terminated employee (discharged or quit) in the prescribed time frame may be assessed a waiting time penalty. The waiting time penalty is an amount equal to the employees daily rate of pay for each day the wages remain unpaid, up to a maximum of 30 calendar days.”

“Thanks” to the great state of California for making this information so clear and concisely available here:

http://www.dir.ca.gov/dlse/FAQ_Paydays.htm

So, 2 weeks owed wages at about 30 hours per week at about $8.50 per hour?

$433 after taxes

Failing to pay the above amount for 7 days?

$476

The “new girl” breaking the law,
Costing your company 2 times what you should have had to pay,
And realizing that this particular ex employee and friends are, unfortunately, not to be the idiots you’ve been hoping they would be?

Priceless

One year aniversary

Well tomorrow is my 1 year anniversary. Thats 1 year happily bickering… er… married 🙂 It’s been a very trying year (not between her and myself, but between us and life.) Heres to another year with, hopefully, less complications.

I love you my sweet!
DK

Once more into the spam filter

I was getting the following error when running “/usr/bin/sudo -u vpopmail -H /usr/bin/spamassassin -D bayes –lint”, per the instructions:

[6932] info: rules: meta test DIGEST_MULTIPLE 
    has undefined dependency 'RAZOR2_CHECK'
[6932] info: rules: meta test DIGEST_MULTIPLE 
    has undefined dependency 'DCC_CHECK'

So, after some digging around I wandered over to /usr/share/spamassassin/20_net_tests.cf and set “meta DIGEST_MULTIPLE” to “PYZOR_CHECK > 1” which seems to have fixed the problem (and knowing my luck and the amount of effort I put into this, which is somewhere near 0 it’ll come back someday to bite me in the butt (thus the blog post)) I was then able to run the following

/usr/bin/sudo -u vpopmail -H \
    /usr/bin/spamassassin -D bayes --lint
/usr/bin/sudo -u vpopmail -H \
    /usr/bin/sa-learn -u vpopmail --force-expire

And to add the following to roots crontab

0 0 * * * /usr/bin/sudo -u vpopmail -H \
    /usr/bin/sa-learn -u vpopmail --force-expire 1>/dev/null

I then reversed what I did here: http://blog.apokalyptik.com/?p=145 (well the spam part, I still have clam=no)

Here’s to hoping.

And regardless of whether it works or not, thanks to Shubes for the comment!

HA EC2 Part #3: What Happens Once You’re Inside the Cloud

Onto what happens inside the cloud!

Since we’re looking to load balance what happens inside the cloud you might be tempted to ask why not use the same sort of method we used for load balancing (well at least fail-over) outside the cloud. And the answer is a resounding YOU CAN! But Rather like a cooking show where you _could_ use water to hydrate something but you could also “bring a little flavor to the party” by using chicken broth or wine, we can find ourselves with the option for a mighty fine set of bonus features — if we’re willing to look past the vanilla DNS round robin load balancing.

Ok, I suppose before I go on I should address you people still scratching your heads that I said you could use DNS for load balancing. I know you’re thinking that you’ll never achieve real balanced load with this method, and you’re right! Like I said more features await! And for everyone wondering why they should use a real load balancer if DNS is going to be “good enough” anyhow: remember that slight problem we mentioned about caching DNS servers? Well that problem would apply here as well, and since previously we couldn’t avoid it but now we can I see no reason not to. Plus it will be more of a headache here than there because the likelihood of a load balancer *LEFT ALONE* failing is a lot less likely than a web or database server which is constantly doing a great many things (and is subject to the whims of people — in the process of development.) Finally you get a single point at which you need to employ firewall configuration instead of X points where X is the number of back-end web servers.

Now as far as load balancers go there seem to be two prevalent kinds. First there is the TCP load balancer, and second there is the proxy.

A TCP load balancer functions on by IP address, Protocol, and Port. For example you may specify a group of servers as the back-end cluster for the IP address a.b.c.d port 80. And thats all you can do with that IP address and that port. It avoids a lot of complication by not caring, in the slightest, why or how you got there, or whether that particular set of web servers is really what you want. For this reason it’s not possible to specify fancy things like all requests on port 80 for example1.com go to cluster A, and example2.com to cluster B. To do that you need another IP address, or to access the cluster on a different port (say a.b.c.d port 81). For former is OK when you have multiple IP’s to work with and you can share IP’s between fail-over load balancers, but neither of those luxuries hold true in the EC2 environment. And the later is fine if you are planning to do this for your development team, but if you’re trying to drive normal web traffic to port 81 it might end up being a little less than convenient for your users.

Which is why I’m going to be focusing on the proxy.

Improperly configured the reverse proxy is a sure ticket to trouble on the internal LAN… Fortunately we aren’t dealing with an internal LAN, we’re dealing with servers which are publicly available anyway. What you can get from a reverse proxy, though, are extra features. You get load balancing, you get fail-over, you get url and host rule based redirection, and you get apache log file aggregation. You get it all… and, I might mention, at a very convenient low low price of FREE! WOO!

Um, er.. >clears throat< yea, never-mind that!

I would say that you have a couple of commonly known programs which can handle reverse proxying in Apache, and Squid. But… lets face it… both of those carry with them a more-so-than-necessarily complicated setup process and are like swatting at a fly with a cannonball. Sure they’ll work, but there re better alternatives for this. Two seemingly popular alternatives are pound and, the newcomer, perlbal. Both of these daemons offer better functionality that LVS (in the back-end server fail-over department) but which do we want to use?

The choice between the two is tough, and I can’t say that I’ve extensively used either, however pound does shine in three areas. First pound does have a notion of the idea of a SESSION and can even manage persistence/affinity (seemingly a LOT better than LVS manages it I might add (the persistence tables *ARE* wiped for a server that goes down!)) However if you have a web application which was developed with a “shared nothing” approach (which is a GOOD thing) this benefit does not really apply, so it’ll be up to the next to to knock your socks off. Second pound does SSL wrapping, taking that load off of your web servers (which is a good thing for responsiveness, isn’t it?). And finally pound offers a logging mode to emulate the apache combined log file format (both with and without virtualhosts.) Which puts pound in a class all by itself (and right up there with the hardware balancers I think). If none of those features matter to you or if (as is very possible) I’m wrong about perlbals feature-set. Then just pick one already (flip a coin, choose the one written in the language you prefer… or… hey… go read their docs and see which one you like better!)

The only drawback that comes to mind right now about using this approach is that you will be making use of text configuration files, so some parsing and rebuilding will end up being necessary for the registering and de-registering of web servers. I’ll add more in the comments if and when I think of more…

So there you have it.

Using a good DNS Service (with a low TTL, and decent API) mixed with a decent reverse proxy you have all the benefits of

  • a load balancer
  • load balancer fail-over
  • rules based request redirector
  • log consolidator
  • back-end server filover
  • and a single point for fire-walling

While this hasn’t exactly been a HOWTO, or a TOASTER, I hope that it’s definitely a pointer in the right direction for people who are looking to scale their applications which have been built on top of the Amazon EC2 (and SQS, and S3) services.

HA EC2 Part #2: Load Balancing the Load Balancer

Lets first address the problem of the dynamic IP address on the load balancer, because it doesn’t matter how good your EC2-side setup is if your clients can no longer reach your load balancer after a reboot. Also complicated because normally you want two load balancers to act as a fail-over pair in case one of them pops for some reason. Which means that we not only need to have the load balancers register with something somewhere we also need a method of de-registering them if, for some reason, they fail. And since downed machines usually don’t do a good job f anything useful we cannot count on them de-registering themselves unless we’re shutting them down manually. Which we don’t really plan on doing, now, do we?!

So here’s the long and short of the situation. Some piece of it, some starting point has to be outside the cloud. Now I know what you’re thinking: “but he just said we weren’t going to be talking about outside the cloud” but no, no, I did not say that; I said that we weren’t going to be talking about running a full proxy outside the cloud. I read that the EC2 team are working on a better solution for all of this, but for right now it’s in a roll your own state, so lets roll our own, shall we?

The basic building block of any web request is DNS. When you type in www.amazonaws.com your machine automagically checks with DNS servers somewhere, somehow, and eventually gets an IP address like this: 72.21.206.80. Now there can be multiple steps in this process, for example when we looked up www.amazonaws.com it *actually* points to rewrite.amazon.com, and finally rewrite.amazon.com points to 72.21.206.80. And this is a process we’re going to take advantage of. But first, some discussion on the possible ramifications of doing this:

DNS (discussed above) is a basic building block of how the internet works. And as such has had a dramatic amount of code written concerning it over the years. And the one type of code which may cause us grief at this stage is the caching proxy server. Now normally when you look up a name you’re asking your ISP’s DNS servers to look the name up for you, and since it doesn’t know it asks one of the primary name servers which server in the internet handles naming for that domain. once it finds that out it asks, a lot like this: “excuse me pdns1.ultradns.net, what is the address for rewrite.amazon.com?” to which your ISP gets a reply a lot like “The address for rewrite.amazon.com is 72.21.206.80 but thats only valid for 5 minutes.” So for 5 minutes the DNS server is supposed to be allowed to remember that information. So after 4 minutes when you ask again it doesn’t go to the source, it simply spouts off what it found out before. However after 5 minutes it’s supposed to check again… But some DNS servers ignore that amount of time (called a Time To Live (TTL)) and cache that reply for however long they feel like (hours, days, weeks?!) And when this happens a client might not get the right IP address if there has been a change and a naughty caching DNS server refuses to look it up for another week.

Alas, there is nothing we can do to fix that. I only mention it so that people don’t come knocking down my door yelling at me about a critical design flaw when it comes to edge cases. And to caution you: when your instance is a load balancer. It’s *ONLY* a load balancer. Don’t use it to run cron jobs, I don’t care if it’s got extra space and RAM, just leave it be. Because the fewer things happening with your load balancer the fewer chances of something going wrong, and the lower the chance of a new IP address, and the lower the chance of running into the above problem if the IP address doesn’t change, right? right!

So when you choose a DNS service you choose one which meets the following criteria:

  • API, you need scriptable access to your DNS service
  • Low (1-2 minutes) TTL
    (so that when something changes you only have 60 or 120 seconds to wait)

Ideally you will have two load balancer images. LB1 and LB2 (for the sake of me not having to type long names every time). You can do this dynamically (i.e. X number of load balancers off the same image), and if you’re a good enough scriptor to be able to do it, then HOW to do it should be fairly obvious.

When LB1 starts up it will automatically register itself at lb1.example.com via your DNS providers API. It will then check for the existence of lb.example.com, if thats not set then it will create it as pointing to itself. If lb.example.com was previously set it till preform a check (HTTP GET (or even a ping)) to make sure that LB2 (which is currently “active” at lb.example.com) is functional. If LB2 is not functional LB1 registers itself as lb.example.com. LB2 performs the same startup sequence, but with lb1 and lb2 switched where necessary.

Now, at regular intervals (lets say 60 seconds), LB1 checks the health of LB2 and vic a versa. If something happens to one of them the other will, if necessary, register itself at lb.example.com.

Well, I think that basically covers the portion of how this would work outside the EC2 cloud, next I’ll deal with what happens inside the EC2 cloud. (piece not written yet… so it’ll take a bit longer than the last two)

HA EC2 Part #1: Identifying the Challenges

I was recently asked to look into load balancing web servers on the Amazon Elastic Cloud Computing Service (EC2). And managing this presents some very interesting problems which need to be worked around. To look at the subject I’ll break it into 3 distinct pieces. #1: Identifying the Challenges (Which you’re currently reading), #2: Load Balancing the Load Balancer, and finally #3 What Happens Once You’re Inside the Cloud. No promises as to how quickly I get these out 🙂

First lets look at what this would normally entail:

You would have a data center, and a router which feeds into a DMZ. On the DMZ you would have a set of load balancers (either hardware or software.) A set so that if one failed the other would take over its job. These load balancers have static IP addresses on the DMZ as well as on the LAN. They also have a shared IP address which they are the balancers for. When one goes down the other takes over the IP address. In a hardware solution this might be accomplished in a fairly elegant and network invisible way. In a software solution this normally entails using IP aliases and forcibly updating the ARP cache on the router.

So the load balancers are the bridge between the DMZ and the LAN. On the LAN, with the load balancers, are a group of web servers. also with static IP addresses. There is a monitoring functionality on the load balancer which detects if a web server is no longer available. When that happens the load balancer updates an internal table and no longer sends requests to that particular web server. When the web server becomes available again the load balancer detects this, updates those internal tables, and begins sending requests to the server once more. All of that happens with varying levels of complexity.

For the scenario of the web servers reply there are multiple possible configurations. The web server may reply to the load balancer and the load balancer would then handle getting the proper response from your data-center to the client (a full reverse proxy). The web server might also reply directly to the client through a network route (in Linux Virtual Server (LVS) terms this is called “Direct Routing” (LVS-DR)) 

  [ WAN ]                                      -> [ Server ]
  [ ROUTER ]                                  |-> [ Server ]
  [ DMZ ] <-> [ Load Balancer ] <-> [ LAN ] <-+-> [ Server ]
                                              |-> [ Server ]
                                               -> [ Server ]

The first thing that jumps out at me is that there is one key assumption in the above setup possibilities, and that is that everything is able to obtain a static IP address. That is that every time a given machine goes down, it comes back up at the same IP address. This is not true of the EC2 service. Your EC2 instances are dynamically allocated new IP addresses (and host-names) each time they are started (and consequently restarted.) So…

  • No static IP for the load balancer
  • No static IP for the web servers

Which means that on top of the challenges of installing and configuring a normal software load balancing solution there are several fold more challenges to overcome to be “successful” in your endeavor.

  • You need to notify your clients if the load balancer address has changed
  • You need to notify your web servers if the load balancer address has changed
  • You need to notify your load balancer if the address of a web server has changed

Now you could, technically, circumvent the first o these challenges by housing the load balancer outside of the EC2 cloud, however this doesn’t make a whole lot of sense seeing as you would end up paying twice for all the bandwidth consumed (You would have to pay for the incoming request at the load balancer, then to make the same request to a web server, then the cost of the reply from the web server to the load balancer, and finally the cost of the reply from the load balancer to the client) so for the sake of this little mental pushup we’ll not even consider that a viable option, only worth mentioning (and we have, so now that thats over…)