Month: August 2006

5 Excellent Uses for Amazon EC2

The Test/Development Environment

One of those things which is always difficult is putting together the right test environment. Traditionally purchasing a seperate set of hardware, , a seperate datacenter, A developers “local copy,” or something as cheesy as different vhosts on the same machine have been the only real answers. Most of those are grossly inneficient as well as expensive. At the best they’re “ok” (because you arent going to put any *real* power in this arena when you have mission critical applications to deploy) and at worst they’re downright dangerous (how many people do you know have never accidentally edited the wrong file in the wrong folder?)

Enter Amazon, wisking you away to a magical place in which its possible to throw as much horsepower as neccesary at a test environment. Where its possible to wipe and reload the entire thing in a *very* short period of time. And where you only have to pay for it when it’s being used. With Amazon EC2 there is no longer any reason that a web app company cannot afford a truly useful test/development environment

Offsite Backups

Do you worry that something is going to happen to your data? Is the only backup copy of your working code on one of those CDR’s sitting in the “server closet”… was it taken more than 3 months ago? If you answered yes to any of these questions Amazon EC2 is right up your alley. With a comple of public keys, and rsync you can put the power of secure backups to work for you! But wait, theres more! Once the backup has completed schedule a shutdown command, and only pay for what you need! Pretend you have 100Gb of data to backup… it changes at 5% per dat, and takes only 4 hours to run the backup remotely… If you setup a once per month full copy, and preform incrimental backups daily you’re looking at in the neighborhood of $130 per month. Beat that with a tape drive… And you dont outgrow the EC2+S3 combo in 3 months of prolific Word Document writing like you would that last tape drive 🙂

Version Control

Closely coupled with the development environment Version control is the heart and soul of any source based application, and theres no reason not to put this on something like an EC2 server. The bandwidth usage is minimal. The gauranteed backed up aspect is a huge load off the restless mind, and the next time you have to run reports on the last three hundred and seventy five thousand revisions… it doesnt bring your real (or test) environment screetching to a rather annoying halt!

Trouble Ticketing

You took the time to setup a nice trouble ticketing system for both your customers and your employees. It’s customized to the teeth. It rocks. It rolls. It just went down with the rest of your infrstructure just now when you blew a circuit breaker. Or did it? Not if you moved this piece of mission critical (and resource light) infrastructure to EC2. You’ll basically be able to run this puppy at the cost of $.10 per hour because the traffic itll cost you will be… what… $3 bucks a month? Change. And the ability to still communicate with all your customers during a major outage? priceless

Monitoring and Alerting

For roughly the same reasons as noted above using EC2 for monitoring your mission critical applications is the right thing to do. Not only do you have a third party verification f availability you have an off network source for things like throughput and responsiveness. If you’re monitor sends SMS through your mail server… and your mail server just died… you wont hear a peep out of your phone until someone brings it back up. And then when it *is* backup not only do you catch flack for not noticing… but all holy hell breaks loose on your phone as 400 SMS messages come in from l;ast nights backlog. Do yourself (and your organization) a favor… EC2 + Nagios + Mon = peace of mind (throw in cacti and you have answers to the dreaded question: “so how is our performance lately anyhow?”). Plus if you use something like UltraDNS which offers a programatic API for their services you can use amazon as the trigger for moving from one ser of servers to the other. Wonderful!

Some comparisons

Lets make some sample comparisons based on 1 year assumptions

High bandwidth (1 Year: 10 servers, 5Tb xfer per month)

  • Low end host – $12,000
  • Mid end host – $21,600
  • Amazon EC2 – $128,880

Low bandwidth (1 year: 10 servers, 200Gb xfer per month)

  • Low end host – $12,000
  • Mid end host – $21,600
  • Amazon EC2 – $8,920

Mixed solution

  • (
  • 8 servers @ amazon for processing (100Gb/mo),
  • 2 servers at a hosting co for bandwidth (2Tb/mo)
  • )
  • EC2 + Low end: $9524 (vs $12,000 hosted, and $9,280 at EC2)
  • EC2 + Mid end: $11,684 (vs $22,800 hosted, and $9,280 at EC2)

Eating Amazon EC2 for lunch :D

At lunch Feedster, and Technorati are neck and neck (well for this past 4 hour stint) with 5 EC2 entries each. Feedster has 3 of the 5 not being junk… Technorati has 5 of 5 not being junk… Of course nothing really new to report on those posts…

However 12f does make an interesting point in his article Innovative Amazon.Com Web Services

Let’s take a hypothetical case – a client that wants a 5 year hosting deal for an ERP solution. They will need servers, storage, and support.

* 20 CPUs
* 1 Terabyte of Storage
* 200 GB of Bandwidth consumption per month

Over 5 years, using Amazon’s infrastructure, your cost will be approximately $100,000. This is going to be far, far less than any comparable quote you would receive from a traditional hosting service.

This is going to save me a lot of headache, and some cash

Seeing as I just had a 75 ft network cable chewed through which connects my office (upstairs) to the internet connection (cable modem down stairs)… and that I just happen to have bought (a long while back) a second openWRT-able router this article: OpenWRT wifi bridge is going to save me headache, time, and money.

First off… buying premade network cable… isnt cheap… and second buying wireless bridges is also not cheap (in the neighborhood of $80 to $160) expecially which are capable of 54Mb/sec…

My last wireless bridge used for this purpose was an old 802.11b jobber which was sorely inadequate… it would choke on the number of connections I have a tendency to hape open… Throughput was *TERRIBLE*… Even though my internet connection is only 6M (8M now) the 11M bridge just could *NOT* cut the cake… And it ran me $80 at the time I bought it…

Heres to hoping that OpenWRT is just as good at bridging as it is at routing

I’ll keep you updated… perhaps I’ll do this tonight (I dont have to do it immediately because I happened to have an old 100Ft cable which is not in use any longer, since my wife dropped her desktop in favor of my old laptop)

I don’t think I agree

Blackfriars’ Marketings post: Amazon.com’s EC2 hosting service: technology diworsification mentions the following:

Amazon.com has fallen into a strategic trap. Instead of investigating more and better ways to serve its retail customers, it has decided to focus on being a technology company instead, presumably because technology companies often command higher valuations. This EC2 service (even the name is geeky) will distract the company’s energy when it should be gearing up for a tough fourth quarter selling season. Amazon survived the dot com bust because people could understand its business and its competitive advantages in retail. Moves like EC2 will make that understanding much harder now.

I agree that if Amazon starts advertising these services on the amazon.com storefront a lot of people will be confused about it… But I dont see Amazon doing that. If Amazon has gotten their infrastructure refined to the point that offering these services to themselves, internally, has not only become easy but trivial then theres really no reason that they cant make a market out of it. If you do something, and you do it well, theres no reason it shouldnt make you money.

I’d go so far as to argue that this move by amazon is probably one of the most gratifying things that most NetOps guys have seen in a long, long, time. Historically the network operation geeks are the atlases upon whose backs the web 2.0 coders tend to stand. And like mythical Atlas holding the world by himself they tend to go unoticed as more high profile names take credit for the entire job… To me, being an ops person at heart, it’s been immensly gratifying to see a company like amazon look at their Ops guys and say "hey, these guys are valuable… we really need these guys… and they can help our company both directly and indirectly"

If you’ve got it… Sell it 😉

Remember when I mentioned that we’ll be seeing some disilusionment?

Well this morning I found my first piece of it. A blog post (which I’ve since closed and dont feel like finding again) mentioned that the “fly in the ointment” was that he wouldnt be able to load an exceptionally large database into the EC2 grid.

The reason for this complaint is the limit of an image to 160Gb (apparently) so minus about 4Gb for the OS and you have 156Gb worth of space on which you could but a database. Now to a lot of you thats seems like a lot of data… and it is… but if you’re processing large amounts of data (research, social stuff, search, etc, etc) 500GB, 1TB, 2TB, 4TB arent unheard of.

Scientific applications aside the most likely place you’ll see people desiring dtabases lartger than 160Gb is probably going to be in the web 2.0 startup group. And I really hav to question the validity of their gripe. If you call in the heavy hitters at MySQL (And I know this, because I’ve worked in a company in which I *HAVE* called in the heavy hitters at MySQL) They’ll tell you, invariably, to spread out your data.

The reasons for spreading our your data are numerous, but the 2 big ones are physical scalability and speed. Lets assume you’re developing your web app… and right now the data is 500Mb. Its fast, its responsive, life is wonderful. Even on your poorly designed monollithic schema. But you take off and start growing at 2x per week! Wonderful! Traffic is GOOD! … … … well…

  • 500Mb
  • 1GB
  • 2GB
  • 4GB
  • 8GB
  • 16GB
  • 32GB
  • 64GB
  • 128GB
  • 256GB
  • 512GB
  • 1TB

Not only does it become increasingly difficult to find a place to store (and backup) your data… its gotten SLOWER… the indexes arent enough, things take a long time… and you dread running an ALTER TABLE more than your mother in law (disclaimer: I like my mother in law just fine)

The solution is to design your schema in such a way that breaking it into pieces is a natural process! If you do this you’ll be ready to maintain your users table in one database cluster on one set of hosts… the log files in another… and various pieces of your data in various places. How to develop a schema which can do this is a whole nother blog post (one which I’ll be very happy to write up if someone requests it)

The point is this: Just because someone has a valid gripe with the service doesnt necessarily mean that its well founded. Try and think through whats being said for yourself — you’ll end up much happier that way.

Oh, and if the OP is reading this… Perhaps I’d be willing to comment/help on your schema? 😉

Condensing the steam (smoke) from this mornings EC2 posts

Generally speaking there is a bit of comparing EC2 to Suns GRID system… A couple mentions of oracle. A link to a screencast about EC2. A few people griping about the idea that (for now) Microsoft Windows might not be an option on EC2 systems…

The most interesting offerings I found this morning were: a look at the EC2 TOS, and how unrestrictive it seems to be, and a walkthrough of how setting up a new instance actually happens, and that screencast (not because it really shows anything… but because it shows something)

And a whole lot of people (like myself last night) regurgitating the specs and prices