I wonder if anybody has put any thought into using Amazons SQS as a backend for apache log file consolidation… It’d work perfectly… i think.

Let me share some perspective on what bandwidth numbers actually mean… Because talking about 1000Gb is a term that doesnt actually connect with anything in people minds

1000Gb/month means that on a 31 day month you are sending out (numbers rated as 1000 equals 1k because thats prevalent)

• 373Kbytes per second…
• 22Mbytes per minute
• 1.3Gbytes per hour
• 32Gbytes per day

or the ability to

• broadcast a 256bkit audio stream to 11.5 users around the clock
  (remember 8 bits == 1 byte)
• send out over 1,500 CDROM images
• send out over 200 DVD images (single sided)
• send out almost 286,000 MP3’s (3.5Mb each)

In otherwords… its a lot of bandwidth… assuming i’m not so tired that my numbers are flawed (which is very possible)

so… at EC2 to send out 5KB/sec for the same month would cost $2.68. 50KB @ $26.80. 500KB @ … well you get the idea.

One last comparison before I head off to bed… At EC2, for $160/month (comparable to a mid-level hosting provider), you get about 365Gb xfer per month… or 136KB/sec for the entire month (or 4 256Kbit audio streams, 547 CDs, 54 DVDs, 101,714 mp3’s, or one 365,000,000 letter text file ;))

Well I, being the looser that I am, Just figured out trackbacks… SO I expect I’ll be using that feature a lot more

At lunch Feedster, and Technorati are neck and neck (well for this past 4 hour stint) with 5 EC2 entries each. Feedster has 3 of the 5 not being junk… Technorati has 5 of 5 not being junk… Of course nothing really new to report on those posts…

However 12f does make an interesting point in his article Innovative Amazon.Com Web Services

Let’s take a hypothetical case – a client that wants a 5 year hosting deal for an ERP solution. They will need servers, storage, and support.

* 20 CPUs
* 1 Terabyte of Storage
* 200 GB of Bandwidth consumption per month

Over 5 years, using Amazon’s infrastructure, your cost will be approximately $100,000. This is going to be far, far less than any comparable quote you would receive from a traditional hosting service.

Seeing as I just had a 75 ft network cable chewed through which connects my office (upstairs) to the internet connection (cable modem down stairs)… and that I just happen to have bought (a long while back) a second openWRT-able router this article: OpenWRT wifi bridge is going to save me headache, time, and money.

First off… buying premade network cable… isnt cheap… and second buying wireless bridges is also not cheap (in the neighborhood of $80 to $160) expecially which are capable of 54Mb/sec…

My last wireless bridge used for this purpose was an old 802.11b jobber which was sorely inadequate… it would choke on the number of connections I have a tendency to hape open… Throughput was *TERRIBLE*… Even though my internet connection is only 6M (8M now) the 11M bridge just could *NOT* cut the cake… And it ran me $80 at the time I bought it…

Heres to hoping that OpenWRT is just as good at bridging as it is at routing

I’ll keep you updated… perhaps I’ll do this tonight (I dont have to do it immediately because I happened to have an old 100Ft cable which is not in use any longer, since my wife dropped her desktop in favor of my old laptop)

Blackfriars’ Marketings post: Amazon.com’s EC2 hosting service: technology diworsification mentions the following:

Amazon.com has fallen into a strategic trap. Instead of investigating more and better ways to serve its retail customers, it has decided to focus on being a technology company instead, presumably because technology companies often command higher valuations. This EC2 service (even the name is geeky) will distract the company’s energy when it should be gearing up for a tough fourth quarter selling season. Amazon survived the dot com bust because people could understand its business and its competitive advantages in retail. Moves like EC2 will make that understanding much harder now.

I agree that if Amazon starts advertising these services on the amazon.com storefront a lot of people will be confused about it… But I dont see Amazon doing that. If Amazon has gotten their infrastructure refined to the point that offering these services to themselves, internally, has not only become easy but trivial then theres really no reason that they cant make a market out of it. If you do something, and you do it well, theres no reason it shouldnt make you money.

I’d go so far as to argue that this move by amazon is probably one of the most gratifying things that most NetOps guys have seen in a long, long, time. Historically the network operation geeks are the atlases upon whose backs the web 2.0 coders tend to stand. And like mythical Atlas holding the world by himself they tend to go unoticed as more high profile names take credit for the entire job… To me, being an ops person at heart, it’s been immensly gratifying to see a company like amazon look at their Ops guys and say "hey, these guys are valuable… we really need these guys… and they can help our company both directly and indirectly"

If you’ve got it… Sell it 😉

Well this morning I found my first piece of it. A blog post (which I’ve since closed and dont feel like finding again) mentioned that the “fly in the ointment” was that he wouldnt be able to load an exceptionally large database into the EC2 grid.

The reason for this complaint is the limit of an image to 160Gb (apparently) so minus about 4Gb for the OS and you have 156Gb worth of space on which you could but a database. Now to a lot of you thats seems like a lot of data… and it is… but if you’re processing large amounts of data (research, social stuff, search, etc, etc) 500GB, 1TB, 2TB, 4TB arent unheard of.

Scientific applications aside the most likely place you’ll see people desiring dtabases lartger than 160Gb is probably going to be in the web 2.0 startup group. And I really hav to question the validity of their gripe. If you call in the heavy hitters at MySQL (And I know this, because I’ve worked in a company in which I *HAVE* called in the heavy hitters at MySQL) They’ll tell you, invariably, to spread out your data.

The reasons for spreading our your data are numerous, but the 2 big ones are physical scalability and speed. Lets assume you’re developing your web app… and right now the data is 500Mb. Its fast, its responsive, life is wonderful. Even on your poorly designed monollithic schema. But you take off and start growing at 2x per week! Wonderful! Traffic is GOOD! … … … well…

• 500Mb
• 1GB
• 2GB
• 4GB
• 8GB
• 16GB
• 32GB
• 64GB
• 128GB
• 256GB
• 512GB
• 1TB

Not only does it become increasingly difficult to find a place to store (and backup) your data… its gotten SLOWER… the indexes arent enough, things take a long time… and you dread running an ALTER TABLE more than your mother in law (disclaimer: I like my mother in law just fine)

The solution is to design your schema in such a way that breaking it into pieces is a natural process! If you do this you’ll be ready to maintain your users table in one database cluster on one set of hosts… the log files in another… and various pieces of your data in various places. How to develop a schema which can do this is a whole nother blog post (one which I’ll be very happy to write up if someone requests it)

The point is this: Just because someone has a valid gripe with the service doesnt necessarily mean that its well founded. Try and think through whats being said for yourself — you’ll end up much happier that way.

Oh, and if the OP is reading this… Perhaps I’d be willing to comment/help on your schema? 😉

Generally speaking there is a bit of comparing EC2 to Suns GRID system… A couple mentions of oracle. A link to a screencast about EC2. A few people griping about the idea that (for now) Microsoft Windows might not be an option on EC2 systems…

The most interesting offerings I found this morning were: a look at the EC2 TOS, and how unrestrictive it seems to be, and a walkthrough of how setting up a new instance actually happens, and that screencast (not because it really shows anything… but because it shows something)

And a whole lot of people (like myself last night) regurgitating the specs and prices

• Feedster: 6 posts, 2 Good, 4 Bad/Iffy
• Technorati: 31 Posts, 15 Good, 16 Bad/Iffy

And while Technorati has picked up all my posts about AEC2… Feedster has not.

http://dan-ball.jp/javagame/mc/