Personal

I couldn’t have put it better myself

Yesterdays penny arcade hit it on the nose. Osx IS more convenient. It IS worth changing your opinions and telling everybody that all that Mac trash talking was about OS9, but OSX radically altered the very fabric of the universe and that you now have to take it all back: the Mac is now a truly reformed beast using its nefarious powers only for the purpose of good and justice (and the occasional profit margin)! Think Ghost Rider meets BSD :).

All geeky references aside. I was once a mac hater. Now I’m a mac lover.  And you know what. I’m okay with that!

One year aniversary

Well tomorrow is my 1 year anniversary. Thats 1 year happily bickering… er… married 🙂 It’s been a very trying year (not between her and myself, but between us and life.) Heres to another year with, hopefully, less complications.

I love you my sweet!
DK

SVN + RoR = Passive Version Controlled Goodness!

While working with both rails and subversion (which I like using to keep my projects under version control) I was irritated by having to go through and add or delete a bunch of files when using the code generation tools. Especially when first putting the project together, there always seemed to be 6 new files to manually add before every commit… So I wrote a script to handle the adding of new, and removing of missing files for a commit.

#!/bin/bash
IGNORE_DIRS="^\?[ ]{6}log$|^\?[ ]{6}logs$|^\?[ ]{6}tmp$"
IFS="
"
for i in $(svn st | grep -Ev $IGNORE_DIRS | grep -E "^\?")
  do
    i=$(echo $i | cut -d' ' -f7-1000)
    svn add $i
done
for i in $(svn st | grep -E "^!")
  do
    i=$(echo $i | cut -d' ' -f7-1000)
    svn del $i
done

Now I just ./rail_svn.sh and then svn ci and everything is always version controlled. Very nice. The only thing you have to watch out for is leaving files laying around (I’ve had a couple commits which, along with code, also involved removing a vim .swp file or two)

I would be willing to bet that this script would be a decent foundation for a passively version-controlled-directory system if anyone were to want to do something like that with svn (think mail stores, or home directories or anything in which files or directories are added or removed often). This is mainly needed because svn was designed to be an active version control system

QMAIL-TOASTER remote redilivery loop problem

I recently switched from my old gentoo server to a new FC5 server. I opted to go with a qmail-toaster setup because, while I’m perfectly capable of manually making my desired qmail+vpopmail setup, I just didn’t want to spend the personal time doing it. So I figured I would give the toaster project a try. And I have to say that I’m fairly impressed.

A lot of the core technological things that it did were done in basically the same way that I would have done them manually (which is bidirectionally gratifying for me) and there are some bells and whistles that are *nice* but I wouldn’t have bothered setting them up on my own (e.g. qmailmrtg graphical log analysis.)

I did (hopefully did and not still do) have one oddball problem with it. After switching over there were certain servers from which I would continuously get the same message over and over from. Everything in my logs showed a successful delivery, and its not as though the messages were stuck in my queue either, the remote servers would actually reconnect and deliver the message again.

Well for a while I had better things to do with my scant time than deal with this one inconvenient (but not critical) issue. Well today I finally cracked. Its probably because I’ve now gotten one particular message something on the order of 30 times now. Thinking about the problem, and examining my logs it seemed that the only time this happened was when a message was processed by simscan for viruses (clamd) and spam (spamd) at the SMTP transmission level. But that was not the complete story because other messages from other servers did not have this problem even though they went through simscan as well.

On a hunch I figured that the sending mail server was probably only designed to wait X number of seconds (or microseconds) after the finished transmission before expecting to get a status code back from my SMTP daemon. If it takes too long then the remote sending server might just assume the connection was lost and re-queue the message for redelivery. So I disabled spam and virus scanning in simscan

#echo ":clam=no,spam=no,spam_hits=12,attach=.mp3:.src:.bat:.pif" \
  > /var/qmail/control/simcontrol
# /var/qmail/bin/simscanmk
# /var/qmail/bin/simscanmk -g
# qmailctl restart

And the problem *seems* to have gone away. I’m not worried about viruses at this point because I’m running OSX as my desktop, and Thunderbird is usually pretty good about spam… so… no loss for me there.

I’m mainly writing this down here so that if someone were to have this problem, and floundering while searching for an answer, they might have a better chance of finding a helpful hint. Searching for things like redelivery and mail loops on google will yield nothing of any value at present.

Cheers
DK

Random Musing: Bluring the Line Between Storage and Database?

As food for thought…

If you had a table `items`

  • itemId char(40),
  • itemName varchar(128),

Another table `tags`

  • tagId char(40),
  • tagName char(40),

And a third table `owners`

  • ownerId char(40),
  • ownerUsername char(40),
  • ownerPassword varchar(128),

It would theoretically be possible to have an S3 bucket ItemsToTags inside which you put empty objects named (ownerId)-(itemId)-(tagId). And a TagsToItems S3 bucket inside which you put empty objects named (ownerIf)-(tagId)-(itemId), it would then be possible to use the Listing Keys Hierarchically using Prefix and Delimiter method of accessing your S3 buckets to quickly determine what items belong to a tag for an owner, and what tags belong to an tag for an owner. You would be taking advantage of the fact that that There is no limit to the number of objects that one bucket can hold, and no impact on performance when using many buckets versus just a few buckets. You could reasonably store all of your objects in a single bucket, or organize them across several different buckets. (both the above links are to quotes taken directly from the S3 API docs provided by amazon themselves)

Using this method it would be possible, I think, to use the S3 datastore in a VERY cheap manner and avoid having to deal with the massive cost of maintaining these kinds of indexes in a RDBMS or on your own filesystems… Interesting. And since the data could be *anything* and you have, by default you have a many to many relationship here you could theoretically store *anything* and sort by tags…

Granted to find a tag related to multiple items you would have to make multiple requests, and weed out the diffs. but. if you’re only talking on the order of 2 or 3 tages per piece of data… it might just be feasible..

Now… Throw in an EC2 front end, and a SQS interface… interesting…

Makes me wonder what the cost and speed would be (if it would be an acceptable tradeoff for not having to maintain a massive database cluster)

Disclaimer: this is a random musing. I’m not advising that anybody actually do this…

I always wanted to be able to draw

DrawSpace is a pretty cool site (read: I havent actually “used it” but I “plan to”) which gives lessons on how to draw… Which is cool because although I’ve always wanted to be artistic… I’ve always proved to be sucessful as an artist in exactly the same measure that your standard brick attains while trying to preform the backstroke…
Perhaps now I’ll not only *NOT* be able to draw, but that I’ve studied and *STILL* can’t draw 😉