Linux

Simple TCP Daemon Example

Using some stuff I’ve covered in the past on my blog here’s a simple way to put up a daytime server (well to put any service onto a tcp port. I haven’t looked into its bi-directional capabilities yet, this was just sort of a proof-of-concept…

$ apt-get install ipsvd
$ wget http://blog.apokalyptik.com/files/daemonize/daemonize.c
$ cc daemonize.c -o daemonize
$ ./daemonize /var/run/daytime.pid /var/log/daytime.log 'tcpsvd 0 13 date'

start/stop and/or monit script are an extremely short jump from there… And kind of trivial/menial… so I leave that as an exercise to you… if you care 🙂

As Close to A Real Daemon As Bash Scripts Get

I’ve written a little something which is gaining some traction internally, and I always intended to share it with the world. So… Here. daemon-functions.sh

What it does is allow you to write a bash function called “payload” like so:

function payload() {
while [ true ]; do
checkforterm
date
sleep 1
done
}
source path/to/daemon-functions.sh

Once you’ve done that it all just happens.  daemon-functions gives you logging of stderr, stdout, a pid file, start, stop, pause, resume, and more functions.  when you start your daemon it detaches completely from your terminal and runs in the background.  Works very simply with monit straight out of the box.  you can have as many daemons as you wish in the same directory and they wont clobber each other (as the pid, control, and log files all are dynamically keyed off of the original script name.)  Furthermore inside your execution loop inside of the payload function place a checkforterm call at any place which it makes sense for your script to be paused, or stopped. it can detect stale pid files and run anyway if the process isnt really running.  As an added bonus you dont actually have to loop inside payload, you can put any thing in there, have a script thats not a daemon, but will take an hour, day, week, month to finish? stick it in, run it, and forget it.

building sed for osx

I work in linux a lot… not bsd. So the OSX (bsd style) implementation of sed really throws me for a loop when I go text-file-spelunking, whats worse is that my scripts using sed aren’t portable between the two OSs.

A quick googling this morning landed me here: http://wiki.octave.org/wiki.pl?OctaveForMac which gives perfectly good instructions on installing sed. except it didnt work. I grabbed the latest version of sed (4.1.5) and got the error 

sed: 1: "install_sh=/Users/apoka ...": command i expects \ followed by text
sed: 1: "install_sh=/Users/apoka ...": command i expects \ followed by text

Ironic, huh? Well taking a guess that at some point sed hadcome to depend on its own functionality to configure itself I jumped back a version… Figuring i replace BSD sed with an out of date GNU sed, and then use the old GNU sed to build the new GNU sed. Which worked great. I Installed first sed-3.0.2, and then 4.1.5 in this manner:

./configure --prefix=/usr/ --with-included-regex --with-included-gettext && make && sudo make install

I’m happy with my -r again…

# date | sed -r s/'[0-9]'/'?'/g
Thu Apr ?? ??:??:?? PDT ????

-v is for verbose, damnit

So, the vast majority of every day administrative command line utilities for Linux use -v as the switch for verbose…. when you use -v you EXPECT verbose. Well sometimes you get that one package which just CANNOT follow the rules. Someone has to think outside the box, someone has to be a unique snowflake. That someone should not be a mass process killing utility! whoever thought up that the argument -v to pkill should INVERT THE MATCH should really take a long slow look at how important being unique really is… because if you’re not aware of this, and you run something like pkill -9 -v nagios…. as root… it’s not going to do what you expect. Nothing good comes of that command.

This has been a PSA

colorizing php cli scripts

It’s pretty common in most scripting languages which center around the command line (bash, perl, etc) to find information on colorizing your shell script output, mainly because those languages are tied very tightly to command line use. It can be difficult, however, to find information about adding this same nice feature to a php cli script. The reason is simple: most people dont use php for cli applications; most cli programmers use something else. It’s not difficult to adapt the same techniques listed in most bash howtos (generally in the section reserved for colorizing your command prompt) for generating colored terminal output for php.

echo "\033[31mred\033[37m
";

echo "\033[32mgreen\033[37m
";

echo "\033[41;30mblack on red\033[40;37m
";

Simple, functional, useful (even if a bit complicated.) I leave it to you to lookup a bash prompt colorization howto to hunt down your own list of escape color codes (call it homework.)

Cheers

Bash Tip: Closing File Descriptors

I recently found that you can close bash file descriptors fairly easily, it goes like this:

    exec 0>&- # close stdin
    exec 1>&- # close stdout
    exec 2>&- # close stderr

Which makes it easy to daemonize things using only bash (lets face it there are times when you JUST don’t need anything more than a simple bash script, you just need it backgrounded/daemonized). Take this example of a daemon that copies any new files created in a directory to another place on the filesystem

#!/bin/bash

##
## Shell Daemon For: Backup /root/
## (poorly coded, quick and dirty, example)
##

PIDFILE="/var/run/rootmirror.pid"
LOGFILE="/log/log/rootmirror-%Y-%m-%d.log"
NOHUP="/usr/bin/nohup"
CRONOLOG="/usr/bin/cronolog"
case $1 in
start)
    exec 0>&- # close stdin
    exec 1>&- # close stdout
    exec 2>&- # close stderr
    $NOHUP $0 run | $CRONOLOG $LOGFILE >> /dev/null &
    ;;
stop)
    /bin/kill $(cat $PIDFILE)
    ;;
run)
    pgrep -f "$0 $1" > $PIDFILE
    while [ true ]; do
        event=$(inotifywait -q -e close_write --format "%f" /root/)
        ( cp -v "/root/$event" "/var/lib/rootmirror/$event" )&
    done
    ;;
*)
    echo "$0 [ start | stop ]"
    exit 0
    ;;
esac

One especially nice detail here is that this wont hang while exiting your SSH session after you start it up (a big pet peeve of mine).

Throttle your Threads…

Lets say you want to run some command, such as /bin/long-command on a set of directories. And you have a lot of directories. You know it’ll take forever to complete serially, so you want to cook up a way to run these commands in parallel. You know the server CAN handle more than one command at once, but you have no idea how many it can handle without keeling over, and you have thousands of commands to run. Running them all at once backgrounded will kill the system for sure. You COULD try and stagger them and let the delay in overlap be a natural throttle, but sometimes the command completes in one minute and sometimes in 10, so thats not a good idea either. So you decide it would be best to set a process concurrency limit. But what if you set that limit too low? too high? restarting in the middle would be bad… you COULD make some sort of completed log and build into your script a skip for completed files, but why? that doesnt seem so elegant. Your car is good at handling variable speed allowances… it goes fast when you say and slow when you say… maybe we can give a simple bash script a gas pedal? That just might work!

echo '5' > /tmp/threads
for i in $(fileroot/*); do  while [ $(pgrep long-command) -ge $(cat /tmp/threads) ]

do

sleep 1

done

( /bin/long-command $i )&

sleep 1

done

Now you can speed it up and throttle it back by adjusting the integer value inside /tmp/threads.

“It was the little old server from Pasadena…”