We use varnish for a lot of our file caching needs, and recently we figured out how to do something rather important through a combination of technologies. Imagine you have backend servers generating dynamic content based on user input. So your users do something that fits the following categories:
- is expensive to generate dynamically, and should be served from cache
- many requests come in for the same objects, bandwidth should be conserved
- doesnt change very often
- once changed needs to take effect quickly
Now wish varnish we’ve been using the Expires header for a long time with great success, but for this we were having no luck. If we set the expires header to 3 weeks, then clients also cache the content for 3 weeks (violating requirement #3.) We can kill the Expires header in varnish at vcl_deliver, but then clients don’t cache at all (#2.) We can add Content-Control, overwrite the Age (otherwise reported Age: will be greater than max-age), and kill the Expires headers in the same place, but this isn’t pretty, and seems like a cheap hack. Ideally we could rewrite the Expires header in varnish, but that doesn’t seem doable.
So what we ended up doing, was header rewriting at the load balancer (nginx.) inside our location tag we added the following:
proxy_hide_header Age;
proxy_hide_header Expires;
proxy_hide_header Cache-Control;
add_header Source-Age $upstream_http_Age;
expires 300s;
Now nginx setsa proper Cache-Control: and Expires: headers for us, disregarding what varnish serves out. Web clients dont check back for 5 minutes (reusing the old object) and varnish can cache until judgment dat because we get wild card invalidation
Isn’t technology fun?!
This is something that has always annoyed me about bash scripts… The fact that it’s difficult to run
/path/to/script.sh --foo=bar -v -n 10 blah -one='last arg'
So I decided to write up a bash function that let me easily (once the function was complete) access this type of information. And because I like sharing, here it is:
#!/bin/bash
function getopt() {
var=""
wantarg=0
for (( i=1; i< =$#; i+=1 )); do
lastvar=$var
var=${!i}
if [ "$var" = "" ]; then
continue
fi
echo \ $var | grep -q -- '='
if [ $? -eq 0 ]; then
## -*param=value
var=$(echo \ $var | sed -r s/'^[ ]*-*'/''/)
myvar=${var%=*}
myval=${var#*=}
eval "${myvar}"="'$myval'"
else
echo \ $var | grep -E -q -- '^[ ]*-'
if [ $? -eq 0 ]; then
# -*param$
var=$(echo \ $var | sed -r s/'^[ ]*-*'/''/)
eval "${var}"=1
wantarg=1
else
echo \ $var | grep -E -- '^[ ]*-'
if [ $? -eq 0 ]; then
# the current one has a dash, so cannot be
# the argument to the last parameter
wantarg=0
fi
if [ $wantarg -eq 1 ]; then
# parameter argument
val=$var
var=$lastvar
eval "${var}"="'${val}'"
wantarg=0
else
# parameter
if [ "${!var}" = "" ]; then
eval "${var}"=1
fi
wantarg=0
fi
fi
fi
done
}
OIFS=$IFS; IFS=$(echo -e "\n"); getopt $@; IFS=$OIFSnow at this point (assuming the above command line parameter and script) I should have access to the following variables: $foo (“bar”) $v (1) $n (10) $blah (1) $one (“last arg”), like so:
OIFS=$IFS; IFS=$(echo -e "\n"); getopt $@; IFS=$OIFS
echo -e "
foo:\t$foo
v:\t$v
n:\t$n
blah:\t$blah
one:\t$one
"
You might be curious about this line:
OIFS=$IFS; IFS=$(echo -e "\n"); getopt $@; IFS=$OIFS
IFS is the variable that tells bash how strings are separated (and mastering its use will go a long way towards enhancing your bash scripting skills.) Anyhow, by default IFS=" " which normally is OK, but in our case we dont want "last arg" to be two seperate strings, but one. I cannot put the IFS assignment inside the function because by that point bash has already split the variable, it needs to be done at a level of the script in which $@ has not been touched yet. So I store the current IFS variable in $OIFS (Old IFS) and set IFS to a newline character. After running the function we reassign IFS to what it was beforehand. This is because I dont know what you might be doing with your IFS. There are lots of reasons you might have already assigned it to something else, and I wouldnt want to break your flow. So we do the polite thing.
And in case the above gets munged for some reason you can see the plain text version here: bash-getopt/getopt.sh
Anyways, hope this helps someone out. If not it's still here for me when *I* need it 