Setting up a Content Caching Server

I love the varnish reverse proxying software: it's very fast, it's very scalable and it lets you perform magic with low end servers which would normally require you to rent expensive hardware costing you thousands of dollars. One of the single best parts of varnish is its wonderful and surprisingly powerful VCL configuration language which is compiled into C and then assembler by varnish - this makes it amazingly quick.

One of the very simplest, yet under-recognised, usages of varnish is as a dumb content caching solution which simply replicates the content of another website. This can allow you to spread the load off of one master website onto a series of round-robin slaves, or it could be used to provide a geo-localised copy of a server such that each region gets much lower page load latencies. This particular example is orientated around Plone, so the cookies it checks for are the Plone login ones - you may need to change these for your particular CMS.

Anyway after an apt-get install varnish, and assuming that you're on Ubuntu or Debian, do nano /etc/default/varnish and replace the uncommented part with something like this:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl -u varnish -g varnish \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

This simply puts varnish on port 80. You may wish to adjust the size of the cache - keep it well under the guaranteed RAM in your VPS unless you like stuttering. Oh, and don't exceed 1.5G if your VPS is 32 bit .

After that do nano /etc/varnish/default.vcl and something similar to:

backend default {
        .host = "europe1.nedproductions.biz";
        .port = "80";
        .first_byte_timeout = 300s; /* varnish v2.0.3 or later only */
        .probe = {
                .url = "/";
                .timeout = 1s;
                .interval = 60s;
                .window = 1;
                .threshold = 1;
        }
}

/* Only permit cluster to purge files from cache */
acl purge {
        "europe1.nedproductions.biz";
        "usa1.nedproductions.biz";
        "localhost";
}

sub vcl_recv {
        /* Before anything else we need to fix gzip compression */
        if (req.http.Accept-Encoding) {
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$")
                {
                        # No point in compressing these
                        remove req.http.Accept-Encoding;
                } else if (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } else if (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        # unknown algorithm
                        remove req.http.Accept-Encoding;
                }
        }

        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                /* Always purge by URL rather than going via vcl_hash
                   as it hashes other factors which break purging */
                purge_url(req.url);
                error 200 "Purged";
        }

        if (req.request != "GET" &&
                req.request != "HEAD" &&
                req.request != "PUT" &&
                req.request != "POST" &&
                req.request != "TRACE" &&
                req.request != "OPTIONS" &&
                req.request != "DELETE") {
                /* Non-RFC2616 or CONNECT which is weird. */
                pipe;
        }
        if (req.request != "GET" && req.request != "HEAD") {
                /* We only deal with GET and HEAD by default */
                pass;
        }
        if (req.http.Cookie) {
                # We only care about the "__ac.*" cookies, used for
                # authentication and special persistent p_* cookies.
                if (req.http.Cookie ~ "__ac.*" ) {
                        pass;
                }
                # Else strip all cookies
                remove req.http.Cookie;
        }
        if (req.http.If-None-Match) {
                pass;
        }
        if (req.url ~ "createObject") {
                pass;
        }

        /* Keep serving if haproxy goes down (ie; Plone is being serviced). Haproxy
        will return a 503 error page if Plone goes down for us */
        if (req.backend.healthy) {
                set req.grace = 120s; /* Only enable if you don't mind slightly stale content */
        } else {
                set req.grace = 24h;
        }

        lookup;
}

sub vcl_hit {
        if (req.request == "PURGE") {
                purge_url(req.url);
                error 200 "Purged";
        }
}
sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache";
        }
}
sub vcl_fetch {
        set obj.grace = 24h; /* Keep at longest used in vcl_recv */
        set obj.http.Via = "usa1.nedproductions.biz";

        if (req.url ~ "\.svgz$") {
                # Add a Content-Encoding to match compressed SVG
                set obj.http.Content-Type = "image/svg+xml";
                set obj.http.Content-Encoding = "gzip";
                remove obj.http.Content-Length;
                set obj.ttl = 86400s;
                set obj.http.Cache-Control = "max-age=3600";
                deliver;
        }
        if (obj.http.Set-Cookie) {
                pass;
        }
        if (req.http.Authorization && !obj.http.Cache-Control ~ "public") {
                pass;
        }
        /* Only use this if you wish to override Plone's CacheFu */
        if (obj.ttl < 3600s) {
                if (obj.http.Cache-Control ~ "(private|no-cache|no-store)") {
                        set obj.ttl = 60s; /* Caching everything anonymous for 60s is handy for being slashdotted :) */
                } else {
                        set obj.ttl = 3600s;
                }
        }
}

sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {" <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html>
  <head>
    <title>"} obj.status " " obj.response {"</title>
  </head>
  <body>
    <div style="background-color:yellow;">
      <h1>This website is unavailable</h1>
      <p>If you are seeing this page, either maintenance is being performed
      or something really bad has happened. Try returning in a few minutes.</p>
      <h2>Error "} obj.status " " obj.response {"</h2>
      <p>"} obj.response {"</p>
      <h3>Guru Meditation:</h3>
      <p>XID: "} req.xid {"</p>
      <address>
         <a href="http://www.nedproductions.biz/">ned Productions Ltd.</a>
      </address>
    </div>
    <div style="position:fixed;top:0;left:0;width:100%;height:100%;z-index:-1;">
    <img alt="" src="http://www.nedproductions.biz/static/BBCTestCard.jpg" style="width:100%;height:100%" /></div>
  </body> </html> "};
    return (deliver);
}

Obviously you'll need to replace the backend with your own - however remember that varnish can take multiple backends and it doesn't really care if they belong to you or not, so you can do such magic as mount parts of other websites as subdirectories of your own website (and cached to boot!) which is a real boon for AJAX programming as it avoids cross-site scripting issues.

What the above config does is to simply cache the entire backend content on the local server. We make use of the grace feature which allows varnish to serve stale content - this is where varnish will immediately serve the cached content whether it is fresh or not and then initiates an asynchronous refresh from the backend. If the backend returns a 302 Not Modified, or new content, the cache entry is reset  as fresh for the next visitor and will expire whenever its Time To Live (TTL) does. The grace factor is set according to the backend's healthiness - if it is responding to requests then varnish will only return cached content for thirty seconds after it has requested a new copy from the backend, and should that request not be answered within thirty seconds then the cached copy is deleted. If it is not healthy then the grace period is extended to twenty-four hours e.g. if the backend goes down for an extended time.

The idea through these settings is that the cache keeps as close as possible to emulating the backend, so if the backend is working but being funny then the cache ought to replicate that funniness instead of smoothing out the kinks.

I have some helpful fixes in these which I have discovered through experience. One is to harmonise Content-Encoding and delete Cookies so the varnish cache doesn't end up with multiple copies of the same data - it hashes both the Content-Encoding and Cookies, so any minor difference at all will cause data duplication which is inefficient. I have the possibility for the cache to be purged, so in Plone you can add a second purge destination in CacheFu such that when Plone purges the primary varnish cache it also purges the dumb cache. One could actually have the primary varnsh cache do this for you by trapping purge requests and firing them at a backend which is actually your dumb cache - this has the convenience of extra transparency and less config hassle in Plone.

Lastly I have a hack in here to remedy compressed SVGZ not working properly in some browsers - I simply retype to SVG and set gzip-encoded. And the last part forces a cache Time-To-Live of one hour if something is set as cacheable but is to be cached for less than one hour, or if it's uncached then it is set to sixty seconds which makes one hell of a difference if you ever get DDoS'd or Slashdotted.

Note that authenticated sessions are ALWAYS passed through unmodified to the backend. This is a dumb cache, so really it's only for anonymous visitors. Most of your scalability issues will always tend to stem from too many logged in users anyway on most websites.