Pure VCL cookie-based sticky sessions in Varnish 2.1

Some time ago I decided to drop Apache httpd from one of my setups. The httpd was no longer used for anything but mod_jk, which only did load balancing (with sticky sessions) between multiple clusters of Tomcat application servers. mod_jk is a particularly nasty kludge and there was no reason for keeping that layer of load balancers – Varnish does LB well enough.

The problem is Varnish 2.1.x does not support cookie-based stickiness. While it’s perfectly possible to implement it in C as a director, I wanted to try a pure VCL solution. One caveat is that req.backend can’t be set from variables in VCL – you have to explicitly enumerate all backends.

The algorithm is similar to mod_jk behavior:

  1. extract app server hostname from the JSESSIONID cookie as set by Tomcat: JSESSIONID=session_id.server_hostname

  2. If a hostname is present and matches one of the app servers, use it as the backend, otherwise use the main director (with all app servers).

  3. If a single backend is set and it’s not healthy, use a cluster-specific director (with all app servers in the same session replication cluster as that backend). The request will go to another app server in the same cluster.

  4. If the cluster-specific director is not healthy (all backends sick), fall back to the main director.

An example VCL snippet:

sub get_lb_worker_from_cookie {
        if (req.http.Cookie ~ "JSESSIONID=") {
          set req.http.X-JS-LBC = regsub(req.http.Cookie,
                                        "^.*?JSESSIONID=([^;]*);*.*$", "\1");

          set req.http.X-JS-LBAS = regsub(req.http.X-JS-LBC,
                                         "^.+\.(.+)$", "\1");
        }

        log "JSESSIONID cookie: " req.http.X-JS-LBC;
        log "LB AS from JSESSIONID: " req.http.X-JS-LBAS;

        unset req.http.X-JS-LBC;
}

sub set_lb_worker_from_cookie_site1 {
        if (req.http.X-JS-LBAS ~ "^as1$") {
          set req.backend = site1_as1;
        } else if (req.http.X-JS-LBAS ~ "^as2$") {
          set req.backend = site1_as2;
        } else if (req.http.X-JS-LBAS ~ "^as3$") {
          set req.backend = site1_as3;
        } else if (req.http.X-JS-LBAS ~ "^as4$") {
          set req.backend = site1_as4;
        } else {
          set req.backend = site1_main_director;
        }

        if (!req.backend.healthy
            && req.backend != site1_main_director) {
          log "Backend " req.backend " sick, forcing to cluster director";

          if (req.http.X-JS-LBAS ~ "^as[12]$") {
            set req.backend = site1_cluster1_director;
          } else if (req.http.X-JS-LBAS ~ "^as[34]$") {
            set req.backend = site1_cluster2_director;
          }
        }

        if (!req.backend.healthy
            && req.backend != site1_main_director) {
          log "Cluster director " req.backend " sick, forcing to main";
          set req.backend = site1_main_director;
        }

        unset req.http.X-JS-LBAS;

        log "Backend set to: " req.backend;
}

sub vcl_recv {
        if (req.url ~ "^/site1/sessions_required_here/") {
          call get_lb_worker_from_cookie;
          call set_lb_worker_from_cookie_site1;
        }
}
This entry was posted in IT and tagged . Bookmark the permalink.

7 Responses to Pure VCL cookie-based sticky sessions in Varnish 2.1

  1. Per Buer says:

    Hi.
    The problem is Varnish 2.1.x does not support cookie-based stickiness
    Actually it does. If you use the client director and set client.identity to (in your case) req.http.X-JS-LBC the same session will always end up on the same backend.

    I’ve tried to clarify the documentation on this now.

    • tgr says:

      You’re partly right – the client director can hash on a header and make backend decisions based on that hash. However, cookie-based stickiness (as used by JK, haproxy and countless other solutions) usually means extracting the backend hostname from a cookie. Therefore, client.identity is actually quite useless in this particular case. Why?

      1. client.identity won’t automagically make the request go to a specific backend. It will only make sure that the session always goes to the same backend (by manipulating a SHA256 hash of client.identity).
      And that’s a big difference, as in this case it’s actually the backend, not the load balancer that determines the destination host (by setting the cookie). I need to explicitly set req.backend based on the cookie contents anyway.

      2. as you can see, this is a bit more than what the client director does. When a backend set from the cookie is sick, the director would just pick another (weighted) backend from the whole set. In this solution only a backend belonging to a specific session replication cluster will be selected – fallback to main director will happen only if the entire cluster is sick. You may have several independent clusters serving the same site.

      Of course, this is not the only way to do this. :-)

  2. cosmih says:

    Hi,

    I am trying to use your mod_jk like sticky session with varnish and I can see that only one tomcat is handling all requests.
    I am wondering if to have the same symptoms in your setup.

    Below is my adaptation of your code:


    backend node01 {
    .host = "node01";
    .port = "8080";
    .probe = { .url = "/helloloadbalancer/index.jsp"; .interval = 5s; .timeout = 2s; .window = 5; .threshold = 3; }
    }

    backend node02 {
    .host = "node02";
    .port = "8080";
    .probe = { .url = "/helloloadbalancer/index.jsp"; .interval = 5s; .timeout = 2s; .window = 5; .threshold = 3; }
    }

    director RRnodes round-robin {
    { .backend = node01; }
    { .backend = node02; }
    }

    sub get_lb_worker_from_cookie {
    if (req.http.Cookie ~ "JSESSIONID=") {
    set req.http.X-JSESSIONID = regsub(req.http.Cookie, "^.*?JSESSIONID=([^;]*);*.*$", "\1");
    set req.http.X-JSESSIONID-NODE = regsub(req.http.X-JS-LBC, "^.+\.(.+)$", "\1");
    }
    unset req.http.X-JSESSIONID;
    }

    sub set_lb_worker_from_cookie {
    if (req.http.X-JSESSIONID-NODE ~ "^node01$") {
    set req.backend = node01;
    } else if (req.http.X-JSESSIONID-NODE ~ "^node02$") {
    set req.backend = node02;
    } else {
    set req.backend = RRnodes;
    }

    if (!req.backend.healthy && req.backend != RRnodes) {
    if (req.http.X-JSESSIONID-NODE ~ "^node01$") {
    set req.backend = node02;
    } else if (req.http.X-JSESSIONID-NODE ~ "^node02$") {
    set req.backend = node01;
    }
    }
    unset req.http.X-JSESSIONID-NODE;
    }

    sub vcl_recv {
    call get_lb_worker_from_cookie;
    call set_lb_worker_from_cookie;
    }

    The setup is something like this:
    - one varnish setting in front of two tomcats
    - the two tomcats configured with the backup manager for session replication

    I am using jmeter to send traffic to varnish and I see that all requests are sent to only one tomcat (not always the same)

    • tgr says:

      No, I don’t have this problem with my setup. Sounds suspicious – if all requests get sent to one Tomcat, even when there’s no JSESSIONID cookie, the problem may not even be in the VCL, but with the Tomcats.

      When you reproduce the problem, please paste (on pastebin, not here) the following data:

      – debug.health from the CLI,
      – full varnishlog output (please re-add the logging statements from my example to your VCL first).

      Post the link here and I’ll take a quick look at it.

  3. Andrea Campi says:

    I came back to your blog to re-read another article (http://monolight.cc/2011/04/content-authorization-with-varnish/) and stumbled upon this.

    What’s funny is that I implemented basically the same last night, with a vmod for Varnish 3.0: https://github.com/zephirworks/libvmod-redis (it’s one of the examples).

    My solution is mostly a toy, while your looks battle-tested. OTOH my approach is extremely flexible: a backend could be writing to Redis too, controlling where users go.

Comments are closed.