I had a couple of interviews long ago which asked me to implemented a last recently used (LRU) cache. A cache itself can simply be implemented using a hash table, however adding a size limit gives an interesting twist on the question. Let’s take a look at how we can do this.

Last Recently Used Cache Eviction

To accomplish cache eviction we need to be easily able to:

• query the last recently used item
• mark an item as the most recently used item

A linked list allows for both operations. Checking for the least recently used item can just return the tail. Marking an item as recently used can be simply removing it from its current position and moving it to the head. The missing puzzle piece is finding this item in the linked list quickly.

Hash tables to the rescue

Looking into our data structure toolbox, hash tables allow us to easily index an object in (amortized) constant time. If we create a hash table from key -> list node, we can find the most recently used node in constant time. The converse is true in that we can also still check for the existence (or lack-there-of) in constant time as well.

After looking up the node we can then move it to the front of the linked list to mark it as the most recently used item.

The Java shortcut

Sometimes knowing less common data structures from the standard library of various programming languages can prove to be of help. Given this hybrid data structure we would have to implement a hash table on top of a linked list. However Java already provides this for us in the form of a LinkedHashMap! It even provides an overridable eviction policy method (removeEldestEntry docs). The only catch is that by default the linked list order is the insertion order, not access. However one of the constructor exposes an option use the access order instead (docs).

Without further ado:

import java.util.LinkedHashMap;
import java.util.Map;

public LRUCache<K, V> extends LinkedHashMap<K, V> {
  private int cacheSize;

  public LRUCache(int cacheSize) {
    super(16, 0.75, true);
    this.cacheSize = cacheSize;
  }

  protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
    return size() >= cacheSize;
  }
}

Previously I had mentioned how to create a simple hello world application in c++ with fcgi. However this simple application doesn’t allow for any inputs. This post we’ll show how retrieve the request uri accessed as well as the POST body content.

Getting the URI and POST content

Looking at the previous app, the output was a very simple hello, world. We will add functionality which spits back out the path that we hit the server with. Our desired functionality looks like:

> curl -d "Chris" http://localhost/foobar
<html>
  <head>
    <title>Hello, world!</title>
  </head>
  <body>
    <h1>Hello Chris from /foobar !</h1>
  </body>
</html>

Notice that we respond with Hello <post content> from <request uri>. In order to do this we will have to capture the request uri and the post content. Let’s start off with the easier one which is the request uri. The request uri is buried within the FCGX_Request’s environment parameters. We can extract it using the FCGX_GetParam function like so:

const char * uri = FCGX_GetParam("REQUEST_URI", request.envp);

Note how we get the param ‘REQUEST_URI’ as named in our nginx configuration.

The slightly trickier part is getting the request content. We will write a helper function to do this for us. We start by determining the CONTENT_LENGTH of the response. If missing we assume it to be 0 since it is not safe to read from the fcgi request if it is missing. We limit the content length to ensure we don’t run out of memory and set it to an arbitrary high value (STDIN_MAX). If we can’t parse the content length we will just assume the max number of bytes.

This starts the function off with:

string get_request_content(const FCGX_Request & request) {
    char * content_length_str = FCGX_GetParam("CONTENT_LENGTH",
                                               request.envp);
    unsigned long content_length = STDIN_MAX;

    if (content_length_str) {
        content_length = strtol(content_length_str,
                                &content_length_str,
                                10);

        if (*content_length_str) {
            cerr << "Can't Parse 'CONTENT_LENGTH='"
                 << FCGX_GetParam("CONTENT_LENGTH", request.envp)
                 << "'. Consuming stdin up to " << STDIN_MAX << endl;
        }

        if (content_length > STDIN_MAX) {
            content_length = STDIN_MAX;
        }
    } else {
        // Do not read from stdin if CONTENT_LENGTH is missing
        content_length = 0;
    }
    ...
}

Now we have the content length we may actually read from standard input, up to the content_length number of characters.

...
    char * content_buffer = new char[content_length];
    cin.read(content_buffer, content_length);
    content_length = cin.gcount();
...

As a caveat of mod_fastcgi (currently not being used but nevertheless it can’t hurt to be a bit defensive) is that we need to consume the rest of stdin for the request because it doesn’t handle the request correctly otherwise.

...
    // ignore() doesn't set the eof bit in some versions of glibc++
    // so use gcount() instead of eof()...
    do cin.ignore(1024); while (cin.gcount() == 1024);
...

Finally we can package up the content into a string and free the allocated memory for our content_buffer.

...
    string content(content_buffer, content_length);
    delete [] content_buffer;
    return content;
}

End Result

Adding some simple changes to call this function and get the uri we end up with a fcgi program that takes http input! Download final version here (main_v2.cpp)

To run this code we recompile with main_v2.cpp and spawn the fcgi process again (link).

If we use curl to post to the site we get:

> curl -d "chris" http://localhost/foobar
<html>
  <head>
    <title>Hello, World!</title>
  </head>
  <body>
    <h1>Hello chris from /foobar !</h1>
  </body>
</html>

And a simple fcgi app in c++ is done!

Prerequisites

This post will assume some familiarity with C++. The setup instructions also assume the server to deploy to is Ubuntu. This tutorial can be applied to other operating systems but I do not provide instructions on installing libraries on other operating systems.

Why FCGI?

FCGI (FastCGI) is a protocol in which web applications can talk to a web server to serve web requests. Long story short FCGI was developed to solve the scalability shortcomings that CGI. Typical CGI applications will fork a new process to deal with the request. While this is convenient, it is terribly inefficient due to the overhead of creating and terminating a process for each web request. The idea with FCGI is that you can spawn a static number of processes to handle web requests. This allows for the overhead of creating and terminating processes to be eliminated as resources for request handling can be reused within one process handling multiple requests.

FCGI easily allows for our application to accept web requests by interacting with stdio. There are alternatives such as implementing your own http server using tools like boost. Simple interface and robust are two of the key points of using fcgi for developing web applications in C++.

There are more reasons to use fcgi but you can check out the official site and the wikipedia page for more info.

Installation

We will need to install the libfcgi++ library, a web server (nignx in our case) and spawn-fcgi to run the fcgi app. We’ll also install curl to help test later on.

sudo apt-get install libfcgi-dev
sudo apt-get install spawn-fcgi
sudo apt-get install nginx
sudo apt-get install curl

The Code

Most of this code is basically a copy paste re-interpretation of the example echo-cpp.cpp from fastcgi.com

#include <iostream>
#include "fcgio.h"

using namespace std;

int main(void) {
    // Backup the stdio streambufs
    streambuf * cin_streambuf  = cin.rdbuf();
    streambuf * cout_streambuf = cout.rdbuf();
    streambuf * cerr_streambuf = cerr.rdbuf();

    FCGX_Request request;

    FCGX_Init();
    FCGX_InitRequest(&request, 0, 0);

    while (FCGX_Accept_r(&request) == 0) {
        fcgi_streambuf cin_fcgi_streambuf(request.in);
        fcgi_streambuf cout_fcgi_streambuf(request.out);
        fcgi_streambuf cerr_fcgi_streambuf(request.err);

        cin.rdbuf(&cin_fcgi_streambuf);
        cout.rdbuf(&cout_fcgi_streambuf);
        cerr.rdbuf(&cerr_fcgi_streambuf);

        cout << "Content-type: text/html\r\n"
             << "\r\n"
             << "<html>\n"
             << "  <head>\n"
             << "    <title>Hello, World!</title>\n"
             << "  </head>\n"
             << "  <body>\n"
             << "    <h1>Hello, World!</h1>\n"
             << "  </body>\n"
             << "</html>\n";

        // Note: the fcgi_streambuf destructor will auto flush
    }

    // restore stdio streambufs
    cin.rdbuf(cin_streambuf);
    cout.rdbuf(cout_streambuf);
    cerr.rdbuf(cerr_streambuf);

    return 0;
}

Breaking code dump down we see that the de-facto interface to stdio (cout/cin/cerr) are hijacked to serve the purposes of request handling. It’s possible to retrieve the iostreams back since we save the stdio stream buffers.

We then initialize the FCGX library with FCGX_Init() and initialize the request object that we share across requests for the lifetime of this process.

We then have a blocking loop accepting fcgi requests with our call to FCGX_Accept_r. The _r version calls the multi-thread safe version of the function, although this is not necessary in our single threaded program. This will cleanup the old request that was passed into FCXG_Accept_r and initialize the new request when it comes in.

We construct the fcgi stream buffers inside the loop, using RAII pattern to ensure the buffers are flushed at the end of the request processing (end of the loop)

Nginx Config

Next up we’ll setup nginx to listen on port 80 for http requests and forward those along the the fcgi process which will listen on port 8000.

The key directive is fastcgi_pass 127.0.0.1:8000 indicates that nginx should forward the fcgi request to port 8000 at localhost. You can replace the address with an upstream directive if you want to want to load balance across many processes. The rest of the fastcgi_param directives are optional and just set appropriate environment variables which get forwarded to the fcgi application.

events {
  worker_connections 1024;
}

http {
  server {
    listen 80;
    server_name localhost;

    location / {
      fastcgi_pass   127.0.0.1:8000;

      fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
      fastcgi_param  SERVER_SOFTWARE    nginx;
      fastcgi_param  QUERY_STRING       $query_string;
      fastcgi_param  REQUEST_METHOD     $request_method;
      fastcgi_param  CONTENT_TYPE       $content_type;
      fastcgi_param  CONTENT_LENGTH     $content_length;
      fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
      fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
      fastcgi_param  REQUEST_URI        $request_uri;
      fastcgi_param  DOCUMENT_URI       $document_uri;
      fastcgi_param  DOCUMENT_ROOT      $document_root;
      fastcgi_param  SERVER_PROTOCOL    $server_protocol;
      fastcgi_param  REMOTE_ADDR        $remote_addr;
      fastcgi_param  REMOTE_PORT        $remote_port;
      fastcgi_param  SERVER_ADDR        $server_addr;
      fastcgi_param  SERVER_PORT        $server_port;
      fastcgi_param  SERVER_NAME        $server_name;
    }
  }
}

Running the code

The system now comprises of three parts. One will be the executable fcgi c++ app. In order to run this we need spawn-fcgi and to listen to a port. Nginx will then forward web requests to this port, translating the http proctocol into the fast cgi protocol.

# run nginx using the provided configuration
sudo nginx -c <path to nginx.conf>

# compile hello_world
g++ main_v1.cpp -lfcgi++ -lfcgi -o hello_world

# spawn the fcgi app on port 8000 with no fork
spawn-fcgi -p 8000 -n hello_world

Then just head to the browser and enter http://localhost/ into the location bar and voilà! Hello, World! See the next part in this tutorial to see how to incorporate the request uri and the request content into this simple app.

C++? Why on Earth?

C++ seems like one of the least likely languages to write a web application in. C++ is associated with complexity, both with language design and the need to manually manage memory. Programming in C++ unleashes a whole class of bugs associated with memory corruption and memory allocation. This is a trade off for giving more power to the developer along generally being much faster than equivalent programs in languages such as Python or Ruby.

Real-time Web as a Use Case

Making a hard real-time application¹ on the web is a futile task. Unpredictable network latencies make this an impossible task. However firm real-time systems are possible². This means that our application can miss a few deadlines without a total system failure.

Deterministic timing is key when designing real-time systems. Systems do not have to be necessarily high performance as long as deadlines are met. Lack of a garbage collector ensures there are no “pauses” in the execution of the application while the garbage collector runs to reclaim unused memory. This is not to say that it is not possible to create real-time applications in managed languages³ but C++ has the advantage of not needing to worry about garbage collection freezing the program.

When might such an application come up? One instance might be if we have a web service that “guarantees”⁴ a response time. These types of services are typically internal services.

Real-Time Bidders

A good example of an external facing application that has a firm real-time requirement are real-time bidders. These servers bid on ads from ad exchanges which act as an auction house for ad real-estate on the web. Each request represents a potential ad placement. Real-time bidders (RTB) typically must respond within a set deadline (including network transfer time). For example an auction server from OpenX must receive the complete response from a RTB within 125ms. RTBs which violate this deadline repeatedly will have less chances to bid on future auctions, reducing the effectiveness of the RTB. This presents a firm real-time problem because:

• failure to meet the deadline is acceptable but quality of service is degraded – not hard real-time
• after the deadline the result (bid response) is worthless – not soft real-time

Side Note About Scheduling (or Lack Thereof)

One topic that hasn’t been covered is scheduling. Assuming the web application is running on a typical flavour of Linux, there is no sense of a real-time scheduler. Depending on the strictness of the real-time constraints it may or may not be necessary to modify the scheduling behaviour of the kernel. Since web application are usually the only main processes living on a single server (or at least a highly dominant one), modifications to scheduling may not be worth the effort versus setting a high priority on the real-time process. These are trade offs I have not run into yet because they usually are only needed under very tight and/or strict real-time deadlines.

Most of the apps I’ve built stand on the shoulder of giants. Open source projects enabled me to jump ahead of building frameworks or webservers and dive straight into building the end product. I’ve always been thankful and felt a need to give back, even in a small way. Recently (while procrastinating from exams) I’ve finally done such a thing.

I’ve recently been working on a flash card-esque site for learning kana. Using redis as the backend I proceeded to use connect-redis to implement sessions. Unfortunately this cluttered my app’s data keyspace with session data with no clear way of being able to separate the two.

Luckily enough for me redis has multiple databases to store data in separate keyspaces. All you had to do was run a select <db num> command. connect-redis even foresaw this issue and has a db option in the RedisStore constructor.

The Ugly ‘Bug’

After running the server for a while I would notice a peculiar bug. It seemed my session data kept resetting after a day or two. Inspecting the redis db I saw that my app data’s database suddenly had session data in it even though I told the RedisStore to use the session db.

Digging around I found that the node redis client doesn’t reselect db’s on reconnection. What was happening was the redis client connection was dropping randomly overnight but was able to re-connect. Good recovery, except now it’s connected to the default db 0 as per the select command docs. Fun stuff.

Working Around It

Since the issue didn’t seem conclusive at the time and this was affecting connect-redis I thought to patch it up with the following code taken from the discussion of issue 86 of node redis.

client.on("connect", function() {
  client.send_anyway = true;
  client.select(db);
  client.send_anyway = false;
});

Ugly? Yes. Necessary? Unfortunately also Yes.

The client.send_anyway dictates whether commands should be enqueued onto the offline queue when the client has not connected. Since we want to select the proper db before the offline queue is executed the send_anyway needs to be set true when issuing the select command.

I added this to my existing redis connections which used other databases and patched it to connect-redis in this pull request. It was picked up by visionmedia in no time (like 2 minutes!) and merged into connect-redis for v1.0.7.

Thoughts and More Ramblings

It was quite satisfying contributing to this library even if though I did almost no work of my own. Although this was a quick patch to ensure the connect-redis always used the db it was passed, a cleaner solution stemming within node redis would be best. Maybe I’ll take my time and invest in providing such an option in the future.

Finally setup a blog. I’ve always wanted to ramble nonsensically on my very own blog but never found the time to roll one out. Most people run the Wordpress route (or Tumblr these days), but I never really dug into Wordpress. It always seem bloated to me and I just wanted something simpler (and editable in vim). In comes Jekyll!

Jekyll is a blog aware static site generator. Paired with github pages, I was set. Github pages has the nice fact that it already supports jekyll site generation on a (specially named) repo push.

The only caveat to using Github Pages is that you can’t really run plugins like haml and sass converters. Jekyll is run with jekyll --safe which prevents haml and sass from running.

Still on the todo list:

• Comments (disqus)
• Search (tapir)
• Prettify (I find this very difficult)
• Get a Logo and Icon
• Post something real