I’m working with Kyle Larson on a project to help Jobster capture and analyze large volumes of data about categorized job impressions and clickthroughs, and wanted to share a useful building block we’ve encountered along the way.   

Dave Nash from our ops team introduced us to Syslog-ng, a drop in replacement for the standard linux syslog daemon. From the Freshmeat project page:

syslog-ng, as the name shows, is a syslogd replacement, but with new functionality for the new generation. The original syslogd allows messages only to be sorted based on priority/facility pairs; syslog-ng adds the possibility to filter based on message contents using regular expressions. The new configuration scheme is intuitive and powerful. Forwarding logs over TCP and remembering all forwarding hops makes it ideal for firewalled environments.

The machine creating the log entry only needs to send it over the network where it’s bufferred and eventually logged on a centralized syslogging server.  The buffering means the web server doesn’t have to wait on a disk or database. Syslog-ng also support load balancing and forwarding if the log traffic exceeds the capacity of a single machine. 

Syslog-ng configuration options allow the log entries to be directed to a variety of destinations (files, named pipes, etc.) based on a fairly rich pattern matching system. One such destination for log entries is mysql by way of a daemon called metricbot, written by Andrew. Metricbot listens on a named pipe to log entries sent by syslog-ng and writes them to a structured mysql database.  Assuming the database can keep up with the insert rate of events, this gives us near real-time import of log entries into the database, without slowing down the rest of the system when the database can’t keep up.

For our purposes, we don’t mind losing log entries in a crash. Syslog-ng allows you to tune how frequently log entries are flushed to disk but doesn’t provide any absolute guarantees that entries will be preserved.