Log Collection Server with Node.js

Recently, I've been thinking about aggregating logs from multiple machines.

Although there are undoubtedly many approaches to this problem, I decided to write a simple HTTP server which would accept logs and then create a script which would monitor files for changes and POST them to the HTTP server.

As an experiment I went ahead and did it with node.js because I hadn't done any projects with it yet, and callbacks struck me as a perfect mechanism for extending an existing command line tool (in this case piping the output of tail -F into HTTP requests to a remote server).

First, implementing the log collection server, which simply echos the received data to stdout.

var sys = require("sys"),
    http = require("http");
var record_message = function(request, msg) {
sys.puts("received: " + msg);
};
http.createServer(function (request, response) {
var content = "";
request.addListener("body", function(chunk) {
content += chunk;
});
request.addListener("complete", function() {
record_message(request, content);
response.sendHeader(200, {"Content-Type": "text/plain"});
response.sendBody("stored message");
response.finish();
});
}).listen(8000);

Next, throwing together the client, which uses a child process to wrap tail -F to monitor a log file.

var sys = require("sys"),
    http = require("http"),
    file = process.ARGV[3],
    hostname = (process.ARGV[4]) ? process.ARGV[4] : "localhost",
    port = (process.ARGV[5]) ? parseInt(process.ARGV[5]) : 8000,
    log_server = http.createClient(port, hostname);
var send_log = function(msg) {
var req = log_server.request("POST", "/",
{"Content-Length":msg.length});
req.sendBody(msg, encoding="ascii")
req.finish();
}
var monitor_file = function(filename) {
var cmd = process.createChildProcess("tail", ["-F", filename]);
cmd.addListener("output", function(data) {
send_log(data);
});
};
monitor_file(file);

Now we can test these two components:

touch log.txt
node logs.js &
node log_watcher.js log.txt &
echo "test" >> log.txt
echo "another test" >> log.txt

At this point the log server is only echoing the received logs to standard out, but it would really be preferable if they were being stored in a file. The simplest way to do this would be to just run the server like this

node logs.js >> centralized.log

but just appending to a file doesn't really allow too much flexibility. Instead, let's throw together a simple mechanism for allowing users to store the logs in arbitrary ways. We'll use appending to a file as an example backend which can later be swapped out for another mechanism.

First, replace the existing definition of record_message in logs.js:

var default = "./log_file_backend",
    backend = (process.ARGV[3]) ? process.ARGV[3] : default,
    record_message = require(backend).record_message;
/*
var record_message = function(request, msg) {
    sys.puts("received: " + msg);
};
*/

Next, we need to throw together the log_file_backend.js file, which will append data to the ./logs.received file in the same directory as the script is being run.

var sys = require("sys"),
    posix = require("posix"),
    filename = "./logs.received",
    fd = posix.open(filename,
                process.O_WRONLY | process.O_APPEND | process.O_CREAT,
                process.S_IRWXU | process.S_IRGRP | process.S_IROTH
                ).wait();
exports.record_message = function(request, msg) {
posix.write(fd, msg);
}

To store the logs in a different way, just make a new module which exports a record_message(request, msg) function and then run logs.js like this:

node logs.js mysql_backend.js
node logs.js couchdb_backend.js

The possibilities are endless. Or at least really really broad.

This is already a nice little utility, and it's actually a bit nicer than one might already realize: while it's been programmed as a fire-and-forget client, the implementation of the node.js HTTP client is such that it will queue up requests when the host isn't available and then send them all in the original order when it next attempts to send a message and discovers the host is available. (This isn't how I expected it to work--I expected to manually implement a message cache for the store-and-forward mechanism--but in this case it ends up working fairly well.)

And that's all there is to it. In less than one hundred lines we have a log collecting server with a pluggable storage backend and a store and forward client which will send changes from monitored files. After doing a few more projects I will definitely have to write up some thoughts about node.js (admittedly more than a few months late to the game), but I can already say it's quite a bit more exciting than I had realized: it's really fun to work with,

Code is available on GitHub.