Using Flot.js with the Digg Streaming API

Last year Digg released our Streaming API, but there still aren't too many tutorials out there, especially on using it for JSONP. I've also wanted to experiment with more dynamic tutorials which run JavaScript examples in-line with the article. So, time to scratch some itches.

In this tutorial we'll look at using Flot, jQuery and the Digg Streaming API to create dynamic graphs.

Install & Setup Flot

First we'll need to download Flot (which comes with a copy of jQuery).

curl http://flot.googlecode.com/files/flot-0.7.tar.gz > flot-0.7.tar.gz
tar -xvzf flot-0.7.tar.gz

Next, create an HTML page in the same directory which these contents:

<html>
<head>
<title>Flot &amp; Digg Streaming API</title>
</head>
<body>
<div id="placeholder" style="width:600px;height:300px"></div>
<!--[if lte IE 8]>
<script type="text/javascript" src="excanvas.min.js"></script>
<![endif]-->
<script type="text/javascript" src="flot/jquery.min.js"></script>
<script type="text/javascript" src="flot/jquery.flot.min.js"></script>
<script>
$(document).ready(function() {
  var data = [[[0,0],[1,1]], [[0,1],[1,0]]]
  $.plot($("#placeholder"), data, { yaxis: { max: 1 } });
});
</script>
</body>
</html>

If everything has downloaded correctly, you should see this example:

Digg Streaming API

The Digg Streaming API makes it possible to see all Diggs, comments and submission occuring on Digg.com in real-time (generally a second or two after they occur, but occasionally a bit longer).

It is available at

http://services.digg.com/2.0/stream

and takes a variety of parameters. For our purposes we'll want to use its JSONP capabilities to perform requests from across a different domain. To do so we'll need to specify format=javascript as well as callback=some_function. All put together the URL will look like:

http://services.digg.com/2.0/stream?types=comment
                                   &format=javascript
                                   &callback=streaming_callback

With that knowledge in mind, let's create a simple script which retrieves recent comments and renders the them. Let's use this HTML fragment as the template for rendering comments:

<div id="digg-comment">
  <span id="digg-author">some author</span>
  <p id="digg-body">some text</span>
</div>

Performing JSONP requests to the Digg Streaming API requires a bit of extra tact, it isn't enough to simply use $.getJSON("http://..."), you'll actually need to write a simple wrapper looking like:

var get_stream = function(callback_func, types) {
  if (types == undefined) {
    types = "digg,submission,comment";
  }
  var domain = "services.digg.com",
    format = "format=javascript",
    types = "types=comment";
  var url = "http://"+domain+"/2.0/stream?"+types+"&"+format;
  $.ajax({url:url,
          dataType:"jsonp",
          crossDomain:true,
          cache:true,
          success:callback_func});
};

More specifically the $.getJSON utility function for JSONP won't work because it attempts to pass extra parameters which the Digg Streamin API doesn't know about and will reject (specifically jQuery passes the _ parameter with the current time as its value as an attempt to bypass caching).

Armed with get_stream we can whip together the code for populating the HTML comment pretty quickly.

var get_comment_stream(callback_func) {
    get_stream(callback_func, "comment");
}

var stream_callback = function(data) {
  var fullname = (data.user.fullname) ? data.user.fullname : data.user.name,
      txt = data.text.substring(0, 45) +
               "<a href=\"" +
               data.item.href +
               "\">&hellip;</a>",
      author_str="<a href=\"http://digg.com/" +
               data.user.name+"\">" +
               fullname +
               "</a>";
  $("#digg-author").html(author_str);
  $("#digg-body").html(txt);
  // schedule a request for retrieving the next comment
  setInterval("get_comment_stream(stream_callback)", 100);
}
$(document).ready(function() {
  get_comment_stream(stream_callback);
}

Once you put those pieces together, it looks a bit like this (well, the version running here has a ten second delay between fetches to avoid wasting too much of your bandwidth, and also stops after sixty requests):

some author

some comment

Okay, so now that we've put together an example of using the Streaming API and have Flot.js installed, let's take a stab at using them together.

Dynamicly Updating Flot

For our example of updating Flot with real-time data from the Streaming API, let's create a chart which will track the number of submissions, comments and Diggs each minute over time.

As usual, the HTML will be trivial.

<div class="js-example" id="exampletwo"></div>

The JavaScript will be a bit more complex this time, but the JSONP aspect will be handled by reusing the get_stream function defined above.

var chart_data_by_minute,
    chart_data,
    options = {
      lines: { show: true },
      points: { show: true },
      xaxis: { tickDecimals: 0, tickSize: 1 },
      yaxis: { tickDecimals: 0, tickSize: 1 },
      legend: { backgroundOpacity: 0, position: "se" }
    },
    graph_callback = function(data) {
      var curr_minute = (new Date()).getMinutes();
          curr_data = (chart_data_by_minute[curr_minute]) ?
                      chart_data_by_minute[curr_minute] :
                      {'submission':0, 'digg':0, 'comment':0},
          minutes = [],
          series = { "submission":[], "comment":[], "digg":[] };

      // update historical data with latest event
      curr_data[data.type]++;
      chart_data_by_minute[curr_minute] = curr_data;

      // find all minutes in historical data and sort them
      $.each(chart_data_by_minute, function(key, val) {
        minutes.push(parseInt(key));
      });
      minutes.sort();

      // generate event-type series for flot
      $.each(minutes, function(index, minute) {
      $.each(["submission", "comment", "digg"], function(index, x) {
        series[x].push([minute, chart_data_by_minute[minute][x]]);
      })});

      // the real magic is here, where we recreate the series
      // every time we get a new event and rerender the graph
      chart_data = [{label:"submission", data:series["submission"]},
                    {label:"comment", data:series["comment"]},
                    {label:"digg", data:series["digg"]}];
      $.plot($("#exampletwo"), chart_data, options);

      // schedule a request for retrieving the next comment
      setInterval("get_stream(graph_callback)", 100);
    };

// get the party started
$(document).ready(function() {
  chart_data_by_minute = {};
  chart_data = [];
  get_stream(graph_callback);
});

Put it together, and you have a real-time graph of comments, Diggs and submissions occuring right now on Digg.

Waiting a

Note that this really isn't an accurate reflection of the total quantity of activity happening on Digg because of the way we're doing the intervals and also because we're missing events. The single-threaded and callback nature of JSONP, along with the lack of support for cursors and bulk response type for all activity since the last cursor mean that you'll need to investigate establishing a persistent to the Digg Streaming API if you really want to see everything. (If you're interested, take a look at this repository on Github for Python examples.)

Certainly not the prettiest of graphs, or the most useful of examples, but working directly with JavaScript examples embedded in the tutorial was certainly an interesting experience, and hopefully slightly more useful than simply seeing screenshots.

As always, hope this was useful, I apologize for any awkwardness in the new format, and let me know your feedback!

All Rights Reserved, Will Larson 2007 - 2014.