Notebook

Auditory Feedback while Live Streaming

17 Feb 2011 at 1 PM

Being awesome online is missing one thing, the applause. We've solved this long time problem for Mario Marathon 4 with the use of AJAX, JSON, and a small Java Applet.

I am sure you have heard about the Mario Marathon fundraiser I host for Child's Play Charity each summer. For those of you living under a rock, or in this case Thwomp, Mario Marathon is a multi-day gaming session broadcast online. Each donation made to Child's Play during the event extends the amount of play. Now it's fourth year the event has raised $125,000 which will be used to provide video games, books, and toys to kids in children's hospitals around the world. Not bad for a few guys sitting on a couch.

A theme of the event is to provide the most interaction between us and viewers distributed around the globe. Each year we try to use technology to bridge this distance. Using services like Ustream.tv, Facebook, Twitter, and IRC chat we've established an environment where people can watch as we play and talk to us through a variety of channels. This year we wanted to add the ability to "hear" the audience react.

The goal was simple, provide viewers with buttons to associated with a few reactions: Applause, Laugh, and Boo. Clicking these buttons would trigger sound effects in our studio which would be heard by everyone involved.

Of course, there were a few limitations. We didn't want non-stop sounds spamming our broadcast, the delay had to be minimal, and the load on our web server and bandwidth had to be reasonable.

How it all works

First, users click a link on our web page to "vote" for a specific reaction to current events on our video stream. JavaScript fires off an AJAX request to the server to log their reaction along with a time stamp and IP address. The same JavaScript code prevents further reactions from being sent for 15 seconds.

Next, we setup a small JavaScript application which interfaces with a Java Applet to play sounds. The application stores a threshold value for each reaction. Periodically (~5 seconds) the system performs an AJAX query to retrieve the number of "votes" for each reaction. When the number of votes exceeds the threshold for a particular reaction a sound is played and the threshold is updated to 110% of the vote count. Over time the threshold decays back to some minimum value (maybe 10).

The result is that the number of clicks required to trigger a sound adjusts based on the number of users currently interacting with the reaction system. If we receive a huge spike of 100 "cheers" the system then requires 110 "cheers" to trigger another sound. This prevents those same 100 people from continuously triggering a sound. As the threshold decays those say 100 people will again be able to cause a cheer.

The turn around time from a click to sound is around 5 - 15 seconds. We do something, the viewer sees what we did (~5-10 second delay), clicks the reaction button, our sound app queries for new reactions (~5 second delay). In practice this time delay seems reasonable. Reactions can easily be tied to events on the stream. Tell a joke, people laugh.

We'll be testing the feature out with a large audience in the near future.

Update: June 2011 - Under heavy load storing votes in a database brought our shared server to its knees. We implemented a quick fix using memcached that solved the load issue.

Update: January 2012 - We recently rewrote this system using node.js which pushed all of the application logic onto the server. We were able to use a single node process to track the sounds and push websocket events to our "player" view. The result is a much more efficient and responsive system.