MediaMath in 4D: Mapping sound and graphics on the XYZ to interpret data

// 07.31.2014 // Data

A while back, I wrote about some of the new opportunities that data sonification can open up for monitoring and analysis. As a quick refresher, data sonification is the process of making data sound like how it looks. I’d like to share a few more examples of data sonification, one “state of the art” example from California Institute of Technology (Caltech), and one experiment from MediaMath.

MediaMath Experiment: MM3D

In 2013, I worked on a project called MM3D. It was an experiment in combining sound and graphics into one single presentation, with multiple MediaMath datasets.


The gist of the app was to show that we could map our bidding algorithm’s data (we call this algorithm “The Brain”), against a month of performance data (which ads led to which purchases, sign-ups, etc) from APAC, EMEA, and the US in a 3D environment. The Brain data was mapped to multiple axes with XYZ, plus size (the data points were spheres) each representing different data points. The performance data was shown as a cluster of bar charts.

The audio portion of the app was in the performance data. Each bar was assigned an associated pitch based on the performance ratio. The better the performance, the higher the pitch, and vice versa. Each bar plays its tone on a mouse-over, so a quick roll of the mouse across your screen immediately identifies which days were great, then a zoom in on the visuals shows you why.


“State of the Art”: Caltech’s AlloSphere

The other example I want to mention here is light-years ahead of where we currently are at MediaMath, and shines a light on where we could be heading. It’s called the AlloSphere, and it’s at Caltech.

The AlloSphere is a huge spherical room with 3D projections and 3D audio, specifically built for advanced “data perceptualization,” of which sound is an integral component. To borrow from the University of Lieden: “Data perceptualization … [is] the translation of signals and information to modalities that appeal to of the human senses. It generalizes the terms “visualization“ and “sonification“ to include all other senses.”

There is a section in this video of the Allosphere where the researchers are projecting a 4D fly-through of actual MRI data of someone’s brain. It’s a spectacular visualization.

Continue watching the video, and the speaker points out that the little wind chime sounds are plotting blood density at each point (around 2:10). She doesn’t make too much of it, but actually that is a bit of genius. Mapping blood flow to sound means a doctor can analyze blood density, while still being able to see the brain tissue. Yet another aspect of data as sound: multiple layers can exist independently, support harmonically, or create discord.

For the computer science folks in the house, Walt gave me this link to a data sonification of various sorting algorithms: Sorting Algorithms.

Data sonification is still an exploratory effort at MediaMath.  It’s a credit to the culture of the company, though, that these kinds of projects not only exist, but are encouraged. Even though data sonification isn’t my “day job” here, I’m still able and encouraged to work on projects like these and will continue experimenting developing alternate methods for interpreting data.

Here’s a short code snippet from MM3D. The code is fairly crude – it was built as a proof of concept – but feel free to poke around:


A Picture of Lance Massey


UI Engineer Lance Massey is a UI Engineer at MediaMath, doing full stack application development for MediaMath’s TerminalOne Marketing OS™. In addition to his programming work, he is also a composer, having written the T-Mobile ringtone along with dozens of other soundtracks and audio designs. His creative sound design firm and passion project NeuroPop is developing health and wellness products including sleep, focus, and analgesia algorithms. His specialties are leading edge front end development, data perceptualization, and using sound to save the world.

Leave a Reply

Your email address will not be published. Required fields are marked *