Tracking audio changes assume that louder scenes (explosions, musical crescendos, shouting) correspond to higher levels of drama. Rapid visual changes (quick motion across the screen, camera motion, or rapid edits) also correspond to action, a quicker tempo, and higher levels of drama. A combined index of audio and visual changes graphed over the length of the movie represents its unique fingerprint, revealing its dramatic highs and lows.

I analyzed forty noteworthy movies and collected the results in this interactive tool. Use it to explore the dramatic profiles for each movie and their corresponding scenes. Do the highest peaks in each profile match the movie’s climactic moments?

Explore the Story Analysis tool, which was used to produce the graphs. Use it to see how each movie’s audio and visuals are analyzed in real-time. You can analyze your own movies (FLV or MP4 format), output the data, and post the results for others to see.

How it was done
First, all the movies had to be converted to the correct Flash-friendly format. Each movie was converted to an MP4 (H.264 codec) file using Handbrake, a free open-source video transcoder. Then I had each movie stream into Flash with the FLVPlayer component.

Tracking the audio changes

Audio levels were analyzed with the ActionScript command, SoundMixer.computeSpectrum(). The command takes a snapshot of the current sound and stores the information as a series of numbers that can be translated visually. While my sound visualization is rather simple, there are countless creative ways to visualize sound. There have even been contests for the most creative visualizations. Since I was most interested in the variation of sound levels throughout the movie, I captured the amplitude (or volume) of the sound every 10 milliseconds and graphed it with a gray line. An average of the sound amplitude was calculated and graphed with a bold white line.

Tracking the visual changes

Every 10 milliseconds, Flash grabbed the image from the video stream with the BitmapData class. The command, getPixel(), gathered the red, green, and blue color information from each pixel. The red, green, and blue color distribution of an image is known as an RGB histogram. My goal was to track changes between histograms that would indicate major visual changes due to camera motion, edits, or subject motion. Much research has been already done on the subject of tracking shot changes for video cataloging, involving complex (and patented) algorithms. I made my calculation quite simple, determined by differences in the histogram area coupled with a dampening function to normalize the extreme values. The resulting index, which reflects visual changes, was graphed as a gray line. An average of the index was calculated and graphed as a bold white line.

Combining audio and visual changes

Combining the audio and visual indices resulted in what I termed, the “drama index”, a measure of the dramatic highs and lows in a movie. The overall shape of the profile, shown in red, can be interactively smoothed out or made more detailed by changing its resolution in the Story Analysis tool.

What does your favorite movie look like?
Analysis of forty distinguished movies–the top ten of all time, the worst ten, the ten highest grossing films, and the previous ten Best Pictures–not surprisingly reveal no common pattern, but it does provide a standard, objective way of tracking a film’s dramatic peaks and valleys–their position, duration, and intensity. This screenshot is a profile of Star Wars. Note the dramatic beginning when Princess Leia’s vessel is boarded, and the slow build-up to the three dramatic peaks at the end: the rescue from the Death Star, the duel between Darth Vader and Obi-Wan Kenobi, and finally the destruction of the Death Star. The analysis works best on modern action films. There is, of course, no consideration for acting, for cinematography, or for the dramatic climaxes that may come in quieter moments (such as the sudden change that crosses an actor’s face with a revelation).

Other movie visualizations
There are many other interesting visualizations of movies. NetFlix recently ran a contest to see if the public could find a more effective way to predict which movies users would prefer based on past ratings. The results of two of the top teams can be visualized as a network of similarities between movies, or as a landscape with similar movies clustered together. (Based on these maps, if you liked Star Wars, then you probably also liked RoboCop).

One recent visualization cleverly plotted the interactions between characters. The hand-drawn map and synthesis of time and geography reminds me a little of Charles Minard’s map of Napolean’s march to Moscow, as discussed and praised by Edward Tufte as a gem of information design.

Finally, the New York Times produced a fascinating look at the box-office revenues of the movies. I love seeing the periodicity in the graph reflecting the predictable huge bumps during the summer blockbuster months and holiday season before the Oscar considerations. Notice also the relatively short, squatter profiles of recent movies compared to the long tails of movies in the past.

What more can we visualize of movies, or the structure of individual stories?