Technological and Computer-based Projects

Scoring Stories: an eye-tracking reading device
Proposed project
Using: Max/MSP, Jitter, OpenCV

Summary: A piece that tracks eye movement to trigger an audio "score" to a written work, keeping music and sound events in sync with the reader's pace. This could deliver audio footnotes, musical score or cues, dialogue interspersed with the written word, ambience based on the characters' mood and location... Imagine a film's score and sound effects, but following along a short story, novel, or poem as you read.

Project Description
As technology has enabled further collaborations among artists in different disciplines— from an organist at the nickelodeon to the score of a film, from Scriabin's 'light organ' to video projection with dance— the written word has so far been largely excluded from these mashups. Reading evokes a state of mind different from that of plays, movies, live music, dance... it is perhaps our most solitary art form. Perhaps this is why it has not yet lent itself to much modernization[1]. But this is not because the medium itself is impossible to collaboritize, if I may coin a phrase; it seems that art which is created alone and enjoyed alone is simply seldom in a position for another artists to say "Hey! Let's try your idea with that other thing we did last week, and see how that goes!"

Scoring Stories is a project to create what might be described as a film score, for a short story, novel or poem. By combining pagination information with eye-tracking, the score will be created in real-time from preexisting sonic material, and kept in sync with the reader whether she speeds up, slows down, backtracks, zones out or stops and starts in the middle of a page.

Many literary works make specific reference to 'real-world' pieces of music. Others evoke exotic or remote locations, situations or cultures. Without taking away from the power of the writer, a sonic element written specifically for the reader's experience could enhance immersion in the story, and extend the ability of the author to express to the reader what, particularly, is important about music they may be referencing. For example, in The Time-Traveler's Wife[2] there is an important scene that takes place at a 1994 Violent Femmes concert in the Aragon Theatre in Chicago's Uptown neighborhood. It is a high-energy punk-tinged concert; and many readers of Niffenegger's book may not be intimately familiar with that particular oeuvre. Add to this the sound of the El train passing as the protagonists and their friends wait in line outside, the sound of the famously surly bouncers at the entrance, the echoic halls as Clare describes the Aragon's "decaying faux-Spanish splendor"... all of this can be subtly paralleled sonically, just as a good film score plays the scene— doesn't overwhelm it, doesn't draw unwanted attention, doesn't make us wonder where the orchestra is and how they got so far out to sea.

While some books have been packaged with a CD readers are invited to play while reading the book, there are of course very limited mechanisms for reminding the reader to play a certain track with a certain part of the book and of course no mechanism at all for keeping the music in sync with the reader. What I propose is to create the first book-score, the first Scoring Stories, to create a sonic environment that can bend and flex with the reader's individual pace, bringing the same immersion into the story-world that we have come to expect in film, but retaining all the wonderful things about books, poems, and stories that keep us coming back to the written word.

1. The relatively recent introduction of the Kindle aside. [jump back up]
2. Audrey Niffenegger, (MacAdam/Cage 2003). [jump back up]