IIIT-H creation plays suitable BGM while reading books, wins award

HYDERABAD: IIIT Hyderabad researchers created a music algorithm that curates and plays appropriate background music as we read books. This independent study project won the 'Brave New Idea Award' at the recently concluded annual conference of the International Society for Music Information Retrieval (ISMIR).

The use of music in factories, gyms or at work, where one must perform mundane chores, has been shown to produce efficient outcomes. A team of IIITH researchers which includes, a Btech student Jaidev Shriram, professor Makarand Tapaswi and professor Vinoo Alluri, attempted to create a similar experience but this time for book reading.

Their patent-pending research culminated in a study titled "Sonus Texere: Automated Dense Soundtrack Construction For Books Using Movie Adaptations" which says that an automatic system can retrieve and weave high-quality instrumental music for the entire length of a book.

The group began by examining rich, narrative-driven movie soundtracks and the books they were originally based on. The Harry Potter series seemed like an obvious choice, for the team owing to their amazing movie adaptations.

To align the book with its movie adaptation, the team resorted to three kinds of matching – Dialogues from the movie were matched with those in the book. A text-to-image-based retrieval matching was done using 'OpenAL's CLIP model.' For example, if the text mentions a snake, an image of a snake in the movie in a potentially relevant segment is found to perform a match. Finally, the movie segments were matched with relevant corresponding music.

However, only about 60 percent of the book was directly matched with the movie adaption. Therefore, the researchers filled in the gaps with an automatic emotion-based retrieval system. "It means that if a particular segment of text evoked feelings of fear, then the system would automatically match it with potential musical segments that could accompany it," explained Jaidev.

To validate their design, the team had conducted a perceptual study on participants who had previously read the Harry Potter book and watched its reel version too. All of them reported that the soundtrack enhanced the reading experience itself making it more immersive.