Many people enjoy live orchestral performances, but those without musical training may find it hard to relate to the music. We have developed a system that helps users by guiding them through the performance using a handheld application in real-time. Using chroma features and dynamic time warping, we attempt to align the live performance audio with that of a previously annotated reference recording. The aligned position is transmitted to users’ handheld devices and pre-annotated information about the piece is displayed synchronously.
Acoustic Features: Chroma
Frequencies are divided into 12 pitch classes corresponding to the 12 notes in the Western chromatic scale (A, A#, etc.). The chromagram reflects the intensity of pitch over time, regardless of octave.
Alignment using Dynamic Time Warping
Dynamic Time Warping (DTW) is related to dynamic programming, and is designed to compare two signals with different timescales and align them. This is used to determine the equivalent time in a reference recording for the music that is currently unfolding in the live performance .
- Chroma are extracted from live audio and compared with chroma from a reference recording.
- A distance matrix and a cost matrix are generated between the two sets of chroma.
- The DTW algorithm aligns the two where this difference is minimized.
Follow along during the performance with real-time LiveNote slides.
- Slides automatically advance with the music, pointing out key highlights, engaging details and images relating to the composition.
- Access additional supplementary content at any time including a digital version of the program notes, a musical glossary and information about The Philadelphia Orchestra.
- This app provides novice and experienced listeners alike the opportunity to learn more about the music while listening to a live performance.
- LiveNote makes it easy to adjust the font size and brightness for optimal viewing during a performance in Verizon Hall. Plus, you can save a copy of the LiveNote content to view at home after the performance is through.
With an iPhone or iPod Touch, audience members use an application that helps guide them through the performance.
- The server pushes the live position to the clients. Users view time-relevant information based on the current position in real time.
- Users can look ahead or back up.
- Multiple information tracks convey different subject matter.
Annotation content is generated through collaborations with Philadelphia universities and organizations. In addition to user feedback, the musicolgists’ feedback helps to tailor the application to the information that they want to convey to their audience.