Visualising Pianists’ Touch: Transcribing Expressive Piano Performance from Audio to Piano Key Motion

Author

Tang, Jingjing and Furuya, Shinichi and Nishioka, Hayato and Shioki, Momoko and Wiggins, Geraint A. and Fazekas, George and Cheung, Vincent K.M.

Abstract

Detailed measurements of piano key motion capture touch, timing, and dynamic control, providing crucial performance insights. Such expressive gestures are overlooked in MIDI, which only records pitch onset, duration, and velocity. Here, we introduce a novel transcription technique that directly maps audio from expressive piano performance to continuous piano key motion. User studies reveal a preference to the transcribed key motion trajectories over MIDI in representing sound, and over 80\% accuracy in matching transcribed trajectories to audio from contrasting piano expressions. Follow-up interviews further indicate that the visualised trajectories can reveal subtle performance nuances and provide actionable guidance for both teaching and practice. An interface example for pedagogy and performance analysis utilising our technique is also illustrated. By providing a physically grounded performance representation that musicians can interpret and act upon, this work establishes a foundation for future interactive tools in music pedagogy, performance feedback, and embodied musical learning.

DOI

https://doi.org/10.1145/3772318.3791621