PianoHandSync: An alignment-based hand pose discrepancy visualization system for piano learning

Author

Liu, Ruofan and Wu, Erwin and Liao, Chen-Chieh and Nishioka, Hayato and Furuya, Shinichi and Koike, Hideki

Abstract

Video-based lessons are becoming a popular way for distance piano education. However, limited by the fixed camera angle, a video is difficult to tell precise 3D hand posture, which is one of the most essential factors for learning piano. This paper presents a visualization system providing the intuitive discrepancy of hand postures in two piano performance videos. Through a motion capture system, the estimated 3D postures are visualized and discrepancies based on distinct metrics are displayed, integrated with modular functions assisting skill acquisition. A pilot study proves that the proposed visualization can be a supplementary means for only video-based lessons in terms of correcting hand postures and fingering.