Jaku-in: A cultural skills training system for recording and reproducing three-dimensional body, eye, and hand movements

Author

Yokoi, Sotaro and Amitani, Kaishi and Hamanishi, Natsuki and Rekimoto, Jun

Abstract

In skill acquisition, it is crucial not only to mimic an expert’s body movements but also to understand their underlying intentions, which can be achieved by observing their hand movements and gaze. However, it was difficult to understand the difference between one’s posture and that of the expert in previous systems that assisted in understanding the intent of the expert. To overcome this limitation, we propose Jaku-in, which displays the expert’s hand movements and gazes in a three-dimensional reconstructed space using a point cloud. The system’s pass-through functionality of head-mounted displays allows novices to observe and emulate a life-size expert from multiple viewpoints: first-person, third-person, and behind-view. This research initially focuses on the Japanese tea ceremony, a practice where the precision and beauty of physical movements are fundamental. This showcases the effectiveness of ’Jaku-in’ in this domain and inspires potential applications in other intricate skill domains.