Sony CSL | Rekimoto

Augmented Interaction: The World Through the Computer

Augmented Interaction is a style of human--computer interface that tries to make computers as transparent as possible. Using this style, a user will be able to interact the real world that is augmented by the computer's synthetic information. The user's situation will be automatically recognized by applying a range of recognition methods, allowing the computer to assist the user without having to be directly instructed by the user. The user's focus is not on the computer, but the real world. The computer's role is to assist and enhance interactions between human and the real world.

As an initial attempt to realize the concept of Augmented Interaction, we are now developing a prototype system called NaviCam. NaviCam is a kind of palmtop computer which has a small video camera to detect real-world environments. This system allows a user to look at the real world with context sensitive information generated by a computer.


The architecture of the NaviCam System


A snapshot of the screen: Computer Augmented Bookshelf

Other (large) images


Video Clip (created 1994) 1'30" 160x120 10fps QuickTime, 2.3MB
NaviCam demonstration at Gallery "MA", Nogizaka, Tokyo (9.11.1996 -- 10.26.1996)
Hightech Shower TV Program
Business Week article


  1. Jun Rekimoto, "The World Through the Computer: A New Human-Computer Interaction Style Based on Wearable Computers", Technical Report SCSL-TR-94-013, Sony Computer Science Laboratories Inc., 1994.
  2. Jun Rekimoto, "Augmented Interaction: Toward a New Human-Computer Interaction Style Based on Situation Awareness", Interactive Systems and Software II (WISS'94 Proceedings), pp.9-17, 1994. (in Japanese) PDF
  3. Jun Rekimoto, "Augmented Interaction: Interacting with the real world through a computer" , HCI International, 1995.
  4. Katashi Nagao and Jun Rekimoto, Ubiquitous Talker, Spoken language interaction with real world objects. Technical Report SCSL-TR-95-003, Sony Computer Science Laboratories Inc., Tokyo, Japan, 1995.
  5. Jun Rekimoto and Katashi Nagao, "The World through the Computer: Computer Augmented Interaction with Real World Environments", User Interface Software and Technology (UIST '95) PDF
  6. Jun Rekimoto, "The Magnifying Glass Approach to Augmented Reality Systems", International Conference on Artificial Reality and Tele-Existence '95 / Conference on Virtual Reality Software and Technology (ICAT/VRST '95). PDF
Sony CSL | Rekimoto