PossessedHand: Controlling Hand Movements with Computer Output

By Emi Tamaki and Jun Rekimoto

NOTE: This is an abstract of the entire article, which appeared in the December 2011 issue of Computer magazine.
Click here to read the entire article.

The PossessedHand human-computer interaction (HCI) system uses computer output in the form of electrical stimulations from multiple, noninvasive electrodes attached to the forearm to control the subject’s hand movements. We envision this technology as having application in systems for learning or guiding execution of various physical skills, such as playing a musical instrument, as well as in rehabilitation systems. The system’s feedback method could be applied to interactive systems in general.

System Overview

As Figure 1 shows, the Possessed-Hand prototype consists of two belts, each with 14 embedded electrodes, that attach to the subject’s forearm; a switching board; a battery; a condenser for generating the electrical pulses; and an Arduino Mega microcontroller connected to a PC via a USB cable.

Figure 1. PossessedHand system prototype.

A GUI on the PC provides the means to vary the stimulation level and thereby control the forearm and wrist muscles connected to the subject’s hand. Proper location of the electrodes is critical – positioning the forearm pads can take a lot of time and requires precise anatomical knowledge. As forearm size differs from person to person, the belts must be custom-fit to each user.

An automatic calibration system identifies the stimulation patterns that control the subject’s hand and selects the optimal stimulation levels for each joint movement.

Experimental Evaluation

To evaluate PossessedHand, we measured finger-joint movement induced by the system on one hand of eight subjects. The average force recorded for all five fingers was 88.75 g. In general, PossessedHand can’t yet induce movement forces sufficient to grasp objects. However, more accurately calibrating the relationship between the electrical stimulation level and amount of bend in the joint could narrow this gap.

Application: Learning to Play the Koto

One application of PossessedHand is helping a beginner learn to play a musical instrument. We developed one example application for the koto, a traditional Japanese stringed instrument that requires complex finger movements to achieve subtle tonal differences.

A koto score includes string numbers, time signatures, and finger numbers, and it’s difficult to read all three simultaneously. PossessedHand “tells” a beginning player which fingers to move and when, enabling the player to focus on the string numbers.

After transcribing several koto scores to PossessedHand stimuli, we had four novice players alternate between normal playing and playing with the aid of our system. Over the course of 36 trials, all four players demonstrated a better sense of rhythm and made fewer errors using PossessedHand.

Future Research

Our immediate goal for future research is to refine the use of PossessedHand in teaching beginners to play music, perhaps expanding to other instruments beyond the koto. We envision using PossesedHand to guide other difficult tasks, such as flying a helicopter or drawing, or to aid in physical rehabilitation. In addition, the system currently controls movements in only one hand, and including both hands could unlock many other possibilities. We’re also considering applying controlled electrical stimulations to different areas of the body, such as the shoulder.

ABOUT THE AUTHOR

Emi Tamaki is a PhD student in the Graduate School of Interdisciplinary Information Studies, University of Tokyo. Contact her at hoimei@acm.org.

Jun Rekimoto is a professor in the Interfaculty Initiative in Information Studies, University of Tokyo, and deputy director of Sony Computer Science Laboratories. Contact him at rekimoto@acm.org.

Editor: Albrecht Schmidt, University of Stuttgart, Germany; albrecht@computer.org