Re: Sensory substitution and the human–machine interface

[문형철]

beyond reason

Sensory substitution and the human–machine interface

Recent advances in the instrumentation technology of sensory substitution have presented new opportunities to develop systems for compensation of sensory loss. In sensory substitution (e.g. of sight or vestibular function), information from an artificial receptor is coupled to the brain via a human–machine interface. The brain is able to use this information in place of that usually transmitted from an intact sense organ. Both auditory and tactile systems show promise for practical sensory substitution interface sites. This research provides experimental tools for examining brain plasticity and has implications for perceptual and cognition studies more generally.

Persons who become blind do not lose the capacity to see. Usually, they lose the peripheral sensory system (the retina), but retain central visual mechanisms. Similarly, persons who become deaf or are without balance usually lose only the peripheral structures relating to sound transduction (the cochlea) or positional orientation (the vestibular apparatus). The input from a sensory substitution system can reach many brain structures including those anatomically and physiologically related to the lost sensory modality [1,2]. Providing information from artificial receptors offers an opportunity to restore function. In the intact visual system, the optical image goes only to the retina, where it is turned into electrical impulses in the optic nerve; the perceived image is re-created in the brain [1]. Many studies have demonstrated that comparable subjective images are experienced by blind persons using one of several sensory substitution systems [3–8]. Whether or not the blind subjects are really seeing is addressed in Box 1. Besides using camera data to restore sight, sensors to replace other lost sensory information, such as sound and balance, have produced auditory [9,10] and vestibular substitution [11,12]. In such restoration of lost senses, information from an artificial receptor is coupled to the brain via a human–machine interface (HMI), replacing information usually carried to the brain from an intact sense organ [13,14]. Sensory substitution is thus only possible because of brain plasticity [1,15,16]. In the past, sensory substitution studies were purely academic; with rare exceptions, none of the devices ever reached the market. However, recent technological advances have led to the possibility of new prosthetic devices being potentially accessible at much lower cost to millions of patients. This is stimulating the interest of both research groups and industry, leading to the establishment of new research and development efforts in many countries. This recent explosion of interest in sensory substitution suggests that now is a good time to review progress in the area. Brain plasticity Brain plasticity can be defined as ‘the adaptive capacities of the central nervous system – its ability to modify its own structural organization and functioning’ [17]. It permits an adaptive (or a maladaptive) response to functional demand. Mechanisms of brain plasticity include neurochemical, synaptic, receptor, and neuronal structural changes [18–21]. Plastic changes in functional representation (usually occurring in response to a combination of a need and training) do not appear to change the original functional representation [22]. The mechanisms of representational changes are not known, but they probably include the unmasking of secondary inputs, such as non-visual inputs to the primary visual cortex [2]. Brain plasticity studies have been reviewed elsewhere [2,23], and are not the focus of this article. Sensory substitution Sensory substitution can occur across sensory systems, such as touch-to-sight, or within a sensory system such as touch-to-touch. In one experiment, the touch sensory information via a glove containing artificial contact sensors was coupled to skin sensory receptors on the forehead of a person who had lost peripheral sensation from leprosy. After becoming accustomed to the device, the patient experienced the data generated in the glove as if they were originating in the fingertips, ignoring the sensations in the forehead [2]. The most successful sensory substitution system to the present is Braille. Information usually acquired visually (reading) is, instead, acquired through the fingertips. We suggest that reading can itself be considered the first sensory substitution system, because it does not occur naturally but rather is an invention that visually presents auditory information (the spoken word). A blind person using a cane is exhibiting another very Corresponding author: Paul Bach-y-Rita (pbachyri@facstaff.wisc.edu). successful simple sensory substitution system; the single point of contact with an object provides a great deal of practical information on object location and identification; the HMI is at the hand where the receptors are activated by the cane’s contact with an object. Stimulation of the hand is experienced at the end of the cane rather than in the hand. In other words, the experience is externalized to the point of contact between the object and the cane. Typically in sensory substitution, the HMI is simply a transducer. It relays the information from artificial sensors to the human sensory interface, which carries the information to the brain. With experience, there is no perception of stimulation at the site of the HMI.

Are blind persons using tactile-vision sensory substitution (TVSS) actually seeing? Heil [4] and Morgan [5] suggest that because blind subjects are being given similar information to that which causes the sighted to see and are capable of giving similar responses, one is left with little alternative but to admit that they are seeing (and not merely ‘seeing’ or having some other sensation). In his essay Molyneux’s Question (an exploration of the question posed in 1688 by William Molyneux to the philosopher John Locke: would a person blind from birth be able to distinguish visually a cube from a globe upon sudden acquisition of vision?), Morgan [5] offers two basic arguments for this position. First, the structural nature of the perceptual system does not offer any criteria for distinguishing seeing from not seeing. The horseshoe crab is offered as an example of a biological system with fewer receptors than most mammals but which can nevertheless see. Morgan’s second argument concerns behavioral equivalence: if blind subjects receive (optical conversions of) optical information that would satisfy criteria for seeing in the sighted, and respond in an indistinguishable manner, one probably has to concede that the blind are seeing. He considered that ‘There is not the slightest reason to think that if two sensory messages give the animal exactly the same information, and lead to exactly the same behavior, they will be perceived differently-even if they come over completely different pathways’ ([5], p. 207). Blind TVSS subjects do use the information in the same way that sighted people do [46]. For example, they exhibit appropriate anticipatory behaviors. In a ball-rolling task, the blind subject will pre-position her hand in response to the therapist’s pre-positioning of her hand. Following the trajectory of the ball provides confirm‎atory and corrective information. This is fully consistent with Freeman’s notion of preafference and reafference [47] or Merleau-Ponty’s notion of the intentional arc [48,49]. The essential point is that blind subjects use the camera data in the intentional arc just as a sighted person would use vision. Hurley and Noe¨ interpret sensory substitution as a form of controlled synesthesia, and state that ‘TVSS effects a new external intermodal mapping from distal sources of visual input to peripheral tactile inputs and on to the somatosensory cortex. As a result, the qualitative expression‎ of somatosensory cortex after adaptation appears to change intermodally, to take on aspects of the visual character of normal qualitative expressions of visual cortex’ ([16], pp. 142–143). Several chapters in Perception et Intermodalite, edited by Proust [50], explore the classical Molyneux question. Pacherie [51] concludes that the TVSS experiments permit a positive answer to the Molyneux question. She considers that TVSS offers the possibility of separating sensations from perception: visual perception is not necessarily based upon visual sensations. She notes that perception is an active process, in which movement plays a part. However, we would add a caveat: for accurate identification and localization in space, the blind person would have to be able to control the movement, which takes minimal training by TVSS subjects. We concluded from our TVSS studies that the movement can under certain conditions be imagined rather than executed and still result in accurate perception [52].

Auditory-vision sensory substitution: seeing via the ears

Substitutive sensory devices for blindness rehabilitation through audition have been studied by DeVolder et al. [24]. Spatial localization and object recognition has been demonstrated by Capelle et al. [25]. The system couples a rough model of the human retina with an inverse model of the cochlea, using a pixel-frequency relationship. A head-mounted TV camera allows on-line translation of visual patterns into sounds. With head or joystick movement, visual frames are grabbed at high-frequency and generate in real-time the corresponding complex sounds which allow recognition. After training, both early blind (EB) and blindfolded sighted subjects were able to identify visual forms [14]. Trained EB subjects acquired rules of visual depth perception and increased their performance in localization of distant objects. Another approach to auditory-vision substitution has been taken by Meijer [12]. He developed a prototype system for a general video to audio mapping, associating height with pitch and brightness with loudness in a left-toright scan of any video frame. This yields a typical resolution of 60 by 60 pixels at a rate of about one frame-per-second.

Tactile-vision substitution: seeing via skin receptors

Tactile-vision sensory substitution (TVSS) studies have been carried out by numerous research groups for over a century. Many electro- and vibrotactile sensory substitution HMIs have been developed which have been applied to various surface areas [26–31]. In particular, the tongue provides a practical HMI [32]. It is very sensitive and highly mobile. Because it is in the protected environment of the mouth, the sensory receptors are close to the surface. The presence of an electrolytic solution, saliva, assures good electrical contact. The Tongue Display Unit (Figure 1), or TDU (which we also identify as a BrainPort because of the similarity to a computer USB port), has been developed to take advantage of these characteristics [33]. The results obtained with a small electrotactile array developed for a study of form perception with a finger tip demonstrated that perception with electrical stimulation of the tongue is somewhat better than with finger-tip electrotactile stimulation, and the tongue requires only ,3% of the voltage (5–15 V), and much less current (0.4–2.0 mA), than the finger-tip [32,34]. The electrodes are separated by 2.34 mm (center to center). The present model has a 12 £ 12 matrix of electrodes. The electrotactile stimulus consists of 40-ms pulses delivered sequentially to each of the active electrodes in the pattern. Bursts of three pulses each are delivered at a rate of 50 Hz with a 200-Hz pulse rate within a burst [32,34]. Early studies are reviewed elsewhere [1,31,35]. In most studies, optical images picked up by a TV camera were transduced into a form of energy (vibratory or direct electrical stimulation) mediated by the skin receptors of one of several parts of the body (abdomen, back, thigh, fingertip, and forehead). Visual information reaches the perceptual levels for analysis and interpretation via somatosensory pathways and structures. After training with TVSS, subjects report experiencing images in space, instead of on the skin. They learn to make perceptual judgments using visual means of interpretation, such as perspective, parallax, looming and zooming, and depth estimates [7]. Once the subject has learned with one motor system (e.g. hand-held camera, using the corresponding kinesthetic system), the camera can be switched to another system (e.g. head-mounted), with no loss of perceptual capacity. And when the HMI, the electro- or vibrotactile array, is moved from one area of skin to another (e.g. from the back to the abdomen or to the forehead), there is no loss of correct spatial localization, even when the array is switched from back to front, because the trained blind subject is not perceiving the image on the skin, but is locating it correctly in space [2]. Subjects using TVSS learn to treat the information arriving at the skin in its proper context. At one moment the information arriving at the skin has been gathered by the TV camera, but at another it relates to the usual cutaneous information (pressure, tickle, wetness, etc.). The subject is not confused; when he/she scratches his/her back under the matrix nothing is ‘seen.’ Even during task performance with the sensory system, the subject can perceive purely tactile sensations when asked to concentrate on these sensations. This does not lead to cognitive overload (Box 2). A poor resolution sensory substitution system can provide the information necessary for the perception of complex images. The inadequacies of the skin (e.g. poor two-point resolution) do not appear as serious barriers to eventual high performance, because the brain extracts information from the patterns of stimulation. It is possible to recognize a face or to accomplish hand-eye coordinated tasks with only a few hundred points of stimulation. An experiment with stationary TVSS displaying the tactile matrix on the subject’s back [2] showed that blind subjects were able to bat a ball as it rolled off a table at a point that had to be predicted by the blind subject. The subject had to identify the rolling ball, estimate the time it would take to reach the edge of the table, estimate the position on the table, identify the location in his ‘visual’ field of the bat, and correctly time his movement of the bat (which was constantly being moved) to bat the ball. The experimental resultwithtwowelltrainedblindsubjectswasthatthemean of the performance was nearly perfect. Recent experiments with motivated subjects show that they can perform hand-’eye’ coordination tasks within the first session.

Tactile-vestibular sensory substitution: balancing via skin receptors

Persons with bilateral vestibular damage (BVD) experience functional difficulties that include postural ‘wobbling’, unstable gait and oscillopsia. This condition presents the unique opportunity to: (i) study a model of an open-loop human control system, and (ii) to re-establish headpostural control by means of vestibular substitution using a head-mounted accelerometer and an electrotactile HMI through the tongue sensory receptors. The use of vestibular sensory substitution produces a strong stabilization effect on head and body coordination in subjects with BVD [11]. BVD patients can stand only with great difficulty, making enormous conscious efforts to integrate a range of visual and tactile cues. Using head-mounted accelerometers coupled via a tongue HMI, subjects are able to adapt to the new source of data in a matter of seconds. Head motion studies of the subjects show that the stability that the victim obtains from the accelerometer data approaches that of the person with normal vestibular function. Other head motion studies show that it is not a placebo effect.

Implanted human–machine technologies

Human–machine interfaces have provided the major challenges to practical sensory substitution. Although this review has emphasized tactile sensory substitution, other techniques are being studied. Mussa-Ivaldi and Miller have reviewed several HMIs including EEG and intra-cortical techniques [36]. They emphasize that feedback is needed for learning and control, requiring the establishment of a ‘closed loop’. Comparably, Suaning and Lovell have developed a 100-channel implanted neurostimulation technique [37]. Nicolelis et al. have studied high-density multielectrode arrays as tools to record from large numbers of individual neurons in the brain, as required to understand complex neuronal processing [38]. Further, such technology may enable direct brain-to-computer interfacing for control of prosthetics or other devices. It has been challenging to maintain functionality of large numbers of electrodes for long periods of time in behaving animals. Between 96 and 704 electrodes were implanted in each of three animals. Single-unit recordings were obtained for as long as 18 months from at least 58 neurons and at earlier time points up to 247 single neurons were recorded in a single session. These results represent a significant advance in the ability to extract large amounts of information from the nervous system. Cochlear implants for deaf persons have been demonstrated to be practical [39]. The techniques reported above may lead to implanted sensory substitution systems for other sensory losses. 

Box 2. Sensory overload
Normal sensory systems do not usually overload. The central nervous system is able to select only the information needed to deal with the situation afforded by the particular context in which it finds itself. In 1970 we found that many attempts to create sensory aids led to the problem of sensory overload [3]. This was the result of setting out to provide a set of maximally discriminable sensations, which almost immediately revealed a sharp limitation in the rate at which people can cope with incoming information when it is presented serially. Visual perception thrives when it is flooded with information, for example, given a whole page of text or a whole scene; however, it falters when the input is diminished or temporally disjointed: when it is forced to read one word at a time, or views the world through a cylindrical tube. At that time, we stated that, ‘It would be rash to predict that the skin will be able to see all the things the eye can behold, but we would never have been able to say that it was possible to determine the identity and layout in three dimensions of a group of familiar objects if this system had not been designed to deliver 400 maximally discriminable sensations to the skin.’ Indeed, the perceptual systems of living organisms are the most remarkable information-reduction machines known. They are not seriously impaired even in situations where a large proportion of the input is filtered out or ignored, but they are invariably handicapped when the input is very impoverished or artificially encoded. Thus, it was clear in 1970 that evidence of overload reflected, at least to some degree, use of inappropriate or poor displays more than a limitation in the information-handling capacities of the sensory system of the perceiver [3].

Neural correlates of sensory substitution In addition to the subjective reports and observed performance of the subjects experiencing sensory substitution, the phenomenon has been investigated with functional neuroimaging. PET studies of sensory substitution mainly rely on cerebral perfusion mapping to reflect brain activity because changes in synaptic activity in the brain result in changes of local perfusion. Several PET studies demonstrated that in cognitive tasks, such as Braille reading and tactile discrimination tasks [40,41], occipital activation was found during pattern recognition in the early blind [42,43]. Another recent study of blind subjects who had six hours of training in letter recognition using a tongue substitution system, showed activation in several ‘visual’ areas, including the cuneus, fusiform gyrus, and inferior medial and superior occipital gyri [44]. Although sensory substitution studies strongly support the capacity of the brain to reorganize [1,2], the actual mechanisms have not been firmly established. Brain imaging studies including PET scans and transcranial magnetic stimulation in Braille, and both auditory and TVSS (see above), and evoked potential studies [2] have provided evidence for reorganization. Receptor up- and downregulation, both synaptic and extrasynaptic mechanisms have been proposed [2], but firm evidence is not yet available. Dynamic and spectrographic analysis of head posturogram accompanying functional vestibular substitution through the tongue can provide clues regarding neural mechanisms of sensory substitution. Power spectra of the head posturogram in normal subjects consist of patterns of discrete peaks in a range from 1–15 Hz. In BVD patients the normal spectral pattern was severely depressed in the 2–15 Hz frequency range, and the power in the spectra shifted to the 0.15–2.0 Hz range. Shift to lower frequency correlated with the severity of the vestibular damage [45]. In some BVD patients using the device, part of the high frequency power spectrum is restored, whereas in others, it is not. In these cases the evidence suggests that the TDU has the effect of canceling some of the noise in the process of maintaining balance with non-vestibular cues. Both processes suggest that the nervous system is integrating the TDU data.

Conclusions

Sensory substitution studies have demonstrated the capacity of the brain to adapt to information relayed from an artificial receptor via an auditory or tactile HMI. With training and with motor control of the input by the subject, percepts are accurately identified and spatially located. Thus, blind persons obtain visual information resulting in visual percepts (e.g. of a ball rolling across a table) and can produce appropriate motor responses (e.g. catching the ball) with a vision substitution system consisting of a TV camera controlled by the subjects movements, the electronic device that transforms the signals from the camera into signals appropriate for the sensory receptors at the HMI, and a tactile array on the tongue. Similarly, persons who have lost vestibular function can regain balance with a tactile vestibular system; in this case the input to the tongue HMI is from an accelerometer which transforms tilt into electrical signals. Although brain imaging and physiological activity correlates of sensory substitution have been described, precise neural mechanisms have not been identified. The field of sensory substitution provides the opportunity to develop practical devices for persons with sensory loss. These will be cosmetically acceptable, miniaturized and inexpensive, and it is expected that they will be widely used. Furthermore, as the experimenter has control of the conditions of the sensory substitution process, they are being developed as tools to explore perceptual, physiological and brain mechanism correlates of a complex cognitive process.

Research up to the present has led to the demonstration of reliable prototypical devices that use sensory substitution to restore lost sensory function. There are three envisioned paths of future development. First, and most urgent, is the development to robust and relatively inexpensive implementations of the technology to make it accessible to a wide range of patients suffering sensory loss. Second, moving beyond the restoration of lost senses, it should be possible to use the same technology to expand human sensibilities, for example, enabling the use of night vision apparatus without interfering with normal vision. Finally, the technology enables a whole range of noninvasive low-risk experiments with human subjects to gain a deeper understanding of brain plasticity and cognitive processes.

Acknowledgements

This research was partially supported in part by the National Science Foundation (Grant no. IIS-0083347). In addition to their university positions, the authors are associated with Wicab, Inc, a company to which the University of Wisconsin has licensed the patented technology for a tongue HMI, which was developed in Bach-y-Rita’s university laboratory under an NIH, NEI RO1 grant. The authors thank Dr Yuri Danilov for his assistance in preparing this article.