Chapter Seven—James J. Gibson and his “Note on the Difference Between a Sensory Modality and a Perceptual System”

More than forty years of contemplation and research by Gibson preceded the publishing of his book The Senses Considered as Perceptual Systems; much of it being provoked by the gentleman who drew this diagram—the “father of Modern Philosophy” René Descartes (1596–1650).

Descartes’ illustration of Dualism. The sensory organs pass on inputs to what he called the “epiphysis” (beyond nature) in the brain and from there to the immaterial spirit. The point in the brain is identical to the position of the pineal gland—and was also known in other cultures as “the third eye.”

If this looks familiar—it’s the same illustration I used in Chapter One when I suggested that our exploration into Dr. Paul Bach-y-Rita’s invention would lead us to a very possible denouement of the alternate name for Descartes’ mind/body dualism—the so-called “mind/body problem”; a perspective that even Paul subscribed to when we first reconnected. Put simply, it is the belief that the mental and physical aspects of a human being are of an entirely different nature and only peripherally connected. Paul’s statement—which I used as a headline for our website: “We don’t see with the eyes—we see with the brain” is a conceptual interpretation of Descartes original theory—and why I titled Part One of this Section “Two Years in a Body/Brain Mindset”—the Brain being part of the Body with no connection to what Paul perceived as a non-material part of the Mind.

The irony is that Paul is the one who first made me aware of Gibson, whose whole professional career was devoted to showing that mental and physical processes are not only directly connected, they actually complement each other; meaning one can’t work without the other.

Gibson’s search for an approach to perception that eliminated—once and for all—any remnants of dualism, conceptually or otherwise, would take almost half a century to fully realize. His explorations became the foundation for it’s own movement in the field of Psychology—which is known as Ecological Psychology.

The term is derived from two Greek words “oikos”—meaning home and “logos” meaning word, since its most basic tenet suggests that one cannot truly study perception without taking into account the specific environment in which the perceiver is embedded. His colleague Professor William H. Mace put it this way: “Ask not what’s inside your head, but what your head’s inside of.”

The ramifications of this approach are almost inexhaustible, beginning with the fact that, according to Gibson, the environment literally contains all the information necessary for any living organism’s survival; thereby totally rejecting the previously held belief that perception only provides some of the information necessary for the act perception and cognition must provide the rest for the act to be complete. I can think of no other statement that puts Gibson’s approach in perspective, than this:

In my view, the claim that we perceive ourselves is coherent only in the framework of a particular theory of perception: James Gibson’s (1966, 1979) ecological approach. In that theory perception has a special status. Gibson often called it “direct,” meaning that the perceptual systems pick up information that invariantly specifies an objectively existing state of affairs. Unlike other forms of knowing, perception is not constructive or inferential. The rich information normally available to vision, for example, enables us to see the environment and our own actions as they really are.—Ulric Neisser, The Perceived Self

There are a number of “eye-opening” facts in this statement. First and foremost, the man who made the statement—Ulric Neisser—is the unofficial “father” of the particular discipline that continues to maintain its dominance in the field of Psychology, while also providing the psychological underpinnings for the field known as Neuroscience: Cognitive Psychology. He actually coined the term! (Neisser has since abandoned Cognitive Psychology as a result of Gibson’s work.)

Secondly, it strikes deeply into the heart of Neuroscience by rejecting its basic premise—that the whole of what we call the human organism can best be understood by ever—increasing closer scrutiny of its parts. Nowhere is this practice more dramatically illustrated than in the work of Paul Bach-y-Rita and his colleagues in their approach to understanding the source of the BrainPort’s therapeutic effect—beginning with the statement that I myself, used as a website headline. “We don’t see with the eyes we see with the brain.”—Bach-y-Rita, 1972.

Gibson would have not only disagreed with this statement he would have said it was incomplete, as indicated here by his own statement: “We are told that vision depends on the eye which is connected to the brain. I shall suggest that natural vision depends on the eyes in the head on a body supported by the ground.”—James. J. Gibson, The Ecological Approach to Visual Perception, 1979

More from Gibson:

The Failure of Correspondences between Nerves and Senses

“Anatomy is an ancient discipline. When the first anatomists discovered fiber bundles leading inward from each eye, each ear and the olfactory membrane, they called them the optic nerves, the auditory nerves, and the olfactory nerve. It was then assumed that each nerve transmitted special messages from the sense organ to the brain.”

(Note: This is precisely as Paul describes it in one of the previous videos.)

“But actually, the correspondence between nerves and senses is more false than true. There is no distinct nerve for taste. There is certainly none for touch. The twelve bilateral pairs of cranial nerves have mixed functions. There are not twelve corresponding modes of sensation, and surely not one mode for each side of a pair. Incoming and outgoing fibers are found in each bundle, so the concept of “sensory” and “motor” nerves is a myth. The thirty one pairs of spinal nerves are even less in correspondence with types of sensation, that is, with qualities of experience.”

And this…

“And the function of the brain when looped with the perceptual organs is not to decode signals, nor interpret messages, nor to accept images. These old analogies no longer apply. The function of the brain is not even to organize the sensory input or to “process the data” in modern terminology. The perceptual systems, including the nerve centers at various levels of the brain, are always seeing information about the environment from the flowing array of ambient energy.

The italics in the last excerpt are mine. Gibson’s term: “the flowing array of ambient energy” is perhaps the most astounding of his discoveries. He asserts that this, in fact, is the form in which all information reaches our eyes; and all our other perceptual systems, as well. More about that in a later Chapter, where we’ll also show it’s relation to David Bohm’s use of the hologram analogy.

No revolution can be accomplished without some help. In Gibson’s case, most of it came from his wife, about whom he wrote the following in this same book: “This whole inquiry has been shared for years with Eleanor J. Gibson, my wife. In 1963 however, we divided the problems between us, and she has concentrated on perceptual learning and development while I concentrated on the senses.” Again, the italics are mine for this reason:

Perceptual learning is what takes place when Paul Bach-y-Rita’s invention is properly used. It is in fact the source—the very foundation of the device’s so-called residual or therapeutic effect.

But Eleanor J. Gibson’s contributions to the Perception Revolution were much more than that. A major force in the Revolution herself, and agreeing as she did with her husband’s statement: “To perceive the world is to co-perceive ourselves,” here’s what she had to say about the inseparability of the perceiver and the perceived during perception:

“Information for perception is contained in the ambient array of energy. In the case of visual perception, we refer to this as an ambient optic array. Energy in this array is structured light and is reflected from surfaces of the environment. An animal positioned in this array can actively use its perceptual systems to detect information specifying the surfaces and events around it. At the same time there is information in the optic array as structured by the body of the perceiver for the self as a unique segment of the array.

“There is information in a stationary optic array of a perceiver that specifies the perceiver, because from a fixed point of observation, the body of the perceiver conceals a portion of the environment round him uniquely, in a way that is proprio-specific, i.e. self-specific. Ernst Mach pointed this out, with a drawing of his view of himself, which he referred to as his “visual ego.” The eyebrows, nose, eyeglasses, mustache, and so on frame the visible vista, and the rest of the world is temporarily hidden.”—Eleanor Gibson, “Ontogenesis of the Perceived Self,” The Perceived Self, p. 25.

Now that you’ve completed my crash course in the work of James J. and Eleanor J. Gibson, here’s the promised “Note on the Difference” in its entirety. You can probably figure out for yourselves where Dr. Paul Bach-y-Rita went wrong in misinterpreting some the contents of the book. Or maybe he was just overwhelmed. This after all was the starting point of the Perception Revolution. And, genius that he was, he did gain enough insight from it to invent his amazing electro-tactile device which was initially introduced to me as the “Tongue Display Unit.” If there are any questions, you know how you can reach me.

February 1972

Note on The Differences between a Sensory Modality and a Perceptual System

J. J. Gibson, Cornell University

The World Wide Web distribution of James Gibson’s “Purple Perils” is for scholarly use with the understanding that Gibson did not intend them for publication. References to these essays must cite them explicitly as unpublished manuscripts. Copies may be circulated if this statement is included on each copy.

Five or six years ago, with the publication of The Senses Considered as Perceptual Systems, the distinction between a sensory modality and a perceptual system was made. It is not clear to everyone. The purpose here is to review and elaborate the distinction. The verb to sense can mean either to have a sensation or to detect something and the two meanings are radically different.

1. A sensory modality is a channel of input from a receptor mosaic along a nerve; a perceptual system is a circular process of input-output between the periphery and the brain.

A “sense” considered as a modality is thus defined anatomically whereas a perceptual system is defined functionally so as to include peripheral adjustments.

2. A sensory modality only allows the possibility of filtering the inputs of the nerve or channel; a perceptual system can modify the inputs of the channel.

Thus a perceptual system can obtain or prevent stimulation of a receptor mosaic, as in looking or not looking and touching or not touching.

3. A sensory modality is studied in the laboratory by imposing stimulation and preventing exploration; a perceptual system can be studied by making available an external array of potential stimulation and permitting its normal activity.

Thus attention can only be a covert activity in the nervous system for a sense modality, whereas it can be an overt observable activity for a perceptual system. Four types of overt activity are exploration, adjustment, orientation, and optimization. But these are not forms of “behavior”.

4. A perceptual system contains components of receptors, receptive surfaces, and adjustable organs which exist in a relation of subordination and superordination; a channel of afferent input is thus only a component of an active perceptual system (cf. Ch. 2 of the Senses Considered).

A bank of receptors, a mosaic, or a sensory surface of energy-transductors, is of course a necessary component of a perceptual system. But the firing of photoreceptors, mechanoreceptors, or chemoreceptors is not sufficient for perception. The perceptual systems, moreover, utilize different combinations of receptor banks at different times, that is, the receptive components overlap.

The fact that a sensory surface like the retina actually contains not only a mosaic of photoreceptors but also a set of “receptive units” is consistent with the above distinction.

5. For a sensory modality a so-called “stimulus” can only consist of the separate local excitations of specific receptors in the mosaic; for a perceptual system there is no “stimulus” and there are no separate adjacent “stimuli”, but instead there is an ordered external array of ambient stimulus energy.

The existing confusion in psychology about stimuli, stimulus patterns, stimulus situations, and stimulus objects is resolved if we distinguish sharply between the local excitation of receptors and the detection of relations in ambient energy.

6. A sensory modality transmits sensory signals to the brain (“sensations” to a “sensorium”); a perceptual system does not normally transmit anything to the brain.

The theory that sensory inputs have to be processed in the brain depends on the analogy of signal-transmission. But if we assume that a perceptual system only operates in this manner, even approximately, when its overt activity is prevented then the theory of input processing (or “information” processing, to beg the question) has only a limited application. We think instead of theories of information “pickup”, or the extracting of “invariants”, or of input-output circuits that can “resonate” to the invariants.

7. The sensory modalities are supposed to have specified phenomenal qualities that corresponds to the sensory nerve and even to the sensory neurons within it, by Johannes Müller’s law; the perceptual systems are not supposed to have specific phenomenal qualities.

The attempt to classify sensations by introspections (to “inventory” them) has not been successful, and hence they cannot be assigned to nerves and neurons. The human perceptual systems, on the other hand, can be classified functionally into five modes of overt attention (looking, listening, feeling, smelling, and tasting) together with the superordinate system of orienting (Ch 3 and 4).

8. The deliverance of a sensory modality are sense data or sense impressions, which are meaningless; the outcome of the activity of a perceptual system is perception, which is meaningful.

Thus, on a sensation-based theory of perception, meaning is something that is added to experience whereas in an information-based theory of perception meaning is something that is discovered in the experience.

9. The concurrent inputs of the two sensory modalities such as sensations of vision and touch aroused by the same object have nothing in common; the concurrent products of two perceptual systems in this situation may be wholly equivalent.

Thus, in the concurrent (and covariant) activities of looking at and feeling a solid object the features of shape and texture emerge in both haptic and optic exploration. The invariants can be detected by either perceptual system (although the sensations vary from moment to moment). The information for shape and texture is the same; it does not have to be learned by association or mediated by words as the supposed equivalence of sensations of vision and touch would have to be (See Ch. 7). It must be admitted that the haptic system cannot detect the color of a surface and the visual system cannot detect the temperature of a surface but this does not detract from the above hypothesis.

A great many experiments on the unsolved puzzles of “touch” and “vision” have to be reinterpreted if this hypothesis is correct.

Conclusion. When a psychologist or a physiologist speaks of sense perception he usually means perception that entails either the having of sense data or the transmission of sensory signals to the brain. But he might mean perception that “senses” the information in ambient light, sound, and contact and thus detects the features of the surrounding world. This kind of perception does not entail either sense data or sensory signals, neither a modality of impressions nor a specific set of neural inputs.