Perception

Perception is not a case of passive recording of sense impressions from outside.

The Process of Perception

• What is happening when I look at (or listen to, or smell, or taste, or feel) something?

Capturing and Converting Sense Data

My sensory apparatus (eyes, ears, nose, mouth, hands) receive some input, which is sent through specialized neural pathways that transmit physical and chemical signals from the environment to specific sensory processing areas in the brain:

• Sight - Photoreceptors (rods and cones) in the retina detect light, and signals are sent down the optic nerve through the optic chiasm and lateral geniculate nucleus of the thalamus, ending in the primary visual cortex of the occipital lobe.
• Hearing - Hair cells in the cochlea of the inner ear detect sound vibrations, and signals are sent via the vestibulocochlear nerve through the brainstem (cochlear nucleus, superior olivary complex) and medial geniculate nucleus of the thalamus to the primary auditory cortex in the temporal lobe.
• Smell - Olfactory sensory neurons in the upper nasal cavity detect airborne chemicals, and signals are sent via the olfactory nerve, bypassing the thalamus, and going straight to the limbic system and primary olfactory cortex.
• Taste - Gustatory receptor cells within taste buds on the tongue and mouth detect chemicals in food, and signals are sent via the facial (anterior tongue), Glossopharyngeal (posterior tongue) or Vagus (throat) nerve to the solitary nucleus in the medulla, passing through the thalamus, and reach the gustatory cortex in the insula.
• Touch - Mechanoreceptors, thermoreceptors, and nociceptors in the skin detect pressure, vibration, temperature, and pain, and signals are sent via the trigeminal nerve (facial) or spinal nerves (body) to the dorsal root ganglia.

Sensory transduction in these processing centers converts physical or chemical stimuli into electrical signals (action potentials) that the brain can process into conscious awareness.

Multisensory Integration

Multisensory integration combines information from the different senses into a single, unified perception of the world.

Specialized cells called multisensory neurons in areas such as the superior colliculus and association cortices (parietal, temporal, and frontal lobes), fire much more intensely when they receive inputs from multiple senses at the same time and place, using the following rules

• The Spatial Rule - Senses are integrated if they come from the same location (e.g., seeing a mouth move and hearing a voice from the same person).
• The Temporal Rule - Senses are integrated if they happen at the same time (e.g., hearing a thunderclap while seeing a flash of lightning).
• The Inverse Effectiveness Rule: Integration is strongest when individual senses are weak or blurry (e.g., relying heavily on lip-reading in a loud, crowded room).

Recognition

A continuous, interactive loop where recognition and multisensory integration constantly feed into one another makes sense of this integrated sensory input:

• Preliminary Feature Extraction - Before the brain can merge different senses, it must first do some basic "bottom-up" sense-making within each isolated channel. Eg The primary visual cortex identifies basic shapes and colors, while the primary auditory cortex identifies pitches and volumes. Your brain does not know what the object is yet, but it extracts enough basic features to realize that a visual flash and a loud boom happened at roughly the same time and place.
• Automatic Pattern Matching - Multisensory integration and the initial stage of making sense of an object happen simultaneously in the brain's association cortices. As the brain merges the sight, sound, and feel of an object, it triggers an automatic "Gestalt" binding. Eg, the texture of a dog's fur, the sound of its bark, and the shape of its body are pulled together into a single neural representation. Integration thus drives recognition. By combining the weak or muddy signals of individual senses into a unified whole, the brain instantly narrows down what the object could possibly be.
• Full Semantic and Emotional Recognition - Knowing the name, meaning, utility, and emotional value of what you are experiencing—happens immediately after multisensory integration is complete. The fully merged sensory package is sent along the "What" pathway (ventral stream) to memory hubs like the hippocampus and frontal lobes to provide conscious recognition. Your brain accesses its database, attaches a linguistic label ("that is my pet dog"), and triggers an emotional response (happiness or fear).

The Ventral Stream: Identification and Consciousness

The Ventral stream is concerned with recognizing objects:

• Signals enter the face and object Library (Inferotemporal Cortex / IT), which contains hyper-specialized modules, such as the Fusiform Face Area (FFA) for faces and the Parahippocampal Place Area (PPA) for scenes. Neurons here respond to whole objects, regardless of size, viewing angle, or lighting. The IT cortex constructs a highly detailed, 3D structural model of the stimulus.
• The IT cortex hands the 3D model over to the medial temporal lobe structures, the perirhinal and entorhinal cortices, which act as the final gatekeeper for object recognition. It compares the incoming 3D model against stored familiarity models and asks: "Have we ever encountered this specific combination of features before?"
• Once the structural model passes the perirhinal gateway, it simultaneously floods the Hippocampus and the Prefrontal Cortex (PFC):
  • The Hippocampus provides semantic retrieval and epistemic binding via your:
    • Declarative memory networks - It cross-references the 3D model with your memories and retrieves the semantic facts associated with the pattern (e.g., "This object is fruit, it grows on trees, it is edible") and hooks into linguistic centers to retrieve its name ("Apple").
    • Episodic memory networks - It retireves past personal experiences (episodic memories), such as the taste of a pie your grandmother baked, instantly giving the sensory object a rich, personal context.
  • The prefrontal cortex is the executive seat of consciousness. It holds the object in working memory, allowing you to deliberate on it, and evaluates it based on your current internal state and goals. If you are hungry, the PFC registers the apple as an immediate food source and initiates motor planning to reach for it. If you are full, the PFC notes its presence but maintains your current focus.

• See:
  • Mark Solms' The Hidden Spring
  • Tom Chivers' Everything is Predictable
  • etc

The Experience of Perception

Perception seems to be an effortless process. When I look out into the garden in front of me, I see the lawn, trees, and my son swinging on a swing.

My image for perception is of many thought traces bubbling or forcing their way to the surface of the mind. Concepts of colors, shapes, borders, biographical memories and scholarly knowledge surge forward to provide meaning to the scene and objects perceived. And this is happening all the time, as your eyes rove over a scene or are watching images on a screen or reading words on a page.

Learning and Perception

When a baby is born and encounters the world for the first time, we must assume that it has very limited organization of what it perceives. Perhaps there is just a sense of more or less light, the smell of the mother, the feel of her skin. But, immediately, we imagine these perceptions begin to be classified and organized. From initial confusion, begins a process of distinguishing objects, initially, perhaps simply as good/agreeable or bad/disagreeable. We know that we are predisposed to pay attention to face-like shapes and by repeatedly seeing the faces of the mother and other family or community members, there begins the process of being able to distinguish them as objects among those available in the world and then to be able to identify them.

A similar process is happening in the auditory world too. The baby hears sounds and among those sounds are words and among those words are some that are repeated more frequently. Caregivers will point to objects and name them, but there is also a general background of words and other sounds that are being organized and eventually can be identified and even repeated. The child's introduction into the linguistic world comes through imitation of the sounds that it hears and the beginning of mastery of the tongue and throat muscles to reproduce what it hears.

I still remember clearly when I was staying in Paris and learning French. I would listen to the radio each morning and suddenly, one morning, what had the previous day been an unbroken chain of sound, now became a stream of words. I'm sure that I didn't identify each word beginning and end immediately, but my mind, immersed in these sounds, had somehow found the knack to split the stream into objects that could be identified and understood.

We can experience this process at a much smaller scale when first learning chess or another game with pieces and positions. Here, of course, the pieces are already identified as separate objects moving in the world of the game but initially we are not able to distinguish between the bishop and the pawn or the king and the queen or to identify a position of check or where a piece is menaced. But after a game or two things become clearer, distinctions can be drawn and each piece is assigned to one or more categories, ranked in order of power and importance etc. When a grandmaster looks at a game in progress even briefly, they can reproduce the positions of all pieces easily because of the depth and quality of their perception - built up over thousands of hours of play.

Mature Perception

But once we have moved beyond the initial identification of objects and on to the effortless recognition of them? What is mature perception?

Perception is a powerful instance of Habit - we do not consciously perceive as the baby or the beginning chess player does. We are able to look around, situate ourselves and identify the objects in our surroundings due to the work of learning that we have done in childhood and to the deep habits we have acquired

X in The Mind is Flat says: "The mind-as-mirror metaphor can't possibly be right - we need a very different viewpoint, that perception requires inference." and suggest the following principles of the operation of perception (and, by extension, thought):

• We see only meaningful organizations (or, at least, the most meaningful organization the brain can find): visual chunks, patterns and whole letters, numbers, words, rather than a random scatter of fragments.
• We see just one meaningful organization at a time.
• Other sensory information that is not part of this meaningful organization is largely or even entirely ignored, to the point of becoming invisible.The brain is continually churning: despite the unaccustomed lack of new input, the brain is desperately attempting to disengage from the current organization, and to find another. When it cannot, the image entirely disappears.
• Perceiving is a type of thinking and perhaps the most important type, with all other types of thought being just powerful extensions of perception.

Reading as Perception

Reading is a particular form of perception which is learned and is peculiar to humans. We can imagine how it is learned:

• Initially writing must be identified as an object amongst those is the world.
• Individual letters are probably identified first, but possibly short words may be instead. The shapes and sounds of letters are identified
• Then groups of letters are identified as words where each word has its own sound (generally derived from the sounds of its component letters) and a meaning. Certain words can be associated with physical objects in the world - "That is a cat" - while others have other less concrete meanings
• Then groups of words are identified as sentences which have a meaning.
• While making all of these identifications, we are additionally, passively, learning that certain sequences of letters are more common than others (and some are never seen), that certain sequences of words are acceptable, while others are not, and a variety of other grammatical rules, many/most of which also apply to our spoken language