Synesthesia, An Inspiring Condition For AI Researchers
The study of sensory perception is key to next-generation AI.
Stephen Schwartz, the famous composer saw colors on each of his piano keys. Tori Amos, the famous singer says her songs appear as images of lights. Arthur Rimbaud, the famous poet associated colors with vowels. All of these people have Synesthesia, a condition where your senses are mixed. You may taste colors, hear textures, and smell shapes, etc.. It’s a condition that’s inspiring researchers who study the connection between sensation and perception.
By understanding how our perceptions work, researchers can better understand how we perceive language, what does it mean to be “conscious”, and how the brain processes our senses. For AI researchers, understanding the link between sensation and perception can help researchers build more sophisticated AI models that can perform complex tasks with a lot less data. It’s also the fundamental research underlying “sentient machines” or machines that have consciousness.
In the US alone, it’s estimated that at least 4% of the population has this condition. There are more than 50 different forms of synesthesia. Interestingly, many young children report experiences of “mixed” senses even if they grow up to be adults without mixed senses. It turns out that in the baby’s brain, there’s a kind of a “blur”. Studies suggest that the sensory areas of the baby’s brain have many cross-activations of neural links.
It’s possible that many of us experienced “mixed” senses when we were babies, then we lose the ability as we grow into adulthood. Studies show that as our brains grow into adulthood, our perceptions become more specific to a particular sense.
Studies also show that adults who experience synesthesia have more physical connections between sensory processing areas of the brain than others. It’s possible that most of us who don’t experience synesthesia in adulthood simply grow out of it in adulthood. This part of us is inhibited.
How Do We Perceive Language
Currently, when we think of the study of language, we often think of semantics or the interpretation of the meaning of words, phrases, sentences, and symbols. People who have synesthesia often see specific letters in colors. Words can invoke specific strong feelings, and sounds. Often, writers who have synesthesia use very strong metaphors in their writing. This is because they experience these metaphors in real life.
This additional metaphor on top of the semantics helps people with synesthesia with memory storage and retention. It also helps people with synesthesia to make additional associations. Studies show that people with synesthesia often learn languages more quickly.
Currently, AI systems, specifically natural language processing systems are mostly concerned with semantics. But, semantics is not all there is in learning a language. A system that can understand metaphors can be very powerful in both making connections between language concepts as well as inferring additional meaning.
What Does It Mean To Be Conscious?
According to Wikipedia, consciousness is the “sentience or awareness of internal or external existence”. Philosophers and Psychologists have been interested in consciousness for a long time. A large part of our learning occurs when we are conscious. It’s also thought that more self-aware people often learn to live better than those are not as self-aware.
When we talk about consciousness in AI, we are often concerned with whether machines can develop consciousness, and whether consciousness help machines learn better.
One area of research has been around the neural correlates of consciousness or finding the relationship between experiences and the activity that originates in the brain. Techniques such as EEG and fMRI are used to examine this relationship. Studying the differences between synesthetes and non-synesthetes can reveal localizations and networks responsible for the dynamic interactions between higher and lower levels of the brain.
If the neural correlates of consciousness can be identified, then there’s a better chance at understanding if machines can develop consciousness and whether consciousness truly help us learn better.
People with synesthesia who identify a sweet taste as “red” experiences both the color red and the sweet taste differently than the non-synesthetes. There might be different neural correlates involved. This type of difference can potentially open up a window to examining the underlying mechanisms of consciousness.
How Our Perceptions Help Us Learn?
Studies of synesthesia suggests that synesthetes learn differently than non-synesthetes. They learn better in categories. For instance, a synesthete might describe their days as, “Blue Mondays, Yellow Tuesday, etc..” depending on the activities and the emotions associated with these activities. In categorizing their days into these broad categories, they can remember details such as ballet lessons happens on Mondays, emotions associated with the ballet lesson, etc..
This way of learning can happen consciously and unconsciously. Using synesthetic associations as a learning strategy is strongly associated with creative learners, stronger verbal cognition, and stronger visual cognition. Research suggests that synesthete learn to use this type of learning strategy due to their “mixed sense”, then they can apply this type of learning strategy to other problems that they face.
Implications for AI
The way that synesthetes learn by categorizing things and relating things to symbols is similar to the Neuro-symbolic AI. Neuro-symbolic AI is not a new concept. From the 1950s to the 1980s, symbolic AI ruled AI research. Learning was viewed as forming an internal symbolic representation of the world, creating rules to deal with concepts, and applying these rules. It’s logic-based. One of the examples of a Neuro-symbolic AI is the SIR or Semantic Information Retrieval. It’s a system that is able to learn from a few logic statements to form its conclusions.
Combining the power of neural networks and neuro-symbolic AI is the next generation in AI development. Neural networks can help symbolic AI systems become smarter by breaking the world into symbols. Then, symbolic AI algorithms can help to incorporate common sense and domain knowledge into deep learning.
This will allow for AI to perform complex tasks such as self-driving and natural language processing with a lot less data.
Conclusion:
Understanding how synesthetes perceive, sense and learn is going to shape a new generation of AI as we move toward developing more complex AI systems by exploring the intersection of neuro-symbolic AI and deep learning.
References:
The Merit for Consciousness Research
Synthesis and release phenomena in sensory and motor grounding. Cases of disinhibited embodiment?
Synesthesia and learning: a critical review and novel theory
