Can AI understand a flower without being able to touch or smell?

The latest generation of artificial intelligence models seem to have a human-level understanding of the world, but it turns out that their lack of sensory information – and a body – places limits on how well they can comprehend concepts like a flower or humour.

Qihui Xu at the Ohio State University and her colleagues asked both humans and large language models (LLMs) about their understanding of almost 4500 words – everything from “flower” and “hoof” to “humorous” and “swing.” The participants and AI models were asked to rate each word for a variety of aspects, such as the level of emotional arousal they conjure up, or their links to senses and physical interaction with different parts of the body.

The goal was to see how LLMs, including OpenAI’s GPT-3.5 and GPT-4 and Google’s PaLM and Gemini, compared with humans in their rankings. It turns out that people and AI have a similar conceptual map of words that don’t relate to interactions with the outside world, but differ greatly when words are linked to senses and physical actions.

For instance, the AI models tended to believe that one could experience flowers via the torso – something that most humans would find odd, preferring to appreciate them visually or with a sniff.

The problem, says Xu, is that LLMs build their understanding of the world from text hoovered-up from the internet, and that just isn’t sufficient to grasp sensual concepts. “They just differ so much from humans,” she says.

Some AI models are trained on visual information such as photos and videos in addition to text, and the researchers found that the results of these models more closely matched the human word ratings, raising the possibility that adding more senses could bring future AI models ever-closer to human understanding of the world.

“This tells us the benefits of doing multi-modal training might be larger than we expected. It’s like one plus one actually can be greater than two,” says Xu. “In terms of AI development, it sort of supports the importance of developing multi-modal models and the importance of having a body.”

Philip Feldman at the University of Maryland, Baltimore County, says that giving AI models a robot body and exposing them to sensorimotor input would probably see ability jump, perhaps substantially, but that we will have to be very careful about how this is done, given the risk of robots causing physical harm to people around them.

Avoiding such risks would mean adding guard rails to robot actions, or only using soft robots that can cause no harm for training, says Feldman – but that would have its own downsides.

“This is going to warp how they understand the world,” says Feldman. “One of the things they would learn is that you can bounce off things, because they have little mass. And so now you try to put that deep understanding that has to do with physical contact [in a real robot with mass] and you have your humanoid robots believing that they can just crash into each other at full speed.
Well, that’s going to be a problem.”