From search engines to voice assistants, computers are getting better at understanding what we mean. This happens thanks to language processing programs that make sense of an astonishing number of words without having to say explicitly what those words mean. Instead, such programs infer meaning based on statistics – and a up-to-date study shows that this computational approach can assign multiple types of information to a single word, much like the human brain.
The test, published April 14 in the journal, was co-led by Gabriel Grand, an electrical engineering and computer science graduate student affiliated with MIT’s Computer Science and Artificial Intelligence Laboratory, and Idan Blank PhD ’16, an assistant professor at the U.S. California in Los Angeles. The work was supervised by a researcher from the McGovern Institute for Brain Research Ew Fedorenko, a cognitive neuroscientist who studies how the human brain uses and understands language, and Francisco Pereira of the National Institute of Mental Health. Fedorenko says the wealth of knowledge her team has uncovered from computational models of language shows how much can be learned about the world through language alone.
The research team began analyzing statistical models of language processing in 2015, when the approach was up-to-date. Such models determine meaning by analyzing how often pairs of words appear in texts and using these relationships to assess the similarity of word meanings. For example, such a program might determine that “bread” and “apple” are more similar to each other than to “notebook” because “bread” and “apple” often appear near words such as “eat” or “snack”, while when the “notebook” is not.
The models were clearly good at measuring the overall similarity of words to each other. However, most words carry many types of information, and their similarity depends on what features are being assessed. “People can come up with different mental scales that help organize their understanding of words,” explains Grand, a former research associate in Fedorenko’s lab. For example, he says, “dolphins and alligators may be similar in size, but one is much more dangerous than the other.”
Grand and Blank, then a graduate student at the McGovern Institute, wanted to know if the models captured the same nuance. And if so, how was the information organized?
To find out how the information in such a model relates to human word understanding, the team first asked volunteers to rate the words on a number of different scales: whether the concepts the words expressed were huge or tiny, unthreatening or risky, saturated or desiccated ? Then, after mapping where people placed different words on these scales, they checked whether the language processing models behaved the same.
Grand explains that distributional semantic models apply co-occurrence statistics to organize words into a huge, multi-dimensional matrix. The more similar words are to each other, the closer they are in this space. The dimensions of the space are huge and its structure has no internal meaning. “There are hundreds of dimensions in these embedded words, and we have no idea what any dimension means,” he says. “We’re really trying to look into that black box and say, ‘Is there structure here?’”
Specifically, they asked whether the semantic scales they asked volunteers to apply were represented in the model. So they checked where words in space were arranged along vectors defined by the extremes of these scales. For example, where did dolphins and tigers go from “big” to “small”? And were they closer together on this line than on the line representing danger (“safe” to “dangerous”)?
Based on more than 50 sets of world categories and semantic scales, they found that the model organized words very similarly to human volunteers. Dolphins and tigers were judged to be similar in size but far apart on scales measuring danger or humidity. The model organized words in a way that represented multiple types of meaning – and did so entirely based on word co-occurrence.
This, says Fedorenko, tells us something about the power of language. “The fact that we can recover so much rich semantic information from just simple word co-occurrence statistics suggests that this is a very powerful source of knowledge about things you may not even have direct perceptual experience with.”