A new neuroimaging study published in eLife reveals that large language models process sentence meaning differently from the human brain when word order, not vocabulary, drives interpretation. The Melbourne-led team found that human brain activity aligns more closely with computational models that explicitly track grammatical roles—such as who is performing an action and who is receiving it—than with standard transformer representations.
Humans do not merely sum word meanings when parsing a sentence. Instead, the brain also tracks syntax and semantic roles to distinguish similar phrases, such as “the dog bit the man” and “the man bit the dog,” despite identical vocabulary. Earlier work pointed to specific regions in the left temporal cortex supporting these role assignments, underscoring the brain’s reliance on structure as well as meaning.
In contrast, researchers note that large language models used in tools like ChatGPT and Claude build their knowledge from massive text corpora, encoding word and sentence meanings as high‑dimensional vectors. To investigate whether these networks mirror brain processing, the team compared several modeling approaches against brain activity measured by fMRI during reading tasks.
Study design and modelling approaches
The researchers designed 108 single-clause English sentences, varied in adjectives, timing, location and substantive words, and included role-swapped constructions to swap grammatical positions while keeping content similar. Across 5,778 sentence pairs, 918 pairs shared much of their vocabulary, so discrimination depended on structure rather than vocabulary.
Four principal modelling approaches were preregistered. Two used numerical vectors: a baseline word-averaging model ignoring word order, and a transformer‑level category averaging five models (ERNIE 2.0, AMRBart, SentBERT, DefSent, and OpenAI embeddings). The other two explicit encodings of structure included a graph-based Abstract Meaning Representation model and a hybrid model that used GPT-4 to tag words by eight semantic roles and built separate vectors for each role with fixed role weights.
The neuroimaging study recruited 39 native English speakers in Melbourne, aged 18 to 40 (average 22). After excluding nine participants for head motion or attention checks, 30 remained. Inside a 7‑Tesla MRI scanner, participants read all 108 sentences four times and answered 40 comprehension questions.
Analyses centred on the middle three seconds of each trial, using representational similarity to compare brain activity patterns with model-derived similarities across all sentence pairs. The focus was on language-network voxels and cortex outside early visual regions. The researchers also tested whether the models’ pairwise similarities tracked grossly with brain responses while controlling for sentence length.
Across the full set of sentences, all four model families showed positive alignment with language-network activity, though the magnitude of alignment was small, as is typical with fMRI data. On average across participants, there was no statistically significant difference between word-averaging and transformer models; the hybrid model showed a modest edge, and the graph-based model aligned best.
When the analysis narrowed to the 918 same‑vocabulary pairs, results shifted markedly. The word-averaging model correlated negatively with language-network activity, because it treated role-swapped sentences as nearly identical, while the brain treated them as distinct. Transformers, while outperforming the word-averaging baseline, still showed an overall negative relationship with brain activity.
By contrast, the hybrid and graph models achieved a modest positive alignment with brain activity on these same-vocabulary pairs, with no significant difference between the two. The researchers argued that the main takeaway is not that one model is superior overall, but that word-averaging and transformer representations fail to capture who-did-what roles as the brain does.
“We found evidence that LLMs process language differently to humans,” said James Fodor, one of the study’s authors. “This suggests that they may not ‘understand’ language in a similar way to humans.”
The correlations observed were small in absolute terms, around -0.05 for the transformer average and 0.07 for the hybrid model on the same-vocabulary pairs, yet the pattern held across participants.
Further analysis showed the Llama 3 transformer only weakly aligned with brain activity in the early layers (2 and 3) and nearly no positive relationship in deeper layers, with visual cortex control dampening even these early connections. The strongest alignment with structured models appeared in temporal regions within the language network, though other areas, including the default mode network, also showed sensitivity to grammar.
On the vocabulary‑matched pairs, the graph model’s strongest brain alignment emerged in the visual cortex, a detail some peers flagged as a potential confound due to text‑level features rather than grammar.
Longer sentences tended to elicit similar brain activity across regions regardless of meaning, a pattern the authors suggest could reflect memory load or deeper processing. Even after accounting for this, the hybrid model remained positively aligned on same‑vocabulary pairs, while transformer and word‑averaging models did not.
The researchers conducted an online validation with 502 native English speakers from Australia and the United States, of whom 486 completed validity checks. Across all 5,778 pairs, transformers and word-averaging models best matched human similarity ratings, but on the 918 same‑vocabulary pairs, the word-averaging model fared worst while the hybrid model aligned most closely with human judgments, followed by transformers and the graph model. Participants’ ratings, however, diverged from the hybrid model in role-swapped pairs, with humans judging them as more similar than the model did.
In an exploratory, non‑preregistered attempt, the researchers fed GPT‑4 prompt text asking it to rate each pair’s similarity on a 0 to 1 scale. GPT‑4’s own ratings aligned more closely with human judgments than any model’s internal embeddings, suggesting that while LLMs can utilise grammar to answer questions about meaning, their internal representations do not mirror brain syntax and semantics.
“Our results do not show that LLMs cannot produce relevant or meaningful language, only that they likely do not understand such language in the way humans do,” Fodor noted. “This arises from differences in the way LLMs are trained and the tasks they are designed to perform.”
The authors caution that the study’s stimuli were highly controlled single-clause sentences, and the generalisability to natural language is an open question. They also emphasise that while some transformer models matched human ratings in some conditions, no single system reproduced brain activity across all measures, and when visual cortex activity was controlled, transformers tended to underperform relative to the structured models.
Commentary accompanying the study noted that the transformers examined were primarily sentence-embedding systems, with Llama 3 among them; the internally wired mechanisms of more sophisticated chatbots like GPT-4 and Claude were not analysed. Online ratings also showed little correlation with fMRI patterns on the same‑vocabulary subset, underscoring potential differences between passive reading and direct pairwise comparison tasks.
Looking ahead, Fodor urged caution in interpreting what the results mean for real‑world AI use. “This is one example of an important point in our age of AI. Just because a computer model can perform a task to a human level, it doesn’t follow that the model is doing so in a human-like way,” he said. “Whether such differences lead to practically important limitations for LLMs and their capabilities is a subject for further research.”
The study, titled Large language models poorly capture how word order shapes representations of sentence meaning in the human brain, was authored by James Fodor, Carsten Murawski, and Shinsuke Suzuki.
