A study into sustained conversational misinformation pressure on large language models found that, when the same false statement was repeatedly fed into a dialogue, the models for some time affirmed it in a fraction of responses. The research tested seven models and tracked how fallibility, persuadability and correctability evolved across conversations, with the findings published in Scientific Reports. Lead author Jordan Rodriguez of the University of Arizona outlined the aim of probing how models respond under continuous misinformation, rather than in a single interaction.
The researchers framed their inquiry around three qualities: fallibility (the tendency to accept falsehoods under pressure), persuadability (the likelihood of becoming more accepting as prompts become more argumentative), and correctability (whether a model could recognise and amend earlier errors when re-evaluated). They aimed to see whether repeating misinformation would alter a model’s stance and whether models could self-correct when given a chance to reconsider previous replies.
The study evaluated seven large language models: ChatGPT 3.5, ChatGPT 4o, ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama-3-70B and DeepSeek-R1. Researchers wrote customised software to interact with each model via its API, preserving full conversation history so each prompt built on prior exchanges. They crafted 100 false statements spanning various subjects, from plainly false assertions to those requiring contextual knowledge to debunk.
To gauge how obscure a statement was, the team recorded the number of Google search results it produced and used this obscurity index to test whether models could refute misinformation on topics with more online information.
The experiments were conducted in three stages. In the first, the false statement was repeated 50 times within a single conversation (ChatGPT 3.5 was also tested with 10 and 25 repetitions). In the second, researchers used 33 increasingly forceful prompts to test persuadability, reintroducing the misstatement at intervals. In the third, responses in which models had previously affirmed the misinformation were used as prompts in new, independent conversations to observe whether the models would recognise and correct the errors without the pressure of the original dialogue.
In the first experiment, ChatGPT 3.5 exhibited four response patterns: it almost always rejected the false statement, sometimes began by rejecting and then agreed, oscillated between agreement and disagreement, or maintained agreement throughout. Over the 5,000 model turns across the study, ChatGPT 3.5 affirmed the misinformation in 12.3 per cent of responses, with other models showing markedly different propensities for endorsement.
Across all models, the rate of affirmation varied considerably. Claude 3.5 Sonnet was least prone to affirming falsehoods, at 0.08 per cent, while Gemini 1.5 Pro reached the highest marked level among the newer models at 3.1 per cent; the remainder stayed under 1.5 per cent.
The researchers found that statements with greater obscurity were more likely to be affirmed under repetition: obscure prompts, though representing 18 per cent of the statements, accounted for 23.4 per cent of affirmations, while common statements, making up 39 per cent of prompts, accounted for 33.3 per cent of affirmations. They suggested models may be more easily misled about topics less represented in their training data.
In the second experiment, most models resisted the argumentative pressure, with affirmation rates equal to or lower than those under simple repetition. ChatGPT 3.5’s rate fell to 2.6 per cent, and ChatGPT 4o and 4o-mini affirmed misinformation in only 0.1 per cent of responses. Claude 3.5 Sonnet’s rate rose to 3.5 per cent, and the influence of statement obscurity disappeared under this testing regime.
DeepSeek, however, behaved as an exception. Its affirmation rate surged from 1.0 per cent under repetition to 22.2 per cent under argumentative prompts, peaking during the severe argument phase before tapering when insults appeared. Annotators noted DeepSeek’s frequent use of sarcasm or satirical storytelling, which made it difficult to judge whether it was genuinely agreeing or merely mimicking the claim.
In the third experiment, four models—ChatGPT 4o, ChatGPT 4o-mini, Gemini 1.5 Pro and DeepSeek—corrected all of their earlier errors when given a second look. Llama-3-70B corrected 96 per cent of the 250 prior errors, while ChatGPT 3.5 corrected 32 per cent. Claude 3.5 Sonnet, which had made only four errors in the first phase, failed to correct any of them, though researchers warned that the small sample size for this model limits conclusions.
“These findings demonstrate that conversational dynamics expose LLM failure modes invisible to standard evaluation, with direct consequences for model selection and use in truth-critical domains,” the study authors concluded.
The study advances understanding of how Generative AI systems respond to misinformation introduced during sustained conversations. It acknowledges that models are evolving rapidly, with the tested versions often superseded by newer releases, meaning results may not generalise to future iterations. It also notes that the experiments used the providers’ default settings, which differ across organisations, and that annotators were affiliated graduate students who did not know which model produced which response. The frequency of such sustained misinformation pressure in real-world use remains an open question for further research.
The paper, Fallibility, persuasibility, and correctability of large language models under sustained conversational misinformation pressure, was authored by Jordan Rodriguez, Zachary Hansen, Luis Anda, Katelyn Rohrer, Camila Grubb, Enrique Noriega-Atala, Mihai Surdeanu, and Marvin J. Slepian.
