AI was told about a fake disease and it didn't take long before chatbots thought it was real

Researchers invented a fake illness called ‘bixonimania’ by creating false studies about the condition, and it wasn’t long before AI chatbots began telling people it was real. 

AI has surged in popularity recently, with millions of people worldwide using tools like ChatGPT and Claude for both work and personal use. 

However, medical researcher Almira Osmanovic Thunström from the University of Gothenburg in Sweden recently led a study that showed why relying on AI might not always get the best results. 

She and her team created a fake medical condition to see if large language models (LLMs) would take on board the misinformation and then give out advice, and the results make for interesting reading. 

AI chatbots began offering medical advice about the fake condition

If you’re an AI chatbot user, then you’re probably already aware of the potential for LLM to hallucinate or to give misinformation.

A recent report from The New York Times found that around one in every 10 AI Overviews in Google searches isn’t accurate, and AI companies even warn users to ‘double-check responses’ as chatbots can make mistakes.

Now, a new study from researcher Osmanovic Thunström and her team has shown just how easy it can be to trick AI into swallowing misinformation. 

The team created a fake medical condition called bixoniamania, writing a Medium blog post about it, as well as uploading two bogus studies written by a made-up researcher to the Preprints.org server. 

The blog post and studies described bixoniamania as a skin condition that was caused by spending too much time staring at screens and then rubbing tired eyes. 

“I wanted to see if I could create a medical condition that did not exist in the database,” Osmanovic Thunström told Nature. 

To make it clear to any humans reading the papers, Osmanovic Thunström inserted lots of red flags, including using a psychiatric term to describe a condition of the eye, claiming the fictitious researcher was from a university that doesn’t actually exist, and including the statement ‘this entire paper is made up’.

However, while a human may have picked up on those glaring warnings, the AI models didn’t. 

Within weeks of the fake studies being shared in 2024, the team began seeing AI chatbots repeating information about bixoniamania as if it were a real condition. 

AI systems, including Microsoft’s Copilot, Google’s Gemini, and ChatGPT, were all found sharing information about bixoniamania, despite it not being real. 

Researchers also spotted another ‘worrying’ issue

As well as being repeated to chatbot users, the researchers also found that the fake studies were being cited in peer-reviewed articles.

Osmanovic Thunström believes that some researchers are relying on AI-generated references without actually reading the original papers. 

“It is worrying when these major claims are just passing through the literature unchallenged, or passing through peer review unchallenged,” Osmanovic Thunström told Nature. 

“I think there’s probably a lot of other issues that haven’t been uncovered.”

Since the study, several of the AI chatbots have evolved and updated, and some now show suspicions about bixonimania. 

In a statement, Google said it was ‘transparent about the limitations of generative AI’ and noted that Gemini recommends its users ‘consult with qualified professionals’. 

While OpenAI said the models that power ChatGPT today are ‘significantly better at providing safe, accurate medical information. 

 Timeline of key AI breakthroughs