E-ISSN 1658-7820
 
2026, Vol: 10, Issue: 1
10 / 1Current Issue Online First Archive Aims and Scope Abstracting & Indexing Most Accessed Articles Most Downloaded Articles Most Cited Articles

Original Article
Online Published: 10 Jan 2026
 


Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models

Ibraheem Altamimi, Mohammed Abdullah Alsaawi, Abdulmalik Alomayyer, Mohamad Hani Temsah, Mohammed Alquhidan, Salem Alshehri, Abdulmajeed Khalid Alzuwayyid, Naif Aljamlan, Yazan Yousef Alghammas.


Abstract
Objective: This study aimed to assess and quantify the referencing hallucination of the four major large language models (LLMs) chatbots.
Methods: This was a cross-sectional study to compare four LLM-chatbots (GPT-4o with internet access, GPT-4 Legacy, GPT-4o, and O1) in referencing hallucination. A total of 400 clinically based prompts (100 prompts for each bot) were made and asked separately to each chatbot. Answers from chatbots were tested using the Reference Hallucination Score (RHS).
Results: This study analyzed 400 prompts (200 basic, 200 advanced) answered by four chatbots (100 per model) using a seven-item hallucination score (KR-20 = 0.808). Reference errors were frequent: 255/400 responses (63.7%) had incorrect or missing publication dates, 240/400 (60.0%) contained broken or misleading links, and only 79/400 (19.8%) provided a correct DOI. GPT-4o Int (286 erroneous vs 414 correct elements) and GPT-4o (317 vs 383) showed the lowest hallucination burden, whereas O1 Preview (621 vs 79) performed worst, with GPT-4 Legacy intermediate.
Conclusion: Medical LLM-based chatbots exhibited different degrees of bibliographic hallucination depending on model type. More recent iterations, such as GPT-4o, have decreased such hallucination, albeit not extinguishing it. Before deploying such tools for clinically relevant communication or decision assistance, reference audits structured within a framework alongside human verification must be conducted.

Key words: Clinical decision support, hallucination, large language models, medical chatbots, medical application.


 
ARTICLE TOOLS
Abstract
PDF Fulltext
How to cite this articleHow to cite this article
Citation Tools
Related Records
 Articles by Ibraheem Altamimi
Articles by Mohammed Abdullah Alsaawi
Articles by Abdulmalik Alomayyer
Articles by Mohamad Hani Temsah
Articles by Mohammed Alquhidan
Articles by Salem Alshehri
Articles by Abdulmajeed Khalid Alzuwayyid
Articles by Naif Aljamlan
Articles by Yazan Yousef Alghammas
on Google
on Google Scholar


How to Cite this Article
Pubmed Style

Altamimi I, Alsaawi MA, Alomayyer A, Temsah MH, Alquhidan M, Alshehri S, Alzuwayyid AK, Aljamlan N, Alghammas YY. Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. IJMDC. 2026; 10(1): 084-090. doi:10.24911/IJMDC.51-1763928059

Web Style

Altamimi I, Alsaawi MA, Alomayyer A, Temsah MH, Alquhidan M, Alshehri S, Alzuwayyid AK, Aljamlan N, Alghammas YY. Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. https://www.ijmdc.com/?mno=298851 [Access: January 25, 2026]. doi:10.24911/IJMDC.51-1763928059

AMA (American Medical Association) Style

Altamimi I, Alsaawi MA, Alomayyer A, Temsah MH, Alquhidan M, Alshehri S, Alzuwayyid AK, Aljamlan N, Alghammas YY. Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. IJMDC. 2026; 10(1): 084-090. doi:10.24911/IJMDC.51-1763928059


Vancouver/ICMJE Style

Altamimi I, Alsaawi MA, Alomayyer A, Temsah MH, Alquhidan M, Alshehri S, Alzuwayyid AK, Aljamlan N, Alghammas YY. Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. IJMDC. (2026), [cited January 25, 2026]; 10(1): 084-090. doi:10.24911/IJMDC.51-1763928059


Harvard Style

Altamimi, I., Alsaawi, . M. A., Alomayyer, . A., Temsah, . M. H., Alquhidan, . M., Alshehri, . S., Alzuwayyid, . A. K., Aljamlan, . N. & Alghammas, . Y. Y. (2026) Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. IJMDC, 10 (1), 084-090. doi:10.24911/IJMDC.51-1763928059


Turabian Style

Altamimi, Ibraheem, Mohammed Abdullah Alsaawi, Abdulmalik Alomayyer, Mohamad Hani Temsah, Mohammed Alquhidan, Salem Alshehri, Abdulmajeed Khalid Alzuwayyid, Naif Aljamlan, and Yazan Yousef Alghammas. 2026. Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. International Journal of Medicine in Developing Countries, 10 (1), 084-090. doi:10.24911/IJMDC.51-1763928059


Chicago Style

Altamimi, Ibraheem, Mohammed Abdullah Alsaawi, Abdulmalik Alomayyer, Mohamad Hani Temsah, Mohammed Alquhidan, Salem Alshehri, Abdulmajeed Khalid Alzuwayyid, Naif Aljamlan, and Yazan Yousef Alghammas. "Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models." International Journal of Medicine in Developing Countries 10 (2026), 084-090. doi:10.24911/IJMDC.51-1763928059


MLA (The Modern Language Association) Style

Altamimi, Ibraheem, Mohammed Abdullah Alsaawi, Abdulmalik Alomayyer, Mohamad Hani Temsah, Mohammed Alquhidan, Salem Alshehri, Abdulmajeed Khalid Alzuwayyid, Naif Aljamlan, and Yazan Yousef Alghammas. "Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models." International Journal of Medicine in Developing Countries 10.1 (2026), 084-090. Print. doi:10.24911/IJMDC.51-1763928059


APA (American Psychological Association) Style

Altamimi, I., Alsaawi, . M. A., Alomayyer, . A., Temsah, . M. H., Alquhidan, . M., Alshehri, . S., Alzuwayyid, . A. K., Aljamlan, . N. & Alghammas, . Y. Y. (2026) Evaluating hallucination behavior in large language models for medical applications: a comparative study of four popular chatbot models. International Journal of Medicine in Developing Countries, 10 (1), 084-090. doi:10.24911/IJMDC.51-1763928059




Most Viewed Articles
Most Accessed Articles

  • Marginal adaptation of fixed prosthodontics
    Shahad Mohammed Halawani, Sahar Amer Al-Harbi
    IJMDC. 2017; 1(2): 78-84
    » Abstract » doi: 10.24911/IJMDC.1.2.7

  • Parasites in central nervous system
    Amal Ahmed Mohamed, Asmaa Mohamed Mansour
    IJMDC. 2017; 1(2): 72-77
    » Abstract » doi: 10.24911/IJMDC.1.2.6

  • Non-diabetic retinopathy: a case report
    Hassan F. Alkwikbi, Nabil Mamon Abdelfattah, Haithem Mamon Abdelfattah
    IJMDC. 2017; 1(1): 33-37
    » Abstract » doi: 10.24911/IJMDC.1.1.6

  • Vitamin D deficiency among high school female students in Riyadh
    Sahar Mohsen Aldhafeeri, Fatemah Ibraheem Alrawaji, Abrar Mubarak Algharbi, Alanoud Abdullah Alhessan, Ayat Sami Qabaja, Ghaleah Abdulaziz Alnassar, Ghaliah Ofayn Alsulami, Mashael Belal Alrubaian, Norah Abdulaziz Alhatim, Salehah Salih Alotaibi, Zahra Ali Alkhamees
    IJMDC. 2017; 1(1): 18-23
    » Abstract » doi: 10.24911/IJMDC.1.1.4

  • Prognosis of chronic complications of diabetes mellitus (DM) after multiple events of diabetic ketoacidosis (DKA)
    Baraa Faiez Rajab, Anwar Essa Alamrim, Ali Essa Alamri
    IJMDC. 2019; 3(5): 474-479
    » Abstract » doi: 10.24911/IJMDC.51-1546551993

    •
    Most Downloaded
    Top Downloaded Articles

  • Perception and attitude of medical students towards trauma and orthopedic teaching: a cross-sectional study
    Nawaf Hameed Almohammadi, Faisal Ali Almesned, Atif Omar Alhawsah, Sami Aoudah Alahmari, Madloul Jassim Alshammari, Hamad Ali Alkhunayfir
    IJMDC. 2019; 3(6): 503-508
    » Abstract » doi: 10.24911/IJMDC.51-1548706801

  • Anti-IL-17 markedly inhibited the in vitro granuloma induced by Schistosoma mansoni soluble egg antigen.
    Asmaa Ezz, Irene S. Gmail, Fatma Elzahraa H. Salem, Rabab Salama, Soad Nady
    IJMDC. 2018; 2(January 2018): 38-46
    » Abstract » doi: 10.24911/IJMDC.2.1.7

  • Vitamin D deficiency among high school female students in Riyadh
    Sahar Mohsen Aldhafeeri, Fatemah Ibraheem Alrawaji, Abrar Mubarak Algharbi, Alanoud Abdullah Alhessan, Ayat Sami Qabaja, Ghaleah Abdulaziz Alnassar, Ghaliah Ofayn Alsulami, Mashael Belal Alrubaian, Norah Abdulaziz Alhatim, Salehah Salih Alotaibi, Zahra Ali Alkhamees
    IJMDC. 2017; 1(1): 18-23
    » Abstract » doi: 10.24911/IJMDC.1.1.4

  • Antibiotic Resistance Crisis
    Nuha Saad Mobarki, Buthaina Abdullah Almerabi, Ahmad Hattan Hattan
    IJMDC. 2019; 3(6): 561-564
    » Abstract » doi: 10.24911/IJMDC.51-1549060699

  • Marginal adaptation of fixed prosthodontics
    Shahad Mohammed Halawani, Sahar Amer Al-Harbi
    IJMDC. 2017; 1(2): 78-84
    » Abstract » doi: 10.24911/IJMDC.1.2.7

    •
    Most Cited Articles
    Most Cited Articles

  • Antibiotic Resistance Crisis
    Nuha Saad Mobarki, Buthaina Abdullah Almerabi, Ahmad Hattan Hattan
    IJMDC. 2019; 3(6): 561-564
    » Abstract » doi: 10.24911/IJMDC.51-1549060699
    Cited : 34 times [Click to see citing articles]

  • Prevalence of depression, anxiety and stress among diabetes mellitus patients in Arar, Northern Saudi Arabia
    AMohammed Elamin Mukrim, Nujud Muteb D. Alshammari, Wurud Muteb D. Alshammari, Maali Subhi T. Alshammari, Yousef Nasif R. Alshammari, Abdulaziz Sael M. Alshammari, Maisa Hamad F. Alanazi, Abdulelah Abdulrhman F. Alzammam, Muharib Mana M. Alshammari, Slwa Khaled Abu Asyah, Aseel Jamal Alzuhayri, Mushref Saeid Alshammari
    IJMDC. 2019; 3(3): 229-233
    » Abstract » doi: 10.24911/IJMDC.51-1542576396
    Cited : 8 times [Click to see citing articles]

  • Knowledge, attitude, and practice of breast self-examination toward breast cancer among female students at King Saud University in Riyadh, Saudi Arabia
    Abdullah Nasser Alomair, Dania Ghazi Felemban, Mohannad Sami Felemban, Jameel Abdullah Awadain, Ammar Saud Altowairqi, Nawaf Fawzan Alfawzan, Fatimah Mohammed Almazayen, Abdulrahman Jalwi Korkoman, Nawaf Saad Alrusayyis
    IJMDC. 2020; 4(2): 429-434
    » Abstract » doi: 10.24911/IJMDC.51-1576668182
    Cited : 8 times [Click to see citing articles]

  • Population awareness about rheumatoid arthritis in Jazan region, Saudi Arabia
    Ahmad Ali Hazzazi, Mohssen Hassen Ageeli, Ahmed Ali Muyidi, Abdulaziz Mohammad Abulgasim, Abdullah Ahmad Yateemi, Nabil Alhakami
    IJMDC. 2020; 4(3): 668-675
    » Abstract » doi: 10.24911/IJMDC.51-1576010943
    Cited : 4 times [Click to see citing articles]

  • Prevalence and factors associated with depression among health care workers in National Guard Hospital in Riyadh, KSA
    Noor Mohammad AlFahhad
    IJMDC. 2018; 2(September 2018): 92-96
    » Abstract » doi: 10.24911/IJMDC.51-1526306040
    Cited : 4 times [Click to see citing articles]

  • Effect of inter-pregnancy interval on pregnancy outcome: a prospective study at Fayoum, Egypt
    Eman M. Mahfouz, Naglaa A. El-Sherbiny, Wafaa Y. Abdel Wahed, Nashwa S. Hamed
    IJMDC. 2018; 2(May 2018): 38-44
    » Abstract » doi: 10.24911/IJMDC.51-1520268317
    Cited : 4 times [Click to see citing articles]

  • Diabetic neuropathy in Saudi Arabia: a comprehensive review for further actions
    Mohammad Mater Aljohani, Amjad Taj Karam, Ayman Abdulaziz Alamri, Mohammed Hamed Manfaloti, Hussain Abbas Alnakhli, Hatem Ahmed Shaqroon
    IJMDC. 2020; 4(11): 2008-2013
    » Abstract » doi: 10.24911/IJMDC.51-1601343665
    Cited : 4 times [Click to see citing articles]

  • Public Awareness of coronary artery disease risk factors in Qassim, Saudi Arabia: a cross-sectional study
    Abdullah Muzil Albadrani, Saleh Sulaiman Al-Ajlan, Abdullah Saer E. Alharbi, Abdulelah Saud Alharbi, Saif Mohammed Radhi Alharbi
    IJMDC. 2020; 4(3): 593-599
    » Abstract » doi: 10.24911/IJMDC.51-1572217065
    Cited : 4 times [Click to see citing articles]

  • Ischemic stroke: prevalence of modifiable risk factors in the Saudi population
    Muhannad Noor Alharbi, Atheer Khalid Alharbi, Mousa Atqan Alamri, Abdulmalik Ayedh Saad Alharthi, Abdulrahman Moneer Alqerafi, Mohamed Noor Alharbi
    IJMDC. 2019; 3(7): 601-603
    » Abstract » doi: 10.24911/IJMDC.51-1548376707
    Cited : 3 times [Click to see citing articles]

  • Prevalence and risk factors for diabetic nephropathy in type 2 diabetic patients, Taif City, Saudi Arabia
    Basel Saad Alzahrani, Turki Hamdan Alzidani, Abdullah Mohammed Alturkistani, Hani Abozaid
    IJMDC. 2019; 3(2): 167-172
    » Abstract » doi: 10.24911/IJMDC.51-1541336905
    Cited : 3 times [Click to see citing articles]

  • Assessment of knowledge and awareness regarding thyroid disorders among Saudi people
    Assem Saleh Ali Almuzaini , Bayan ahmad I Alshareef , Sundos Hamoud O Alghamdi, Ayman Adnan Munshy, Abdulmajeed Khalid M. Aljarallah, Saud Abdulaziz A Salman, Khalid Saud Alroqi, Reem Dayel A Alkhaldi
    IJMDC. 2019; 3(12): 1070-1076
    » Abstract » doi: 10.24911/IJMDC.51-1568037206
    Cited : 3 times [Click to see citing articles]

  • Assessment of knowledge, attitude, and practice in relation to use of isotretinoin among Al-Madinah population, Saudi Arabia
    Amr Molla, Hassan Abdullah Alrizqi, Emtinan Mohammed Salem Alruhaili, Saad Abdullah Alrizqi, Arwa Musaad Alsubhi
    IJMDC. 2020; 4(1): 107-112
    » Abstract » doi: 10.24911/IJMDC.51-1571752077
    Cited : 3 times [Click to see citing articles]

  • Ethical considerations in doctors & pharmaceutical industries relationship: a narrative review
    Mohammed Saud Almasri, Yousef Rafat Bukhari, Bandar Saleh Alzuair, Mansour Khalid Almadi, Amro Khalid Bin Abdulrahman
    IJMDC. 2020; 4(1): 244-252
    » Abstract » doi: 10.24911/IJMDC.51-1572794591
    Cited : 3 times [Click to see citing articles]

  • Assessment of the knowledge of pregnant women regarding the effects of GDM on mothers and neonates at a Maternal and Children hospital in Najran, Saudi Arabia
    Amina Abdo Salhi, Medical Intern, Majed Saeed Alshahrani, Maram Mohammed Alyamin, Waad Ahmed Hamdi, Sarah Raja Alyami, Albatool Saleh Almagbool, Nujoud Hamad Almoqati, Seham Hamad Almoqati, Esra'a Amer Nasser Al-Saaed, Hessa Salem Al Habes, Batool Mubarak Abdullah Alwdei, Ahlam Yhya Saeed Alyami, Wesal Ahmed Alassiri
    IJMDC. 2019; 3(4): 370-375
    » Abstract » doi: 10.24911/IJMDC.51-1545817090
    Cited : 3 times [Click to see citing articles]

  • Awareness of the Saudi population about the role of the Emergency Rooms
    Zead Ibrahim A. Alhussain, Mohammed Ali A. Alghamdi, Rana Mohamed Ahmed, Abdullah Yahya Aldhban, Abdulrahman Ali Alghamdi, Naif Shalan J. Alalyani
    IJMDC. 2019; 3(11): 957-962
    » Abstract » doi: 10.24911/IJMDC.51-1568717752
    Cited : 3 times [Click to see citing articles]

    •

    About International Journal of Medicine in Developing Countries

    International Journal of Medicine in Developing Countries is a peer-reviewed, open-access international medical journal. Journal Title Internat ... Read more.



    For best results, please use Internet Explorer or Google Chrome.

    Contact Information

    Publisher: Waraqa Publishing House, Riyadh, Saudi Arabia



    Hosted by: Discover STM Publishing Ltd. 164 Moyville. Rathfarnham, Dublin, D16Y4E, Ireland.

    Hours:  M-F: 7:30am - 4:30pm 



    Emails:

    Journal Editorial Office:  contact@ijmdc.com

    All queries regarding the publishing or acceptance of the manuscript should be directed to the Journal Editorial Office at Riyadh, Saudi Arabia. 

    Hosting service is provided by Discover STM (Ireland):  contact@discoverpublish.com
    POLICIES & JOURNAL LINKS
    Author Login
    Reviewer Login
    Author LoginReviewer LoginAbout PublisherEditorial PoliciesEditorial Review PolicyPeer Review PolicyEditorial & Peer Review ProcessAuthor's Rights and ObligationsPublication Ethics and Publication Malpractice StatementConflict of Interest PolicyPlagiarism PolicyProtection of Research Participants (Statement On Human And Animal Rights)Corrections, Retractions & Expressions of ConcernSelf-Archiving PoliciesAdvertising PolicyLicense InformationCopyright Information