500 - Assessment of Chatbots’s knowledge in Special Needs Dentistry

Purpose: Chatbots, powered by big language models, use artificial intelligence to mimic real human chats; however, their ability for decision-making in dentistry hasn't been investigated much. This study aimed to evaluate the accuracy and consistency of chatbots in answering questions related to special needs dentistry.

Methods: Nine publicly accessible chatbots, including Google Bard, ChatGPT 4, ChatGPT 3.5, Llama, Sage, Claude 2 100k, Claude-instant, Claude-instant-100k, and Google Palm, were evaluated on their ability to answer a set of 25 true/false questions related to special needs dentistry and 15 questions for syndrome diagnosis based on their oral manifestations. Each chatbot was asked independently 3 times at 3-week intervals from November to December 2023 and the responses were evaluated by pediatric dentistry residents. Using SAS software, the comparison of accuracy rates among the chatbots was conducted employing the exact Wilcoxon test, a well-established statistical procedure renowned for its robustness in scientific analyses. Cronbach’s alpha was utilized to measure the consistency of the chatbots' responses.

Results: Chatbots had an average accuracy of 55%±4% in answering all questions, 37%±6% in diagnosis, and 67%±8% in answering true/false questions. No significant difference (p>0.05) of the accuracy rates was detected between any pairwise chatbot comparison. All chatbots demonstrated acceptable reliability (Cronbach alpha >0.7), with Claude instant having the highest reliability of 0.93.

Conclusion: Chatbots exhibit high consistency in responding to all questions, and a higher accuracy in responding to true/false questions than diagnostic questions.