Evaluation of large language models for clinical sign-based oral assessment in dogs compared with veterinary practitioners

This study aimed to evaluate the diagnostic performance of Large Language Models (LLMs) in canine oral assessment by comparing their accuracy and agreement with a panel of expert veterinary academicians. Sixty lateral oral photographs from clinical canine cases were evaluated for nine oral clinical signs by novice veterinarians, a panel of expert veterinary academicians, and four LLMs (ChatGPT-5.1, ChatGPT-5.1 Thinking, Claude-Sonnet 4.5, Gemini-Pro 3). Expert consensus served as the reference standard. Intra-model consistency testing demonstrated substantial reliability for Claude-Sonnet 4.5 (90 % full agreement; kappa = 0.75), moderate reliability for Gemini-Pro 3 (70 %; kappa = 0.57) and ChatGPT-5.1 (50 %; kappa = 0.43), and weak reliability for ChatGPT-5.1 Thinking (30 %; kappa = 0.33). Overall diagnostic accuracy across evaluators ranged from 33.3 % to 95 %. Novice veterinarians achieved statistically significant agreement in six clinical signs (e.g., calculus kappa = 0.577; p < 0.001; pigmentation kappa = 0.378; p < 0.001), outperforming all LLMs in diagnostic breadth. Among LLMs, ChatGPT-5.1 and ChatGPT-5.1 Thinking reached significant weak-to-moderate agreement for several clinical signs (e.g., calculus kappa = 0.356-0.485; p <= 0.001). LLMs outperformed novices only in traumatic lesions (Gemini-Pro 3 kappa = 0.290; p = 0.001). All evaluators exhibited chance-level performance for tooth fractures. In conclusion, novice veterinarians demonstrated higher diagnostic consistency and agreement with experts than all evaluated LLMs. Although ChatGPT-5.1 models showed the strongest AI performance, current LLMs remain insufficient for independent diagnostic use in veterinary dentistry.

Journal

VETERINARY JOURNAL

Year

2026

Volume

317

ISSN

1090-0233

eISSN

1532-2971

DOI

10.1016/j.tvjl.2026.106611

WoS UID

WOS:001705078200001