Guideline-Grounded Rheumatology Chatbots Receive Positive Feedback, but Safety Validation Remains Essential
Published in Artificial Intelligence, Journals, Tools.
A study published on October 5, 2026 in the Journal of Medical Systems evaluated ten disease-specific chatbots grounded in German rheumatology guidelines. They used GPT-4o with retrieval-augmented generation and were disseminated through 13 rheumatology centres and six patient organisations.
Between September 2025 and January 2026, the study recorded 6,291 question–answer exchanges across 1,534 sessions. Users rated 2,671 responses, of which 92.9% received positive feedback. The systems declined to answer in 4.2% of exchanges.
An automated language-model evaluator rated 79.1% of answers completely correct and 95.3% completely safe. These are evaluator scores, not proof of clinical safety. Agreement with physician assessments of guideline adherence was weak; outdated guideline content also produced incorrect statements about drug approvals.
The study did not establish improved clinical outcomes. No real-time clinical safety monitoring or escalation pathway was implemented. The authors call for independent validation and studies of educational effectiveness before broader reliance on these tools.
Source: Wilhelmi and colleagues, Journal of Medical Systems, October 5, 2026.