A Policy-guided LLM Pipeline for Safer Clinical Decision Support: Error Analysis on High-risk Medication Queries

Keywords:
LLM Safety, Clinical Decision Support, Guardrails, Medication Safety, Error Analysis.

Abstract:
Large language models can generate fluent and often convincing answers in medical contexts, but in high-risk settings, fluency alone is not enough. This paper presents a policy-guided LLM pipeline for medication-related clinical decision support, designed to classify prompts into safety-oriented decision categories: ACCEPT, WARN, DEFER, ESCALATE, or REFUSE. The system combines rule-based risk detection, guardrail routing, and a decision policy layer to handle prompts involving dosage, pregnancy, drug interactions, self-adjustment, and other safety-critical contexts. The pipeline was evaluated on a benchmark of 55 medication-related questions using two baseline models. Both models produced the same overall system accuracy of 60.0%, with a false accept rate of 16.4%, suggesting that the main limitations are not model-specific but structural. Three recurring failure modes emerged: false acceptance of implicitly risky prompts, over-escalation of educational or professional-context questions, and under-escalation of self-adjustment or dangerous-intent cases. These findings make the system useful not only as a prototype, but also as a transparent framework for studying where safetyoriented LLM pipelines succeed and where they still fail.

CITATION:

IEEE format

Copy to clipboard

A. Stevanović, M. Jovanović, “A Policy-guided LLM Pipeline for Safer Clinical Decision Support: Error Analysis on High-risk Medication Queries,” in Sinteza 2025 - International Scientific Conference on Information Technology, Computer Science, and Data Science, Belgrade, Singidunum University, Serbia, 2026, pp. 27-31. doi:10.15308/Sinteza-2026-27-31

APA format

Copy to clipboard

Stevanović, A., Jovanović, M. (2026). A Policy-guided LLM Pipeline for Safer Clinical Decision Support: Error Analysis on High-risk Medication Queries. Paper presented at Sinteza 2025 - International Scientific Conference on Information Technology, Computer Science, and Data Science. doi:10.15308/Sinteza-2026-27-31

BibTeX format
Download

RefWorks Tagged format
Download