Clinically Valid & Ethically Compliant healthcare small-language models

This study has been written by Yma Health Engineering team and covers its 2025 endevours to build an cost-effective and portable (offline) model for frontier healthcare or personal fully-private use. Learn more about us and our work at https://lnk.yma.health/2grngqje

Data Collection & AI Insights

Where general interpretation and non-medical advice became the expected norm for users to query LLMs, an effective clinical appendage operated by a clinical institution is a challenging endeavour — from the point of accuracy (and resulting responsibility for the advice given) as well as from the point of customer data liability in terms of it being intepreted by external APIs of commercial models.

For AI models to deliver reliable healthcare recommendations, they require accurate, structured, and diverse datasets. One of Yma Health’s operating intents to exist has been to integrate patient-reported data, clinical records, and external medical studies to create a comprehensive AI-driven medical intelligence system.

Medical Knowledge Curation & AI Training

Our approach to curating and training AI models relied on selecting high-quality, domain-relevant content that aligns with the following principles:

1. Domain Relevance

All articles must pertain to the specific domain or field being addressed. They should provide accurate, contextually appropriate information tailored to the intended audience.

2. Evidence-Based Content

We prioritized articles grounded in the strongest possible evidence. This included peer-reviewed studies, systematic reviews, clinical guidelines, and meta-analyses to ensure reliability and scientific validity.

3. Comprehensive Language

Articles must be written in a clear, concise, and comprehensive manner. The language should be accessible to diverse audiences, allowing both professionals and laypersons to understand the content effectively.

4. Tone of Voice

The tone of voice used in the training content is an essential component of our model’s development. We taught our AI to adopt an empathetic, professional, and user-friendly tone to enhance communication and engagement.

A knowledge database in a specific domain is continuously updated with the latest scientific advancements. Our approach focused on manually curating and regularly updating high-quality data.

Data Sources

From Patients:

• Symptom tracking, telemedicine inquiries, and digital health assessments.
• Wearable data (e.g., heart rate, glucose monitoring).
• Medication adherence and lifestyle changes.

From Healthcare Providers:

• Electronic health records (EHRs), physician notes, and clinical imaging reports.
• Lab test results and diagnostic outcomes.
• Structured guidelines from top medical organizations (WHO, ADA, ESC, etc.).

Approach

Before gathering references for model development, a team of healthcare professionals and biomedical research experts defined the key aspects of the domain. For each aspect, we compiled a detailed list of relevant topics, ensuring comprehensive coverage with high-evidence information.

We use our standard approach to literature review and data collection.

Additionally, we identified related subdomains to enhance the modelʼs breadth of knowledge within the field. For example, in the case of GLP-1 receptor agonist (GLP-1RA) knowledge, an adjacent subdomain was cardiometabolic diseases. We covered this subdomain with up-to-date medical guidelines and comprehensive systematic reviews to ensure a well-rounded understanding of the field. Figure 1 presents a graphical representation of the database dedicated to the GLP-1RA domain.

The domain is divided into two subdomains. The first covered knowledge about existing indications for use. For each indication (an aspect of the subdomain), we extracted the most detailed, high-evidence information regarding topics such as efficacy and safety. For every aspect and its corresponding topics, we researched literature on all existing GLP-1RA drugs. Additionally, we included recent studies on drugs under development, along with updated information on currently approved medications.

The second subdomain included information about cardiometabolic disorders, as GLP-1RAs are used to treat diseases within this category. For the most prevalent of these conditions, we have included information about risk factors, prevention, lifestyle influences, and more.

Overall, our GLP-1RA domain knowledge included over 120 manually curated, evidence-based entries.

How we prepared data for model training

Large language models often process billions of parameters, but they lack depth in specialized domains. In medicine, current and verified data is critical. Thatʼs why we structured our dataset into several key layers:

1. Medical Guidelines & Protocols

We compiled around 100 documents related to diabetes and GLP-1 medications, including recommendations from leading U.S. endocrinology associations (Singhal et al., 2023b). Since clinical guidelines can be up to 100 pages long, we selected only the latest versions (published within the last 2–3 years) to ensure our AI incorporates the most up-to-date treatment approaches.

2. Scientific Studies & Reviews

We sourced approximately 200 research publications covering topics such as the side effects of semaglutide and related drugs, efficacy across different age groups, and drug interactions. A significant portion of our insights comes from large-scale meta-analyses, ensuring high data reliability.

3. General Medical Knowledge

To establish a foundational understanding of medical terminology and pathophysiology, we included medical textbooks and open-source datasets like MedMCQA.

4. Real-World Medical Conversations (Anonymized)

AI models that donʼt understand the nuance of doctor-patient communication often produce responses that are too formal, detached, or ethically questionable. To address this, we incorporated transcripts of actual medical consultations (with patient consent), enabling our AI to adopt a more natural, empathetic tone. Research by Johri et al. (2023) indicates that this approach increases user satisfaction with AI-driven tools by 1.5–2 times.

5. Specialized “Safety” & “Awareness” Datasets

These datasets help our AI handle sensitive or provocative medical inquiries more cautiously. If the AI lacks confidence in its response, it is trained to redirect users to a qualified healthcare professional (Bedi et al., 2024).

All data was carefully cleaned and formatted through validation, duplicate removal, and expert review. The final dataset includes around 100,000 structured records (question-answer pairs or dialogue scenarios), meticulously annotated with contextual insights (Clusmann et al., 2023).

Model Training Process

The training process was structured into several key phases to optimize medical accuracy, responsiveness, and efficiency.

1. Base Fine-Tuning

Initially, we fine-tuned the Qwen 2.5 Instruct 7B model on a corpus of medical texts without enforcing a specific response style. This step allowed the model to deepen its understanding of medical terminology and diabetes treatment logic.

During the Pre-Training (PT) phase, we used unstructured medical texts, adapting methodologies for instruction-tuned models.

Inspired by Yizhong Wang et al.ʼs “Self-Instruct: Aligning Language Models with Self-Generated Instructions”, we applied the Self-Instruct technique to generate synthetic queries, enabling the model to integrate new medical knowledge without forgetting its instruction-following capabilities.

2. SFT (Supervised Fine-Tuning)

We created a question-answer dataset based on real (or expert-simulated) medical cases related to GLP-1 therapy.

The model was trained to generate concise yet medically accurate responses, ensuring clinical reliability.

3. DPO (Direct Preference Optimization)

Using Direct Preference Optimization (Dubey et al., 2024), we refined how the model understands and adapts to audience preferences.

The model was trained to adjust response styles based on different patient categories, ensuring clarity, accessibility, and empathy.

4. Additional Optimization

We experimented with quantization techniques (4-bit, 6-bit) and adapter-based methods like LoRA (Touvron et al., 2023).

The goal was to compress the model while maintaining accuracy, allowing for efficient deployment without sacrificing performance.

As highlighted by Chen et al. (2023), this cascaded training approach (RAW → SFT → DPO + adaptation methods) provides optimal results in medical AI applications, where tone calibration and an extra layer of safety measures are critical.

AI Benchmarking & Model Validation

Objective

This guide focused on systematically evaluating and comparing the performance of a specialized AI agent, tailored to a specific domain of interest and related healthcare topics, against another model. The evaluation measured both general AI performance and domain-specific expertise. This workflow was then reviewed with the use of one of our key focus areas as an example: Glucagon-Like Peptide-1 Receptor Agonists (GLP1-RA).

Preliminary Benchmarks for Medical Knowledge Assessment approach

After training the model, the first step in verifying its performance was assessing its level of medical knowledge. For this task, we used the LM as Judge approach to test domain knowledge, we assessed the modelʼs ability to understand the medical field using common benchmarks:

Multiple-Choice Benchmarks

MEDQA4 A benchmark for medical question answering, derived from the USMLE (United States Medical Licensing Examination) questions covering multiple medical specialties. Tests the model’s ability to answer complex, clinically relevant multiple-choice questions.

MedMCQA5 A large-scale dataset of multiple-choice questions covering a wide range of medical topics, including anatomy, pharmacology, and pathology.

MMLU (Medical Category)6 A general-purpose benchmark that includes a medical subset with questions from various domains, including clinical knowledge, anatomy, and genetics. Tests both medicine-specific and general reasoning abilities.

PUBMEDQA9 A biomedical question-answering dataset derived from PubMed abstracts, containing expert-annotated yes/no/maybe answers to research-related questions.

Open-Response Benchmarks

K-QA (Knowledge-QA)7 A benchmark for open-ended medical question answering, requiring detailed explanations or reasoning. Evaluates the model’s ability to generate accurate, coherent, and contextually appropriate responses. It measures two evaluation metrics: Hallucination rate — measures how many of the gold statements contradict the modelʼs answer. Comprehensiveness — measures how many of the clinically crucial claims are entailed from the predicted answer.

MedSafetyBench8 MedSafetyBench evaluates the medical safety of large language models (LLMs) based on ethical and safety principles derived from the American Medical Association’s Principles of Medical Ethics.

Our preliminary Benchmarks for Medical Knowledge Assessment results

In addition to our open-source benchmarks, we developed a custom benchmark comprising 700 questions derived from select medical articles and guidelines. Each question required the model to pick one correct answer from four possible choices.

The benchmark was divided into two parts:

Clinical Cases 400 questions — presenting patient details such as age, gender, medical characteristics, and clinical histories.

General Knowledge 300 questions — focussing on foundational theory and reference information within the relevant medical domain.

These tasks focused on metabolic health topics, including diabetes, prediabetes, obesity, GLP-1 receptor agonists, nutrition, and mental health. Experimental evaluations produced accuracy scores for both open-source and proprietary models, with the latter accessed through subscription-based APIs (e.g., OpenAI, Anthropic).

To ensure fair comparisons across diverse architectures and parameter scales, we reported the best accuracy each model achieved under various testing scenarios, ranging from zero-shot to three-shot and five-shot learning.

Below is a concise summary of the test results:

Small Models (≤10B Parameters)

• Yma 7B 89.14% held the highest accuracy among small models.
• Meta-Llama-3.1–8B 81.57% and Mistral-7B 79.14% trail behind with a notable gap.
• Gemma-2–9B 79.29% is comparable to Mistral-7B.
• DeepSeek-R1-Distill-Qwen-7B 68.86% and MedAlpaca-13B 67.43% (despite its parameter count) show notably lower scores.
• Llama3-OpenBioLLM-8B 33.14% is a clear outlier with especially poor performance.

Large Models (>10B Parameters)

• Claude 3.5 Sonnet 91.29% ranks at the top across all models.
• Qwen2.5–72B 90.72% and GPT-4o 90.71% are close contenders.
• Phi-4 90.57% also scores highly, confirming general-purpose models can excel.
• Mixtral-8–7B 88.14% falls slightly below Yma 7B, underscoring competitive performance.
• Most large models achieve between 85–91%, with MedAlpaca-13B 67.43% as a stark exception.

Overall, Yma 7B performed on par with high-achieving large models such as GPT-4o and Claude 3.5 Sonnet, showing no statistically significant difference in accuracy within this dataset.

Human-evaluated metrics approach

Comprehensive assessment by human evaluator is the gold standard for LLM applications in healthcare. Regardless of the agentʼs scope of expertise, the following metrics should be assessed for evaluation of the SLM model.

After training, all AI models were rigorously benchmarked using a structured evaluation process to ensure they meet specific quality parameters. This process involves randomized comparisons and assessments by both clinicians and users, as shown in the workflow (Figure 2).

This evaluation framework is based on the analysis of modern, relevant literature, including but not limited to references (1–3).

The evaluation framework consisted of three parts: A, B, and C (Figure 2). Part A described the expert evaluation of the model, while Parts B and C were focussed on user evaluation.

Protocol

A set of 100 questions about GLP-1RA agonists and related topics was created by a biomedical research expert and subsequently evaluated and refined by an experienced clinician-endocrinologist with over 15 years of experience to align with general patient concerns.

Fifty percent of the questions are strictly related to GLP-1RA therapy, while the other half pertain to diabetes and obesity. Twenty questions involve simulated patient histories reflecting the risk factor distribution of general patients on GLP-1RA, requiring context analysis in the response.

Two independent endocrinologists were recruited to provide answers to all 100 questions. Their responses were crafted to simulate a conversational tone (with compassion and empathy) and were based on relevant clinical guidelines. The median response length was 150 words.

Module A: Clinical Evaluation

To ensure evaluation reliability, three expert annotators from the field of endocrinology with experience in treating diabetes and obesity participated. Responses from two models (Yma and ChatGPT) were generated in advance, along with responses from clinicians. The 100 questions were randomly split into a main set (80 questions) and a ‘held-out’ set (20 questions).

The main set of 80 questions were randomly assigned to three clinicians, yielding approximately 80 independent evaluations (ChatGPT, Yma, and physician responses distributed (33% each). The response order were randomized, with one question-response pair per page in the evaluation interface. This randomization minimized biases related to model response recognition and ensures independent evaluation.

The ‘held-outʼ set of 20 questions were evaluated by all three clinicians with all three responses (ChatGPT, Yma, and physician responses). The order of questions were randomized. This approach allows for the calculation of inter-annotator agreement metrics, which assess evaluation reliability. In cases of annotator disagreement, evaluators were asked to reach a consensus post-evaluation, increasing the statistical power of findings.

Additionally, this set will help determine overall response preferences, with clinicians selecting their preferred response.

Annotators were unrestricted time for evaluation. Responses from both models and physicians were blinded, and all evaluations remained confidential until completion.

Evaluation Metrics 5-Point Likert Scale

Each metric was then assessed using the question, “Do you agree with the statement about the response?” with the following grading scale:

• Strongly disagree
• Disagree
• Neither agree nor disagree
• Agree
• Strongly agree

In cases where responses were rated as unsafe or incorrect, clinicians were asked to provide additional comments explaining their grading, helping to identify areas for future improvement.

Statistical Analysis After evaluation, the null hypothesis (H₀) stated that there is no significant difference between the two models (ChatGPT vs. Yma) regarding the measured metrics.

Each metric were compared using the Mann-Whitney statistical test, as Likert scale scores are not normally distributed. A p-value < 0.05 will indicate significant differences between models (HA). The same method compared clinician responses to model responses. Pairwise comparisons (ChatGPT vs. Yma, Yma vs. Doctor, ChatGPT vs. Doctor) applied Benjamini-Hochberg p-value adjustments. Inter-annotator agreement was assessed using Fleiss’ Kappa index, where a value ≥0.8 indicates good agreement. Additionally, Spearman correlation coefficients were calculated for each metric.

Preference Estimation

An additional method involved selecting the preferred response among the two models and the physicianʼs response. For each question in the ‘held-outʼ set: Input was processed by both models. Responses were displayed simultaneously on the same screen, with random order assignment (“Response 1” or “Response 2” or “Response 3”). Evaluators selected the preferred response. The preference distribution was visualized using stacked bar plots.

Module B: Layperson Assessment

This module evaluated the modelʼs ability to provide readable, empathetic, and relevant responses. It ensured that the model not only provided accurate information but also communicated effectively to non-expert users.

From a total set of 100 questions, 50 questions were randomly selected for user evaluation: out of these, 20 questions were used to assess 5-point Likert Scale metrics, while the remaining 20 questions were used for preference ranking.

Three lay-person evaluators (without a medical background but familiar with healthcare topics) were recruited. Responses from both models and physicians were blinded to ensure unbiased evaluation.

Statistical Analysis

Each metric was compared using the Mann-Whitney test, with Benjamini-Hochberg correction applied for pairwise comparisons. The same approach as in clinician evaluation was then used for determining general response preferences.

Module C: Real-Time User Evaluation

This module involved real-time evaluation with laypersons familiar with GLP-1RA agonists, including real patients on GLP-1RA therapy. Evaluators were asked 15 questions, and responses were generated in real-time.

Evaluation Design

The user asks a question of interest:

Input was processed by both the developed model and the competitor model.

Responses were displayed simultaneously on the same screen, randomized in order.

Evaluators assessed responses based on Readability, Empathy, and Relevance, followed by a general preference choice.

Statistical Analysis For metric comparisons, the Wilcoxon Signed-Rank Test was used, as responses were generated for the same questions. Preference analysis followed the same methodology as in previous modules.

Benchmarking results

The evaluation results provided critical insights into the model’s performance. These insights helped us refine the AI’s capabilities, ensuring that it delivered responses that are scientifically accurate, empathetic, and user-friendly.

Our human-evaluated results

The set of 100 questions and corresponding 300 responses (Yma, ChatGPT-4o, and physicians’ responses) on GLP1-RA therapy, diabetes, and obesity management was randomized according to the protocol described above. Twenty questions were retained for preference evaluation.

Three clinicians independently evaluated the responses using metrics of correctness, safety, completeness, and conciseness (Figure 3).

There was no statistical difference in any metric other than conciseness, as assessed by the Wilcoxon rank-sum test. Yma’s responses were more concise than ChatGPT-4o’s (Benjamini-Hochberg adjusted p-value = 0.024); physicians’ responses were significantly more concise than those from ChatGPT-4o (adjusted p-value = 4.8 × 10^-4). No significant differences were observed between physicians’ and Yma’s responses (adjusted p-value = 0.18).

These results indicate that the responses from the Yma small language model are not inferior to those from either the advanced large language model ChatGPT-4o or the doctors in terms of response quality.

User evaluation

A set of 50 questions, excluding medical histories, was selected for evaluation by three non-experts to make the assessment easier for regular users. Twenty questions were retained for preference evaluation.

On the set of 30 responses from each model and physicians, ChatGPT-4o demonstrated higher empathy than the physicians’ responses (Benjamini-Hochberg adjusted p-value = 0.02). No significant difference in empathy was found between Yma’s and ChatGPT-4o’s responses (adjusted p-value = 0.07). Additionally, no significant differences were observed in readability or relevance in pairwise comparisons (Figure 4).

Preference evaluation

Next, users and physicians were asked to choose which of the three presented answers to the same question they preferred. There were 20 questions in total for both clinicians and users.

Overall, clinicians chose doctors’ responses more often (38% for doctors’, 37% for ChatGPT’s, and 25% for Yma’s responses). Users, on the other hand, preferred responses from ChatGPT (72% for ChatGPT’s, 23% for doctors’, and 5% for Yma’s responses) (Figure 5).

In conclusion, these results suggested that our SLM-based solution is on par with ChatGPT-4o and physicians in key metrics of medical appropriateness (correctness, safety, completeness, and conciseness) of the provided responses.

Moreover, our model delivers empathetic, relevant, and well-readable answers, as evaluated by three laypersons. However, our solution remains inferior to the responses of ChatGPT and physicians in terms of general preferences. This may be due to various factors; for instance, the length of the response could have influenced users’ preferences, as ChatGPT provided longer responses on average (median response length: Yma — 1140 symbols, physicians — 1026 symbols, ChatGPT — 2819 symbols).

A limitation of this evaluation approach is the difficulty in assessing the impact of selecting a particular answer. To address this, we plan to survey both users and clinicians to identify and correct the weaknesses of our model. In future, following improvements, we aim to involve more clinicians and users in preference assessments to obtain a more comprehensive evaluation. Additionally, we will implement real-time response evaluation, where users will be able to ask questions and receive anonymized answers from both models simultaneously.

Overall, the results highlighted the potential of SLM-based approaches in domain-specific topics.

Outcomes & Next Steps

We have successfully developed a compact AI model eliminating the need to send patient data to external APIs. The model demonstrated a level of medical accuracy comparable to large LLMs, but with significantly lower computational costs.

From a safety perspective, the model exhibits fewer hallucinations in medical tasks, as its entire training foundation is built on domain-specific guidelines and real-world, physician-validated data.

Final Takeaway: The Future of SLMs in Healthcare

This domain-specific approach to training Small Language Models (SLMs) has proven to be highly effective for real-world medical applications.

• Support patients with accurate, AI-driven medical guidance
• Reduce the workload of healthcare professionals
• Maintain strict data privacy by running locally in the frontier or locally on users’ devices
• Significantly lower infrastructure costs compared to large-scale LLMs

These results reinforce our belief that SLMs represent the most practical and efficient path for deploying generative AI in healthcare.