Introducing the First Intelligent AI Knowledge Agent in Healthcare, Using Patients’ Medical Records for More Accurate Information and Routing

By Allon Bloch and Ran Shaul
May 13, 2024

Introducing the First Intelligent AI Knowledge Agent in Healthcare, Using Patients’ Medical Records for More Accurate Information and Routing

We started K Health with the mission of creating access to high quality medicine

After seven years, we have an AI system that’s used in clinical settings, based around K’s AI-driven medical chat. Furthermore, we have integrated our offering with a world-class medical primary care offering that is now embedded into leading health systems.

And now, we have built the first Generative AI (GenAI) offering in healthcare that is more accurate and personalized to you.

A Major Departure From Other LLM-Driven AI Information Applications

We’re now adding an AI Knowledge Agent to our offering, with plans to be live soon. You can request access to demo the Knowledge Agent here.

The Agent, composed of an array of large language models (LLMs) enriched by our in-house algorithms, is a major departure from other LLM-driven AI information applications, such as ChatGPT, in the following ways:

  • Personalized to you: Its answers to questions take into account relevant aspects of a patient’s medical history and electronic medical record (EMR), to provide patients with more personalized information.
  • Doctor’s office digital front door: It’s built into our direct-to-consumer affiliated virtual clinic, and embedded into major health systems, to provide an intelligent “digital front door” to route patients to the right place to resolve their medical needs–whether they’re big or small, acute, chronic or preventative. 
  • Patient navigator: Our AI Agent can navigate you through the healthcare system to be in front of the right information, doctors and specialists. This massively empowers patients while giving doctors ‘AI super powers’ to diagnose and treat patients 24X7. This also means we can help you reach primary care and speciality doctors, labs and tests all within a single health offering.
  • More accurate: K Knowledge Agent is purpose-built to provide more accurate medical information (see below)–with less hallucinations than other GenAI tools built on LLMs.

Read the article about K’s AI Knowledge Agent in Digital Health Wire

Hallucinations Don’t Have a Place in Healthcare

GenAI LLMs have a tendency to hallucinate–or essentially make up answers. 

Hallucinations are fine if you’re writing a humorous wedding toast, but not in medicine given the need for highly relevant and accurate information. This is doubly important given the complexity and subtle nature of medicine. 

This is why we took a different approach when creating our AI Knowledge Agent:

Optimized for accuracy

We’re optimized to provide full and more accurate answers by using select high quality health sources. If the answer to your question cannot be found in the sources, our AI will tell you it doesn’t know the answer. We also use a number of agents to verify the answer is adherent to the sources. These mechanisms dramatically reduce the number of potential mistakes.

Smartly integrated to the EMR

Integrating information from a patient’s medical history is key as it gives our models the capability to give personalized information based on a patient’s detailed medical information. However, this wealth of complicated information poses a new challenge to LLM models, and when this is done naively, it leads to inaccuracies.

In order to generate more accurate, personalized answers we used a multiple-agent approach: one agent filters only the parts of the EMR relevant to the question, and a different agent answers the question based on the filtered EMR.

How Does Our AI Knowledge Agent Stack up

We measured our Knowledge Agent against other LLMs for specific medical questions, and we looked at two crucial metrics: 

  1. Hallucination Rate: This metric measures the number of mistakes, i.e. the number of times the LLM contradicts a statement that was written in the gold standard answer.
  2. Comprehensiveness: This metric measures how many of the clinically crucial statements are included in the predicted answer.

We also looked at coverage, which measures how many questions the LLM decided to answer.

In a manuscript currently under review for publication in a top-tier natural language processing (NLP) conference (preprint), we evaluated our model’s answers against state of the art models on a dataset containing 202 highly curated questions and answers. It is shown that our non-personalized answers are 9% more comprehensive and hallucinate 36% less than GPT-4, our best competitor.

Here are the full results:

MethodCoverage ComprehensivenessHallucination Rate
K Health Knowledge Agent 96% 62.915.4

Lastly, when comparing our AI Knowledge Agent to physicians who work in K Health’s affiliated clinics we can see where our Agent is particularly strong. 

Preliminary results on 50 personalized clinical questions grounded in the patient’s EMR show that while both our Knowledge Agent and the physicians had a single mistake, our Knowledge Agent was more comprehensive. While physician answers reached a score of 0.40 in comprehensiveness, our answers reached a score of 0.62 (55% better).

Specific Examples of Demo Patients Simulating Real-Life Medical Situations

The following are specific examples of demo patients simulating real-life medical situations. Our AI pulls out the pertinent data points from the patient’s medical history to answer the question the user asks.

We included both the Knowledge Agent’s answer as well as the perspective on the Agent’s answer from Dr. Zehavi Kugler, Vice President Medical Sciences at K Health and Primary Care Physician.

In all cases, a user can click a button in the same experience to engage with K’s clinical expert AI and be in front of a doctor in a matter of minutes:

Let’s start with a user who is brand new to our Knowledge Agent. We do not have any knowledge about their medical history, therefore the Agent’s answer will be accurate but cannot be personalized. Next we ask the Agent the same question, but ground it with the patient’s EMR. Now, you get a highly tailored answer:

Here is a different patient, with a different medical history, asking the same question:

The Knowledge Agent surfaces smoking-related cough as the first relevant medical condition. It also surfaces the need for imaging as part of the evaluation.

And next, the same question but to a patient with a different set of background diseases:

The Knowledge Agent identifies the patient’s iron deficiency as anemia–a condition associated with certain gastrointestinal issues like gastritis and GERG–and makes the link between reflux-induced cough and anemia, listing it as a potential cause of the cough.

Patient with a history of pulmonary embolism asking about a medication that could potentially increase risk:

The most important part of this answer is that patients with a background of pulmonary embolism should avoid combined oral contraceptives that contain estrogen due to the higher risk of blood clots.

Here is another common symptom of higher heart rate for an asthmatic with pulmonary embolism. This could be due to several medical reasons. In all cases, a user can click a button in the same experience to engage with K’s clinical expert AI and be in front of a doctor in a matter of minutes:

The Agent takes the patient’s history of pulmonary embolism into account as a possible sign of her racing heartbeat, as well as her of albuterol for asthma, which may cause fast heartbeats.

And here is another common symptom of fatigue for a patient with complex background diseases. Again, the user can easily be in front of a doctor in minutes 24X7:

Agent references two significant potential causes of patient’s fatigue: her Hashimoto’s thyroiditis and anemia. It also referenced other common causes, and offers a helpful treatment plan.

When people take multiple medications and supplements, you must take into account the potential of different active ingredients clashing. Here is a great example of the Knowledge Agent taking drug<>drug interactions into account:

When people take multiple medications and supplements, you must take into account the potential of different active ingredients clashing. The Knowledge Agent takes into account the interactions between medications the patient is on when answering the question.

When providing medical knowledge, you must also account for patients that have symptoms that may be side effects of medications: 

It is essential to be aware of the side effects of simvastatin and the Agent addresses them first. It also suggests potential diagnoses based on the patient’s medical history.

Want to test the K Health Knowledge Agent?

K Health articles are all written and reviewed by MDs, PhDs, NPs, or PharmDs and are for informational purposes only. This information does not constitute and should not be relied on for professional medical advice. Always talk to your doctor about the risks and benefits of any treatment.

Allon Bloch and Ran Shaul

Allon Bloch the Co-founder and CEO of K Health and Ran Shaul is the Co-founder and Chief Product Officer of K Health.