Part 3 - What can AI say?
Data collection
As we’ve explained previously, one of the most important enablers for developing AI use-cases is the availability of high-quality, relevant data. Many hospitals have specific sources of information that they consider credible and helpful to their patients, either produced themselves or by others. These may include factsheets, images and possibly videos which outline best medical practice in a variety of areas, from eating well to managing specific conditions, as well as links to external sources of data validated by the hospital. Care teams will be able to upload the resources that will be helpful as a basis to inform their patients into their own proprietary knowledge base, which will then drive the bespoke conversational interaction with that hospital’s patients.
In addition to this body of information, they had care plans for each patient. As explained above, these care plans are a unique set of guidelines for how patients should manage their diseases. When a user sets up an account on Avatr their care plan is loaded into the chatbot, such that its responses draw from the database of verified medical information, and the specifics of the care plan. The use of this data ensures that chatbot is generating responses which are personalised to the users' needs.
Data pre-processing
How do you integrate these data sources with an LLM, such that it is able to draw on that information to provide useful responses?
To build a quality “RAG” process, you want to make sure you are retrieving the most relevant information from your data source for the specific user query. You could perform a basic keyword search, which would match word or word stems, but keyword search is limited especially for larger datasets. Retrieving data that is not relevant for the question can easily confuse the LLM and make it provide an answer that is incorrect or confusing to the user. For example, say the search includes the keyword “bank”, do we mean “river bank” or “financial bank” ? The solution to this is to use vectors, more specifically “dense vectors”. So your database would include the text but also the vectorized version of that text. Vectorizing or embedding a sentence means translating it to a set of numbers in a multidimensional space to represent its semantic “meaning” .. so the word “river bank” and “financial bank” would be quite far away from each other in this vector space.
One of the first steps for the team to develop the solution was to create a vector database which will hold the knowledge base (factsheets, images etc.) as embeddings or vectors. The embeddings allow it to measure the similarity of the request to different pieces of information in the knowledge base, and pick out the most relevant resources to share with the patient. By restricting that search to a specific body of information the hospital team can ensure that the information they were providing people is verified medical guidance.
Creating the model
The creation of the model differs from many of the others which we have explored throughout the module, because the development process does not include a fine-tuning step. You might have noticed that we haven’t been referring to the data in the system as training data. The resources such as factsheets and images aren’t being fed into the LLM to train it to better understand the nuances of medical language like how the images of sunsets and fires helped the Yolo algorithm distinguish between the two.
Rather, the LLM has been pre-trained to process unspecialised language and generate human-like responses, and the retrieval process draws on specific information drawn from the knowledge bank. As such there is not a fine-tuning step, but rather a process of iterating and refining the retrieval process and prompts to make sure that it is producing information-rich, context-relevant responses to user requests, that reflect the knowledge a well-trained doctor would provide.
Impact
Initial testing of Avatr technology has shown its potential to support doctors by providing them with data on how people are interacting with their care plans, giving doctors a more informed basis to make interventions to encourage adherence. In resource-constrained hospitals, where doctors are overstretched, these resources can allow them to meet the needs of their patients and deliver faster and better care. For patients, initial testing has already started to reduce re-admittance rates. Access to high-quality medical information has empowered outpatients to better follow their care plans and manage their diseases on their own terms.
Avatr GenAI aims to deliver even higher levels of adherence, leading to health and economic improvements. The Avatr system generates granular data at personal and population levels to measure and evaluate adherence to care plans. Aligned with the Health Belief Model, impact measures include changes in beliefs and behaviours leading to health and economic improvements. The evidence informs further iterative development of the Avatr GenAI system, specifically, optimization of the embedding, retrieval and generation steps. There will also be various complexities of transitioning from a development environment to a production environment which is suitable for scaling up of the solution.