LLMs in International Development - A Recipe for Success?
A blog by Seb Mhatre, FCDO Pioneer and Olivier Mills of Baobab Tech, a Frontier Tech implementing partner
Pilot: Using LLMs as a tool for international development professionals
An illustrative diagram of the recipe-based framework. Credit: Olivier Mills, Baobab Tech
Introduction
Have you ever tried a recipe box? A recipe box is the concept that selected ingredients combined with selected recipes are neatly packaged into a box delivered to your home to make it easy for you to create a bunch of tasty dishes. The idea is that giving people a choice of recipes that are tailored to the available ingredients and putting it all in a box makes it easier for them to choose what they want to eat and easier for them to cook those dishes. This is what we are trying to do but with data and an LLM tool instead of ingredients and a kitchen.
Over the past year, we have been exploring the use of Large Language Models (LLMs) to support international development professionals in accessing, analyzing, and synthesizing knowledge from the vast corpus of development-related documents. Our pilot, which has been documented in previous blogs, has focused on leveraging LLMs to enhance decision-making, improve efficiency, and generate information from IATI data that development professionals can use in their work.
Classic chatbot limitations
LLM chatbots, such as ChatGPT, Claude or Gemini, give you huge freedom and flexibility. You can ask them to do anything and they will try and do it. Our initial LLM tool had the same feature. But for many people, knowing what sequence of prompts to give is a real challenge. Users of the tool faced the problem of not having a clear idea of what they were aiming to create and not knowing how their prompts would interact with the RAG architecture and the underlying IATI database to create their desired knowledge product.
To address these challenges, we have taken a recipes approach that shows the user examples of what knowledge products are possible and then guides the user through the series of steps required to create that knowledge product.
Guided workflows
The recipe-based framework divides each recipe into three key stages:
1. Building a Collection of Relevant and Useful Information
Users define the scope of their search using filters (e.g., thematic areas, geographic focus). AI-powered search assists in retrieving and curating relevant programmes and documents. This forms the knowledge base, which we call a “collection” for the next steps.
2. Intermediary outputs, e.g. Creating a Bespoke Summary
Extract or synthesis stage of the collected entities. A common action is for users to request customized summaries focusing on specific aspects (e.g., lessons learned, effectiveness, key indicators) for each item in the collection. LLMs will generate structured summaries for each item, providing intermediary knowledge products that can be part of the collection, and acts as a basis for the last stage.
3. Generating a Knowledge Product
Generate structured outputs that are directly useful for decision-making. Users define the final output format (e.g. a comparative table, thematic synthesis or portfolio summary). AI assists in structuring and formatting the product, ensuring usability and consistency, and only using the content from the collected items from the intermediary outputs.
A recipe is a set of clear instructions for each stage of the process. Users can look at examples of previous recipes for different knowledge products, e.g. a portfolio summary of FCDO’s programmes in Kenya or a guidance note for designing health programme indicators. They can then follow that recipe, modify the recipe (portfolio summary of FCDO’s programmes in Uganda) or create an entirely new recipe. This recipe approach has both advantages and limitations. The main advantages are simplicity of use, leading to faster results, and increased replicability, leading to easier evaluation. The main disadvantages are that users sometimes need more freedom to explore queries that aren’t facilitated by the current structure.
We are excited to launch the tool next month to a first cohort of users, evaluate the accuracy and relevancy of the outputs produced, and work with other IATI publishers to expand the knowledge base beyond FCDO IATI data. If you are interested in joining our initial cohort of users, please get in touch by e-mailing jenny.prosser@dt-global.com.
*We are seeing other initiatives following a recipe-based approach such as the Humanitarian AI assistant by Datakind, see this post.
If you’d like to dig in further…
🚀 Explore this pilot’s profile page
📚 Learn more about our community of practice
