What might a Community of Practice in LLMs for knowledge management look like, and why is it needed?
A blog post by Seb Mhatre, FCDO Pioneer.
Pilot: Using LLMs as a tool for International Development professionals
Humans rely on information to make decisions, but in a world flooded with data, how do we ensure decision-makers access the right insights precisely when they need them?
The answer is good knowledge management. It’s a prerequisite for effective international development because making the right decision about how to allocate or use scarce resources depends on having the right knowledge to make that decision. That means having reliable, relevant, usable and trusted knowledge available at the point when you are making that decision.
We know that Large Language Models (LLMs) will transform what good knowledge management looks like. In recent years LLM tools have already shown that they can reduce the cost and efforts needed to find and analyse information, and the technology is continuing to develop rapidly. Amidst developments we’re asking: what can we do to make sure that this change is as beneficial to the international development sector as possible?
We can’t predict future effects of a rapidly changing technology like LLMs, but despite the uncertainty, we do know that the following four factors will play a significant role.
To maximise the potential of LLMs for knowledge management in the development sector, we recommend building a community of practice to share learning, insights and ideas to work towards a future where LLMs are working for the benefit of development. Please read on to learn more and express your interest in joining this community here.
Four factors that will determine the future of LLMs in knowledge management in international development:
Underlying LLM technology
The underlying LLM technology consists of the foundational large language models themselves, such as GPT, Claude, PaLM and LLaMA, the libraries for connecting to those models and frameworks for developing new applications. This includes new capabilities like web search, retrieval augmented generation and agentic functionality for enabling multi-step processes. This underlying technology is evolving quickly, and is likely to continue to do so, in response to a high level of investment and competition in the wider tech market as a result of potentially huge rewards. And as that technology becomes more capable and cheaper it is likely to drive a high level of demand from consumers, which in turn will lead to further investment and innovation.
Availability and quality of the data
The effectiveness of LLM tools depends on high quality data. A significant fraction of the data used in international development is generated by international development actors. For example, strategy documents, policy documents, programme documents, evaluation reports, guidance notes, research, grey literature, contracts and tenders. However, both the accessibility of this data and the quality of this data are highly variable. There are good examples like IATI data that has both a common standard, has an API and an established governance structure and there are established principles like the FAIR principles for Findable, Accessible, Interoperable and Reusable data (https://www.go-fair.org/fair-principles/) . But there are also many examples where useful data is not published or not published to a common standard and doesn’t have an API or an established governance structure. The bad news is that data accessibility and quality are not improving quickly. However, the good news is that this is something within the gift of the international development sector if we have the will to make it a priority.
Application to problems
Whatever the current state of the underlying LLM technology, making it useful in international development depends on innovators understanding the problems and figuring out where LLM tools can add value. These innovators could be internal teams within large development actors, small start-ups or even big tech. In each case innovators will depend on a clear demand signal. A clear demand signal depends on a recognition of the value of the product (which we are starting to see but it's still early days) and a resource allocation model and procurement structure that enables organisations to invest in innovation. For example, small innovation budgets, overly centralised IT resource allocation and onerous procurement rules will slow innovation and worse risk an uncompetitive market. If we get the demand signal right, we can both enable a competitive market leading to a diverse range of LLM tools and can shape that market to incentivise interoperability and reuse that will in turn stimulate further innovation.
Learning and Capability
Both the previous factors depend on the capability of development organisations to understand the data they both need and produce and the capability to stimulate innovation. Both kinds of capability have been a challenge for development actors for many years. Data capability means having capability across the entire data value. See this article for a description of the data value chain focused on official statistics. https://data2x.org/wp-content/uploads/2019/08/Data_Value_Chain.pdf .
LLM tools change every part of the data value chain as they change the value of unstructured data. This means that development actors need to rethink what types of data can add the most value to which problems, how they collect, process and add value to that data, how they publish that data, how that data is managed and governed and how users incorporate data into their decision-making processes. Therefore, development actors need to build the capability to do this type of thinking and to enact these types of processes. For example, building senior capability to understand the value of data and how to incentivise innovation, building capability in users to make effective use of LLM tools which in turn creates demand or building capability to support effective cross-organisational data governance structures. Building capability is slow, so it is likely that it will take years to shift the baseline levels. On the positive side, building capability is an area where development actors can work together, sharing best practice and showcasing what’s possible.
Build a community of practice with us
Over the last year, I have been working with Robbie, Olivier, Jenny and Johannes to build an LLM tool that would be useful for people who work in international development and to learn about what we can about the potential and risks of LLM tools for the sector. See these previous blog posts on our pilot profile for some of what we have learnt so far. Using LLMs as a tool for International Development professionals — Frontier Tech Hub
Knowledge management in international development has many different actors each with different perspectives, capabilities and incentives. So, to maximise the potential of LLMs for knowledge management, development organisations need to support sharing and coordination between those actors. This means encouraging the growth of a community to support the effective sharing of evidence on the value of knowledge management given advances in LLMs and the sharing of ideas to encourage innovation. Then using that community to create a common vision that inspires action, common standards to support interoperability and best practice in data governance to enable sustainable, effective coordination between the public and private sector.
