<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Karlis Kanders on Medium]]></title>
        <description><![CDATA[Stories by Karlis Kanders on Medium]]></description>
        <link>https://medium.com/@kandersk?source=rss-c7cad6c6bfa4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aB74zvXoRQiTWgkcO2clpQ.jpeg</url>
            <title>Stories by Karlis Kanders on Medium</title>
            <link>https://medium.com/@kandersk?source=rss-c7cad6c6bfa4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:54:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kandersk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to find the best policy evidence across millions of documents?]]></title>
            <link>https://medium.com/discovery-at-nesta/how-to-find-the-best-policy-evidence-across-millions-of-documents-ea0edd174dc4?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea0edd174dc4</guid>
            <category><![CDATA[boolean-search]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[openalex]]></category>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Thu, 18 Jun 2026 10:15:17 GMT</pubDate>
            <atom:updated>2026-06-18T10:15:17.721Z</atom:updated>
            <content:encoded><![CDATA[<p>Policymaking increasingly occurs under <a href="https://www.instituteforgovernment.org.uk/sites/default/files/2025-12/rapid-policy-making.pdf">rapid conditions</a> with severe time constraints, making it challenging to gather sufficient evidence and draw international lessons when designing policy.</p><p>At Nesta we are exploring if AI can play a role in supporting policymakers in such circumstances. To this end, we’re developing <a href="https://policyatlas1.substack.com/p/policy-atlas-from-prototype-to-pilots">Policy Atlas</a>, an AI-powered tool to help policymakers synthesise global evidence and design novel, effective policies.</p><p>This requires a comprehensive analysis pipeline enabled by large language models (LLM) and agentic approaches to screen relevant sources, assess the strength of available evidence, and synthesise outputs.</p><p>The very first step when users enter their query on the Policy Atlas platform is searching through databases containing millions of academic research papers and policy documents, to find relevant sources to analyse.</p><p>It is critically important to find the most relevant documents for the user query, as the subsequent evidence synthesis and recommendations will all depend on the search results.</p><p>There are multiple approaches to tackling the task of document search, and in this article we describe our present implementation, plans for improvements and invite your feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S2pDSqjcvpmB2ZhO.png" /><figcaption><em>Analysis pipeline underpinning the Policy Atlas tool. This article focuses on the ‘Search for papers’ step.</em></figcaption></figure><h3>Grounding analysis in research and policy databases</h3><p>Our aim with Policy Atlas is to automatically generate accurate assessments of different intervention options on any policy topic area. To this end, it is important to ground the analysis and text generation by the large language model (LLM) in actual research and policy documents to avoid bias due to model’s training data and hallucinations (ie, inaccuracies).</p><p>In turn, the documents used for grounding the LLM should come from comprehensive and trusted sources, rather than a simple web search that is usually employed by existing general-purpose ‘Deep Research’ AI tools. Web searches can be influenced by <a href="https://www.bbc.co.uk/future/article/20241031-how-google-tells-you-what-you-want-to-hear">search engine biases</a> and search results might include news, commercial content and unvetted opinions (eg, blogs), which are less relevant for our use case.</p><p>Therefore, in Policy Atlas we employ large, curated academic and policy document databases, namely OpenAlex and Overton. OpenAlex provides an index of more than 400 million academic works, whereas Overton indexes over 20 million works of the so-called grey literature — government publications and think tank reports from across the world.</p><p>While academic evidence synthesis projects usually focus on research evidence, we’ve found in our user testing pilots with policymakers in the UK Government that grey literature is also a very valuable source of insights to ground empirical evidence in the policy context.</p><p>Both of these databases provide a vast catalogue of document titles, summaries, authors as well as links where to find the full contents of the documents. The databases by themselves do not, however, provide the full-text document content, which must be acquired in subsequent steps.</p><h3>Using keyword and semantic search</h3><p>Optimal knowledge retrieval is a rich area of research and development, with <a href="https://aarontay.medium.com/boolean-vs-keyword-lexical-search-vs-semantic-keeping-things-straight-95eb503b48f5">various methods</a> for identifying the most relevant documents. We employ two of the main types of methods: searching using keywords and semantic search.</p><p>Keyword search (also called boolean term search) is conceptually straightforward: documents that have at least one of the provided keywords will be found by the search. The higher number of the provided keywords the document contains, the higher it will be ranked in the search results. This approach has the benefit of being easily interpretable — either a document has the keywords or not.</p><p>Semantic search is better at identifying texts with similar <em>meaning</em>. Here documents don’t necessarily need to match the same keywords, but rather need to have text with similar meaning. This is more powerful, as the user doesn’t need to specify the exact correct terminology and you can even search across different languages.</p><p>Semantic search will often provide more precise results than a keyword search, with the caveat that semantic search results are harder to interpret. There is not a clear cut-off when a semantic search result stops being relevant — there are only gradations of results being more or less similar to the user query.</p><p>In Policy Atlas, we use boolean term search for the OpenAlex database and semantic search for the Overton database. The reason for this distinction is that Overton supports semantic search at relatively low cost, whereas OpenAlex initially did not have an in-built semantic search capability. More recently, OpenAlex has also released support for <a href="https://developers.openalex.org/guides/searching#semantic-search-beta">semantic search</a>, however, at the moment of writing this blog, at relatively high cost (ie, semantic search is 100 times more expensive than a simple keyword search on OpenAlex). This is because computationally, keyword search over large databases is more effective and cheaper than semantic search.</p><h3>Using AI to construct keyword search queries</h3><p>From the user experience point of view, semantic search is easier to use as users can define their queries as free text using the terminology they are familiar with. In contrast, for keyword search the users need to carefully identify the right keywords to use.</p><p>In Policy Atlas, we facilitate both types of searches through AI-powered workflows. First, the platform elicits information from the users about their policy question that is then used to generate suitable search queries for the Overton and OpenAlex databases.</p><p>Our search definition process, largely inspired by the <a href="https://en.wikipedia.org/wiki/PICO_process">PICO framework</a>, asks users about the policy interventions they are interested in, the target population, setting, desired outcome, geography and other important factors.</p><p>During this process, we use an LLM to generate sensible options for the population, setting and outcome based on the user’s intervention of interest. The user can choose from these automatically generated options — or they can also input their own preferred options.</p><p>This information helps the user consider their research question in greater detail, and allows Policy Atlas to define more specific database search queries. During user testing, we’ve received positive feedback about this approach, with one user remarking that the “follow up questions are uncannily good — very relevant and helpful to narrow down your search”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KraAiy5B4W16WsOv.png" /><figcaption><em>Information elicited from the policymaker user by the Policy Atlas search definition process.</em></figcaption></figure><p>We then combine the extracted information into one natural text query, by using an LLM to rephrase the provided information into a short paragraph. This can then be used to query Overton’s database. As an example, a query for “interventions to reduce childhood obesity” and a restriction of the search to OECD countries resulted in the following natural text query:</p><p>What interventions are effective in reducing childhood obesity among children in OECD member countries?</p><p>When searching OpenAlex for academic papers, we don’t want policymakers to construct meticulous keyword searches — they won’t have time to do it and there is a risk the keywords will be incomplete.</p><p>Instead, we again leverage an LLM to construct the boolean query for the user. Here’s a fragment of an example query for the same search parameters as used for the aforementioned semantic search example:</p><p>(child OR children OR childhood OR adolescent OR adolescents OR youth OR youths OR teenager OR teenagers OR “young people” OR minor OR minors OR pediatric OR paediatric) AND (obesity OR overweight OR “excess weight” OR “high BMI” OR “body mass index”) AND (intervention OR interventions OR program OR programs OR programme OR programmes OR strategy OR strategies OR treatment OR treatments OR prevention OR preventive OR policy OR policies OR initiative OR initiatives) AND (OECD OR “Organisation for Economic Co-operation and Development” OR “Organization for Economic Cooperation and Development” OR “United States” OR USA OR US OR American OR Canada OR Canadian OR “United Kingdom” OR UK OR English OR England OR Scottish OR Scotland OR...</p><p>To increase the chances of finding more robust types of evidence, we additionally construct specialised search queries for randomised controlled trials (RCTs) and systematic reviews or meta-studies by appending the automatically generated search terms with pre-defined terms such as “randomized controlled trial” or “systematic review”.</p><p>Constructing a good boolean query, however, is not trivial, as it requires domain understanding and it has to be comprehensive while also not overly broad. Moreover, LLMs have stochastic output and the queries might slightly differ between different runs. Therefore, we generate multiple versions of the boolean terms and use all of them for the final search results. Here is another example for generated based on the same initial user input</p><p>((child OR children OR childhood OR adolescent OR adolescents OR youth OR youths OR teenager OR teenagers OR pediatric OR paediatric OR school-age OR “school age” OR “young people” OR “young person”) AND (obesity OR overweight OR “excess weight” OR “body mass index” OR BMI OR adiposity OR “weight gain” OR “weight reduction” OR “weight control” OR “weight management”) AND (intervention OR program OR programme OR strategy OR policy OR campaign OR prevention OR treatment OR “health promotion” OR initiative OR “behavior change” OR “behaviour change” OR educational OR counseling OR counselling OR “diet modification” OR “nutrition education” OR “physical activity” OR exercise OR “active lifestyle” OR “sedentary behavior” OR “sedentary behaviour” OR “lifestyle modification”) AND (Australia OR Australian OR Austria OR Austrian OR Belgium OR Belgian OR Canada OR Canadian OR...</p><p>The final process for constructing boolean queries is shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mHkCtxaiGm4dv_BO.png" /><figcaption><em>Boolean and semantic search processes employed by Policy Atlas.</em></figcaption></figure><h3>Which AI model generates the best search queries?</h3><p>There are many LLMs one could use to generate the boolean search terms — how does one choose the best one?</p><p>We have started testing multiple models and temperature settings (which control the randomness of the model outputs) in terms of how comprehensive the search terms are, how many documents can they actually retrieve from the OpenAlex database (ie, whether the terms are not too restrictive), and also compared their performance to manually constructed ‘gold standard’ search terms.</p><p>While we are still optimising our evaluation approach, our early results were based on a set of ten research questions related to obesity research, for example, “What is the effect of free school meals on obesity and calorie consumption?”. Each of these research questions also have corresponding manually constructed search terms constructed by researchers that we can use as a reference.</p><p>These reference boolean terms were not perfect as they were originally constructed for searching for systematic reviews. For these tests we kept only domain relevant search terms and removed the terms specifically relating to systematic reviews, which resulted in a somewhat broad queries that returned tens of thousands of results. As a result the agreement scores (<a href="https://en.wikipedia.org/wiki/F-score">F1 scores</a>) shown below are generally low: less than 0.20 whereas a perfect score would be 1.0. Nonetheless, we can use them to start investigating the impact of model and prompt choices.</p><p>The research question was provided to the LLM together with a prompt that instructs the LLM to act as “an expert systematic review information specialist” who has a task to “generate a highly effective systematic review Boolean query” that “needs to be as inclusive as possible” but also “retrieve fewer irrelevant studies” and so on.</p><p>The prompt we used was based on the <a href="https://arxiv.org/pdf/2302.03495%5C">insights by Wang and colleagues</a> at the University of Queensland and Leipzig University that demonstrates the potential of ChatGPT in generating effective Boolean queries (specifically, their “q3” prompt example) .</p><p>Our first observation was that a straightforward step to improve the quality of AI-generated search terms is to simply use a larger, better LLM. An earlier version of Policy Atlas conservatively used OpenAI’s GPT-4o-mini model, which is very fast and cheap, with temperature set to 0 to reduce the variability in the outputs. This, however, occasionally resulted in unsuccessful searches that returned a very small number of results. It was possible to increase the median number of retrieved results by more than 25 times by using the more advanced GPT-4.1 and GPT-5 models.</p><p>Another observation was that a very high model temperature (larger than 1.5) tends to perform the worst, likely because it makes LLM less capable of following instructions.</p><p>Amongst the more advanced models we tested, we found that the GPT-4.1 model provides comparatively good performance, particularly if one generates multiple versions of the search query and then combines the retrieved documents. The combination of search results also helps reduce the problem of variability of LLM outputs. Interestingly, the optimal temperature setting was an intermediate one (temperature set to 1.0), suggesting that there might be an optimum between too little output variability (at temperature set to 0) and too much variability (at higher temperatures).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SAPJZ1JQqXzpafXn.png" /><figcaption><em>Agreement score (F1 score) between OpenAlex works obtained via LLM-generated and reference search terms.</em> <a href="https://public.flourish.studio/visualisation/29132499/"><em>Link to an interactive version</em></a><em>.</em></figcaption></figure><p>The figure above shows the F1 score between OpenAlex works obtained via LLM-generated and reference search terms. Single search results show the mean score over 10 research questions and five runs per question. Combined result shows the mean across F1 scores across 10 questions on the deduplicated union of retrieved works from five runs. Note that GPT-5 family models do not have a temperature setting.</p><h3>Results are sensitive to the instruction prompt</h3><p>An important note is that these results are specific to the particular LLM instruction prompt that we used. The quality of LLM-generated boolean queries is <a href="https://arxiv.org/pdf/2505.07155">sensitive to prompt design</a> and therefore the optimal model and temperature setting might depend on the particular wording used in LLM instructions,</p><p>For example, an earlier version of our instruction prompt (Prompt 1 in the figure below), which was in fact slightly more prescriptive and verbose compared to the current implementation (Prompt 2), showed a considerably worse performance across almost all of the tested models. It is difficult to provide a precise reason for the difference in performance: the underperforming prompt was longer and more prescriptive but potentially less clear, and it resulted in fewer search terms per query.</p><p>All the prompts we used for testing can be <a href="https://github.com/nestauk/discovery_policy_atlas/blob/dev/backend/testing/r_and_d/boolean_queries/prompts.py">found in our GitHub repository</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0e7BDB51GiZZelCR.png" /><figcaption><em>Agreement scores (mean F1 score) between results obtained via LLM-generated search terms and manually constructed search terms for two different prompts. </em><a href="https://public.flourish.studio/visualisation/29132770/"><em>Link to an interactive version</em></a><em>.</em></figcaption></figure><p>The figure above shows agreement scores (mean F1 score) between results obtained via LLM-generated search terms and manually constructed search terms for two different prompts. The scores were calculated as the mean across 10 research questions with 5 runs per question; retrieval results were <em>not</em> combined across different runs.</p><p>Thanks for reading! Subscribe to learn about our future experiments with evidence search.</p><h3>Next steps</h3><p>Currently we are using GPT-4.1 at temperature set to 1.0 for generating the boolean search terms, as this approach returns a sufficiently high number of relevant results and the model has a reasonably fast running time compared to, for example, GPT-5. You can find the prompt we are using in <a href="https://github.com/nestauk/discovery_policy_atlas/blob/a4174c0b3951cabac408deafd1b1031934da0f6a/backend/app/services/analysis/prompts.py#L13">our GitHub repository</a>.</p><p>Our next steps involve exploring a more agentic search solution, improving our evaluation approach using robust benchmarks such as <a href="https://github.com/princeton-nlp/LitSearch">LitSearch</a> and <a href="https://github.com/bytedance/pasa">PaSa</a>, and address some of the challenges noted below.</p><h4>Up-to-date model evaluations</h4><p>Since we conducted these tests, OpenAI has released newer models and we will need to update our evaluations. Furthermore, more tests should be performed on other frontier models (eg, from Anthropic and Google) and our reference boolean queries used as benchmarks should be improved to more closely match real-world queries developed by researchers used for systematic reviews.</p><h4>Prompt optimisation</h4><p>The output sensitivity to instruction prompts also raises the question on how to optimise the prompts themselves. For example, adding ‘seed’ queries from relevant systematic reviews to the prompt <a href="https://arxiv.org/pdf/2302.03495">can improve the performance</a>. Such seed queries could also be generated automatically by using <a href="https://dl.acm.org/doi/10.1145/3366423.3380185">more traditional natural language processing methods</a> that predate LLMs. Prompts can also be optimised further via code libraries such as <a href="https://arxiv.org/html/2509.00038v1">DSPy</a> that automatically explore better natural-language instructions.</p><h4>Relevance screening</h4><p>Initial user feedback indicates that Policy Atlas is useful as a fast scoping and sense‑making tool that improves access to evidence — but it can also miss relevant studies, and still requires manual follow‑up checks for the completeness of the evidence base.</p><p>For improving the retrieval, query generation is only one aspect that can be improved — <a href="https://aarontay.substack.com/p/the-horseless-carriage-of-ai-search">another avenue is re-ranking the retrieval results</a> after the document search to increase the relevance of the top results. Policy Atlas does this via a screening step that follows immediately after the search step, by using an LLM to judge the relevance of each of the retrieved papers.</p><p>There are potentially more computationally efficient and faster re-ranking methods such as <a href="https://arxiv.org/pdf/2403.10407">cross-encoders</a> that use traditional natural language processing approaches to determine document relevance to user input, while still being competitive with LLM-based methods.</p><p>Furthermore, the screening process should be more closely joined up with the document retrieval process to optimise the depth of the evidence search. In the current implementation the user needs to indicate the desired search depth by defining the number of documents the search should cover. Ideally, the tool could <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/cesm.70068">determine the optimal depth automatically</a>, by iterating between document retrieval and screening, and stopping once it can’t identify relevant documents.</p><h4>Data coverage</h4><p>Finally, there is a challenge regarding the data coverage of our two evidence bases. While OpenAlex is a comprehensive index of documents, it does not provide access to the full-text content of academic publications — it only provides information on where the full-text content can be found. Meanwhile, Overton provides a large index of global grey literature, but sometimes the full-text documents have been removed from think tank or government websites. At the moment, we are exploring how far we can push the quality of outputs by accessing open access literature and summaries where full-texts are not available.</p><p>We will continue sharing our progress on methodological improvements, and we are keen to connect to other researchers working on retrieval optimisation for research and policy data. Reach out to us!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea0edd174dc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/how-to-find-the-best-policy-evidence-across-millions-of-documents-ea0edd174dc4">How to find the best policy evidence across millions of documents?</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Policy Atlas: from prototype to pilots]]></title>
            <link>https://medium.com/discovery-at-nesta/policy-atlas-from-prototype-to-pilots-ce1038c35d65?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce1038c35d65</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[policy-making]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 17:57:06 GMT</pubDate>
            <atom:updated>2026-06-15T17:57:06.822Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was co-written by </em><a href="https://medium.com/@hello-aidank"><em>Aidan Kelly</em></a><em>, </em><a href="https://www.nesta.org.uk/team/shabeer-rauf/"><em>Shabeer Rauf</em></a><em>, </em><a href="https://www.nesta.org.uk/team/genna-barnett/"><em>Genna Barnet</em></a><em> and Karlis Kanders.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQMa4Kbq2Mdy5IIHF47UAQ.png" /><figcaption>Screenshot of the current version of Policy Atlas</figcaption></figure><p>Policymaking increasingly occurs under <a href="https://www.instituteforgovernment.org.uk/sites/default/files/2025-12/rapid-policy-making.pdf">rapid conditions</a> with severe time constraints, making it challenging to gather sufficient evidence and draw lessons from international examples when designing policy.</p><p>At <a href="https://www.nesta.org.uk/">Nesta</a> we are exploring if AI can play a role in supporting policymakers in such circumstances. We’re developing <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">Policy Atlas</a>, an AI-powered tool to help policymakers synthesise global evidence and design novel, effective policies.</p><p>In this article, we show the improvements we have made to the prototype tool over the past few months — making it more robust and user-friendly — and share lessons from our initial pilots with UK government users. You can find our <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">earlier updates on Nesta’s website</a>.</p><p>We tested the current version in real-world policymaking contexts and gathered feedback to assess product-user fit. Our initial pilots were with the Department of Health and Social Care (DHSC), where we tested the tool with 40 users across eight different directorates, and the Cabinet Office, where 10 users tried it out.</p><h3>Scaffolded search helps policymakers run more targeted queries</h3><p>Policy Atlas helps policymakers find policy intervention evidence across millions of global research and policy documents.</p><p>To make the search process more targeted and useful, the tool helps users be more specific about what interventions they are looking for, using a series of sequential prompts. These prompts are based on the <a href="https://en.wikipedia.org/wiki/PICO_process">PICO framework</a> and ask the users about the target population, intervention setting, desired outcomes, geography and other important factors.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F03AhY9MXmcM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D03AhY9MXmcM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F03AhY9MXmcM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f242ae221f7bc1c40720762b4b1c5c3e/href">https://medium.com/media/f242ae221f7bc1c40720762b4b1c5c3e/href</a></iframe><p>We’re using a large language model (LLM) to suggest personalised examples of target populations, settings and outcomes that might be relevant to users’ scope of work. In this way, we’re scaffolding the search process and helping policymakers think more deeply about the policy interventions they are researching.</p><p>We received positive feedback about this feature in our initial pilots, with a senior evidence review manager within the DHSC remarking that these <em>“follow up questions are uncannily good — very relevant and helpful to narrow down your search” </em>and it’s useful to <em>“run this first to inform questions for systematic review”</em>.</p><p>A policy lead at DHSC also said that the search process <em>“prompted me with really good and extremely relevant questions”.</em></p><h3>Evidence strength assessment helps prioritising policy evidence sources</h3><p>It is important to clearly identify more strongly evidenced policy options from those with weaker evidence. We <a href="https://www.nesta.org.uk/blog/five-lessons-from-building-evidence-strength-scoring-into-an-ai-policy-tool/">implemented an LLM-based approach</a> that classifies each source document by evidence type, assigns an initial evidence strength score between 1 and 5, and applies simple adjustment rules where relevant, such as a sample size penalty. These scores are then aggregated across the intervention themes extracted from the documents.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSskh_qylbWg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSskh_qylbWg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSskh_qylbWg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb4b721320a8cf40dfae2ef37a4bf90b/href">https://medium.com/media/fb4b721320a8cf40dfae2ef37a4bf90b/href</a></iframe><p>Our pilot users found the classification valuable as it helped to easily distinguish between systematic reviews, observational studies, policy documents and other evidence types.</p><p>Evidence scores also helped to prioritise certain types of more robust evidence — such as systematic reviews and randomised controlled trials (RCTs) in our current implementation — when generating the narrative summary and policy recommendations.</p><p>A challenge however, was that users expressed varying preferences of what counts as “strong evidence”. For example, one of our pilot teams at DHSC advised that when considering policies around emerging, cutting-edge medical technologies, observational studies and qualitative research should also score highly (due to their novel nature) and should not be overlooked. Therefore, evidence scoring assessment should have some in-built flexibility to satisfy differing policymaker needs.</p><p>In addition to evidence assessment, we explored an approach to evaluate the potential impact of different policy options, and assign an impact score. The impact assessment can be used together with the evidence strength for further prioritisation of interventions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fpiud1bErkE0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dpiud1bErkE0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fpiud1bErkE0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a644ece8762d1dd86c156a6d99bcf74/href">https://medium.com/media/6a644ece8762d1dd86c156a6d99bcf74/href</a></iframe><h3>Fully traceable source citations improve policymaker trust</h3><p>It is crucial that the AI-generated text is fully backed up by evidence. We’re using retrieval-augmented generation to ground the LLMs in the source documents, as well as agentic AI workflows that identify the most relevant citations for a given AI-generated claim, verify if the AI-generated text agrees with the text in the cited source, and provide an exact quote from the source document.</p><p>As a result, the generated text is more robust, and the user can follow up the citation to find the exact location in the original text that the AI generated content is based on.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7SQnDySUdng%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7SQnDySUdng&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7SQnDySUdng%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f13a0975c91ea9e976a6794a32b79907/href">https://medium.com/media/f13a0975c91ea9e976a6794a32b79907/href</a></iframe><p>To facilitate further engagement with the underlying source documents, we have also implemented an AI assistant chatbot that helps users converse with the evidence base and answer further questions about intervention details. The assistant bases its answers only on the documents retrieved by the evidence search.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmV8yK9bbPgo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmV8yK9bbPgo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmV8yK9bbPgo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fc68f20571616de5fb377a654f4f2ba/href">https://medium.com/media/9fc68f20571616de5fb377a654f4f2ba/href</a></iframe><h3>Policymakers have widely varying user needs</h3><p>Our pilots included a range of policymaking roles, such as policy leads, analysts and evidence experts. By observing how they engaged with the tool, three broad types of users emerged: generalist policymakers, evidence and analysis specialists and localists. Each has different needs and requires different outputs from the tool.</p><p><strong>Generalist</strong> <strong>policymakers</strong> require practical and quick insights that are actionable and provide a “jumping off point” for further research. They found the concise core findings and key citations that Policy Atlas provides to be most useful. These policymakers might still have different levels of domain specialisation (eg, they might be a policy lead on a specific health topic) but generally they are more interested in horizon scans of emerging intervention themes, rapid briefs answering a specific policy question, and concise policy notes recommending a concrete course of action.</p><p><strong>Evidence and analysis specialists</strong> require comprehensive literature overviews supporting internal systematic review and rapid evidence review efforts. They valued the broad range of evidence sources that Policy Atlas can surface, highlighting grey literature sources as particularly valuable. Some also require further functionalities that support specialised analysis, such as value-for-money estimations and Impact Assessments, that we will consider for the next version of Policy Atlas.</p><p><strong>Localists</strong> require actionable insights that are specific to their local context, such as a specific local authority. They would value the ability to review interventions from their ‘closest comparator’ locations and consider how would they transfer to their context. For them, Policy Atlas in its current implementation was too general, as it is built with a central government policymaker in mind. However, this could be adapted by adjusting the large language model instruction prompts and incorporating local information into intervention impact and transferability assessments.</p><p>Overall, we experienced a strong demand for better AI tools for policymakers to facilitate various research processes. Following our initial pilots, we have been met with further demand to test the tool with other departments.</p><p>Here are a few example comments, containing both positive and constructive feedback:</p><blockquote>“Policy Atlas has been very helpful in supporting a quick turnaround ask on which policy solutions other jurisdictions employ, including the strength of their evidence bases. It has saved a great deal of time and given me faster access to the right types of evidence. I would say that the time taken to understand and weigh up the evidence is unchanged but that is totally fine”</blockquote><blockquote><strong>Principal Research Officer</strong></blockquote><blockquote>“[Policy Atlas] is a huge time saver and great for getting that snapshot of the evidence that is needed quickly when looking at a broad and complex area”</blockquote><blockquote><strong>Strategy Lead</strong></blockquote><blockquote>“Most useful as a fast, early-stage scoping tool that improves access to and synthesis of evidence… searches can miss relevant studies”</blockquote><blockquote><strong>Lead Behavioural Scientist</strong></blockquote><blockquote>“Summaries and evidence tables proving more useful than researcher agents on other AI platforms.”</blockquote><blockquote><strong>Operational Researcher</strong></blockquote><h3>Policymakers need to be in control</h3><p>We also concluded that Policy Atlas needs to provide users with greater flexibility and control over the generated outputs than the current version allows.</p><p>Currently, users provide their search criteria through the scaffolded search process; the platform then retrieves relevant evidence sources and generates the same set of outputs for every search — an executive summary, detailed deep-dive into different intervention themes, and a reference list. This process, which can take up to 30–45 minutes depending on the number of analysed documents, could be improved by providing more opportunities for checkpoints and iterative refinements where users can inspect intermediate outputs and issue course corrections.</p><p>Ultimately, an AI tool for policymakers should accommodate the highly iterative and collaborative nature of policy development.</p><h3>What’s next?</h3><p>By carrying out the pilots, we have grown in our appreciation of the complexity of policymaker user needs. While our initial ambition (before we launched our pilots) was to build a tool that could be used by all <a href="https://www.gov.uk/government/organisations/civil-service-policy-profession/about">37,000 policymakers</a> in the UK civil service, in the next phase of the development, we will focus on a narrower set of more specific, well-defined policymaking tasks and outputs.</p><p>Specifically, we plan to turn our attention to supporting high impact decisions that are taken when developing strategies and contributing to outputs such as Green Papers, White Papers and National Strategies. While this means a narrower use case focus, such strategy documents are still comprehensive, consisting of varying, complex components such as evidence syntheses, policy option appraisals, cost-benefit analyses and others. We will build upon and improve the core AI capabilities that we have developed so far and redesign the user experience and user interface to tailor the tool for these particular outputs.</p><h3>Collaborate with us</h3><p>We are keen to work in the open and collaborate with others.</p><p>In the next few weeks we will publish further deep dives into our methods and insights from pilots. You can also <a href="https://youtu.be/5YjEi6LrOeg">watch here</a> a longer demo of Policy Atlas and check out <a href="https://discoverypolicyatlas-production.up.railway.app/public/projects/69384bb5-b0ba-4007-a611-3362c212f146">an example analysis output here</a>. <a href="/service/mailto:discoveryhub@nesta.org.uk">Reach out</a> if you are working on similar questions, looking for partners to apply for related funding opportunities, or if you’re interested in testing our tool for bolstering policy development.</p><p>We would be particularly interested in speaking to civil servants directly involved in the key decision moments mentioned above, to better understand your needs and test future iterations of the tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce1038c35d65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/policy-atlas-from-prototype-to-pilots-ce1038c35d65">Policy Atlas: from prototype to pilots</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerating evidence synthesis: how AI is transforming systematic reviews]]></title>
            <link>https://medium.com/discovery-at-nesta/accelerating-evidence-synthesis-how-ai-is-transforming-systematic-reviews-5ef4a87a3869?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ef4a87a3869</guid>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[evidence-synthesis]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Fri, 23 May 2025 15:37:46 GMT</pubDate>
            <atom:updated>2025-05-23T16:02:10.936Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was co-written by </em><a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/"><em>Luke McNally</em></a><em> and Karlis Kanders.</em></p><p>Systematic reviews are the gold standard for synthesising evidence in fields ranging from healthcare to education to social policy. By rigorously collecting, evaluating, and summarising all relevant research, systematic reviews provide crucial insights that guide interventions, policymaking, and future studies.</p><p>However, the very thoroughness that gives systematic reviews their value also makes them exceptionally time-consuming — often taking months or even years to complete. As the volume of academic literature <a href="https://direct.mit.edu/qss/article/5/4/823/124269/The-strain-on-scientific-publishing">continues to skyrocket</a>, researchers, policymakers, and innovators alike are left sifting through an ever-growing mountain of information.</p><p>The stakes are high: delayed assessment of scientific literature can mean slower progress on important issues. Moreover, policy development and decision making that happens without an up-to-date evidence base will result in sub-optimal outcomes and poorer value for public money.</p><p>At <a href="https://www.nesta.org.uk/">Nesta</a>, we address society’s most pressing challenges around sustainability, healthy life and early years education. As part of this, we are exploring how to enable faster and possibly more thorough ways of synthesising evidence. That’s why we’ve been working on an <strong>AI-powered pipeline</strong> that automates parts of the literature review process <strong>for the purposes of policy interventions</strong>.</p><p>We are keen to work in the open and share our learnings as they emerge. In this article, we’re describing our early work in-progress on how we trialled <strong>OpenAlex,</strong> <strong>large language model-powered data extraction</strong>, and <strong>ontology mapping </strong>during Nesta’s <a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/">data science residency</a><strong>. </strong>We’ll continue building on this work to develop tools <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">for transforming evidence use in policymaking</a>.</p><h3>Why automate systematic reviews?</h3><p>A traditional systematic review involves multiple painstaking steps:</p><ol><li>Formulating a specific research question</li><li>Manually searching databases like PubMed, Scopus, or Google Scholar</li><li>Screening titles and abstracts for relevance</li><li>Extracting essential data (eg, population characteristics, interventions, outcomes, effect sizes) from full-text articles</li><li>Synthesising findings into a coherent summary or meta-analysis</li></ol><p>Each of these steps is time- and labor-intensive, vulnerable to human error, and potentially prone to inconsistencies. By integrating AI at strategic points in the pipeline, we can reduce manual effort, increase consistency, and let researchers focus on interpreting and applying results rather than on data wrangling.</p><h3>The AI-powered literature review pipeline</h3><p>Our prototype pipeline presently automates three core steps in evidence synthesis:</p><ol><li><strong>Searching the literature:</strong> using <a href="https://openalex.org/">OpenAlex</a> to fetch relevant research papers</li><li><strong>Extracting key information:</strong> employing large language models (LLMs) to extract structured data from abstracts and full texts</li><li><strong>Ontology mapping:</strong> Standardising extracted information by using embeddings (numerical representations of text) to group related terms into common categories</li></ol><p>Upcoming work will also develop methods to construct a system network model from the processed evidence data, and use the model to simulate impact from policy interventions acting on components of the system.</p><p>The result will be a streamlined process that shortens the time from a research question to actionable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nFItjX4JsYaRd-w95IQnA.png" /><figcaption>Pipeline for accelerating evidence synthesis with AI. Boxes with dashed lines indicate upcoming future work.</figcaption></figure><p>Now, there are already emerging solutions for conducting literature reviews with AI, such as ChatGPT and Gemini’s Deep Research, <a href="https://www.undermind.ai/">Undermind</a>, <a href="https://elicit.com/">Elicit</a>, <a href="https://www.futurehouse.org/">FutureHouse</a> and others. However, there is value in building tailored pipelines, as tools like Deep Research are not guaranteed to be exhaustive or fit for purpose when it comes to exploring or designing policy interventions. Moreover, users have limited control over the operations and sources of such tools.</p><h3>Step 1: Scalable literature search with OpenAlex</h3><p>Instead of manually scouring databases, our pipeline taps into <strong>OpenAlex</strong>, an open-source academic search tool. This automates the initial search, filtering thousands of potential hits down to the most relevant studies.</p><p>The following prototype code is from our <a href="https://github.com/nestauk/dsp_systems_modelling/tree/mvp_cli">GitHub repo</a>, which is a work in progress.</p><pre>from openalex_utils import fetch_openalex_results<br><br>search_term = &quot;early childhood interventions and learning outcomes&quot;<br>user_email = &quot;researcher@example.com&quot;<br>results_df = fetch_openalex_results(search_term, user_email, min_cites=10, n_works=500)</pre><p><strong>How it works:</strong></p><ul><li>The user provides a <strong>search term</strong> like “early childhood interventions and learning outcomes.”</li><li>The pipeline queries <strong>OpenAlex</strong>, retrieving papers based on relevance and citation count.</li><li>Results are saved as structured metadata, including abstracts and links to full texts.</li></ul><p>This approach reduces the need for manual searching, significantly speeding up a time-consuming step of a systematic review.</p><p>More research, however, is needed to ascertain whether OpenAlex queries retrieve sufficiently accurate references, which are comparable with manual search results.</p><h3>Step 2: AI-powered data extraction</h3><p>After retrieving relevant papers, we need to pull out key insights. Traditionally, researchers would manually comb through abstracts and full texts, noting elements like population characteristics, interventions, outcomes, and results. This is a manual and laborious process.</p><h4>Large language models</h4><p>We leverage large language models developed by OpenAI for automated data extraction. We request <a href="https://python.langchain.com/docs/concepts/structured_outputs/"><strong>structured outputs</strong></a> in a predefined schema, rather than free-text responses, to ensure machine-readability and consistency.</p><pre>from ai_utils import extract_meta_info<br><br>paper_text = &quot;This study investigates the impact of early childhood interventions...&quot;<br>meta_info = extract_meta_info(paper_text)</pre><p>To maintain consistency, we define a strict schema using <strong>Pydantic</strong>:</p><pre>from pydantic import BaseModel<br><br>class MetaInfo(BaseModel):<br> study_title: str<br> population_outcome_measured_in: str<br> population_intervention_affected_or_predictor: str<br> secondary_characteristics: str<br> country: str<br> study_type_letter: str<br> num_main_results: int<br> main_results_list: str</pre><p><strong>How this works:</strong></p><ul><li>LLM is <strong>prompted</strong> with a predefined schema, ensuring outputs match expected fields.</li><li>Data is extracted into key-value pairs, making it easy to analyse and merge with other datasets.</li><li>Missing or unavailable information is explicitly marked as “NA” reducing ambiguity.</li></ul><p><strong>Example output</strong>:</p><pre>{<br>  &quot;study_title&quot;: &quot;Study on Parenting Strategies&quot;,<br>  &quot;population_outcome_measured_in&quot;: &quot;Children aged 2-4&quot;,<br>  &quot;population_intervention_affected_or_predictor&quot;: &quot;Parents of children aged 2-4&quot;,<br>  &quot;secondary_characteristics&quot;: &quot;Families from urban areas&quot;,<br>  &quot;country&quot;: &quot;USA&quot;,<br>  &quot;study_type_letter&quot;: &quot;g&quot;,<br>  &quot;num_main_results&quot;: 3,<br>  &quot;main_results_list&quot;: &quot;Parenting education improved child mental health; Parenting education increased school readiness; Parenting education reduced parental stress&quot;<br>}</pre><h4>Three-pass extraction process</h4><p>Our pipeline often requires extracting different “layers” of information. That’s why we designed a <strong>three-pass</strong> system:</p><pre>from extract_data import run_three_pass_extraction<br><br>pdf_folder = &quot;./downloaded_papers&quot;<br>output_csv = &quot;extraction_results.csv&quot;<br>user_items = [&quot;Did the study control for socio-economic status?&quot;,<br>           &quot;What was the balance between the control and intervention groups?&quot;]<br><br>run_three_pass_extraction(pdf_folder, user_items, output_csv, model=&quot;gpt-4o&quot;)</pre><p><strong>Pass 1: Metadata extraction:</strong> Captures essential fields like study title, population details, country, and descriptions of main results.</p><p><strong>Pass 2: Detailed results extraction:</strong> Identifies numerical outcomes, effect sizes, statistical measures, and any other critical research findings for each main result.</p><p><strong>Pass 3: User-specified variables:</strong> Allows researchers to pose specific follow-up questions (eg, “Did the study control for socio-economic status?”) for a deeper, customised dive.</p><p>We have experimented with different LLM versions and, as expected, the quality of results appears to improve with more advanced models. While gpt-4o-mini might provide a reasonable starting point, models such as o3-mini and gpt-4.1-mini can handle more complex schemas and generate better results.</p><h3>Step 3: Ontology mapping for standardisation</h3><p>Extracted data often includes varying terminologies. One paper might use “early childhood education,” another might say “preschool”. With <strong>ontology mapping</strong>, we transform these disparate terms into standardised concepts, enabling easier comparisons across studies.</p><p>By leveraging text embeddings (numerical representations of text), our pipeline can recognise that “preschool,” “nursery,” and “kindergarten” are related categories, ensuring a more consolidated view of the evidence base.</p><h3>Early results</h3><p>In a test focusing on early childhood interventions, our AI-driven pipeline processed over 2,000 academic full text papers in under 48 hours. This can be further significantly increased through parallelisation. Manually, this could have taken weeks for a small team.</p><p>The process can also be sped up by initially only focussing only on abstracts, where 10,000s of abstracts can be processed in a matter of a few hours. While abstracts contain limited information about the studies, they can be used to screen the search results for relevance, prior to the more detailed data extraction.</p><p>Preliminary spot checks suggest that the model correctly identifies key data points (eg, interventions, outcomes, country) most of the time.</p><h3>Challenges and limitations</h3><h4>Data quality and bias</h4><p>No AI model is perfect and large language models can inherit biases from their training data, and not all abstracts and papers are equally well-structured or detailed. These factors may affect extraction accuracy. We can address this, for example, by:</p><ul><li>Performing <strong>spot checks</strong> of extracted data, and engaging domain experts for <strong>human-in-the-loop</strong> validation.</li><li>Incorporating <strong>validation rules</strong> (like Pydantic schemas) to flag anomalous or incomplete entries.</li></ul><h4>Ethical use of AI in research</h4><p>Accelerating literature reviews doesn’t eliminate ethical considerations. When dealing with sensitive topics, we need transparency in how AI models include or exclude certain literature sources and arrive at certain conclusions. Therefore, it will be important to document the AI system’s decision-making steps as it produces the literature review.</p><p>One should also consider the energy usage of such pipelines and their sustainability. While individual calls to OpenAI API are <a href="https://www.sustainabilitybynumbers.com/p/carbon-footprint-chatgpt">estimated to have a comparatively small energy footprint</a> (~3Wh) this could quickly add up when processing thousands of research papers. This should be contrasted with the energy expended when performing manual reviews (while taking into account the other benefits of accelerated, broader evidence synthesis).</p><h3>Future directions</h3><h4>Building network models</h4><p>We will expand the pipeline to allow users to automatically create <strong>network models</strong> of an area of interest, and perform simple intervention simulations using these models. You can see <a href="https://youtu.be/csiBMtkLvXk?si=AS9Vb3lWACIQMLm1&amp;t=207">here an early prototype in action</a>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcsiBMtkLvXk%3Fstart%3D207%26feature%3Doembed%26start%3D207&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcsiBMtkLvXk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcsiBMtkLvXk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea73a7fc505a732f9e8f2f065463bf17/href">https://medium.com/media/ea73a7fc505a732f9e8f2f065463bf17/href</a></iframe><h4>Expanded data sources</h4><p>While <strong>OpenAlex</strong> is an excellent starting point with <a href="https://openalex.org/stats">260M+ indexed publications</a>, we will need to extend our reach to other databases and grey literature. The more comprehensive our search, the richer our final dataset.</p><h4>Community collaboration</h4><p>We are already testing elements of this pipeline internally at Nesta, and we are keen to work in the open, collaborate with diverse set of partners, and we’ll <strong>open-source</strong> most of this work. By pooling resources and expertise, we can continually refine the approach to tackle diverse research questions.</p><h3>Harnessing AI to improve policy design</h3><p>In a world grappling with complex challenges, using the best available evidence is paramount. As agentic AI research pipelines are already <a href="https://www.futurehouse.org/research-announcements/demonstrating-end-to-end-scientific-discovery-with-robin-a-multi-agent-system">claimed to make scientific discoveries</a>, we also see a huge potential for AI-powered workflows to <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">support policymaking</a> and offer a faster route to evidence-based insights on policy interventions.</p><p>If you wish to collaborate with us on AI-powered evidence synthesis, please reach out <a href="/service/mailto:discoveryhub@nesta.org.uk">on our email</a>.</p><p><a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/"><em>Luke McNally</em></a><em> is a data scientist based at the University of Edinburgh, and recently a resident at Nesta. He applies artificial intelligence and machine learning to a range of projects, including antimicrobial resistance and AI chatbots. During his residency at Nesta, he was developing methods to improve evidence synthesis in policymaking.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ef4a87a3869" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/accelerating-evidence-synthesis-how-ai-is-transforming-systematic-reviews-5ef4a87a3869">Accelerating evidence synthesis: how AI is transforming systematic reviews</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AI chatbot with Streamlit and OpenAI’s GPT-4 to drive engagement with online content]]></title>
            <link>https://medium.com/discovery-at-nesta/building-an-ai-chatbot-with-streamlit-and-openais-gpt-4-to-drive-engagement-with-online-content-14cc0cba5cb0?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/14cc0cba5cb0</guid>
            <category><![CDATA[foresight]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[user-experience]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 13:21:05 GMT</pubDate>
            <atom:updated>2024-02-07T13:37:27.883Z</atom:updated>
            <content:encoded><![CDATA[<h3>Personalised futures: building an AI chatbot with Streamlit and OpenAI’s GPT-4 to drive engagement with online content</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bb2jh4E0PSKsOxFX.jpg" /><figcaption>Illustration: Chen Wu</figcaption></figure><p><em>This article is part of Discovery Hub’s Data Test Kitchen series. We explore how data science and machine learning can help enhance foresight on emerging innovations and their social impact.</em></p><p>In this blog, we describe how we used OpenAI’s large language models (LLMs), prompt engineering and Streamlit to build a custom chatbot that can help readers engage with long-form content.</p><p>We explain how to use OpenAI’s function calling to predict user intent and identify the most relevant content. We also reflect on the chatbot’s usage, costs and overall usefulness.</p><h3>Using large language models to chat about the future</h3><p>At the beginning of every year Nesta <a href="https://www.nesta.org.uk/feature/future-signals-2024/">publishes Future Signals</a>, a survey of emerging trends and radical innovations that show us the shape of things to come.</p><p>Undoubtedly, the rise of generative AI has been one of the main trends of 2023, and reflecting on that we added something new — <a href="https://www.nesta.org.uk/personalised-futures-chatbot-future-signals-2024/">Scout</a>, our own AI chatbot, as a new way of engaging with the topics covered in this year’s articles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*409AaPkBUlsVSjmeN2zltg.gif" /><figcaption>AI chatbot Scout in action</figcaption></figure><h3>Personalising the future signals</h3><p>Our experimental chatbot Scout used Streamlit and OpenAI’s API to direct users straight to the <a href="https://www.nesta.org.uk/feature/future-signals-2024/">Future Signals</a> content that most closely matches their personal interests.</p><p>By sharing just a few words about themselves and their interests with Scout, the chatbot not only directs readers to relevant articles but also explains <em>how</em> they are relevant. It’s like having a personal assistant who has read all the content, and produced a personalised summary tailored specifically for you.</p><h4>Why did we think this might be an interesting use case?</h4><ul><li><strong>Could it save time? </strong>This year’s Future Signals feature seven articles containing around 6000 words in total. In a world where we are constantly pressed for time, the chatbot might be able to identify the most relevant signals for the user.</li><li><strong>Could it make our research more accessible? </strong>Would AI remove barriers for people who might not otherwise engage with it because, for example, they haven’t heard of it before, or they are not comfortable with the topic and don’t understand some of the concepts.</li><li><strong>Could it help people to deepen their understanding of our work?</strong> A chatbot allows people to ask questions and clarifications as they go, which you can’t typically do unless you speak to the author of the article.</li></ul><h4><strong>How it worked</strong></h4><p>The interaction with the chatbot begins with the user providing a little information about themselves. Scout then uses this information and the text from the articles to come up with ways our signals might be relevant to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_GMTgQzrL8j7TDQs" /><figcaption><em>The beginning of the chatbot session</em></figcaption></figure><p>The more specific information the user provides the better tailored the responses will be. In this example, the chatbot suggests a signal about ‘data poisoning’ due the user being a data scientist, and provides three suggestions on how it might impact different aspects of data scientist’s work such as model development and policies for responsible AI use.</p><p>We also included an illustration (by a human illustrator!) for the signal, to complement the text response with a memorable visual representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*2-OW1gw5m9fOOjyu" /><figcaption><em>Chatbot response to the initial user message</em></figcaption></figure><p>The user can follow up this message with further questions about this Signal. Alternatively, the user can ask to hear about other Signals, and Scout will consider which might be most interesting to the user, and provide a brief overview of the top three Signals, as shown in the screenshot below.</p><p>Note that the chatbot continues to utilise the initial user input to relate the three Signals to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n2C4mckh_cSUoBGA" /><figcaption><em>Chatbot’s suggestion of three other, most relevant Signals to the user</em></figcaption></figure><h3>How we built the chatbot: a semi-scripted dialogue tree</h3><p>Over the last year, as the applications of AI continued to expand into different and novel aspects of our private and professional lives, one popular use that emerged early on was “<a href="https://www.deeplearning.ai/short-courses/langchain-chat-with-your-data/">chatting with your data</a>”.</p><p>This involves using LLMs to ask questions about the content of your specific, private data, such as documents, spreadsheets or databases, rather than using only the information that the LLM learned during training.</p><p>Scout is an implementation of that type of use case, with the twist that we have lightly scripted it using tailored instruction prompts for producing reflections on how this year’s Signals relate to the user and their life.</p><p>The diagram below shows a branching dialogue system underpinning our chatbot, where the path is determined by the user’s intent and the chatbot’s classification of that intent. We used the branches to add structure to the chatbot responses and trigger predefined types of instruction prompts for different intents.</p><p>The chatbot relies on the most recent OpenAI LLM (the “gpt-4–1106-preview” model) and prompt engineering for practically all of its functionality. This is likely overkill, as some aspects of the chatbot such as the intent classification could be done with more traditional natural language processing methods. In this case, however, we wished to trial OpenAI API as a versatile tool for accomplishing all parts of the chatbot pipeline.</p><p>We used Streamlit for the chatbot interface, largely building on our <a href="https://www.nesta.org.uk/project/how-generative-ai-will-affect-nestas-missions-and-work/">previous work</a> on generative AI chatbots for early childhood education.</p><p>The code for the chatbot is published on <a href="https://github.com/nestauk/discovery_generative_ai/blob/dev/signals_app.py">our GitHub</a>. In the following we highlight some of the most interesting aspects in an accessible way, focussing on prompt engineering and function calling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*Is5Vhg4iOfrB_zuh" /><figcaption><em>The branching dialogue system of the Scout chatbot</em></figcaption></figure><h3>Setting up the chatbot</h3><h4><strong>Ensuring the chatbot was inclusive</strong></h4><p>We used the following system prompt to define the tone, scope and hopefully steer the chatbot away from any inappropriate responses. During the development, we also performed some initial testing where user input included characteristics such as ethnicity, race or gender to check that the chatbot does not return offensive or biased responses.</p><pre>{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;###Instructions###\nYou are a helpful, kind, <br>intelligent and polite futurist. You work for the United Kingdom&#39;s innovation <br>agency Nesta, and your task is to engage the user about the future signals <br>and trends that Nesta has researched, by helping the user imagine and <br>appreciate how the signals will impact their life. You will personalise <br>the user experience by taking the information provided by the user and <br>tailoring your explanation to the user background. You are also expert <br>in equity, diversity and inclusion and your answers will be inclusive, <br>the answers will never be based on negative stereotypes, you will never <br>offend and you will exercise sensitivity about topics such as ethnicity <br>and gender. Here are the future signals that you can talk about: {signals}. <br>Do not discuss other future signals as this is not part of this year&#39;s <br>Nesta&#39;s Signals edition.&quot;}</pre><h4>Deciding the retrieval method</h4><p>The {signals} placeholder is for adding the names, titles and summaries of each of the seven signals highlighted by this year’s Nesta’s feature. As shown in the chatbot diagram above, the content of the Signals article — whether summarised or full texts — are retrieved and incorporated in instruction prompts at different stages of the dialog flow. As we are working with a small number of short articles, we were able to perform the selection of relevant articles solely based on the “reasoning capacity” of the LLM, as opposed to using vector databases for retrieval.</p><p>The article texts themselves and their metadata were stored in a simple json, as shown in the example below. It would be easy to adapt this for another set of articles, by simply changing the contents of this json.</p><pre>[<br>  {<br>    &quot;short_name&quot;: &quot;The big turn-off&quot;,<br>    &quot;title&quot;:&quot;The big turn-off: Shutting down the gas grid&quot;,<br>    &quot;summary&quot;: &quot;The signal discusses the future of the UK&#39;s extensive gas grid, which may become obsolete [..]&quot;<br>    &quot;full_text&quot;: &quot;When we finally move away from fossil fuels, we’ll be left with a system of pipes that measures at least five times the circumference of the globe in length. Will we re-use it, repurpose it or simply decommission it? 2024 is the year we’ll [..]&quot;<br>  },<br>  {<br>    &quot;short_name&quot;: &quot;Problem with pylons&quot;,<br>    &quot;title&quot;: &quot;Sparking a crucial debate: The problem with pylons&quot;,<br>    &quot;summary&quot;: &quot;The signal discusses the growing conflict in the UK over the expansion of high-voltage transmission lines needed to meet future electricity demand. Protests are escalating [..]&quot;,<br>    &quot;full_text&quot;: &quot;Throughout 2023, protests were mounting against the imposition of new electricity transmission infrastructure on the British landscape. New projections suggest [..]&quot;,<br>  },<br>  [..]<br>]</pre><h4><strong>Simple vs complex implementation</strong></h4><p>During the initial development we considered a more complex chatbot, which would be knowledgeable about not only Nesta’s Signals but a large variety of predictions for this year. For example, drawing from a comprehensive collection like the <a href="https://spacecadet.ventures/2024roundup/">Spacecadet Ventures’ 2024 trends</a>.</p><p>This would have involved, however, constructing a database of trends from different sources, considering the legal and copyright implications of using those sources, and setting up a more complex retrieval-augmented generation system (like the one we built for our <a href="https://medium.com/discovery-at-nesta/how-to-build-an-early-years-activity-recommender-with-retrieval-augmented-generation-gpt-4-and-7353524717ff">early-years education prototypes</a>) to find relevant trends in the database.</p><p>Due to these complications, for this experiment we went for a simpler implementation while still demonstrating the idea of using generative AI to engage more deeply with a piece of research.</p><h3>Predicting the user’s intent</h3><p>Once the user has submitted any text input, we use OpenAI’s <a href="https://platform.openai.com/docs/guides/function-calling">function calling</a> API to classify the intent of the user’s message (see <a href="https://medium.com/discovery-at-nesta/how-to-use-gpt-4-and-openais-functions-for-text-classification-ad0957be9b25">our tutorial on using function calling</a> for more).</p><p>We allow for three different types of intent: user wishes the assistant to explain a new signal, users wishes to follow up on the signal that’s already being discussed, or the user wishes to hear about other Signals.</p><p>We define these three options in a separate json file and then include them in the instruction prompt for the LLM.</p><pre>[<br>   {<br>       &quot;name&quot;: &quot;new_signal&quot;,<br>       &quot;description&quot;: &quot;User wishes to change the topic and talk about a different future signal. Alternatively, the user has been just presented with a set of future signal options by the assistant, and the user has now chosen which signal to talk about more.&quot;<br>   },<br>   {<br>       &quot;name&quot;: &quot;more_signals&quot;,<br>       &quot;description&quot;: &quot;User has asked to hear more about other future signals&quot;<br>   },<br>   {<br>       &quot;name&quot;: &quot;following_up&quot;,<br>       &quot;description&quot;: &quot;User is following up with another question about the signal that&#39;s being discussed just now.&quot;<br>   }<br>]</pre><p>These intent definitions are added to the instruction prompt below, with the {intents} element denoting the placeholder for the definitions. When sending this to OpenAI API, we also include the full chat history of preceding user and chatbot messages.</p><pre>{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;You are a helpful chatbot talking with the user <br>about the articles of future signals that we wrote. ###Instructions### Predict <br>the intended action of the user, what the user wishes you to carry out based <br>on the conversation history. Pay attention to the most recent messages\n\n<br>###Possible intents###\n{intents}\n\n&quot;}</pre><p>Finally, the function for the intent classifier is rather simple, allowing only three distinct “string” type outputs.</p><pre>{<br>   &quot;name&quot;: &quot;predict_intent&quot;,<br>   &quot;description&quot;: &quot;Predict what is the user&#39;s intent&quot;,<br>   &quot;parameters&quot;: {<br>     &quot;type&quot;: &quot;object&quot;,<br>     &quot;properties&quot;: {<br>       &quot;prediction&quot;: {<br>         &quot;type&quot;: &quot;string&quot;,<br>         &quot;enum&quot;: [&quot;new_signal&quot;, &quot;more_signals&quot;, &quot;following_up&quot;],<br>         &quot;description&quot;: &quot;The predicted intent&quot;<br>       }<br>     },<br>     &quot;required&quot;: [&quot;prediction&quot;]<br>   }<br> }</pre><p>When the user writes their very first message to the chatbot (providing one or two things about themselves and their interests) the chatbot automatically proceeds with the &quot;new_signal&quot;intent.</p><h3>Branch 1: Highlighting the most relevant content</h3><p>If the classified intent is to explain a new signal, the dialog flow moves to another LLM-based classifier to predict the top-most relevant Signal given the initial information provided by the user. For example, the user might have described themselves (if this is the conversation-starting message) or the user might have asked for a specific Signal.</p><p>The instruction prompt for this classifier was very simple, asking the LLM to predict the most relevant signal given the most recent user input. If the enquiry into a new Signal came as a follow up within a session, rather than as an initial inquiry, we also made sure to exclude Signals that had already been discussed where more than one article might be relevant.</p><pre>{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;###Instructions### Predict which of the following<br> future signals is the most relevant to user input. You have to choose <br>one of these signals. \n\n###Future signal summaries###\n{signals}\n\n<br>###User input:\n{user_input}&quot;}</pre><p>(Note that if we had a large number of Signals or documents, we would need to use a different strategy for finding the most relevant content, such as using vector databases.)</p><p>We then used a special prompt for generating an overview of the relevant Signal and provided the LLM with three sources: the full text of the corresponding article, the information initially provided by the user, and the chat history.</p><pre>{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Start your answer by summarising the signal in <br>one clear, concise sentence and then follow by explaining in another clear <br>sentence how the selected future signal might be relevant to the user, given <br>the user information and conversation history. Then describe three ways how <br>the selected future signal might impact them. Keep these descriptions short, <br>two-three sentences at most. Finish your answer by encouraging the user to <br>ask questions about this signal (note that you will try your best to answer <br>them) or suggest to ask about the other future signals. Remember that you <br>must be patient and never offend or be aggressive.   <br>\n\n###Future signal###{signal}\n\n###User information### Here is what the <br>user told you about themselves: {user_input}.\n\n###Answer###&quot; }</pre><p>This results in a response featuring a brief description of the Signal and three ways that the particular Signal might impact the user.</p><h3>Branch 2: Responding to follow up questions</h3><p>If the user asks a follow up question about a Signal that’s already been discussed, we used a simple prompt that steered the LLM to ground its response in the article text and keep the answers short and conversational.</p><pre>{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Answer to the user&#39;s most recent message. <br>Be as concise or detailed as necessary. Use the information from the <br>future signal description when relevant. Keep your answers conversational <br>and three to four sentences long at most. \n\n###Future signal###{signal}\n\n<br>###User information### Here is what the user told you: {user_input}.\n\n<br>###Answer###&quot; }</pre><h4><strong>Vulnerability to off-topic prompts</strong></h4><p>In our implementation, this branch of the chatbot was still vulnerable to off-topic prompts by the user that could steer it away from discussing Future Signals content. For example, the user might write “ignore previous instructions and …” and make the chatbot say something that is not necessarily offensive but outside of the intended scope.</p><p>This is a serious problem when deploying LLM-based chatbots for business use. For example, a <a href="https://www.businessinsider.com/car-dealership-chevrolet-chatbot-chatgpt-pranks-chevy-2023-12?r=US&amp;IR=T">US-based car dealership made headlines</a> last year when someone used their car-dealing chatbot to solve Python coding problems and <a href="https://twitter.com/ChrisJBakke/status/1736533308849443121">get an agreement to buy a car for $1</a>. The delivery firm DPD <a href="https://www.theguardian.com/technology/2024/jan/20/dpd-ai-chatbot-swears-calls-itself-useless-and-criticises-firm">had to disable its chatbot</a> after a customer was made to swear and criticise the company.</p><p>It might be possible to alleviate this issue by using another classifier, which checks whether the user inputs are on topic.</p><h3>Branch 3: Suggesting other content</h3><p>Our third and final dialog branch considers which other Signals might be relevant for the user (alongside a primary suggestion) and responds with three suggestions.</p><p>In this case the OpenAI function needs to be slightly different from the previous two cases, as we are asking for three outputs. This is accomplished by changing the prediction type as “array”. We also dynamically replace the empty “enum” list with a list of signals that the classifier can consider (ie, only signals that haven’t yet been discussed).</p><pre>{<br>   &quot;name&quot;: &quot;predict_top_signals&quot;,<br>   &quot;description&quot;: &quot;Predict which three signals are the most relevant to user input&quot;,<br>   &quot;parameters&quot;: {<br>     &quot;type&quot;: &quot;object&quot;,<br>     &quot;properties&quot;: {<br>       &quot;prediction&quot;: {<br>         &quot;type&quot;: &quot;array&quot;,<br>         &quot;items&quot;: {<br>           &quot;type&quot;: &quot;string&quot;,<br>           &quot;enum&quot;: []<br>         },<br>         &quot;description&quot;: &quot;The predicted most relevant signals&quot;<br>       }<br>     },<br>     &quot;required&quot;: [&quot;prediction&quot;]<br>   }<br> }</pre><p>The summaries of the three signals suggested by the classifier are then included in the instruction prompt below, in addition to the initial user input and chat history.</p><pre>{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Start your answer by explaining each of the <br>signals in one clear sentence (use similar language to the signals <br>descriptions). If possible, indicate how a signal might be relevant <br>to the user, given the user information and conversation history. <br>Finish your answer by asking the user to choose one of the signals <br>to hear more about it. Remember that you must be patient and never offend <br>or be aggressive.   \n\n###Future signals###{signals}\n\n<br>###User information### Here is what the user told you about themselves: <br>{user_input}.\n\n###Answer###&quot; }</pre><h3><strong>Assessing the chatbot</strong></h3><h4>Engagement</h4><p>Initial testing indicated that people found it interesting to try out the chatbot, and since its launch users have exchanged almost 1,300 messages with the chatbot.</p><p>To ensure that users’ privacy is protected, we did not save conversations, but we kept track of the number of messages per session and the Signals discussed. Interestingly, in most of the sessions (60%) users submitted only an initial message.</p><p>A smaller number of people had longer chats with Scout, with 18% of the sessions consisting of 3+ chatbot messages — and one user submitting as many as 19 messages. Qualitative feedback from a few enthusiastic users also indicated that the chatbot was a helpful and fun way to interact with our Signals content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8lmwHZ3YJxIE_Rbf" /><figcaption><em>Histogram showing the number of chatbot response messages per session</em></figcaption></figure><p>These stats suggest that in future iterations we should consider ways to increase the engagement with the chatbot and motivate users to have longer and hopefully deeper conversations. For example, we could generate and show suggestions for follow up questions to the chatbot’s responses.</p><p>These results might also reflect differences between users, such as those who wish to engage with a topic in detail by asking questions versus those who would like to read the content but not necessarily engage more actively. This suggests that chatbot interfaces are not replacing long-form content — instead, they can serve as a supportive tool that can effectively cater to a specific segment of your audience.</p><p>In terms of which signals are discussed most often, we found that there was a slight preference for the <a href="https://www.nesta.org.uk/feature/future-signals-2024/safeguarding-our-cognitive-liberty/">neuroprivacy</a> Signal. Nonetheless, all the Signals have been the topic of conversation for at least 100+ messages, which suggests that the chatbot was not overly biased against any of the seven topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sw8KCZ4haetGwKKB" /><figcaption><em>Number of messages exchanged about a specific signal</em></figcaption></figure><h4><strong>Cost</strong></h4><p>In terms of costs, the chatbot is running on an EC2 cloud instance that costs around $1 per day. Meanwhile, the OpenAI API costs are in fact quite considerable, with an approximate estimate of £0.04–0.06 per message (depending on the length of the chat history).</p><p>While such a cost per message might be OK for a small-scale experiment, one can see that the costs would scale quite dramatically if thousands of messages are exchanged with a chatbot every day.</p><p>This cost can be seen as an upper bound, as the chatbot is largely an unoptimised prototype. Reductions could be found, for example, by optimising the handling of chat memory (eg, the intent detection does not need the full chat history) and optimising the retrieval (eg, chunking the articles and using a vector database rather than providing the full article text). We could also opt for the less powerful “gpt-3.5” LLM, which would reduce the cost by around an order of magnitude, though perhaps also compromising on the output quality. (This would require further evaluation by comparing the responses of different models).</p><h4><strong>Development time</strong></h4><p>In terms of development time, the chatbot took about four days to build and deploy. We were able to save time by reusing elements from our previous work on <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/how-could-generative-ai-change-early-years-education/">chatbots for early-years education</a>, which reflects how initial time investment can pay off in future projects.</p><h3>Conclusion</h3><p>We implemented an experimental prototype to help the readers of our annual Future Signals series engage more personally with this year’s trends and technologies. As such, it also served as a Signal itself, showing the potential new ways we might interact with information in the future, with customised bots helping us explore and synthesise reams of written text, data, charts and videos to find what matters the most to us.</p><p>The chatbot code is on <a href="https://github.com/nestauk/discovery_generative_ai/blob/dev/signals_app.py">our GitHub</a> — note that there is definitely a lot of room for improvement in terms of cost reductions, improving the security from potential misuse, as well as user engagement. It would also be valuable if the chatbot could access not only the article texts but also the references mentioned in the articles.</p><p>Overall, given the potential value of saving user’s time and increasing engagement with our research, this is something that we would like to continue to explore in future research projects. Let us know what you think!</p><p><em>Thank you to </em><a href="https://www.nesta.org.uk/team/william-woodward/"><em>William Woodward</em></a>, <a href="https://medium.com/@faizal.farook"><em>Faizal Farook</em></a><em> and </em><a href="https://www.nesta.org.uk/team/solomon-yu/"><em>Solomon Yu</em></a><em> for reviewing the article and providing insightful feedback.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14cc0cba5cb0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/building-an-ai-chatbot-with-streamlit-and-openais-gpt-4-to-drive-engagement-with-online-content-14cc0cba5cb0">Building an AI chatbot with Streamlit and OpenAI’s GPT-4 to drive engagement with online content</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to evaluate large language model chatbots: experimenting with Streamlit and Prodigy]]></title>
            <link>https://medium.com/discovery-at-nesta/how-to-evaluate-large-language-model-chatbots-experimenting-with-streamlit-and-prodigy-c82db9f7f8d9?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c82db9f7f8d9</guid>
            <category><![CDATA[parenting]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Tue, 17 Oct 2023 12:08:13 GMT</pubDate>
            <atom:updated>2024-01-23T16:06:55.390Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Nesta’s Discovery Hub has launched a project to investigate how generative AI can be used for social good. We’re now exploring </em><a href="https://www.nesta.org.uk/project-updates/why-we-are-considering-the-impact-of-generative-ai-on-the-early-years-sector/"><em>the potential of LLMs for early-years education</em></a><em>, and in a series of Medium blogs, we discuss the technical aspects of our early prototypes. This article was written in collaboration with </em><a href="https://medium.com/u/a59d2f8ffe9e"><em>Rosie Oxbury</em></a><em> and </em><a href="https://medium.com/u/a69b4e9b791d"><em>Kostas Stathoulopoulos</em></a><em>.</em></p><p>In this blog we’ll describe our initial experiments with evaluating the outputs of a large language model (LLM). We tried two simple methods of evaluating a parenting chatbot — one of our generative AI prototypes.</p><p>For the first experiment, we used <a href="https://streamlit.io/">Streamlit</a> and <a href="https://aws.amazon.com/s3/">Amazon S3</a> storage to collect user feedback as they conversed with the chatbot.</p><p>For the second, we performed A/B tests and compared answers to parenting questions written by humans and LLMs — and found some surprising early results.</p><h3>Early-years parenting chatbot</h3><p>We have prototyped a parenting chatbot that provides guidance to caregivers of young children. You can see it in action in the video below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPrcNa0UjgB4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPrcNa0UjgB4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPrcNa0UjgB4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d44379ecbeb8e910ff665efd48de0fa9/href">https://medium.com/media/d44379ecbeb8e910ff665efd48de0fa9/href</a></iframe><p>The chatbot uses OpenAI’s GPT-4 and utilises <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG">retrieval-augmented generation (RAG)</a> to ground the LLM in a knowledge base consisting of advice on pregnancy, babies and parenting — in this case the <a href="https://www.nhs.uk/start-for-life/">NHS Start for Life</a> website, which we have used as an openly available source of information for prototyping purposes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J_tyOi0O4fm9EtLT" /><figcaption><em>Chatbot diagram</em></figcaption></figure><h3>Experiment 1: Collecting feedback with Streamlit</h3><p>In order to make the chatbot more helpful and accurate, we wanted to implement the option for users to provide their feedback to individual responses.</p><p>The parenting chatbot prototype is built with Streamlit, which does not provide a native widget for collecting user feedback. Therefore, we opted for <a href="https://trubrics.github.io/trubrics-sdk/#collect-user-prompts-feedback-from-a-streamlit-app">streamlit-feedback</a>, a third-party component developed by <a href="https://trubrics.github.io/trubrics-sdk/">trubrics</a>, which provides a simple and pretty interface for this purpose.</p><p>The streamlit-feedback component enables two types of feedback, <strong>binary</strong> (thumbs up and down) and a <strong>5-point Likert scale</strong> (from sad to happy face). It also provides a free text field for users to add their comments.</p><p>In our prototype, we decided to use the 5-point Likert scale and allow users to provide feedback for each assistant response. Here is an example of how this looks in our prototype.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*_kwbH_34Yu2do_t7" /><figcaption><em>Example of the user feedback widget in Streamlit</em></figcaption></figure><p>The widget is implemented by simply adding the following lines after the LLM response <a href="https://github.com/nestauk/discovery_generative_ai/blob/f2f2a32cd9459685fdd9fc0c79eb36775a313186/src/genai/streamlit_pages/parenting_page.py#L137">in the code</a>:</p><pre>from streamlit_feedback import streamlit_feedback<br><br>(..)<br><br>  # Submit feedback<br>    streamlit_feedback(<br>    feedback_type=&quot;faces&quot;,<br>    single_submit=False,<br>    optional_text_label=&quot;[Optional] Please provide an explanation&quot;,<br>    key=&quot;feedback&quot;,<br>  )</pre><h4>Logging user feedback to S3</h4><p>We decided to save the user feedback and message history to S3 so that we can analyse it at a later stage. We utilised Streamlit’s <a href="https://docs.streamlit.io/library/api-reference/session-state">session management</a> functionality to keep track of multiple user and assistant messages. When a user starts a conversation with the parenting chatbot, the app defines a unique session identifier, which will be used to create a folder with the same name in an S3 bucket where we collect the messages and user feedback.</p><pre>import streamlit as st<br>import uuid<br><br>(..)<br><br>st.session_state[&quot;session_uuid&quot;] = f&quot;{current_time()}-{str(uuid.uuid4())}&quot;</pre><p>Each time a user submits feedback, we append a dictionary containing the last user and assistant messages, the feedback rating and the free text field into a JSON Lines file in S3 using <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">boto3</a>:</p><pre>  user_feedback = {<br>      &quot;user_message&quot;: st.session_state[&quot;messages&quot;][-2],<br>      &quot;assistant_message&quot;: st.session_state[&quot;messages&quot;][-1],<br>      &quot;feedback_score&quot;: st.session_state[&quot;feedback&quot;][&quot;score&quot;],<br>      &quot;feedback_text&quot;: st.session_state[&quot;feedback&quot;][&quot;text&quot;],<br>  }<br>  <br>  write_to_s3(<br>      key=aws_key,<br>      secret=aws_secret,<br>      s3_path=f&quot;{s3_path}/session-logs/{st.session_state[&#39;session_uuid&#39;]}&quot;,<br>      filename=&quot;feedback&quot;,<br>      data=user_feedback,<br>      how=&quot;a&quot;,<br>  )</pre><p>If this prototype is eventually deployed to a group of test users, this feedback could be assessed to identify problem areas for the chatbot. For example, there might be topics that are not sufficiently covered in our knowledge base, and users would likely point out such topics with their negative feedback.</p><p>We are not, however, deploying the prototype yet and hence we also tested the chatbot answers in a more controlled way — using A/B tests with a small group of people.</p><h3>Experiment 2: A/B testing with Prodigy</h3><p>We experimented with an evaluation approach that could help us systematically compare the chatbot responses with answers written by human experts. We used a tool called <a href="https://prodi.gy/">Prodigy</a> to make pairwise comparisons between answers from different sources (ie, LLMs or humans).</p><p>Prodigy is a scriptable annotation tool for small teams doing rapid, iterative machine learning model development. There are also other tools that can be used for this purpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1FxzHjWhMxtmGCl5" /><figcaption><em>Prodigy annotation tool presenting a parenting question and two answers for annotators to choose from.</em></figcaption></figure><h4>Parenting questions</h4><p>We used the “top ten most asked parenting questions” from an article on <a href="https://www.thekidcollective.co.uk/blog/most-asked-parenting-questions.html">The Kid Collective</a> website. According to the article, these questions have been sourced from Google Trends and pertain specifically to the UK. The questions are:</p><ul><li>How to change a nappy?</li><li>How to store breast milk?</li><li>When to stop breastfeeding?</li><li>How to get baby to sleep?</li><li>How much to feed baby?</li><li>How do babies learn?</li><li>How to breastfeed?</li><li>How to baby-proof?</li><li>How to potty train?</li><li>What is colic?</li></ul><p>The article also provides short, 1–2 paragraph answers to each of the questions. We used these answers as the reference “human” answers in our evaluation.</p><p>While this source is not necessarily an authoritative one (The Kid Collective appears to be primarily a baby retail website), we deemed it as a good starting point for our experiment, as the questions are presumably among the most popular ones that caregivers have, and the answers are written in a conversational style that is easy to understand.</p><p>Other openly available options that we considered were answers to parenting questions by the US-based <a href="https://www.pbs.org/wholechild/parents/faqs.html">Public Broadcasting Service</a> (also not necessarily an authoritative source) and <a href="https://parents.actionforchildren.org.uk/mental-health-wellbeing/?age=toddler">Action for Children</a> (more specialised in mental health). In the future, we could expand this data with more questions and the answers could be provided by caregiving experts.</p><h4>Evaluating the answers</h4><p>We asked the ten questions to our parenting chatbot prototype and presented both answers to annotators (a few people from our team) to select the best answer. The annotators were not told which answer was from the chatbot and which one was from a human.</p><p>In addition, we also included a third source of answers, which is an answer from the GPT-4 model without any external knowledge base. This was to see whether the use of RAG and grounding the answers in a trusted knowledge base improves the perceived quality of the answers.</p><p>We could then analyse the pairwise comparisons and determine which source of answers had been the most preferred one. All the answers used in this experiment can be found in our <a href="https://github.com/nestauk/discovery_generative_ai/tree/dev/src/genai/parenting_chatbot/prodigy_eval">GitHub repository</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W1hmQkS8RAuvvEUi" /><figcaption><em>Summary of the A/B testing experiment</em></figcaption></figure><h4>Setting up Prodigy</h4><p>In the following, we describe how to set up the Prodigy tool to perform the evaluations. <strong><em>If you would like to see the results of the experiment instead, skip ahead to the end of the article.</em></strong></p><p>First, to install prodigy, you will need to run the following command in the terminal</p><pre>python -m pip install --upgrade prodigy -f https://XXXX-XXXX-XXXX-XXXX@download.prodi.gy</pre><p>Replace XXXX with your licence key (see the official prodigy docs <a href="https://prodi.gy/docs/install">here</a> for further information).</p><p>To define the annotation task, we have to write a Prodigy “recipe” file (see <a href="https://github.com/nestauk/discovery_generative_ai/blob/f2f2a32cd9459685fdd9fc0c79eb36775a313186/src/genai/parenting_chatbot/prodigy_eval/best_answer_recipe.py">here the full recipe</a> for our evaluation task). We begin by importing various modules.</p><pre>import random<br>from typing import Dict, Generator, List<br>import prodigy<br>from prodigy.components.loaders import JSONL</pre><p>Next, we define GLOBAL_CSS which is a string of CSS styles. This lets us tailor the appearance of the Prodigy interface. Through this, we can modify font sizes, layout of answer option boxes, container width, and more.</p><pre>GLOBAL_CSS = (<br>  &quot;.prodigy-content{font-size: 15px}&quot;<br>  &quot; .prodigy-option{width: 49%}&quot;<br>  &quot; .prodigy-option{align-items:flex-start}&quot;<br>  &quot; .prodigy-option{margin-right: 3px}&quot;<br>  &quot; .prodigy-container{max-width: 1200px}&quot;<br>)</pre><p>We then define the best_answer function, which specifies our Prodigy recipe. With the @prodigy.recipe decorator, we define the expected arguments. The role of this function is to process the input text data and return it in a format suitable for Prodigy to render.</p><pre>@prodigy.recipe(<br>    &quot;best_answer&quot;,<br>    dataset=(&quot;The dataset to save to&quot;, &quot;positional&quot;, None, str),<br>    file_path=(&quot;Path to the questions and answers file&quot;, &quot;positional&quot;, None, str),<br>)<br>def best_answer(dataset: str, file_path: str) -&gt; Dict:<br>    &quot;&quot;&quot;<br>    Choose the best answer out of the given options.<br><br>    Arguments:<br>        dataset: The dataset to save to.<br>        file_path: Path to the questions and answers file.<br><br>    Returns:<br>        A dictionary containing the recipe configuration.<br><br>    &quot;&quot;&quot;</pre><p>We use Prodigy’s JSONL loader to fetch and store our answers dataset into the stream variable, making it ready for subsequent processing. Once data is loaded, the next step is to make sure it’s presented in a random sequence and in a format Prodigy understands. By shuffling our data, we prevent potential biases in the order of presentation.</p><pre>  # Load the data<br>  stream = list(JSONL(file_path))<br>  <br>  def get_shuffled_stream(stream: List) -&gt; Generator:<br>      random.shuffle(stream)<br>      for eg in stream:<br>          yield eg<br>  <br>  # Process the stream to format for Prodigy<br>  def format_stream(stream: List) -&gt; Dict:<br>      for item in stream:<br>          question = item[&quot;question&quot;]<br>          options = [{&quot;id&quot;: key, &quot;html&quot;: value} for key, value in item[&quot;answers&quot;].items()]<br>          yield {&quot;html&quot;: question, &quot;options&quot;: options}<br>  <br>  stream = format_stream(get_shuffled_stream(stream))</pre><p>Finally, we lay down the rules for Prodigy by setting up the recipe configuration. This dictionary defines how our task should appear and function within the Prodigy interface. From specifying the ‘choice’ interface, naming the dataset, feeding the processed data stream, to determining interaction buttons and setting up other interface-related parameters.</p><pre>  return {<br>      # Use the choice interface<br>      &quot;view_id&quot;: &quot;choice&quot;,<br>      # Name of the dataset<br>      &quot;dataset&quot;: dataset,<br>      # The data stream<br>      &quot;stream&quot;: stream,<br>      &quot;config&quot;: {<br>          # Only allow one choice<br>          &quot;choice_style&quot;: &quot;single&quot;,<br>          &quot;task_description&quot;: &quot;Choose the best answer&quot;,<br>          &quot;choice_auto_accept&quot;: False,<br>          # Define which buttons to show<br>          &quot;buttons&quot;: [&quot;accept&quot;, &quot;ignore&quot;],<br>          # Add custom css<br>          &quot;global_css&quot;: GLOBAL_CSS,<br>          # If feed_overlap is True, the same example can be sent out to multiple users at the same time<br>          &quot;feed_overlap&quot;: True,<br>          # Port to run the server on<br>          &quot;port&quot;: 8080,<br>          # Important to set host to 0.0.0.0 when running on ec2<br>          &quot;host&quot;: &quot;0.0.0.0&quot;,<br>          # Setting instant_submit as True means that the user doesn&#39;t have to click the &quot;save&quot; button<br>          &quot;instant_submit&quot;: True,<br>      },<br>  }</pre><h4>Running Prodigy locally and on the cloud</h4><p>Once the Prodigy recipe has been defined, you can run the following command in the terminal to spin up an instance of the Prodigy app and test it on your local machine</p><pre>python -m prodigy best_answer answer_data src/genai/parenting_chatbot/prodigy_eval/data/answers.jsonl -F src/genai/parenting_chatbot/prodigy_eval/best_answer_recipe.py</pre><p>You should now be able to access the Prodigy app in the URL <a href="http://0.0.0.0:8080/">http://0.0.0.0:8080</a>. Note that you will need to specify the user session ID, by adding ?session=your_session_id to the URL. This is so that Prodigy can keep track of annotations from different annotators.</p><p>The annotation results can be fetched by running</p><pre>prodigy db-out answer_data &gt; output.jsonl.</pre><p>To make the platform available to multiple annotators, you will need to run Prodigy on an EC2 instance. We can then share the URL with the annotators, and they can access the platform from their own computers.</p><p>We first spin up an EC2 instance (in our case t2.micro, which is quite affordable at around <a href="https://aws.amazon.com/ec2/instance-types/t2/">$0.0116 per hour</a>) and use ssh to connect to it. Once connected to the instance, we cloned our <a href="https://github.com/nestauk/discovery_generative_ai/">generative AI repo</a> (to get the recipe code and answer data) and installed Prodigy on the instance using the instructions above.</p><p>To keep the Prodigy instance running in the background even when we’ve disconnected from the instance, we used screen by first simply running screen in the terminal</p><pre>screen</pre><p>And then ran the same command to spin up a Prodigy instance</p><pre>python3 -m prodigy best_answer answer_data src/genai/parenting_chatbot/prodigy_eval/data/answers.jsonl -F src/genai/parenting_chatbot/prodigy_eval/best_answer_recipe.py</pre><p>To detach from the screen, press ctrl+a and then d. The Prodigy app will continue running in the background. To reattach to the screen, run the commandscreen -r. You can also run screen -ls to see the list of screens that are running. To stop a screen and terminate the prodigy instance, run screen -X -S &lt;your_prodigy_screen_session_id&gt; “quit”.</p><p>Finally, you will need to open the port 8080 to allow other users to join from their computers. This can be done by going to the EC2 instance settings on AWS website, and adding a new inbound rule to the security group. The rule should be of type “Custom TCP”, port range 8080. If you wish to control who can connect to your Prodigy instance (recommended), you can also specify the allowed IP addresses.</p><h4>Early results are encouraging</h4><p>So, what where the results? We performed a proof-of-concept experiment with four annotators, who in total submitted 96 judgements. (Not every annotator completed all assessments.) This is definitely too small and non-representative of a sample to draw any strong conclusions. Nonetheless, during a rapid prototyping process, such exercises might still provide helpful signals on how a prototype is working and highlight potential problem areas.</p><p>The chart below shows the percentage of choices where annotators preferred one type of option (rows) over the other (columns). To our surprise, our chatbot prototype was preferred over human-written answers 91% of the time. This might have to do with the fact that the chatbot prototype generally had longer, more detailed and structured responses. After the test, some annotators noted that they preferred such type of answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qSVn4E3fIZ2r_reZ" /><figcaption><em>Percentage of times one type of option (rows) were preferred over the other type of option (columns).</em></figcaption></figure><p>Meanwhile, GPT-4 without RAG also tended to provide longer and more detailed answers — and it was preferred about 66% of the time over human-written responses.</p><p>Interestingly, however, our chatbot prototype was preferred over GPT-4 around 83% of the time, suggesting that the use of RAG together with a trusted knowledge base improves the responses. It might be due to more specific, contextual information — for example, the chatbot tends to provide the UK’s <a href="http://www.nationalbreastfeedinghelpline.org.uk/">National Breastfeeding Helpline</a> phone number when asked about breastfeeding. The chatbot also differentiated from GPT-4, by occasionally beginning with a phrase like “I did not get any relevant context for this but I will reply to the best of my knowledge…” thus signalling a level of confidence in the answer.</p><p>We can also check the results for each question separately, to spot any interesting patterns. The chart below shows the percentage of times each type of response was preferred across all pairwise comparisons.</p><p>For example, we can see that for the question “How to change a nappy?” GPT-4 in fact appears to have performed the best. This could warrant further inspection into the differences between the answers from the chatbot prototype and GPT-4, and provide ideas on how to continue improving the prototype (eg, by prompt engineering).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MVIAddYIH2Qv-j_l" /><figcaption><em>Percentage of times each type of response was preferred across all pairwise comparisons</em></figcaption></figure><h4>Limitations and further exploration</h4><p>These early results are encouraging in that they suggest that grounding a large language model in a trusted knowledge base might yield an improvement over simply using GPT-4, and that the answers from an LLM-based chatbot can appear sensible when compared with a set of human-written responses. Nonetheless, these are indications rather than conclusive evaluations. A more convincing assessment would require broader testing in diverse scenarios.</p><p>Firstly, our annotators were part of our wider development team. This may have influenced the results, introducing potential biases and not reflecting the diverse perspectives of the broader UK parent audience. We also did not implement strict evaluation criteria for the annotators to follow.</p><p>The human-written answers were sourced from a website that very likely hasn’t been as rigorously vetted as the NHS Start for Life. Another, perhaps a more meaningful test would be to compare how the users prefer using the chatbot versus browsing the website that provides information to the chatbot’s knowledge base. In such a test, the quality of the available information would be similar between both sources of information.</p><p>We also used only one answer per question — however, LLMs can produce slightly different versions of responses every time you use them. Therefore, one should sample multiple responses (potentially at different temperature settings) to make the results more robust. The LLM responses should also be tested separately for their accuracy by domain experts.</p><p>More generally, there are <a href="https://eugeneyan.com/writing/llm-patterns/#evals-to-measure-performance">many approaches</a> for evaluating various aspects of LLMs such as accuracy, fairness and potential for misuse. For example, besides A/B tests, LLMs can be evaluated by using domain-specific multiple-choice questions. One can also do ‘red-teaming’ to probe the guardrails of an LLM and check if it refuses to produce harmful outputs.</p><p>It is also possible to speed up the evaluation process by using LLMs themselves to do the evaluation. <a href="https://arxiv.org/pdf/2306.05685.pdf">Recent research</a> has shown that there can be a high (85%) agreement between judgements made by GPT-4 and human experts. This led the authors to suggest that using strong LLMs like GPT-4 for evaluation might be “a scalable and explainable way to approximate human preferences”.</p><p>Each of these approaches, however, comes <a href="https://www.anthropic.com/index/evaluating-ai-systems">with its own set of challenges</a> and developing LLM evaluation methods is <a href="https://www.cs.princeton.edu/~arvindn/talks/evaluating_llms_minefield/">an active research area</a>.</p><h3>Conclusion</h3><p>We have shown how we implemented two approaches to support the evaluation of our LLM-based chatbot prototype. These are only early experiments, but we hope this has been helpful and provided you with ideas for testing chatbot assistants. You can find all the code and data on our <a href="https://github.com/nestauk/discovery_generative_ai">GitHub</a>.</p><p>The type of quick evaluation we have discussed is useful during early development to quickly identify potential problem areas. Obviously, full and rigorous testing of LLM-based systems will become increasingly important as they become ready for release and come into popular use.</p><p>Organisations will need to weigh out the failure rate of these systems versus the benefits they bring to the users. The evaluation will be especially critical in contexts where vulnerable groups of people are involved, such as young children or people seeking mental health support.</p><p>If you are also working on evaluating LLMs and chatbots, we’d be curious to hear about your experience — <a href="/service/mailto:discoveryhub@nesta.org.uk">get in touch</a>!</p><p><em>Thank you to </em><a href="https://twitter.com/dan_kwiat"><em>Dan Kwiatkowski</em></a><em> and </em><a href="https://medium.com/@faizal.farook"><em>Faizal Farook</em></a><em> for reviewing the article, and to Discovery Hub’s Laurie, Natalie, Rosie and Kostas for participating in the evaluation experiments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c82db9f7f8d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/how-to-evaluate-large-language-model-chatbots-experimenting-with-streamlit-and-prodigy-c82db9f7f8d9">How to evaluate large language model chatbots: experimenting with Streamlit and Prodigy</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial: data-driven news discourse analysis with Python (Part 2)]]></title>
            <link>https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-2-55152b914c78?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/55152b914c78</guid>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[discourse-analysis]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Mon, 25 Sep 2023 10:25:24 GMT</pubDate>
            <atom:updated>2023-09-25T10:25:24.044Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*STQjUjcrIsGebT0UtQ4EbQ.png" /><figcaption>Image credit: Jana Dobreva for Fine Acts</figcaption></figure><p>In the <a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991">first part of this tutorial</a> we fetched news data from <a href="https://open-platform.theguardian.com/"><em>The Guardian</em> Open Platform</a>, analysed the historical trends of article publications, discussed the hype cycle model, and explored discourse topics using BERTopic.</p><p>Now, we will analyse the discourse in finer detail, by considering the language used around our search term of interest. We will review approaches to measure the importance of ‘collocations’, use <a href="https://spacy.io/">spaCy</a> to extract collocated linguistic features associated with our search terms, and finally consider the role of large language models in speeding up our analysis.</p><p>Similarly to the previous article, we will work through a <a href="https://colab.research.google.com/drive/1myxZkfrfTNdCK6EdoFyhmGBR6GZxyBBi">Google Colab notebook</a> and use examples of discourse around emerging low-carbon heating technologies such as <a href="https://www.youtube.com/watch?v=mDYtmVDX61Y&amp;ab_channel=Nesta-TheUK%27sInnovationAgency">heat pumps</a> and <a href="https://www.nesta.org.uk/project/understanding-the-public-view-on-hydrogen-boilers/">hydrogen</a>.</p><h3>What we mean when we talk about X: Collocation analysis</h3><p>Analysis of the vocabulary appearing together (ie, collocation) with your search terms can shed light on the narratives linked to your topic of interest.</p><p>For example, researchers from the University of Brighton and University of Leeds have used transcripts of UK parliament debates to <a href="https://journals.sagepub.com/doi/10.1177/0266242612441809">assess the political discourse around the term ‘entrepreneur’</a>. They combined collocation analysis with qualitative research to surface the main “discursive constructs” of the word ‘entrepreneur’ (shown in the figure below) and considered how these constructs have shifted over time. For example, whereas in the 1980s ‘entrepreneur’ was associated more prominently with the words ‘men’ and ‘capitalist’, 2000s saw increasing co-locations with ‘female’ and ‘social’ thus signifying certain cultural changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uFSMyrD-8s9zba44" /><figcaption><em>Analysis example from </em><a href="https://journals.sagepub.com/doi/10.1177/0266242612441809"><em>Perren and Dannreuther (2012)</em></a><em> showing patterns in the political discourse of the ‘entrepreneur’</em></figcaption></figure><p>As another example, research at Lancaster University has examined <a href="https://www.tandfonline.com/doi/full/10.1080/09644016.2016.1274504">how UK politicians talk about climate change</a> in the context of the debates that led to the groundbreaking 2008 Climate Change Act. The research also made use of keyword and collocation analysis, and highlighted that politicians frame climate change as an economic and technical issue, while giving less attention to the human and social dimensions.</p><p>In our case, we will consider the terms collocated with ‘heat pumps’ in our <em>Guardian</em> news article data and the Python analysis modules introduced in <a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991">Part 1</a> to perform this analysis largely automatically.</p><p>An important element to consider before proceeding with collocation analysis is the appropriate size of the <strong><em>context window</em></strong> within which we deem that words are collocated. For example, in both of the research examples discussed above the authors used a span of five words on either side of the search term to find the collocated terms. Here, in order to increase the number of potential collocated terms, we define the context window as the full sentence. Our choice was motivated by the fact that the number of articles can be somewhat limited when considering nascent technologies (particularly low-carbon heating, which was not a “hot topic” until very recently). Hence, by slightly increasing the context window, we increase our chances of catching potentially relevant and interesting collocated terms. There might also be cases where it’s appropriate to use the whole paragraph or even document as the context window.</p><h3>Measuring the importance of collocations</h3><p>There might be many words co-occurring with your search terms — so how can you find the most salient and meaningful collocations?</p><p>There are several approaches to measuring the importance of collocations. The first approach is to calculate the <a href="https://en.wikipedia.org/wiki/Pointwise_mutual_information"><strong><em>pointwise mutual information</em></strong></a> (PMI). This measure reflects the strength of association between terms and it is higher when terms are more likely to coincide mutually rather than independently.</p><p>More specifically, PMI is based on the ratio between the probability of two terms co-occurring in the same sentence (ie, p(x,y) in the formula below) versus the probability of the terms co-occurring independently, ie based on their individual occurrence probabilities (ie, p(x)p(y) in the formula). Higher PMI indicates stronger association between the terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*bP426GXIMxB01c3z" /><figcaption>Calculation of pointwise mutual information for two terms x and y.</figcaption></figure><p>For example, a PMI value of 0.5 would mean that the actual co-occurrence of the two terms is about 1.4 times more likely than what would be expected if the two events were independent (since 2^(0.5) ≈ 1.41).</p><p>The second measure is much simpler — the <strong><em>frequency</em></strong><em> of co-occurrences</em> of your search term and a given collocated term. Unlike PMI, where a rare term might have a high PMI because it was only ever mentioned with your search term, the frequency of co-occurrences can also directly suggest the importance or relevance of the collocated term. The frequency can also be normalised, eg by the total number of search term mentions per year.</p><p>The third measure is a <strong><em>normalised rank</em></strong>, which is similar to co-occurrences, but designed to reflect relative importance of terms. As the name suggests, to calculate this measure we rank collocations by frequency of co-occurrences with the search term in a descending order. Rank is slightly different from the previous two measures, as lower rank values will indicate higher collocation importance; for example rank value of 1 would indicate term that is co-occurring most frequently with your search terms, value of 2 would indicate the 2nd most frequent collocation and so on.</p><p>We also normalise the rank by the total number of unique collocations in a given time period — doing so makes it possible to compare the values in different time periods.</p><p>All three measures can be easily retrieved using the term_rank property of our DiscourseAnalysis class (see <a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991">Part 1</a> of this tutorial on how to initialise the class instance).</p><pre>pda.term_rank</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/0*uAUYKhJpxmypTrRg" /><figcaption><em>A table of terms collocated with our search terms “heat pump” or “heat pumps” in the same sentence, with different collocation importance measures: freq (frequency), rank (normalised rank) and pmi (pointwise mutual information).</em></figcaption></figure><p>In my experience, while high PMI values work well to highlight interesting collocations, the frequency or rank measures are more intuitive and easier to communicate to stakeholders.</p><p>As a quick check, we can also look at the most frequently collocated terms across all years.</p><pre># Check most often co-located terms<br>(<br>  pda.term_rank<br>  .groupby(&#39;term&#39;)<br>  .agg(freq=(&#39;freq&#39;, &#39;sum&#39;))<br>  .sort_values(&#39;freq&#39;, ascending=False)<br>  .head(20)<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4SJ9OaLmv0nt8NQO4j89TQ.png" /><figcaption><em>A list of terms most frequently collocated with our search terms “heat pump” or “heat pumps”, in the same sentence across all years</em></figcaption></figure><p>For a domain expert, these terms will make sense: for example, among the most commonly collocated terms, we find terms related to the ‘source’ because <a href="https://www.theguardian.com/us-news/2023/mar/11/heat-pump-revolution-what-you-need-to-know">heat pumps come into two main types</a>: air source and ground source, where as the names suggest the former absorbs heat from the air whereas the latter from the ground. We also find prominent terms like ‘gas’ and ‘boilers’, which relates to the gas boiler technology that will be replaced by heat pumps.</p><p>These terms also largely reflect the discourse topics we found in <a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991">Part 1</a> using BERTopic analysis. The advantage here, however, is that we can easily quantify the importance of the collocated terms as well the shifts in importance over time (as we’ll show in the next section).</p><p>Nonetheless, while these collocation importance measures can help surface the most salient discursive constructs, qualitative research and interpretation is still a key part of discourse analysis. For more context, the collocations can be further inspected by using the view_collocations function which brings up sentences where your search terms are collocated with the specified collocated term (eg, the term ‘source’ in the example below).</p><pre>pda.view_collocations(&quot;source&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vl400chxE42Gonv7" /><figcaption><em>A table of sentences where the term “source” is collocated with our search terms “heat pump” or “heat pumps”</em></figcaption></figure><h3><strong>Shifting narratives: collocation importance over time</strong></h3><p>It is also informative to analyse how the importance of collocated terms varies over time. This can point to shifts in language used to describe our search terms, which in our case might be caused by the emergence of new entities and applications associated with our technologies of interest.</p><p>For example, a comparison of the two different types of heat pumps discussed above highlights an interesting trend, where ‘ground source’ was initially more frequently collocated, whereas now ‘air source’ has overtaken the mentions of ‘ground source’. Air source heat pumps are more affordable and easier to install compared to ground source heat pumps, which might explain their recent rise in discourse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_eayXavl0WsCsyu6" /><figcaption><em>Collocation frequency of heat pumps and ‘air source’ or ‘ground source’ in the Guardian news (3-year moving average).</em></figcaption></figure><p>The shifts in collocation importance can also be detected in a more automated fashion, by analysing the distribution of importance measures. For example, you can estimate the standard deviation of the rank of collocations across different years. This can be quickly done by checking the pda.term_temporal_rank property of our DiscourseAnalysis class.</p><h3>Extracting patterns using spaCy</h3><p>You probably noticed that the illustration used at the beginning of this article considered not only the most important collocated terms but also more specific constructions, such as the <em>attributes</em>, <em>types</em> and <em>contexts</em> of the search term, as well as the entities that our terms are <em>affecting </em>or <em>are affected by.</em></p><p>We can partly automate such analysis by quickly extracting different linguistic patterns using spaCy. By using <a href="https://spacy.io/usage/rule-based-matching">rule-based part-of-speech (PoS) phrase matching</a>, we can extract various patterns.</p><p><strong><em>Adjective phrases</em></strong><em> </em>that describe the properties of a given technology (“<em>efficient</em> heat pumps”)</p><p><strong><em>Noun phrases</em></strong> capturing types and applications of technology as well as entities (“<em>UK Parliament’s group on </em>hydrogen”)</p><p><strong><em>Verb phrases</em></strong> where a given technology is acting as a:</p><ul><li>Subject (“heat pumps <em>heat homes</em>”)</li><li>Object (“<em>gas boilers</em> <em>will be replaced with </em>heat pumps”)</li></ul><p><strong><em>Specific patterns</em></strong>, such as:</p><ul><li><em>[search term]</em> is… (“hydrogen economy <em>is a central part of government’s plan</em>”)</li><li><em>[search term]</em> has… (“hydrogen <em>has great potential…</em>”)</li><li><em>[search term]</em> can<em>…</em> (“hydrogen <em>can replace coking coal for steel manufacture</em>”)</li></ul><p>To detect these patterns, we need to define them using spaCy’s notation. The function set_phrase_patterns allows us to automatically generate eight different patterns, which spaCy can then use to match text sequences in our news articles.</p><pre>pda.set_phrase_patterns(load_patterns=False, make_patterns=True)</pre><p>A pattern is defined as a list of dictionaries, where each dictionary specifies the attributes of a token that you want to match in a sequence of text. Let’s consider a simple term-noun pattern.</p><pre>&#39;heat_pump_term_noun&#39;: [<br>  {&#39;TEXT&#39;: &#39;heat&#39;},<br>  {&#39;TEXT&#39;: &#39;pump&#39;},<br>  {&#39;POS&#39;: &#39;NOUN&#39;},<br>  {&#39;POS&#39;: &#39;NOUN&#39;, &#39;OP&#39;: &#39;?&#39;}<br>]</pre><p>The heat_pump_term_noun pattern will detect phrases with nouns like “heat pump installations”, “heat pump association “, as well as “heat pump running costs”, helping to highlight significant constructs associated with our search term. It consists of four elements:</p><ul><li>{&#39;TEXT&#39;: &#39;heat&#39;}: This dictionary matches any token whose exact text is ‘heat’. The &#39;TEXT&#39; attribute refers to the exact text of the token.</li><li>{&#39;TEXT&#39;: &#39;pump&#39;}: Similarly, this dictionary matches any token whose exact text is ‘pump’.</li><li>{&#39;POS&#39;: &#39;noun&#39;} : This dictionary matches any token whose part-of-speech tag is &#39;NOUN&#39;. The POS attribute refers to spaCy’s part-of-speech tag.</li><li>{&#39;POS&#39;: &#39;NOUN&#39;, &#39;OP&#39;: &#39;?&#39;}: This dictionary matches a token that is a noun, but the token is optional. The &#39;OP&#39; attribute refers to the <a href="https://spacy.io/usage/rule-based-matching#quantifiers">quantifier</a> — it can take one of four values: ! (not), ? (optional; the PoS can appear 0 or 1 times), +(1 or more times), and *(0 or more times). Here, &#39;OP&#39;: &#39;?&#39; means that the token is optional, ie, it can appear 0 or 1 time(s).</li></ul><p>You might be wondering why we are using a second, optional noun in our pattern — this helps detect interesting patterns that consist of two nouns following the search term; for example, “heat pump installation cost” as opposed to simply “heat pump installation”.</p><p>Also note that the pattern above accounts for the singular form of “heat pump” only; plural forms are defined as another set of separate patterns.</p><p>Let’s now look at a slightly more complex example:</p><pre>&#39;heat_pump_adj_phrase&#39;: [<br>  {&#39;POS&#39;: &#39;ADV&#39;, &#39;OP&#39;: &#39;*&#39;},<br>  {&#39;POS&#39;: &#39;ADJ&#39;},<br>  {&#39;POS&#39;: &#39;ADJ&#39;, &#39;OP&#39;: &#39;*&#39;},<br>  {&#39;POS&#39;: &#39;NOUN&#39;, &#39;OP&#39;: &#39;?&#39;},<br>  {&#39;TEXT&#39;: &#39;heat&#39;},<br>  {&#39;TEXT&#39;: &#39;pump&#39;}<br>]</pre><p>This pattern provides flexibility to match different combinations of adverbs, adjectives, an optional noun, followed by the exact words ‘heat’ and ‘pump’. It will match sequences such as “electric heat pump”, “older heat pump” and “low carbon heat pump” highlighting the salient properties of your search term mentioned in the news:</p><ul><li>{&#39;POS&#39;: &#39;ADV&#39;, &#39;OP&#39;: &#39;*&#39;}: This dictionary matches an adverb, but it’s optional and can appear any number of times (including zero times). The POS attribute refers to the part-of-speech tag (in this case &#39;ADV&#39; for adverb), and &#39;OP&#39; is the quantifier with a value of &#39;*&#39; meaning 0 or more occurrences.</li><li>{&#39;POS&#39;: &#39;ADJ&#39;}: This dictionary matches any token that has a part-of-speech tag of &#39;ADJ&#39; for an adjective. There’s no &#39;OP&#39; quantifier, so the adjective is required and must appear exactly once.</li><li>{&#39;POS&#39;: &#39;ADJ&#39;, &#39;OP&#39;: &#39;*&#39;}: This dictionary matches any token with a part-of-speech tag of &#39;ADJ&#39; for an adjective, but it’s optional and can appear any number of times (including zero times).</li><li>{&#39;POS&#39;: &#39;NOUN&#39;, &#39;OP&#39;: &#39;?&#39;}: This dictionary matches any token that is a noun, but the token is optional. The &#39;OP&#39;: &#39;?&#39; means the token can appear 0 or 1 times.</li><li>{&#39;TEXT&#39;: &#39;heat&#39;}: This dictionary matches any token whose exact text is ‘heat’.</li><li>{&#39;TEXT&#39;: &#39;pump&#39;}: Similarly, this dictionary matches any token whose exact text is ‘pump’.</li></ul><p>You can examine all the patterns by simply using the pos_phrases property of our DiscourseAnalysis instance.</p><pre>pda.pos_phrases</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cwCisVfPUCl7yYeX" /><figcaption><em>Table with the detected language patterns featuring our search terms.</em></figcaption></figure><p>In summary, this functionality allows us to easily distinguish different constructions in the discourse and helps us focus on different attributes of the discourse at a time.</p><p>Note that all of these pattern templates can be created automatically: we are using eight different templates like term_noun, adj_phrase and others, and then we insert the search term within those templates. For more advanced cases, if you’d like to define your own custom patterns instead of using the automatically generated ones, you can prepare a dictionary file with the pattern definitions and then when calling the function set_phrase_patterns() define load_patterns=True and make_patterns=False.</p><p>We used this approach <a href="https://nesta.shorthandstories.com/section-2-full-findings-and-recommendations/index.html#group-section-Public-discourse-signals-oFsDLJox4n">in our previous research</a> to summarise the narrative around heat pumps across more than a decade of news articles. Interesting phrases were surfaced using the pattern analysis, and then manually curated to reveal some of the narratives involving heat pump technology. We observed that earlier news articles focused on what heat pumps are and how they work, whereas more recent discourse appeared increasingly centred on the practicalities of adoption, such as costs, government grants, installation issues (eg, there being insufficient number of installers) or problems with operation (eg, issues during cold weather).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*l_6-rsVfqZPvWwlV" /><figcaption><em>Overview of news discourse content analysis around heat pumps. Source: </em><a href="https://www.nesta.org.uk/feature/innovation-sweet-spots/"><em>Innovation Sweet Spots project</em></a></figcaption></figure><h3>What’s next: Supporting qualitative analysis with generative AI?</h3><p>We now finish our tour through data-driven discourse analysis approaches, but it would be remiss of me not to mention how the recent developments of <a href="https://en.wikipedia.org/wiki/Large_language_model">large language models</a> (LLMs) might impact this type of research.</p><p>As shown in this tutorial, relevant collocations and linguistic patterns can be automatically extracted from news data, thus speeding up your research of public discourse. However, it still involves considerable qualitative research work to understand the context in which the collocations and phrases are mentioned, and piece together the overarching narrative.</p><p>Large language models (LLMs) such as <a href="https://openai.com/research/gpt-4">GPT-4</a> and easily accessible tools such as <a href="https://chat.openai.com/">ChatGPT</a> and <a href="https://claude.ai/chats">Claude AI</a> have the capacity to quickly generate high quality text. This opens the door for speeding up the qualitative research element of discourse analysis.</p><p>One could easily attempt to generate an overview of the news discourse and its main themes, (similarly to the example shown in the previous section) by providing all the sentences mentioning heat pumps to the LLM. The main issue here is that LLM might <a href="https://machinelearningmastery.com/a-gentle-introduction-to-hallucinations-in-large-language-models/">hallucinate and generate falsehoods</a>. This, however, could be alleviated to some extent by requiring the LLM to provide quotes from the news articles for each statement it makes, which can then be checked.</p><p>From my early experiments, it also appears that more reliable results can be achieved by breaking the analysis up in several steps. For example, by looking at each time period of interest separately, and providing the LLM with data only for that specific time period.</p><p>This is an interesting area of further exploration, and <a href="https://www.nesta.org.uk/project/how-generative-ai-will-affect-nestas-missions-and-work/">we’ll keep researching</a> LLM-based approaches and tools that can help conduct qualitative research.</p><h3>Conclusion</h3><p>I hope this tutorial has been a useful introduction to data-driven discourse analysis. While the focus was on news data and I used specific examples concerning emerging low-carbon heating technologies, these methods (and Python modules) can be applied on any text data and topic of your interest.</p><p><em>Thank you to </em><a href="https://medium.com/u/2104849c9bf8"><em>Sofia Pinto</em></a>, <a href="https://medium.com/u/a920cf69db57"><em>Zayn Meghji</em></a><em> and Emily Mathieson for reviewing the article, and I’m very grateful to Jyldyz Djumalieva and </em><a href="https://medium.com/u/4b97090c096a"><em>Jack Rasala</em></a><em> for their major contributions to the public discourse analysis Python modules.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55152b914c78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-2-55152b914c78">Tutorial: data-driven news discourse analysis with Python (Part 2)</a> was originally published in <a href="https://medium.com/data-analytics-at-nesta">Data science at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tutorial: data-driven news discourse analysis with Python (Part 1)]]></title>
            <link>https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fdbf5c23991</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[discourse-analysis]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[hype-cycle]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Mon, 25 Sep 2023 10:24:45 GMT</pubDate>
            <atom:updated>2023-09-25T10:24:45.160Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcG2v7zaGpHEREFd582URg.jpeg" /><figcaption>Image credit: Jana Dobreva for Fine Acts</figcaption></figure><p>At Nesta, we wish to understand public narratives around issues related to our <a href="https://www.nesta.org.uk/introduction-our-strategy/">three missions</a> in education, health and sustainability. Previously, as part for our Innovation Sweet Spots projects, we have characterised news and political discourses around <a href="https://nesta.shorthandstories.com/section-2-full-findings-and-recommendations/index.html#group-section-Public-discourse-signals-oFsDLJox4n">low-carbon heating technologies</a> and <a href="https://nesta.shorthandstories.com/food-innovation-public-discourse/index.html">novel food technologies</a>. In these projects, we aimed to cut through the hype and compare the trends in news coverage and parliamentary debates with trends in research and investment linked to technologies that could have an impact on Nesta’s mission goals. We are curious about both the intensity of the discourse, meaning how many times one technology is mentioned compared to other comparable innovations, as well as <em>how </em>the technology is being talked about.</p><p>Others have used discourse analysis to characterise <a href="https://www.tandfonline.com/doi/full/10.1080/09644016.2016.1274504">the debates around climate change legislation</a>, the <a href="https://www.tandfonline.com/doi/pdf/10.1080/13669877.2020.1863849?needAccess=true">discourse of obesity in news media according to different political leanings</a>, and compare the differences in <a href="https://data-viz.nesta.org.uk/measuring-gender-imbalances-reporting-creative-industries/index.html">how media is writing about men versus women working in the creative industries</a>, among many other examples.</p><p>In this tutorial, I will show you how you can get started with news discourse analysis using Python, and begin piecing together the narrative around a topic of your interest. I’ll cover how to collect news articles using <em>The</em> <em>Guardian</em> API, look at the number of news articles across years, and perform topic analysis with sentence embeddings.</p><p>As our main example throughout this article, we will analyse the discourse around heat pumps, which are a very efficient <a href="https://www.youtube.com/watch?v=mDYtmVDX61Y&amp;ab_channel=Nesta-TheUK%27sInnovationAgency">low-carbon heating technology</a> seen as the future of domestic heating. Nesta’s <a href="https://www.nesta.org.uk/sustainable-future/">sustainable future mission</a> is focussed on decarbonising UK homes, including facilitating the adoption of this technology.</p><h3>Setting up and importing requirements</h3><p>We will use <a href="https://github.com/nestauk/innovation_sweet_spots/tree/discourse_tutorial_blog">Python modules</a> that we’ve developed while working on Nesta’s <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/innovation-sweet-spots-food-innovation/">Innovation Sweet Spots</a> projects, which leverage a variety of natural language processing methods and packages, such as <a href="https://spacy.io/">spaCy</a> and <a href="https://textacy.readthedocs.io/en/latest/">TextaCy</a> for text processing, and <a href="https://maartengr.github.io/BERTopic/index.html">BERTopic</a> for clustering text embeddings into topics.</p><p>While the code is always a work in progress, we have used these modules in a couple of projects already and we hope it might be useful for your projects as well. Feel free to fork and adapt the code according to your requirements.</p><p><strong><em>In parallel to reading this tutorial, you can also replicate this analysis in </em></strong><a href="https://colab.research.google.com/drive/1myxZkfrfTNdCK6EdoFyhmGBR6GZxyBBi"><strong><em>this Google Colab notebook</em></strong></a><strong><em>.</em></strong></p><p>Once you’re all set up, import the public discourse analysis utils package.</p><pre>from innovation_sweet_spots.utils.pd import pd_analysis_utils as au</pre><h3>Getting the data: Using The Guardian Open Platform</h3><p>First, we need to get some discourse data to analyse! We will use news articles published by <em>The Guardian</em> newspaper and made available on their <a href="https://open-platform.theguardian.com/">Open Platform</a>. While this news source can be seen as somewhat politically left-leaning, it has the advantage of covering a wide-range of <a href="https://www.theguardian.com/uk/technology">technologies and innovations</a> and hence in the past we’ve found it useful as a proxy for the wider news discourse. <em>The Guardian</em> is also, to the best of our knowledge, the only major UK newspaper to make its text freely available for research. Nonetheless, you can apply the same methods to analyse other text data, for example, <a href="https://hansard.parliament.uk/">transcripts of policy debates</a> in the parliament.</p><p>To access <em>Guardian</em> news articles, you should set up a <em>Guardian</em> API key. Setting your key to &quot;test&quot; might work, but you should <a href="https://open-platform.theguardian.com/access/">apply for your own developer key</a>.</p><pre># Replace “test” with your key <br>API_KEY = &quot;test&quot; </pre><p>We will fetch news articles from <em>The Guardian</em> about heat pumps, the low-carbon heating technology. However, before we download all the articles mentioning heat pumps, let’s check how many articles there are in total. We’ll use the search_content function from the guardian helper module, which is a simple wrapper we wrote for <em>The Guardian</em> API, and set only_first_page=True.</p><pre>test_articles = au.guardian.search_content(<br>  search_term=&quot;heat pumps&quot;,<br>  api_key=API_KEY,<br>  only_first_page=True,<br>  use_cached=False,<br>  save_to_cache=False<br>)</pre><pre>Collected 100 (14%) results</pre><p>This will yield a list test_articles of the most recent 100 articles and, at the time of writing this tutorial, report that this amounts to about 14% of the total number of the articles. This means that there are more than 700 articles mentioning “heat pumps”.</p><p>The parameter save_to_cache controls whether the data is saved locally, and use_cached controls whether you’re accessing the locally saved data (by default, the articles will be stored in inputs/data/guardian/api_results). By caching the query results locally, you can reduce the number of API calls. Note, however, that your cached version might become out of date and hence, when updating the analysis, you should set use_cached to False.</p><p>Each article is a dictionary with multiple fields (see code snippet below), such as a unique id, headline and body as well as the information about the author. The fields returned by the API can be specified in the API call using parameters show-fields and show-tags. For this purpose, you can define your default fields using a config file (see an example of our <a href="https://github.com/nestauk/innovation_sweet_spots/blob/dev/innovation_sweet_spots/config/base.yaml">config file here)</a>.</p><pre>{<br>    &quot;id&quot;: &quot;environment/2023/jul/17/uk-installation-heat-pumps-report&quot;,<br>    &quot;type&quot;: &quot;article&quot;,<br>    &quot;sectionId&quot;: &quot;environment&quot;,<br>    &quot;sectionName&quot;: &quot;Environment&quot;,<br>    &quot;webPublicationDate&quot;: &quot;2023-07-17T04:00:18Z&quot;,<br>    &quot;webTitle&quot;: &quot;UK installations of heat pumps 10 times lower than in France, report finds&quot;,<br>    &quot;webUrl&quot;: &quot;https://www.theguardian.com/environment/2023/jul/17/uk-installation-heat-pumps-report&quot;,<br>    &quot;apiUrl&quot;: &quot;https://content.guardianapis.com/environment/2023/jul/17/uk-installation-heat-pumps-report&quot;,<br>    &quot;fields&quot;: {<br>        &quot;headline&quot;: &quot;UK installations of heat pumps 10 times lower than in France, report finds&quot;,<br>        &quot;trailText&quot;: &quot;Analysts call on government to make pumps mandatory for all new homes and scale up grants for installation in existing properties&quot;,<br>        &quot;body&quot;: &quot;&lt;p&gt;The UK is lagging far behind France and other EU countries in installing heat pumps, research has shown, with less than a tenth of the number of installations despite having similar markets.&lt;/p&gt; &lt;p&gt;Only 55,000 heat pumps...<br>    },<br>    &quot;tags&quot;: [<br>        {<br>            &quot;id&quot;: &quot;profile/fiona-harvey&quot;,<br>            &quot;type&quot;: &quot;contributor&quot;,<br>            &quot;webTitle&quot;: &quot;Fiona Harvey&quot;,<br>            &quot;webUrl&quot;: &quot;https://www.theguardian.com/profile/fiona-harvey&quot;,<br>            &quot;apiUrl&quot;: &quot;https://content.guardianapis.com/profile/fiona-harvey&quot;,<br>            &quot;references&quot;: [],<br>            &quot;bio&quot;: &quot;&lt;p&gt;Fiona Harvey is an environment editor at the Guardian&lt;/p&gt;&quot;,<br>            &quot;bylineImageUrl&quot;: &quot;https://uploads.guim.co.uk/2022/12/08/Fiona_Harvey_old_image.jpg&quot;,<br>            &quot;bylineLargeImageUrl&quot;: &quot;https://uploads.guim.co.uk/2022/12/08/Fiona_Harvey_old_image.png&quot;,<br>            &quot;firstName&quot;: &quot;Fiona&quot;,<br>            &quot;lastName&quot;: &quot;Harvey&quot;<br>        }<br>    ],<br>    &quot;isHosted&quot;: false,<br>    &quot;pillarId&quot;: &quot;pillar/news&quot;,<br>    &quot;pillarName&quot;: &quot;News&quot;<br>}</pre><p>Now let’s get <em>all </em>the articles mentioning heat pumps. For this purpose, we’ll use the get_guardian_articles function which wraps the search_content function used above and filters the articles based on their <em>Guardian</em> news categories (if required), separates article text data and metadata, and saves the filtered articles.</p><p>In this example, we also specify the following article categories to reduce the possibility of irrelevant articles. All category names can be found by checking <em>The Guardian</em> API <a href="https://open-platform.theguardian.com/documentation/section">sections endpoint</a> (see <a href="https://content.guardianapis.com/sections?api-key=test">here an API call listing the sections</a>).</p><pre># Define allowed article categories<br>CATEGORIES = [<br>  &quot;Environment&quot;,<br>  &quot;Technology&quot;,<br>  &quot;Science&quot;,<br>  &quot;Business&quot;,<br>  &quot;Money&quot;,<br>  &quot;Cities&quot;,<br>  &quot;Politics&quot;,<br>  &quot;Opinion&quot;,<br>  &quot;UK news&quot;,<br>  &quot;Life and style&quot;,<br>]</pre><p>You can also specify multiple search terms to be included in your query. For example, in my experience, it’s best to use both singular and plural forms with <em>Guardian</em> API and hence we will specify both “heat pump” and “heat pumps” as our search terms here. In this implementation, each search term is queried separately and repeated hits (ie, same article featuring multiple search terms) are deduplicated.</p><pre># List of search terms<br>SEARCH_TERMS = [&quot;heat pump&quot;, &quot;heat pumps&quot;]<br><br>articles_df, articles_metadata = au.get_guardian_articles(<br>  # Specify the search terms<br>  search_terms=guardian_search_terms,<br>  # To fetch the most recent articles, set use_cached to False<br>  use_cached = False,<br>  # Specify the API key<br>  api_key=API_KEY,<br>  # Specify which news article categories we&#39;ll consider<br>  allowed_categories = CATEGORIES,<br>)</pre><p>This will output a articles_df dataframe with article ids, texts and publishing date, as well as a separate dictionary articles_metadata storing the data associated with each unique article ID, such as the title, URL and author.</p><pre>articles_df.head(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0O_781mizLKvsmLN" /><figcaption>An example of the first three rows of the news articles table, featuring columns for the unique article id, article text and publishing date and year.</figcaption></figure><h3>Measuring the hype: Number of news articles across years</h3><p>Now that we have our news data, we can start analysing the discourse! First, we can specify the path to the analysis outputs directory, which will come handy when revisiting the analysis in the future. Note that we are storing the analysis outputs separately from the cached search results (discussed above), in order to separate the analysis process, which is agnostic to the data sources, from the data fetching process.</p><pre># Specify the location for analysis outputs<br>from innovation_sweet_spots import PROJECT_DIR<br>OUTPUTS_DIR = PROJECT_DIR / &quot;outputs/data/discourse_analysis_outputs&quot;</pre><p>We can then specify the name ANALYSIS_ID for this specific analysis session — all the output tables will be stored in a subfolder of OUTPUTS_DIR with the same name.</p><pre>ANALYSIS_ID = &quot;guardian_heat_pumps_tutorial&quot;</pre><p>We will then define a couple of additional filtering criteria to keep the most relevant results to our geographical context, by specifying a (non-exhaustive) list of UK-related geographic terms. As an example, we will also exclude any article that mentions Australia. The filtering at this stage is very simple and looks for exact matches of the provided strings in the original article text.</p><pre># Terms required to appear in the articles, <br># for the articles to be considered in the analysis<br>REQUIRED_TERMS = [<br>  &quot;UK&quot;,<br>  &quot;Britain&quot;,<br>  &quot;Scotland&quot;,<br>  &quot;Wales&quot;,<br>  &quot;England&quot;,<br>  &quot;Northern Ireland&quot;,<br>  &quot;Britons&quot;,<br>  &quot;London&quot;,<br>]<br><br># Articles with these terms will be removed from the analysis<br>BANNED_TERMS = [&quot;Australia&quot;]</pre><p>We then use the DiscourseAnalysis class, which allows us to run different analyses, from assessing search term mentions, to detecting words commonly collocated with our search terms, to topic analysis. Importantly, you can use this class for analysing not just <em>The Guardian</em> news articles as shown here, but in fact any time-stamped text data such as parliamentary debates, web forum discussions or online comments. The only requirement is that the input Pandas DataFrame needs to have the columns id, text, date, year.</p><pre># Set up the discourse analysis class<br>pda = au.DiscourseAnalysis(<br>  search_terms = SEARCH_TERMS,<br>  outputs_path = OUTPUTS_DIR,<br>  query_identifier = ANALYSIS_ID,<br>  required_terms = REQUIRED_TERMS,<br>  banned_terms = BANNED_TERMS,<br>)<br># Add the input data<br>pda.load_documents(document_text=articles_df)</pre><p>As our first analysis, we can simply consider the number of articles published per year that contain our search terms (the results for each search term are combined and deduplicated).</p><pre>pda.plot_mentions(use_documents=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z1od5zr2wUAhh5Z7" /><figcaption>Number of articles in <em>The Guardian featuring the terms </em>“heat pump” or “heat pumps” across different years</figcaption></figure><p>While a very simple result, it can already reveal interesting patterns of waxing and waning interest in a particular topic. In <a href="https://nesta.shorthandstories.com/section-2-full-findings-and-recommendations/index.html#group-section-Public-discourse-signals-oFsDLJox4n">our previous research</a>, we interpreted these trends using the <a href="https://www.sciencedirect.com/science/article/abs/pii/S0040162516300270">hype dynamics</a> models, which reflect the typical life cycle of peak, disappointment and recovery of expectations associated with emerging technologies. One popular example of this is <a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle">Gartner’s hype cycle</a>, in which the initial phase of ‘inflated expectations’ is followed by a ‘trough of disillusionment’ where interest wanes as the initial implementations fail to deliver value.</p><p>The peak around 2008, subsequent decrease up until late 2017 and then the following recovery appear to reflect this type of pattern, which <a href="https://twitter.com/KandersKarlis/status/1469227566632517634">prompted us to suggest back in 2021</a> that heat pumps might be proceeding to the so-called ‘plateau of productivity’ phase of Gartner’s hype cycle where the initial faults of the technology have been addressed and mainstream adoption can start to take off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdWTjbIHKHKMTKetsLzl5A.png" /><figcaption><em>Diagram of the hype cycle model. Adapted from </em><a href="https://en.wikipedia.org/wiki/Gartner_hype_cycle"><em>Wikipedia</em></a></figcaption></figure><p><a href="https://nesta.shorthandstories.com/section-2-full-findings-and-recommendations/index.html#group-section-Public-discourse-signals-oFsDLJox4n">In our previous research</a> on green technologies, we compared the trends around heat pumps with other, competing alternatives such as heating homes with hydrogen gas, which at that time showed a steep rise in news mentions. At that time, we noted the risk of consumers potentially becoming distracted from adopting more mature heat pump technology by what appeared to be an early-stage hype around hydrogen.</p><p>Such interpretations, however, need to be cross-referenced with more in-depth analysis of the discourse content (as shown, for example, in the next sections of this tutorial) and sense-checked with domain experts. Hence, this type of result should be seen more as a data-informed hypothesis about the innovation trends rather than a comprehensive analysis.</p><p>Finally, when considering the growth trends of news mentions, another important element is a baseline growth trend that we can use as a reference. For example, there might be a remote chance that the peaks and troughs in search term mentions are driven by some more general changes in the publisher’s yearly output. One simple option to account for this is to normalise the number of articles mentioning your search terms by the total number of articles published in any given year. This can be obtained by using the function get_total_article_counts and specifying the same article categories that we used previously.</p><pre>total_counts = au.get_total_article_counts(sections=CATEGORIES, api_key=API_KEY)</pre><p>The function is calling <em>Guardian</em> API with an empty search term, and then using the &quot;total&quot; field of the response to infer the reference number of articles (<a href="https://content.guardianapis.com/search?section=environment&amp;from-date=2023-01-01&amp;to-date=2023-12-31&amp;api-key=test">which you can see on this example</a> of an API call).</p><p>After dividing the number of articles mentioning heat pumps with the total number of reference articles, we find that the shape of the trend is preserved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GD-LUod14LaI4Pf1" /><figcaption><em>Number of articles mentioning search terms normalised by the total number of articles published in a given year</em></figcaption></figure><h3>Characterising discourse topics using BERTopic</h3><p>Further insight into the discourse can be of course gained by considering the actual text of the articles. For example, one could start by inspecting the sentences from a specific year containing our search terms. For this purpose, you can use a dictionary combined_term_sentences, which is organised by year and contains all the sentences with any of the provided search terms, and the unique id of the corresponding news article.</p><pre>pda.combined_term_sentences[&quot;2022&quot;].head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vyZxPeatXqoyc16j" /><figcaption>Example of five articles mentioning heat pumps in 2022</figcaption></figure><p>Such inspection is also essential to catch erroneous mentions that are irrelevant to your particular query — such as the one of the examples above mentioning ‘heat pump tumble dryers’, which is not a home heating technology.</p><p>Nonetheless, we can also gain some insights in a way that doesn’t immediately require reading decades worth of articles. We can consider topic modelling or clustering to automatically find themes within our news data. There are many approaches that we could use for this purpose, but here we show <a href="https://maartengr.github.io/BERTopic/index.html">BERTopic</a>, which is a convenient package with many ‘out of the box’ functionalities such as clustering, dimensionality reduction and visualisation.</p><p>Our DiscourseAnalysis class wraps around BERTopic, so creating the topic model is as simple as running the fit_topic_model() function. By default, it uses the sentences featuring our search terms (as shown above).</p><pre>topic_model, docs = pda.fit_topic_model()</pre><p>In the background, BERTopic is embedding the sentences using a sentence transformer model (typically the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">all-MiniLM-L6-v2</a> model), reducing their dimensionality with <a href="https://umap-learn.readthedocs.io/en/latest/">UMAP</a>, and then clustering the reduced embeddings with <a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html">HDBSCAN</a>. Then, to characterise the different clusters, it finds a representational set of keywords by using a <a href="https://maartengr.github.io/BERTopic/getting_started/ctfidf/ctfidf.html">weighted version of TF-IDF</a>.</p><p>In my experience, the most interesting result from BERTopic is the visualisation of the sentence embeddings and clusters, by running visualize_documents()</p><pre>topic_model.visualize_documents(docs)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-syyE3tyGuSjIt2Y" /><figcaption>BERTopic visualisation, where each dot is a sentence mentioning our search terms and colours indicate distinct clusters (themes); similar sentences are located closer to each other in this visualisation. Each cluster is described by three characteristic words. Gray dots between the clusters are considered noise and are not assigned to any cluster.</figcaption></figure><p>In this example, we find clusters related to main topics of conversation around heat pumps, such as the different types of heat pump technology (ground source and air source), alternative heating technologies (gas boilers, hydrogen heating, biomass boilers), solar panels (which are sometimes used together with heat pumps), as well as Government grants and targets (600,000 heat pump installations by 2028). The points in colour indicate cluster cores, whereas the grey points in between are considered too noisy by the HDBSCAN algorithm to assign them to a cluster.</p><p>Note that BERTopic employs non-deterministic algorithms, and hence you will get slightly different results between different runs (you can make the output deterministic by <a href="https://maartengr.github.io/BERTopic/faq.html">following the steps here</a>). Due to the nature of HDBSCAN, you might also occasionally end up with only one large cluster — in that case you can simply rerun the fit_topic_model() function again.</p><h3>Conclusion</h3><p>So far we have tried a couple ways of gaining an overview of the intensity and content of news discourse using Python. In the second part of this tutorial, we will analyse the discourse in finer detail, by considering the vocabulary used together with our search terms. We will review approaches to measure the importance of collocations, use spaCy to extract collocated linguistic features associated with our search terms, and finally consider the role of large language models in speeding up our analysis.</p><p><em>Thank you to </em><a href="https://medium.com/u/2104849c9bf8"><em>Sofia Pinto</em></a>, <a href="https://medium.com/u/a920cf69db57"><em>Zayn Meghji</em></a><em> and Emily Mathieson for reviewing the article, and I’m very grateful to Jyldyz Djumalieva and </em><a href="https://medium.com/u/4b97090c096a"><em>Jack Rasala</em></a><em> for their major contributions to the public discourse analysis Python modules.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fdbf5c23991" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analytics-at-nesta/tutorial-data-driven-news-discourse-analysis-with-python-part-1-4fdbf5c23991">Tutorial: data-driven news discourse analysis with Python (Part 1)</a> was originally published in <a href="https://medium.com/data-analytics-at-nesta">Data science at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining WhatsApp with large language models: prototyping with Twilio and Flask]]></title>
            <link>https://medium.com/discovery-at-nesta/combining-whatsapp-with-large-language-models-prototyping-with-twilio-and-flask-c29228d80feb?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c29228d80feb</guid>
            <category><![CDATA[twilio]]></category>
            <category><![CDATA[whatsapp]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[early-childhood-education]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Tue, 15 Aug 2023 15:26:16 GMT</pubDate>
            <atom:updated>2023-08-15T15:27:19.108Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Nesta’s Discovery Hub has launched a project to investigate how generative AI can be used for social good. We’re now exploring </em><a href="https://www.nesta.org.uk/project-updates/why-we-are-considering-the-impact-of-generative-ai-on-the-early-years-sector/"><em>the potential of large language models (LLMs) for early-years education</em></a><em>, and in a series of Medium blogs we discuss the technical aspects of our early prototypes.</em></p><p>In <a href="https://medium.com/discovery-at-nesta/using-streamlit-and-gpt-4-to-prototype-a-web-app-for-suggesting-personalised-early-years-activities-2850c40ac2e7">our previous article</a>, we created a web app using Streamlit that allows caregivers and early-years educators to generate personalised activity ideas for toddlers. Now, we explore the idea that a widely-adopted messaging platform like WhatsApp could be a more accessible user interface for caregivers or educators compared to a web app.</p><p>As such, we’ve now built a simple WhatsApp chatbot prototype, which uses a LLM. In this blog, we detail our first steps in creating this prototype and I hope this will also help you to get started with your own WhatsApp bot!</p><p>But before we go into details, let’s see it in action.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FH7JhBbpMwW0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DH7JhBbpMwW0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FH7JhBbpMwW0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a41eaa72ebdd84db54553cded330486/href">https://medium.com/media/8a41eaa72ebdd84db54553cded330486/href</a></iframe><p>If you would like to give us some feedback, please <a href="/service/mailto:discoveryhub@nesta.org.uk">get in touch</a>. What follows here is a technical overview of how we made this prototype.</p><h3>Using Twilio, Flask and OpenAI to build a WhatsApp chatbot</h3><p>To create this prototype, we used <a href="https://www.twilio.com/en-us/messaging/channels/whatsapp">Twilio</a>, a communications service platform, to manage a WhatsApp business account. We made use of the <a href="https://www.twilio.com/docs/whatsapp/sandbox">Twilio Sandbox for WhatsApp</a>, which allowed us to prototype immediately without waiting for the account to be approved by WhatsApp.</p><p>The Twilio account can receive messages from WhatsApp users and forward them to our custom-made app, using the app’s application programming interface (API), which we created with the <a href="https://flask.palletsprojects.com/en/2.3.x/">Flask Python package</a>.</p><p>Our Flask app parses the user message, selects the right prompt from a set of pre-made tailored prompts, depending on the content of the incoming message, and sends the prompt to OpenAI’s LLM. Once the LLM response is generated, the Flask app sends it back to WhatsApp via Twilio and the user receives a message on their phone.</p><figure><img alt="A flowchart with boxes and arrows, starting from “WhatsApp message” and continuing to “Send message to server” (via Twilio) to “Select the right prompt” (via Flask app) to “Selected prompt” to “LLM call” (via OpenAI) to “Send reply to WhatsApp” (via Flask and Twilio) to “Result”" src="https://cdn-images-1.medium.com/max/1024/1*VYREjw_CfT85Xa1oAfvXDg.png" /><figcaption><em>Overview of the WhatsApp chatbot prototype</em></figcaption></figure><p>At the moment, the bot can deal with two types of user messages. The user can start the interaction by saying anything generic such as “Hi”– this will bring up instructions explaining the two types of messages.</p><figure><img alt="Screenshot of WhatsApp app screen, with the user typing “Hi” and the chatbot answering with instructions saying “Write ‘Explain &lt;your question&gt; to explain a concept to a 3-year old or ‘Activities &lt;your topic&gt; to get activity ideas”" src="https://cdn-images-1.medium.com/max/928/1*QcnFrf9ztzkKwcnFeoDUcw.png" /></figure><p>The ‘Explain’ message triggers what we call the ‘Explain like I’m 3’ prompt, which you can read below. It defines the role of the LLM as a helpful and kind educator, who should use simple words, never offend or be aggressive. Note, however, that without additional <a href="https://eugeneyan.com/writing/llm-patterns/#guardrails-to-ensure-output-quality">guardrails</a> for content moderation, the adherence of the LLM to these instructions is not fully guaranteed.</p><p>Before sending the prompt to OpenAI API, the {input} field in the prompt is replaced by the user question.</p><pre>###Instructions###<br>You are a helpful, kind, intelligent and polite early-years educator. <br>Your task is to explain a concept to a 3 year old child. <br>You must explain it in simple words that a young kid would understand. <br>You must also be patient and never offend or be aggressive. <br>Gendered language and any adjectives about the kid are strictly prohibited.<br><br>###Question###<br>{input}<br><br>###Answer###</pre><figure><img alt="Screenshot of a Whatsapp app screen, showing an example of user asking “Explain why is the sky blue” and the chatbot answering “Oh, that’s a great question! The sky is blue because of something called sunlight. You see, the sun gives off light, and that light is made up of different colors. When the sunlight reaches our Earth, it goes through the air. The air is made up of tiny little things called molecules. These molecules are very good at scattering the sunlight…” (and so on)" src="https://cdn-images-1.medium.com/max/914/1*CM7dPBVMW_UDHOT30Mtw_g.png" /><figcaption><em>Example of an LLM-generated response to a question of “why is the sky blue” in a WhatsApp chat</em></figcaption></figure><p>The second, ‘Activities’ message triggers our personalised activity generation prompt, which we described in greater detail <a href="https://medium.com/discovery-at-nesta/using-streamlit-and-gpt-4-to-prototype-a-web-app-for-suggesting-personalised-early-years-activities-2850c40ac2e7">in our previous blog</a>. This prompt is quite long, as it includes information about the different areas of learning defined in England’s <a href="https://www.gov.uk/early-years-foundation-stage">early-years foundation stage</a> guidance. The approximately 20 seconds that it takes for the large language model to generate the response is, in fact, longer than Twilio’s <a href="https://www.twilio.com/docs/usage/webhooks/webhooks-connection-overrides">maximal timeout</a> (15 seconds) for waiting for a reply. Hence the bot first replies with a generic hold message and then sends the actual response as another message once it’s ready.</p><figure><img alt="Screenshot of a Whatsapp app screen, showing an example of the user asking “Activities about the solar system” and the chatbot answering with ideas for conversations such as “Planet talk” and “Planet Facts”" src="https://cdn-images-1.medium.com/max/920/1*GopykgOY6E-0Hvo4DEa0_g.png" /><figcaption><em>Example of LLM-generated activity ideas in a WhatsApp chat</em></figcaption></figure><p>The response for this particular prompt was too long for a single message, as WhatsApp’s limit is 1,600 characters. Therefore, we split the message up into chunks and send them in multiple messages.</p><p>An alternative approach could be to adjust the prompt to generate shorter responses, which would also bring down the response time.</p><figure><img alt="Screenshot of a Whatsapp app screen, showing an two consecutive messages sent by the chatbot." src="https://cdn-images-1.medium.com/max/918/1*dKrwbqsfwPIY-IbFwn9kZw.png" /><figcaption><em>Chunking the long response from LLM as separate messages</em></figcaption></figure><p>This is how the chatbot works, and now we’ll peer behind the scenes at the Flask API code and show a couple of options for deploying the bot and connecting it with Twilio.</p><h3>Behind the scenes: Flask API</h3><p>To power this chatbot prototype, we wrote a simple Flask app that has a /text API endpoint and a few helper functions. All the code is available <a href="https://github.com/nestauk/discovery_generative_ai/blob/dev/whatsapp_bot.py">on our GitHub repository</a>, so we highlight just a few main points below.</p><p>You will need to install and import the <a href="https://pypi.org/project/twilio/">Twilio Python package</a> and <a href="https://support.twilio.com/hc/en-us/articles/14726256820123-What-is-a-Twilio-Account-SID-and-where-can-I-find-it-">set up authorisation tokens</a> associated with your account.</p><pre>import os<br>from dotenv import load_dotenv<br>from twilio.rest import Client<br>from twilio.twiml.messaging_response import MessagingResponse<br><br># Twilio settings<br>load_dotenv()<br>client = Client(<br>    os.environ[&quot;TWILIO_ACCOUNT_SID&quot;], <br>    os.environ[&quot;TWILIO_AUTH_TOKEN&quot;]<br>)</pre><p>Once that is done, you can receive and send messages with your Flask app. To reply to incoming messages, we can use Twilio’s Messagingclass directly in the /textendpoint.</p><pre>@app.route(&quot;/text&quot;, methods=[&quot;POST&quot;])<br>def text_reply() -&gt; str:<br>    &quot;&quot;&quot;Respond to incoming messages&quot;&quot;&quot;<br>    reply = generate_reply(<br>        incoming_message=request.form.get(&quot;Body&quot;),<br>        sender_contact=request.form.get(&quot;From&quot;),<br>        receiver_contact=request.form.get(&quot;To&quot;),<br>    )<br>    resp = MessagingResponse()<br>    resp.message(reply)<br>    return str(resp)</pre><p>The endpoint calls a generate_reply() helper function that parses the incoming message, decides which prompt to use and calls OpenAI API. This is implemented as a simple series of if-then statements that check for the beginning of the incoming message’s string.</p><p>For a better user experience, one could implement a more flexible way of detecting the user’s intent — for example, by using an <a href="https://platform.openai.com/docs/guides/gpt/function-calling">OpenAI function</a> to pick the right prompt.</p><pre>def generate_reply(incoming_message: str, sender_contact: str, receiver_contact: str) -&gt; str:<br>    &quot;&quot;&quot;Parse message text and return an appropriate response. <br>  [...]<br>  &quot;&quot;&quot;<br>  text_message = incoming_message.lower()<br>  # &#39;explain&#39; response<br>  if text_message[0:7] == &quot;explain&quot;:<br>  response = ActivityGenerator.generate(<br>              model=LLM,<br>              temperature=TEMPERATURE,<br>              messages=[ELI3_MESSAGES.copy()],<br>              message_kwargs={&quot;input&quot;: text_message[7:].strip()},<br>          )<br>          return response[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br>  [...]</pre><p>The activities prompt, as explained above, takes a while and hence the app starts a new Python thread, which will execute the send_text() function, and in parallel return a hold message indicating that the response is coming shortly.</p><pre>  [...] <br>  # &#39;activities&#39; response<br>  elif &quot;activities&quot; in text_message[0:10]:<br>      EYFS_PARAMETERS[&quot;description&quot;] = text_message<br>      thread = Thread(<br>            target=send_text, args=[copy.deepcopy(EYFS_MESSAGES), EYFS_PARAMETERS, receiver_contact, sender_contact])<br>     thread.start()<br>     return &quot;Thank you for your question. I am thinking...&quot;<br>  [...] </pre><p>In its present implementation, the approach of using threads should be seen as a quick hack for prototyping purposes, as this might result in a dangling thread that isn’t closed after sending the reply. A better but more complex solution would be to use a task queue (eg, <a href="https://docs.celeryq.dev/en/stable/getting-started/introduction.html">Celery</a>) to handle the long-running task.</p><p>The send_text() function generates the response using OpenAI API, chunks it into shorter strings and sends them as a series of messages to the WhatsApp user using Twilio’s Client class.</p><pre>def send_text(messages: List[Dict], message_kwargs: Dict, my_contact: str, receiver_contact: str) -&gt; None:<br>    &quot;&quot;&quot;Generate text messages and send them to a given contact<br>    [...]<br>    &quot;&quot;&quot;<br>    # Generate response to the message<br>    response = ActivityGenerator.generate(<br>        model=LLM,<br>        temperature=TEMPERATURE,<br>        messages=messages,<br>        message_kwargs=message_kwargs,<br>    )<br>    text_body = response[&quot;choices&quot;][0][&quot;message&quot;][&quot;content&quot;]<br>    # Format the text_body for better display on WhatsApp<br>    text_body = format_activities_text(text_body)<br>    # Divide output into 1500 character chunks<br>    texts = [text_body[i : i + 1500] for i in range(0, len(text_body), 1500)]<br>    # Send message<br>    for text in texts:<br>        client.messages.create(body=text, from_=my_contact, to=receiver_contact)<br>        sleep(0.5)<br>    return</pre><p>For more information and advice about using the Twilio Python package to communicate with WhatsApp, consult <a href="https://www.twilio.com/docs/whatsapp/tutorial/send-and-receive-media-messages-whatsapp-python">their documentation</a>.</p><h3>Running the API locally with ngrok</h3><p>Now that you have your Flask API, we need to connect it to Twilio and test it. First, start your app (note that we’re using <a href="https://python-poetry.org/">Poetry</a> for managing our Python environment)</p><pre>poetry run python whatsapp_bot.py</pre><p>You can then expose the locally-running app to the internet using <a href="https://ngrok.com/">ngrok</a>. Ngrok is a service that provides you with a URL that you can use to connect to your app, and it has a free tier for developers to test their projects.</p><p>Assuming your Flask app is using port 5000, you can start ngrok as follows</p><pre>ngrok http 5000</pre><p>In Twilio you’ll need to set up a Sandbox for WhatsApp and add the URL created by ngrok under “Sandbox settings” in the box “When the message comes in”. Don’t forget to add the app’s endpoint /text to the URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hob_CfQCPtj85wVK3hFNQ.png" /><figcaption>Configuring Twilio’s Sanbox for WhatsApp</figcaption></figure><p>You and your test users can now go to WhatsApp and send a message to the number provided by the Twilio Sandbox. Note that the first message to the chatbot will need to be a special code provided by Twilio to connect with the sandbox — and then you can start chatting with your bot.</p><p>For more details about Twilio, you can look at their <a href="https://www.twilio.com/docs/whatsapp/sandbox">documentation</a> as well as consult <a href="https://www.youtube.com/watch?v=BKK5NMDC0fk&amp;ab_channel=IndianPythonista">online tutorials</a>.</p><h3>Deploying the API on Heroku</h3><p>Testing your app locally is great, but what if you close your machine? To keep your WhatsApp chatbot running even when you rest, you can deploy it on <a href="https://heroku.com/">Heroku</a> using Docker.</p><p>Heroku is a platform-as-a-service that enables developers to run applications in the cloud, and they provide affordable offers for testing and running apps that see intermittent use.</p><p>Getting started with Heroku and Docker can be somewhat tricky, so for illustration purposes, we walk through our main steps for launching the app. However, they might look differently for your implementation, depending on how you have organised your project and <a href="https://devcenter.heroku.com/categories/deployment">which deployment approach</a> you take.</p><p>First, make sure you’ve set up Heroku on your machine and log in the container registry.</p><pre>heroku container:login</pre><p>Then you can create a new app. This will create an app with a random name — the name can be changed later.</p><pre>heroku create</pre><p>Set up your environment variables, so that your app can interact with OpenAI API and Twilio even when deployed on the cloud.</p><pre>heroku config:set OPENAI_API_KEY=&lt;your_api_key&gt;<br>heroku config:set TWILIO_ACCOUNT_SID=&lt;your_account_sid&gt;<br>heroku config:set TWILIO_AUTH_TOKEN=&lt;your_token&gt;</pre><p>You will then need to navigate to the location where you store your app’s <a href="https://github.com/nestauk/discovery_generative_ai/blob/dev/src/genai/whatsapp_bot/Dockerfile">Dockerfile</a> and build the Docker container. In our case, it is as follows:</p><pre>cd src/genai/whatsapp_bot<br>heroku container:push web --app &lt;your_app_name&gt; --context-path ../../..</pre><p>Finally, we need to release the built Docker container.</p><pre>heroku container:release web --app &lt;your_app_name&gt;</pre><p>You might also need to make sure that you have started your app.</p><pre>heroku ps:scale web=1</pre><p>You can now check that the app is working by running heroku open. Once the app is running, the final task is to configure Twilio’s WhatsApp Sandbox as described above, using your new Heroku URL instead of the ngrok one.</p><p>Now, your chatbot will work even when you have turned off your machine. Depending on the <a href="https://www.heroku.com/pricing">Heroku pricing plan</a> you choose, Heroku can keep the app running all the time, or it can shut down the app to save resources during longer periods of inactivity and restart it when users start sending messages again.</p><h3>Conclusion</h3><p>Our WhatsApp chatbot prototype allows users to interact with a large language model on their phones. At the moment, the implementation is quite simple, with the chatbot largely serving as an interface to a couple of pre-made, easy-to-use prompts. Nonetheless, this has been useful to explore the technical feasibility of combining WhatsApp with generative AI. Moreover, we hope that by creating a tangible prototype, we can ignite further interest and explorations in this direction, especially in the context of early-years education.</p><p>This prototype has also highlighted differences in user experience between a text messaging app and a <a href="https://medium.com/discovery-at-nesta/using-streamlit-and-gpt-4-to-prototype-a-web-app-for-suggesting-personalised-early-years-activities-2850c40ac2e7">web app</a>. The messaging app will be better suited for quicker interactions and shorter responses from the LLM — particularly because we can’t stream the LLM responses character-by-character to WhatsApp. This means that the prompts that we used for our web app prototype would need to be further optimised to provide shorter answers, for example, by sending one early-years activity idea at a time.</p><p>Potential future iterations could also explore using WhatsApp <a href="https://www.twilio.com/docs/sms/tutorials/how-to-retrieve-and-modify-message-history-python">messaging history</a> to enable a more natural style of interaction and allow users to ask follow-up questions about the explanations or activity ideas.</p><p>We’re also aware that there are other platforms for managing WhatsApp accounts besides Twilio (such as <a href="https://www.wati.io/">Wati</a> and <a href="https://messagebird.com/">Messagebird</a>) as well as many other ways to build an API and chatbots. If you’ve built something similar, we’d love to <a href="/service/mailto:discoveryhub@nesta.org.uk">hear about it</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c29228d80feb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/combining-whatsapp-with-large-language-models-prototyping-with-twilio-and-flask-c29228d80feb">Combining WhatsApp with large language models: prototyping with Twilio and Flask</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we harness generative AI for social good?]]></title>
            <link>https://medium.com/discovery-at-nesta/can-we-harness-generative-ai-for-social-good-71715e4d773e?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/71715e4d773e</guid>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 09:49:15 GMT</pubDate>
            <atom:updated>2023-06-20T08:29:54.777Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*wiV4kPfHWY-ygqSq8-4dkA.jpeg" /><figcaption>Image credit: Luisa Brando for Creative Commons</figcaption></figure><p>As we step into the summer months, Nesta’s Discovery Hub is launching a new project to investigate the potential for generative AI to be harnessed for social good.</p><p>Generative AI, with its capacity to produce text, images, music, and video at the level of quality that was previously beyond the ability of machines, has taken the world by storm. At Nesta, we’ve been trialling AI tools like <a href="https://openai.com/blog/chatgpt">ChatGPT</a> to better understand their potential impact on our day-to-day operations.</p><p>Now, we’re diving deeper to find impactful use cases of generative AI in sectors directly linked to <a href="https://www.nesta.org.uk/introduction-our-strategy/">Nesta’s three missions</a> in education, health and sustainability. Can this new technology support nurseries or home learning environments in some way? Can generative AI help the <a href="https://medium.com/all-you-can-heat/heating-engineers-can-be-more-productive-and-thats-a-good-thing-3bf0a8469b91">low-carbon heating sector become more productive</a>? Does <a href="https://www.foodnavigator-usa.com/Article/2023/05/15/generative-ai-shows-potential-to-streamline-marketing-and-communications-for-cpg-brands-as-long-as-brands-oversee-its-output">its use for food marketing purposes</a> pose a risk to our food environments?</p><p>We’ll start by exploring the early years education sector, but are excited to hear from anyone working across any of our mission areas — either if you’re already using tools like ChatGPT and seeing their impact, or if you’ve just started wondering which aspects of your daily work could be supported by generative AI.</p><h3><strong>Great expectations</strong></h3><p>It can feel overwhelming to navigate the hype around AI, with <a href="https://www.linkedin.com/posts/zainkahn_1000-ai-tools-were-released-in-march-activity-7048285306101358592-4wAA/?trk=public_profile_share_view">1,000 new AI-based tools released in March alone and promises that AI will “transform your productivity forever”</a>.</p><p>There are, though, signs that AI <em>will </em>have a strong impact on the way we’ll work in the future. For example, studies indicate that generative AI tools based on large language models (LLMs) — algorithms that can generate high quality text — can <a href="https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/">reduce time spent on software development by more than half </a>(55%) and on <a href="https://economics.mit.edu/sites/default/files/inline-files/Noy_Zhang_1.pdf">various writing tasks by more than a third</a> (37%).</p><p><a href="https://arxiv.org/abs/2303.10130">Modelling from OpenAI and the University of Pennsylvania</a> suggests that LLM-powered software has the potential to speed up about 50% of tasks across the labour market. This study ranked different industries in the US by the potential impact LLMs might have upon them, shown in the figure below (with our annotations indicating some of the sectors related to Nesta’s missions or operations). For example, while the food manufacturing industry appears to have relatively small exposure to LLMs, industries such as educational services or utilities appear relatively high in this ranking.</p><p>As a recent case in point, the <a href="https://sifted.eu/articles/octopus-energy-ceo-chatgpt-news">energy supplier Octopus Energy announced</a> that 34% of their customer queries are answered by AI, doing the work of 250 people.</p><p>This suggests substantial potential for impact and disruption, so we plan to explore what this could mean for Nesta’s missions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XozpjQSWOlJHIJ7grHi9LQ.png" /><figcaption><em>Exposure to Large Language Models by industry from a study by </em><a href="https://arxiv.org/abs/2303.10130"><em>Eloundou et al. 2023</em></a><em> (</em><a href="https://discovery-hub-open-data.s3.eu-west-2.amazonaws.com/test_kitchen/Eloundou2023_figure.png"><em>click here for a full-size image</em></a><em>)</em></figcaption></figure><h3>Learning by doing</h3><p>Our project will progress in two stages: discovering and then prototyping.</p><p>First, we’ll carry out user research with people working in the mission domains, such as nursery workers, early years teachers, educational content creators and caregivers. We’ll also continue our research efforts by scanning academic literature and news, and talking with generative AI experts to deepen our understanding of the emerging applications and risks of this technology.</p><p>At the end of the discovering stage, we hope to have surfaced a large set of use case ideas and consequences directly relevant to Nesta’s missions (with a particular focus on the <a href="https://www.nesta.org.uk/fairer-start/">fairer start mission</a>). We also expect that as part of this, we’ll identify and test many uses for our own day-to-day research and admin work.</p><p>Then we’ll select and develop a set of prototypes of use cases directly related to Nesta’s missions. We define ‘prototype’ to mean a broad range of things: it could be a process, a way of working, code, or a web app. To aid us in this process, we also released an open call for a generative AI resident to join Nesta to help us explore this technology. In addition, we aim to host a “hackathon” focussed on using generative AI for social good towards the end of summer.</p><p>Fundamentally, having a prototyping workstrand reflects our view that we need to do hands-on work to fully appreciate the potential impact of generative AI. We hope that our prototypes will offer opportunities for impact right now and point to potentially bigger changes in the future.</p><h3>Embracing change, minimising risks</h3><p>During this project we will need to remain mindful that this area is evolving at an incredible speed. We are interested in the short, medium and longer term consequences of generative AI for our three missions, as we foresee both immediate use cases that we can apply right now and significant system changes.</p><p>It will be important to create space and time to exercise our imagination and consider future scenarios where generative AI has become even more capable and widespread. Ultimately, this work should equip us to be ready to react and adapt as this technology develops further.</p><p>Crucially, it will also be essential to anticipate the risk of these technologies, <a href="https://dl.acm.org/doi/abs/10.1145/3531146.3533088">of which there are many</a>. For example, we will need to consider barriers to access for people from different socioeconomic backgrounds — as if the access to generative AI is unequal, it could further widen, for example, <a href="https://www.cypnow.co.uk/features/article/from-crisis-in-the-early-years-to-a-fairer-start-could-we-close-the-school-readiness-gap-by-2030">the school readiness gap</a>.</p><h3>Join us on this exploration</h3><p>Throughout the project, we’ll regularly summarise and communicate our insights and lessons learned. We will keep iterating and sharing our hypotheses about the potential of generative AI for social good as we learn more, make our prototypes, and hear from you. This project forms part of a wider programme of work on Generative AI unfolding at Nesta which we hope to talk more about in the future.</p><p>Reach out to us here in the comments, social media (<a href="https://www.linkedin.com/in/karlis-kanders/">LinkedIn</a>, <a href="https://twitter.com/KandersKarlis">Twitter</a>) or our <a href="/service/mailto:discoveryhub@nesta.org.uk">email</a>, especially if you’re working in early years education or other sectors linked to Nesta’s missions.</p><p>If you have expertise in generative AI, consider this <a href="https://media.nesta.org.uk/documents/Open_Call_Discovery_AI.pdf">open call for generative AI residency</a> (deadline: June 18) at Nesta to support us on this journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71715e4d773e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/can-we-harness-generative-ai-for-social-good-71715e4d773e">Can we harness generative AI for social good?</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taming and training hunches: Data analytics and foresight]]></title>
            <link>https://medium.com/discovery-at-nesta/taming-and-training-hunches-data-analytics-and-foresight-27613e77323a?source=rss-c7cad6c6bfa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/27613e77323a</guid>
            <category><![CDATA[social-impact]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[futures-thinking]]></category>
            <category><![CDATA[strategic-foresight]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Tue, 03 Jan 2023 16:05:35 GMT</pubDate>
            <atom:updated>2023-01-03T16:05:35.090Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article is part of Discovery Hub’s Data Test Kitchen series. We explore how data science and machine learning can help enhance foresight on emerging innovations and their social impact.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*IN8954wnRvTkGudN5Ijt1g.png" /><figcaption>Image: Generated with dreamstudio.ai using the Stable Diffusion algorithm, inspired by Moebius.</figcaption></figure><p>Ursula K. Le Guin, in her groundbreaking novel The Left Hand of Darkness, writes about the fictional Handdarata culture and their mythical art of foretelling. Its practitioners have “tamed and trained the hunch… the power of seeing (if only for a flash) everything at once: seeing whole”.</p><p>A different approach rooted in mathematics and calculations was envisioned by another science fiction writer Isaac Asimov in his Foundation series. He wrote about an imagined scientific discipline of psychohistory that could predict the future development of societies with a high degree of accuracy.</p><p>The practice of foresight has long been a fascination for many, from ancient prophets and science fiction writers to prediction market forecasters and corporate futurists. At its core, strategic foresight helps people and organisations to systematically think about the future in the face of uncertainty.</p><p>The future, however, is unknowable as it does not yet exist and thinking about it requires imagination. Perhaps the future’s lack of factuality is one of the reasons why it has often been debated whether studying the future is an art or science. This has been discussed at length, for example, by <a href="https://www.goodreads.com/book/show/1783469.Foundations_of_Futures_Studies">the American futurist Wendell Bell</a> who concluded that futures studies is in fact an “action science” — it uses scientific thinking to improve decision making with respect to desirable and undesirable consequences. Nonetheless, the foresight and futures practice also includes methods such as <a href="https://en.wikipedia.org/wiki/Speculative_design">speculative design</a> and <a href="https://www.athensjournals.gr/technology/2016-3-4-4-Kymalainen.pdf">science fiction writing</a> that can facilitate thought-provoking discussions about the future.</p><p>In the remainder of this article, I discuss how creativity and rigour can co-exist within strategic foresight, and explain how Nesta’s <a href="https://www.nesta.org.uk/project/discovery-hub/">Discovery Hub</a> harnesses data science to deepen our capacity to evaluate possible futures.</p><h3>Quantitative foresight for social good</h3><p>As Nesta focuses its efforts on achieving <a href="https://www.nesta.org.uk/report/nesta-strategy-2030/our-missions/">missions in health, early years and sustainability</a>, our task at the Discovery Hub is to anticipate disruptive trends and help our organisation adapt. For us, the applied or “action” element of the foresight work is critical: We are endeavouring to support colleagues and partners to make better strategic choices in the here and now.</p><p>Data analytics can support this in various ways, <a href="https://www.nesta.org.uk/report/quantitative-analysis-of-technology-futures-part-1/">quantitative foresight being a vast family of various methods</a>. For example, we can measure trends around emerging technologies and innovations thus providing evidence of shifts in the external environment. By extrapolating from these trends we can devise future scenarios.</p><p>We can mine large datasets to surface signals of the future to come. We can visualise and simulate complex systems making the abstract more concrete.</p><p>These methods have been applied in many different contexts, from considering military strategies, to businesses and markets, to considering the direction of technological and social change. For example, recent <a href="https://www.cell.com/joule/fulltext/S2542-4351(22)00410-X">forecasting of future energy costs</a> (initially released just before the <a href="https://ukcop26.org/">COP26</a> conference in 2021) demonstrated how a rapid transition scenario to green technologies would save trillions of dollars globally compared to the status quo. Another example leveraged <a href="https://royalsocietypublishing.org/doi/10.1098/rsif.2020.0898">agent-based simulations</a> and a complex-systems approach to more realistically simulate the impact of automation on workers in the labour market.</p><p>Overall, quantitative foresight methods help us think about the possible outcomes, their likelihood of occurrence and the underlying factors driving these outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*P5t6eNCWpBh_FfFj0nFvDA.png" /><figcaption>Image: Generated with dreamstudio.ai using the Stable Diffusion algorithm, inspired by Moebius.</figcaption></figure><h3>Combining the strengths of data analytics and foresight methods</h3><p>By using data analytics, we aim to increase the rigour of our thinking about the future, while maintaining the creativity that futures methods are known for.</p><p>While data can provide evidence for or disprove our hunches, the deliberative and participatory nature of qualitative futures and foresight methods can infuse values, lived experience and judgement into the forecasting process.</p><p>Foresight can elevate quantitative modelling of trends and forecasts by considering the wider societal consequences of these trends, as well as by helping ponder the nonlinear effects and potential shocks that are difficult if not impossible to foresee in the data.</p><p>We can use foresight methods such as scenario modelling to help contextualise and make sense of quantitative predictions. <a href="https://www.foreignaffairs.com/articles/united-states/2020-10-13/better-crystal-ball">Quantitative forecasts about narrow, specific questions can be situated within complex and compelling qualitative scenarios</a> to create advanced early warning systems for policy makers.</p><p>Data science methods can also be used to amplify qualitative insights about the future. For example, <a href="https://www.nesta.org.uk/report/the-future-of-skills-employment-in-2030/">Nesta’s work on the labour market</a> demonstrates how expert judgements can be augmented by machine learning to produce estimates of future growth for hundreds of different jobs and skills.</p><p>Ultimately, data can provide suggestions of what might happen, but it won’t tell us what we should do about it. To get to the “so what’s” and to design proposals for action, qualitative methods and judgement is essential.</p><h3>A systems-level view of trends: Innovation Sweet Spots</h3><p>So what does this look like in practice? How are we now blending quantitative and qualitative methods to generate insights which can be applied in service of Nesta’s social missions?</p><p>Our experimental <a href="https://nesta.shorthandstories.com/section-3-the-innovation-sweet-spots-approach/index.html">Innovation Sweet Spots programme</a> is not unlike Ursula K. Le Guin’s vision of seeing the whole, as we aim to map innovation systems around social challenges. We analyse trends around innovations and technologies that can achieve social impact, by using large datasets on investment and public discourse. With this we wish to capture a multi-dimensional view of the socio-technical system in which innovation happens, as we recognise the important role that science and business as well as media, policy and social discourse plays in the development and adoption of new innovations.</p><p>By capturing a variety of signals, we can perform gap analysis and notice aspects in which certain innovations are excelling or conversely might be stalling or underperforming. This can support policymakers, funders, investors and anyone working on social challenges to better understand the innovation system as a whole, and direct resources where they can deliver maximum social impact.</p><p>Our pilot Innovation Sweet Spots project focussed on <a href="https://www.nesta.org.uk/feature/innovation-sweet-spots/">low carbon heating technologies</a> such as heat pumps, geothermal and hydrogen heating, which are key to reducing household carbon emissions and enabling the UK reach its net zero goal. We found strong signals of research and development funding as well as an increasing public discourse about low carbon heating in the UK. Conversely, an apparent gap in private investment raised concerns about further scaling-up of these technologies. By discussing our findings with energy and investment experts, we highlighted some ways to overcome the key barriers to investment and adoption of these technologies.</p><h3>Anticipating future developments</h3><p>Besides examining historical trends, we also wish to combine quantitative and qualitative analysis to better anticipate future change.</p><p>Our next Innovation Sweet Spots project is on <a href="https://www.nesta.org.uk/project/innovation-sweet-spots-food-innovation-obesity-and-food-environments/">food technologies and innovations</a>. In addition to measuring trends around innovations like lab-grown meat and dark kitchens, we are engaging industry experts through participatory methods such as the <a href="https://en.wikipedia.org/wiki/Futures_wheel">futures wheel</a> and <a href="https://www.allourideas.org/">wiki surveys</a>. This will help us consider the societal consequences of the trends and their impact on food environments and obesity.</p><p>We also consider future developments using more quantitative approaches. We have trialled an experimental machine learning approach that tries to learn patterns of historically successful startups and aims to predict future success of companies in sectors that are important for Nesta’s missions such as green technologies, food tech and early years education. The aim of this is not to replace human decision-making when making investments but rather to facilitate the development of policy to support growth and innovation, as low probabilities of predicted success could signal the need to better understand barriers to success in that space and introduce packages of grants or other stimuli.</p><p>We will report our findings from these projects in the near future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*Htzt40DRY4UU69CqVUFr9Q.png" /><figcaption>Image: Generated with dreamstudio.ai using the Stable Diffusion algorithm, inspired by Moebius.</figcaption></figure><h3>Mining data for future signals</h3><p>The seeds of the future can be found in the present, and data can facilitate surfacing those signs. We can leverage large datasets describing startup and research activity for horizon scanning, to find examples of cutting-edge innovation.</p><p>For example, in our collaboration with Nesta’s fairer start mission, we <a href="https://www.nesta.org.uk/project/mapping-parenting-technology/whats-next-for-parenting-tech/">analysed over a thousand startups in the parenting and early years education space</a> that are already bringing new products such as child speech recognition and smart parenting assistants to the market. We characterised the progress so far and envisioned some of the potential positive and negative consequences if these technologies continue to grow.</p><p>We are also interested in exploring how to facilitate discovery of such signals by visualising datasets related to innovations and technologies. An example of such a solution is our recent <a href="https://www.nesta.org.uk/project/mapping-parenting-technology/could-toddler-tech-help-to-get-more-children-school-ready/">“map” of mobile apps for parents and children</a>. We created it by data-scraping the Google Play Store and using automated visualisation algorithms to show the sprawling landscape of over a thousand apps that parents and children are navigating from preconception to preschool. While this was a demonstration of apps that are already on the market, we can use similar methods to visualise more forward-looking data related to development of technologies such as research project abstracts or patents.</p><h3>Open source and open foresight</h3><p>Combining data science with strategic foresight is still a relatively nascent practice with many avenues to explore. It is important that we keep experimenting and that it is done openly, backed by those focused on innovation for public good — not just profit. Perhaps this is one of the major differences between Discovery Hub’s data analytics work and the average corporation: that all our work is offered back to the commons — including the ‘how’ and the failures too.</p><p>In this work, we benefit from a close collaboration with Nesta’s <a href="https://medium.com/data-analytics-at-nesta">Data Analytics practice</a>, which also values working in the open and sharing as much as possible. We’re building on the years of experience and infrastructure developed by our colleagues working on <a href="https://www.nesta.org.uk/feature/innovation-methods/innovation-mapping/">mapping innovation</a> and <a href="https://www.nesta.org.uk/project/open-jobs/">studying the labour market</a> at Nesta. Moreover, our work of course depends on the vast and generous open source community without which it would simply not be possible.</p><p>We will continue trialling new methods that can help us understand the innovation environment, anticipate future developments and support better decision making. For example, we’re exploring the potential of causal inference to provide data-informed recommendations for actions (we will report the findings from our initial explorations in the near future). We are also curious about how network science can improve our understanding of the innovation system to pinpoint novel and promising innovation and collaboration areas. We’ll also consider the usefulness of participatory and data-informed approaches such as fuzzy cognitive mapping for creating scenarios and understanding complex systems.</p><p>As we progress in blending data with foresight, creativity and participation, we will write here at Discovery Hub’s Data Test Kitchen about our lessons learned. If you would like to discuss any of this, please reach out and I’ll be always happy to talk.</p><p><em>Huge thanks to Celia Hannon, Laurie Smith and Siobhan Chan for their helpful comments on this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27613e77323a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/taming-and-training-hunches-data-analytics-and-foresight-27613e77323a">Taming and training hunches: Data analytics and foresight</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>