<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Discovery at Nesta - Medium]]></title>
        <description><![CDATA[Nesta’s Discovery Hub scans for emerging trends, promising technologies and interventions - Medium]]></description>
        <link>https://medium.com/discovery-at-nesta?source=rss----f5a47887598f---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Discovery at Nesta - Medium</title>
            <link>https://medium.com/discovery-at-nesta?source=rss----f5a47887598f---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:22:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/discovery-at-nesta" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to find the best policy evidence across millions of documents?]]></title>
            <link>https://medium.com/discovery-at-nesta/how-to-find-the-best-policy-evidence-across-millions-of-documents-ea0edd174dc4?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/ea0edd174dc4</guid>
            <category><![CDATA[boolean-search]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[openalex]]></category>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Thu, 18 Jun 2026 10:15:17 GMT</pubDate>
            <atom:updated>2026-06-18T10:15:17.721Z</atom:updated>
            <content:encoded><![CDATA[<p>Policymaking increasingly occurs under <a href="https://www.instituteforgovernment.org.uk/sites/default/files/2025-12/rapid-policy-making.pdf">rapid conditions</a> with severe time constraints, making it challenging to gather sufficient evidence and draw international lessons when designing policy.</p><p>At Nesta we are exploring if AI can play a role in supporting policymakers in such circumstances. To this end, we’re developing <a href="https://policyatlas1.substack.com/p/policy-atlas-from-prototype-to-pilots">Policy Atlas</a>, an AI-powered tool to help policymakers synthesise global evidence and design novel, effective policies.</p><p>This requires a comprehensive analysis pipeline enabled by large language models (LLM) and agentic approaches to screen relevant sources, assess the strength of available evidence, and synthesise outputs.</p><p>The very first step when users enter their query on the Policy Atlas platform is searching through databases containing millions of academic research papers and policy documents, to find relevant sources to analyse.</p><p>It is critically important to find the most relevant documents for the user query, as the subsequent evidence synthesis and recommendations will all depend on the search results.</p><p>There are multiple approaches to tackling the task of document search, and in this article we describe our present implementation, plans for improvements and invite your feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S2pDSqjcvpmB2ZhO.png" /><figcaption><em>Analysis pipeline underpinning the Policy Atlas tool. This article focuses on the ‘Search for papers’ step.</em></figcaption></figure><h3>Grounding analysis in research and policy databases</h3><p>Our aim with Policy Atlas is to automatically generate accurate assessments of different intervention options on any policy topic area. To this end, it is important to ground the analysis and text generation by the large language model (LLM) in actual research and policy documents to avoid bias due to model’s training data and hallucinations (ie, inaccuracies).</p><p>In turn, the documents used for grounding the LLM should come from comprehensive and trusted sources, rather than a simple web search that is usually employed by existing general-purpose ‘Deep Research’ AI tools. Web searches can be influenced by <a href="https://www.bbc.co.uk/future/article/20241031-how-google-tells-you-what-you-want-to-hear">search engine biases</a> and search results might include news, commercial content and unvetted opinions (eg, blogs), which are less relevant for our use case.</p><p>Therefore, in Policy Atlas we employ large, curated academic and policy document databases, namely OpenAlex and Overton. OpenAlex provides an index of more than 400 million academic works, whereas Overton indexes over 20 million works of the so-called grey literature — government publications and think tank reports from across the world.</p><p>While academic evidence synthesis projects usually focus on research evidence, we’ve found in our user testing pilots with policymakers in the UK Government that grey literature is also a very valuable source of insights to ground empirical evidence in the policy context.</p><p>Both of these databases provide a vast catalogue of document titles, summaries, authors as well as links where to find the full contents of the documents. The databases by themselves do not, however, provide the full-text document content, which must be acquired in subsequent steps.</p><h3>Using keyword and semantic search</h3><p>Optimal knowledge retrieval is a rich area of research and development, with <a href="https://aarontay.medium.com/boolean-vs-keyword-lexical-search-vs-semantic-keeping-things-straight-95eb503b48f5">various methods</a> for identifying the most relevant documents. We employ two of the main types of methods: searching using keywords and semantic search.</p><p>Keyword search (also called boolean term search) is conceptually straightforward: documents that have at least one of the provided keywords will be found by the search. The higher number of the provided keywords the document contains, the higher it will be ranked in the search results. This approach has the benefit of being easily interpretable — either a document has the keywords or not.</p><p>Semantic search is better at identifying texts with similar <em>meaning</em>. Here documents don’t necessarily need to match the same keywords, but rather need to have text with similar meaning. This is more powerful, as the user doesn’t need to specify the exact correct terminology and you can even search across different languages.</p><p>Semantic search will often provide more precise results than a keyword search, with the caveat that semantic search results are harder to interpret. There is not a clear cut-off when a semantic search result stops being relevant — there are only gradations of results being more or less similar to the user query.</p><p>In Policy Atlas, we use boolean term search for the OpenAlex database and semantic search for the Overton database. The reason for this distinction is that Overton supports semantic search at relatively low cost, whereas OpenAlex initially did not have an in-built semantic search capability. More recently, OpenAlex has also released support for <a href="https://developers.openalex.org/guides/searching#semantic-search-beta">semantic search</a>, however, at the moment of writing this blog, at relatively high cost (ie, semantic search is 100 times more expensive than a simple keyword search on OpenAlex). This is because computationally, keyword search over large databases is more effective and cheaper than semantic search.</p><h3>Using AI to construct keyword search queries</h3><p>From the user experience point of view, semantic search is easier to use as users can define their queries as free text using the terminology they are familiar with. In contrast, for keyword search the users need to carefully identify the right keywords to use.</p><p>In Policy Atlas, we facilitate both types of searches through AI-powered workflows. First, the platform elicits information from the users about their policy question that is then used to generate suitable search queries for the Overton and OpenAlex databases.</p><p>Our search definition process, largely inspired by the <a href="https://en.wikipedia.org/wiki/PICO_process">PICO framework</a>, asks users about the policy interventions they are interested in, the target population, setting, desired outcome, geography and other important factors.</p><p>During this process, we use an LLM to generate sensible options for the population, setting and outcome based on the user’s intervention of interest. The user can choose from these automatically generated options — or they can also input their own preferred options.</p><p>This information helps the user consider their research question in greater detail, and allows Policy Atlas to define more specific database search queries. During user testing, we’ve received positive feedback about this approach, with one user remarking that the “follow up questions are uncannily good — very relevant and helpful to narrow down your search”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KraAiy5B4W16WsOv.png" /><figcaption><em>Information elicited from the policymaker user by the Policy Atlas search definition process.</em></figcaption></figure><p>We then combine the extracted information into one natural text query, by using an LLM to rephrase the provided information into a short paragraph. This can then be used to query Overton’s database. As an example, a query for “interventions to reduce childhood obesity” and a restriction of the search to OECD countries resulted in the following natural text query:</p><p>What interventions are effective in reducing childhood obesity among children in OECD member countries?</p><p>When searching OpenAlex for academic papers, we don’t want policymakers to construct meticulous keyword searches — they won’t have time to do it and there is a risk the keywords will be incomplete.</p><p>Instead, we again leverage an LLM to construct the boolean query for the user. Here’s a fragment of an example query for the same search parameters as used for the aforementioned semantic search example:</p><p>(child OR children OR childhood OR adolescent OR adolescents OR youth OR youths OR teenager OR teenagers OR “young people” OR minor OR minors OR pediatric OR paediatric) AND (obesity OR overweight OR “excess weight” OR “high BMI” OR “body mass index”) AND (intervention OR interventions OR program OR programs OR programme OR programmes OR strategy OR strategies OR treatment OR treatments OR prevention OR preventive OR policy OR policies OR initiative OR initiatives) AND (OECD OR “Organisation for Economic Co-operation and Development” OR “Organization for Economic Cooperation and Development” OR “United States” OR USA OR US OR American OR Canada OR Canadian OR “United Kingdom” OR UK OR English OR England OR Scottish OR Scotland OR...</p><p>To increase the chances of finding more robust types of evidence, we additionally construct specialised search queries for randomised controlled trials (RCTs) and systematic reviews or meta-studies by appending the automatically generated search terms with pre-defined terms such as “randomized controlled trial” or “systematic review”.</p><p>Constructing a good boolean query, however, is not trivial, as it requires domain understanding and it has to be comprehensive while also not overly broad. Moreover, LLMs have stochastic output and the queries might slightly differ between different runs. Therefore, we generate multiple versions of the boolean terms and use all of them for the final search results. Here is another example for generated based on the same initial user input</p><p>((child OR children OR childhood OR adolescent OR adolescents OR youth OR youths OR teenager OR teenagers OR pediatric OR paediatric OR school-age OR “school age” OR “young people” OR “young person”) AND (obesity OR overweight OR “excess weight” OR “body mass index” OR BMI OR adiposity OR “weight gain” OR “weight reduction” OR “weight control” OR “weight management”) AND (intervention OR program OR programme OR strategy OR policy OR campaign OR prevention OR treatment OR “health promotion” OR initiative OR “behavior change” OR “behaviour change” OR educational OR counseling OR counselling OR “diet modification” OR “nutrition education” OR “physical activity” OR exercise OR “active lifestyle” OR “sedentary behavior” OR “sedentary behaviour” OR “lifestyle modification”) AND (Australia OR Australian OR Austria OR Austrian OR Belgium OR Belgian OR Canada OR Canadian OR...</p><p>The final process for constructing boolean queries is shown in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mHkCtxaiGm4dv_BO.png" /><figcaption><em>Boolean and semantic search processes employed by Policy Atlas.</em></figcaption></figure><h3>Which AI model generates the best search queries?</h3><p>There are many LLMs one could use to generate the boolean search terms — how does one choose the best one?</p><p>We have started testing multiple models and temperature settings (which control the randomness of the model outputs) in terms of how comprehensive the search terms are, how many documents can they actually retrieve from the OpenAlex database (ie, whether the terms are not too restrictive), and also compared their performance to manually constructed ‘gold standard’ search terms.</p><p>While we are still optimising our evaluation approach, our early results were based on a set of ten research questions related to obesity research, for example, “What is the effect of free school meals on obesity and calorie consumption?”. Each of these research questions also have corresponding manually constructed search terms constructed by researchers that we can use as a reference.</p><p>These reference boolean terms were not perfect as they were originally constructed for searching for systematic reviews. For these tests we kept only domain relevant search terms and removed the terms specifically relating to systematic reviews, which resulted in a somewhat broad queries that returned tens of thousands of results. As a result the agreement scores (<a href="https://en.wikipedia.org/wiki/F-score">F1 scores</a>) shown below are generally low: less than 0.20 whereas a perfect score would be 1.0. Nonetheless, we can use them to start investigating the impact of model and prompt choices.</p><p>The research question was provided to the LLM together with a prompt that instructs the LLM to act as “an expert systematic review information specialist” who has a task to “generate a highly effective systematic review Boolean query” that “needs to be as inclusive as possible” but also “retrieve fewer irrelevant studies” and so on.</p><p>The prompt we used was based on the <a href="https://arxiv.org/pdf/2302.03495%5C">insights by Wang and colleagues</a> at the University of Queensland and Leipzig University that demonstrates the potential of ChatGPT in generating effective Boolean queries (specifically, their “q3” prompt example) .</p><p>Our first observation was that a straightforward step to improve the quality of AI-generated search terms is to simply use a larger, better LLM. An earlier version of Policy Atlas conservatively used OpenAI’s GPT-4o-mini model, which is very fast and cheap, with temperature set to 0 to reduce the variability in the outputs. This, however, occasionally resulted in unsuccessful searches that returned a very small number of results. It was possible to increase the median number of retrieved results by more than 25 times by using the more advanced GPT-4.1 and GPT-5 models.</p><p>Another observation was that a very high model temperature (larger than 1.5) tends to perform the worst, likely because it makes LLM less capable of following instructions.</p><p>Amongst the more advanced models we tested, we found that the GPT-4.1 model provides comparatively good performance, particularly if one generates multiple versions of the search query and then combines the retrieved documents. The combination of search results also helps reduce the problem of variability of LLM outputs. Interestingly, the optimal temperature setting was an intermediate one (temperature set to 1.0), suggesting that there might be an optimum between too little output variability (at temperature set to 0) and too much variability (at higher temperatures).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SAPJZ1JQqXzpafXn.png" /><figcaption><em>Agreement score (F1 score) between OpenAlex works obtained via LLM-generated and reference search terms.</em> <a href="https://public.flourish.studio/visualisation/29132499/"><em>Link to an interactive version</em></a><em>.</em></figcaption></figure><p>The figure above shows the F1 score between OpenAlex works obtained via LLM-generated and reference search terms. Single search results show the mean score over 10 research questions and five runs per question. Combined result shows the mean across F1 scores across 10 questions on the deduplicated union of retrieved works from five runs. Note that GPT-5 family models do not have a temperature setting.</p><h3>Results are sensitive to the instruction prompt</h3><p>An important note is that these results are specific to the particular LLM instruction prompt that we used. The quality of LLM-generated boolean queries is <a href="https://arxiv.org/pdf/2505.07155">sensitive to prompt design</a> and therefore the optimal model and temperature setting might depend on the particular wording used in LLM instructions,</p><p>For example, an earlier version of our instruction prompt (Prompt 1 in the figure below), which was in fact slightly more prescriptive and verbose compared to the current implementation (Prompt 2), showed a considerably worse performance across almost all of the tested models. It is difficult to provide a precise reason for the difference in performance: the underperforming prompt was longer and more prescriptive but potentially less clear, and it resulted in fewer search terms per query.</p><p>All the prompts we used for testing can be <a href="https://github.com/nestauk/discovery_policy_atlas/blob/dev/backend/testing/r_and_d/boolean_queries/prompts.py">found in our GitHub repository</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0e7BDB51GiZZelCR.png" /><figcaption><em>Agreement scores (mean F1 score) between results obtained via LLM-generated search terms and manually constructed search terms for two different prompts. </em><a href="https://public.flourish.studio/visualisation/29132770/"><em>Link to an interactive version</em></a><em>.</em></figcaption></figure><p>The figure above shows agreement scores (mean F1 score) between results obtained via LLM-generated search terms and manually constructed search terms for two different prompts. The scores were calculated as the mean across 10 research questions with 5 runs per question; retrieval results were <em>not</em> combined across different runs.</p><p>Thanks for reading! Subscribe to learn about our future experiments with evidence search.</p><h3>Next steps</h3><p>Currently we are using GPT-4.1 at temperature set to 1.0 for generating the boolean search terms, as this approach returns a sufficiently high number of relevant results and the model has a reasonably fast running time compared to, for example, GPT-5. You can find the prompt we are using in <a href="https://github.com/nestauk/discovery_policy_atlas/blob/a4174c0b3951cabac408deafd1b1031934da0f6a/backend/app/services/analysis/prompts.py#L13">our GitHub repository</a>.</p><p>Our next steps involve exploring a more agentic search solution, improving our evaluation approach using robust benchmarks such as <a href="https://github.com/princeton-nlp/LitSearch">LitSearch</a> and <a href="https://github.com/bytedance/pasa">PaSa</a>, and address some of the challenges noted below.</p><h4>Up-to-date model evaluations</h4><p>Since we conducted these tests, OpenAI has released newer models and we will need to update our evaluations. Furthermore, more tests should be performed on other frontier models (eg, from Anthropic and Google) and our reference boolean queries used as benchmarks should be improved to more closely match real-world queries developed by researchers used for systematic reviews.</p><h4>Prompt optimisation</h4><p>The output sensitivity to instruction prompts also raises the question on how to optimise the prompts themselves. For example, adding ‘seed’ queries from relevant systematic reviews to the prompt <a href="https://arxiv.org/pdf/2302.03495">can improve the performance</a>. Such seed queries could also be generated automatically by using <a href="https://dl.acm.org/doi/10.1145/3366423.3380185">more traditional natural language processing methods</a> that predate LLMs. Prompts can also be optimised further via code libraries such as <a href="https://arxiv.org/html/2509.00038v1">DSPy</a> that automatically explore better natural-language instructions.</p><h4>Relevance screening</h4><p>Initial user feedback indicates that Policy Atlas is useful as a fast scoping and sense‑making tool that improves access to evidence — but it can also miss relevant studies, and still requires manual follow‑up checks for the completeness of the evidence base.</p><p>For improving the retrieval, query generation is only one aspect that can be improved — <a href="https://aarontay.substack.com/p/the-horseless-carriage-of-ai-search">another avenue is re-ranking the retrieval results</a> after the document search to increase the relevance of the top results. Policy Atlas does this via a screening step that follows immediately after the search step, by using an LLM to judge the relevance of each of the retrieved papers.</p><p>There are potentially more computationally efficient and faster re-ranking methods such as <a href="https://arxiv.org/pdf/2403.10407">cross-encoders</a> that use traditional natural language processing approaches to determine document relevance to user input, while still being competitive with LLM-based methods.</p><p>Furthermore, the screening process should be more closely joined up with the document retrieval process to optimise the depth of the evidence search. In the current implementation the user needs to indicate the desired search depth by defining the number of documents the search should cover. Ideally, the tool could <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/cesm.70068">determine the optimal depth automatically</a>, by iterating between document retrieval and screening, and stopping once it can’t identify relevant documents.</p><h4>Data coverage</h4><p>Finally, there is a challenge regarding the data coverage of our two evidence bases. While OpenAlex is a comprehensive index of documents, it does not provide access to the full-text content of academic publications — it only provides information on where the full-text content can be found. Meanwhile, Overton provides a large index of global grey literature, but sometimes the full-text documents have been removed from think tank or government websites. At the moment, we are exploring how far we can push the quality of outputs by accessing open access literature and summaries where full-texts are not available.</p><p>We will continue sharing our progress on methodological improvements, and we are keen to connect to other researchers working on retrieval optimisation for research and policy data. Reach out to us!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea0edd174dc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/how-to-find-the-best-policy-evidence-across-millions-of-documents-ea0edd174dc4">How to find the best policy evidence across millions of documents?</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Policy Atlas: from prototype to pilots]]></title>
            <link>https://medium.com/discovery-at-nesta/policy-atlas-from-prototype-to-pilots-ce1038c35d65?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/ce1038c35d65</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[policy-making]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 17:57:07 GMT</pubDate>
            <atom:updated>2026-06-15T17:57:06.822Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was co-written by </em><a href="https://medium.com/@hello-aidank"><em>Aidan Kelly</em></a><em>, </em><a href="https://www.nesta.org.uk/team/shabeer-rauf/"><em>Shabeer Rauf</em></a><em>, </em><a href="https://www.nesta.org.uk/team/genna-barnett/"><em>Genna Barnet</em></a><em> and Karlis Kanders.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQMa4Kbq2Mdy5IIHF47UAQ.png" /><figcaption>Screenshot of the current version of Policy Atlas</figcaption></figure><p>Policymaking increasingly occurs under <a href="https://www.instituteforgovernment.org.uk/sites/default/files/2025-12/rapid-policy-making.pdf">rapid conditions</a> with severe time constraints, making it challenging to gather sufficient evidence and draw lessons from international examples when designing policy.</p><p>At <a href="https://www.nesta.org.uk/">Nesta</a> we are exploring if AI can play a role in supporting policymakers in such circumstances. We’re developing <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">Policy Atlas</a>, an AI-powered tool to help policymakers synthesise global evidence and design novel, effective policies.</p><p>In this article, we show the improvements we have made to the prototype tool over the past few months — making it more robust and user-friendly — and share lessons from our initial pilots with UK government users. You can find our <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">earlier updates on Nesta’s website</a>.</p><p>We tested the current version in real-world policymaking contexts and gathered feedback to assess product-user fit. Our initial pilots were with the Department of Health and Social Care (DHSC), where we tested the tool with 40 users across eight different directorates, and the Cabinet Office, where 10 users tried it out.</p><h3>Scaffolded search helps policymakers run more targeted queries</h3><p>Policy Atlas helps policymakers find policy intervention evidence across millions of global research and policy documents.</p><p>To make the search process more targeted and useful, the tool helps users be more specific about what interventions they are looking for, using a series of sequential prompts. These prompts are based on the <a href="https://en.wikipedia.org/wiki/PICO_process">PICO framework</a> and ask the users about the target population, intervention setting, desired outcomes, geography and other important factors.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F03AhY9MXmcM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D03AhY9MXmcM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F03AhY9MXmcM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f242ae221f7bc1c40720762b4b1c5c3e/href">https://medium.com/media/f242ae221f7bc1c40720762b4b1c5c3e/href</a></iframe><p>We’re using a large language model (LLM) to suggest personalised examples of target populations, settings and outcomes that might be relevant to users’ scope of work. In this way, we’re scaffolding the search process and helping policymakers think more deeply about the policy interventions they are researching.</p><p>We received positive feedback about this feature in our initial pilots, with a senior evidence review manager within the DHSC remarking that these <em>“follow up questions are uncannily good — very relevant and helpful to narrow down your search” </em>and it’s useful to <em>“run this first to inform questions for systematic review”</em>.</p><p>A policy lead at DHSC also said that the search process <em>“prompted me with really good and extremely relevant questions”.</em></p><h3>Evidence strength assessment helps prioritising policy evidence sources</h3><p>It is important to clearly identify more strongly evidenced policy options from those with weaker evidence. We <a href="https://www.nesta.org.uk/blog/five-lessons-from-building-evidence-strength-scoring-into-an-ai-policy-tool/">implemented an LLM-based approach</a> that classifies each source document by evidence type, assigns an initial evidence strength score between 1 and 5, and applies simple adjustment rules where relevant, such as a sample size penalty. These scores are then aggregated across the intervention themes extracted from the documents.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSskh_qylbWg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSskh_qylbWg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSskh_qylbWg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb4b721320a8cf40dfae2ef37a4bf90b/href">https://medium.com/media/fb4b721320a8cf40dfae2ef37a4bf90b/href</a></iframe><p>Our pilot users found the classification valuable as it helped to easily distinguish between systematic reviews, observational studies, policy documents and other evidence types.</p><p>Evidence scores also helped to prioritise certain types of more robust evidence — such as systematic reviews and randomised controlled trials (RCTs) in our current implementation — when generating the narrative summary and policy recommendations.</p><p>A challenge however, was that users expressed varying preferences of what counts as “strong evidence”. For example, one of our pilot teams at DHSC advised that when considering policies around emerging, cutting-edge medical technologies, observational studies and qualitative research should also score highly (due to their novel nature) and should not be overlooked. Therefore, evidence scoring assessment should have some in-built flexibility to satisfy differing policymaker needs.</p><p>In addition to evidence assessment, we explored an approach to evaluate the potential impact of different policy options, and assign an impact score. The impact assessment can be used together with the evidence strength for further prioritisation of interventions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fpiud1bErkE0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dpiud1bErkE0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fpiud1bErkE0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a644ece8762d1dd86c156a6d99bcf74/href">https://medium.com/media/6a644ece8762d1dd86c156a6d99bcf74/href</a></iframe><h3>Fully traceable source citations improve policymaker trust</h3><p>It is crucial that the AI-generated text is fully backed up by evidence. We’re using retrieval-augmented generation to ground the LLMs in the source documents, as well as agentic AI workflows that identify the most relevant citations for a given AI-generated claim, verify if the AI-generated text agrees with the text in the cited source, and provide an exact quote from the source document.</p><p>As a result, the generated text is more robust, and the user can follow up the citation to find the exact location in the original text that the AI generated content is based on.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7SQnDySUdng%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7SQnDySUdng&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7SQnDySUdng%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f13a0975c91ea9e976a6794a32b79907/href">https://medium.com/media/f13a0975c91ea9e976a6794a32b79907/href</a></iframe><p>To facilitate further engagement with the underlying source documents, we have also implemented an AI assistant chatbot that helps users converse with the evidence base and answer further questions about intervention details. The assistant bases its answers only on the documents retrieved by the evidence search.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmV8yK9bbPgo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmV8yK9bbPgo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmV8yK9bbPgo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fc68f20571616de5fb377a654f4f2ba/href">https://medium.com/media/9fc68f20571616de5fb377a654f4f2ba/href</a></iframe><h3>Policymakers have widely varying user needs</h3><p>Our pilots included a range of policymaking roles, such as policy leads, analysts and evidence experts. By observing how they engaged with the tool, three broad types of users emerged: generalist policymakers, evidence and analysis specialists and localists. Each has different needs and requires different outputs from the tool.</p><p><strong>Generalist</strong> <strong>policymakers</strong> require practical and quick insights that are actionable and provide a “jumping off point” for further research. They found the concise core findings and key citations that Policy Atlas provides to be most useful. These policymakers might still have different levels of domain specialisation (eg, they might be a policy lead on a specific health topic) but generally they are more interested in horizon scans of emerging intervention themes, rapid briefs answering a specific policy question, and concise policy notes recommending a concrete course of action.</p><p><strong>Evidence and analysis specialists</strong> require comprehensive literature overviews supporting internal systematic review and rapid evidence review efforts. They valued the broad range of evidence sources that Policy Atlas can surface, highlighting grey literature sources as particularly valuable. Some also require further functionalities that support specialised analysis, such as value-for-money estimations and Impact Assessments, that we will consider for the next version of Policy Atlas.</p><p><strong>Localists</strong> require actionable insights that are specific to their local context, such as a specific local authority. They would value the ability to review interventions from their ‘closest comparator’ locations and consider how would they transfer to their context. For them, Policy Atlas in its current implementation was too general, as it is built with a central government policymaker in mind. However, this could be adapted by adjusting the large language model instruction prompts and incorporating local information into intervention impact and transferability assessments.</p><p>Overall, we experienced a strong demand for better AI tools for policymakers to facilitate various research processes. Following our initial pilots, we have been met with further demand to test the tool with other departments.</p><p>Here are a few example comments, containing both positive and constructive feedback:</p><blockquote>“Policy Atlas has been very helpful in supporting a quick turnaround ask on which policy solutions other jurisdictions employ, including the strength of their evidence bases. It has saved a great deal of time and given me faster access to the right types of evidence. I would say that the time taken to understand and weigh up the evidence is unchanged but that is totally fine”</blockquote><blockquote><strong>Principal Research Officer</strong></blockquote><blockquote>“[Policy Atlas] is a huge time saver and great for getting that snapshot of the evidence that is needed quickly when looking at a broad and complex area”</blockquote><blockquote><strong>Strategy Lead</strong></blockquote><blockquote>“Most useful as a fast, early-stage scoping tool that improves access to and synthesis of evidence… searches can miss relevant studies”</blockquote><blockquote><strong>Lead Behavioural Scientist</strong></blockquote><blockquote>“Summaries and evidence tables proving more useful than researcher agents on other AI platforms.”</blockquote><blockquote><strong>Operational Researcher</strong></blockquote><h3>Policymakers need to be in control</h3><p>We also concluded that Policy Atlas needs to provide users with greater flexibility and control over the generated outputs than the current version allows.</p><p>Currently, users provide their search criteria through the scaffolded search process; the platform then retrieves relevant evidence sources and generates the same set of outputs for every search — an executive summary, detailed deep-dive into different intervention themes, and a reference list. This process, which can take up to 30–45 minutes depending on the number of analysed documents, could be improved by providing more opportunities for checkpoints and iterative refinements where users can inspect intermediate outputs and issue course corrections.</p><p>Ultimately, an AI tool for policymakers should accommodate the highly iterative and collaborative nature of policy development.</p><h3>What’s next?</h3><p>By carrying out the pilots, we have grown in our appreciation of the complexity of policymaker user needs. While our initial ambition (before we launched our pilots) was to build a tool that could be used by all <a href="https://www.gov.uk/government/organisations/civil-service-policy-profession/about">37,000 policymakers</a> in the UK civil service, in the next phase of the development, we will focus on a narrower set of more specific, well-defined policymaking tasks and outputs.</p><p>Specifically, we plan to turn our attention to supporting high impact decisions that are taken when developing strategies and contributing to outputs such as Green Papers, White Papers and National Strategies. While this means a narrower use case focus, such strategy documents are still comprehensive, consisting of varying, complex components such as evidence syntheses, policy option appraisals, cost-benefit analyses and others. We will build upon and improve the core AI capabilities that we have developed so far and redesign the user experience and user interface to tailor the tool for these particular outputs.</p><h3>Collaborate with us</h3><p>We are keen to work in the open and collaborate with others.</p><p>In the next few weeks we will publish further deep dives into our methods and insights from pilots. You can also <a href="https://youtu.be/5YjEi6LrOeg">watch here</a> a longer demo of Policy Atlas and check out <a href="https://discoverypolicyatlas-production.up.railway.app/public/projects/69384bb5-b0ba-4007-a611-3362c212f146">an example analysis output here</a>. <a href="/service/mailto:discoveryhub@nesta.org.uk">Reach out</a> if you are working on similar questions, looking for partners to apply for related funding opportunities, or if you’re interested in testing our tool for bolstering policy development.</p><p>We would be particularly interested in speaking to civil servants directly involved in the key decision moments mentioned above, to better understand your needs and test future iterations of the tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce1038c35d65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/policy-atlas-from-prototype-to-pilots-ce1038c35d65">Policy Atlas: from prototype to pilots</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mapping Heat Pump Innovation via LLMs: A Circular Economy Approach to Sustainable Heating]]></title>
            <link>https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-91bc023a2481?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/91bc023a2481</guid>
            <category><![CDATA[ghg-emissions]]></category>
            <category><![CDATA[circulareconomy]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[heat-pumps]]></category>
            <dc:creator><![CDATA[Diego Pascual Villegas]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 09:23:00 GMT</pubDate>
            <atom:updated>2025-12-08T09:22:59.330Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 2: Results</h4><p><em>Diego Pascual Villegas collaborated with Nesta’s </em><a href="https://www.nesta.org.uk/discovery/"><em>Mission Discovery</em></a><em> team over the Summer 2025 on his Master thesis “Heat Pump Innovation Through a Circular Economy Lens: Analysing Technological Pathways for Sustainable Heating Systems”.</em></p><p>Heat pumps are a mature heating technology, with decades of engineering and innovation behind them. However, there is still room for incremental improvements that reduce installation and running costs, make retrofits easier, and cut embodied carbon and refrigerant related impacts over the whole lifecycle, including end of life. This research, therefore, focuses on mapping where current patent and research activity is concentrated across these and other areas to highlight the emerging technological pathways.</p><p>Building on the LLM-based approach developed with Nesta’s Discovery team (<a href="https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-36e65f2c6f0d">Blog Part I</a>), the analysis examines over 47,000 documents from 2020 to 2025. It shows how research efforts are spread across five key categories: “Traditional Components”, “Non-Traditional Technologies”, “System Design”, “Other Improvements” and “Circular Economy”.</p><p>Getting a clear picture of these patterns can help us understand the most active areas of innovation, where the most likely future improvements for heat pumps might originate from.</p><p><strong>General findings</strong></p><p>The innovation landscape shows interesting patterns in where research and development is headed by analysing sources, trends, categories, and timelines. This contributes to decisions on where to concentrate further innovation, investments, and policy efforts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tlITu7gtTU6fCVqpluJdmg.png" /></figure><p>Our initial data was around 46,000 patents from Google Patents and 34,000 academic papers published in 2020–2025 from OpenAlex that mention “heat pump” in their title or abstract. After performing automated screening with a large language model (LLM) to filter out irrelevant results, the relevance step described in <a href="https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-36e65f2c6f0d">Part I </a>reduced this initial pool to 47,136 documents that are primarily about heat pumps. Of these, around 45,015 were patents, and 2,121 were papers.</p><p>The fact that patents are more dominant probably has to do with how the LLM relevance filter works and the differences in source material: patents tend to highlight devices right in their abstracts, while research papers often cover broader systems and concepts. This might reflect both the content and possible methodological biases. Therefore, the following analysis results reflect mostly patenting trends.</p><p><strong>Heat pump application and type</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zk1Nl9WbXV2VFIXWkTihwg.png" /></figure><p>This stacked bar chart offers a detailed look at how different types of heat pumps are used across various applications. The data reveals that air source heat pumps (ASHPs) are the most popular, especially for domestic use, thanks to their lower costs and easier retrofitting compared to ground source heat pumps (GSHPs). The study also shows that while the majority of research and patents focus on ASHPs, nearly half of all innovations are labelled as ‘unclear’ regarding the specific heat source, indicating that the patent summary discussed generic components without specifying the heat source, and we’d need to improve the analysis by analysing the full text.</p><p><strong>Heat pump innovation category distribution and timeline</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rh8ByCtlfYs293dfI4tiyA.png" /></figure><p>This category distribution bar chart breaks down where heat pump innovations are happening across five main categories, highlighting the industry’s ongoing emphasis on incremental improvements to established technologies.</p><p>The data only includes documents with categories aligned with the standard taxonomy. These categories are made up of specific subcategories.</p><p><strong>· Traditional components</strong> form the foundation of heat pump performance, including compressors, refrigerants, heat exchangers, motors, and lubrication systems.</p><p><strong>· Non-traditional technologies</strong> include emerging solid-state approaches such as elastocaloric, electrocaloric, magnetocaloric, ionocaloric, barocaloric, and thermoelectric heat pumps, which use alternative principles to vapour compression technologies and can operate without moving parts or conventional refrigerants.</p><p><strong>· System design</strong> focuses on how heat pumps are commissioned, controlled and integrated into buildings or networks.</p><p><strong>· Other improvements</strong> target specific operational challenges, such as flexible cycles, defrosting and thermal storage.</p><p><strong>· Circular economy</strong> principles capture more systemic design choices, including disassembly, modularity, repairability, recycling, refrigerant recovery and related strategies to extend lifetimes and reduce waste across the whole lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QogDi4dWsj-TISQEA3Ow5A.png" /></figure><p>This plot breaks down the innovation timeline by category from 2020 to 2024, based on filtered data. Traditional components still account for the majority, but their numbers have grown by 46.6% in absolute terms. System design has seen the most considerable increase, with a 77.7% jump. While CE remains somewhat niche, it has increased by 42.1%. Non-traditional technologies have increased by 52%, while other improvements have seen a more modest growth of 32.9%.</p><p><strong>Heat Pump Innovation: Traditional Components</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SK6splr7TVbBPyRR0vmXeg.png" /></figure><p>This horizontal bar chart shows how innovation effort is distributed across traditional heat pump components, with heat exchangers and compressors accounting for around 60% of the documents in this category. Refrigerants occupy a smaller share than might be expected given their central role in performance and regulation; fully understanding this pattern would require further work to separate refrigerant specific inventions from those labelled under other categories (more detail in ‘The Refrigerant Challenge’ section below).</p><p>Components such as motors, drives and lubrication systems appear smaller, which could mean there is less patenting explicitly focused on these elements, but it may also reflect a limitation of this analysis, since it only includes documents that mention “heat pump” in the title or abstract. Relevant for performance is how these components work together. An example is the pairing of lubricants and refrigerants, where dissolved refrigerants affect lubricant behaviour in the compressor, and conversely, how oil presence in the circulating refrigerant influences system efficiency. Yet, these interactions can be overlooked in system design, offering new opportunities to research the intersection of components.</p><p>More broadly, this component-based analysis does not capture the primary intent behind each innovation. For example, whether a given document mainly targets lower upfront costs, higher seasonal efficiency, reduced embodied emissions, smaller unit size, or easier installation. Mapping that intent across components and categories would be a valuable extension for future work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CtPMY6aaaMOhNWfiDNVOQ.png" /></figure><p>This timeline shows how research on traditional heat pump components has evolved from 2020 to 2024. All areas have seen growth, with Motors &amp; Drives increasing 126.1%, although from a relatively small starting count (222). These inventions focus on devices that power and modulate compressor speed, using more advanced capacity control to reduce on–off cycling and improve seasonal efficiency compared with traditional single speed systems. Meanwhile, compressors, heat exchangers and refrigerants remain key players in heat pump technology, with steady growth of 41% to 47%, reflecting ongoing incremental improvements in these established components.</p><p><strong>The Circular Economy and its role in heat pump systems</strong></p><p>The Circular Economy (CE) is about shifting away from the “take-make-dispose” mindset to a regenerative system that emphasises making better use of resources, cutting down waste, and supporting long-term sustainability. Its key principles are avoiding waste and pollution, making products last longer, and supporting the recovery of natural systems.</p><p>Heat pumps are expected to be deployed on a larger scale over the coming decade as countries decarbonise building heating. This is why the intersection with the CE is important; choices about materials, refrigerants, service models, and end-of-life routes are directly linked to environmental impacts. Incorporating circular principles can help <strong>ensure </strong>that the climate benefits of electrifying heating are not compromised by avoidable material use, leakage, or waste.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T6Dijq4mz0hyv_0y6T3EPw.png" /></figure><p>The analysis shows that most circular economy labelled innovations are focused on the “refrigerant recovery and reclamation” subcategory. This likely reflects regulatory and climate pressure around Fluorinated gases (F-gases). Other circular strategies, such as design for disassembly, modular architectures, repairability, recycling and reuse of components, appear less frequently. The bar chart, therefore, shows a narrow focus on one part of the lifecycle.</p><p><strong>A systems approach: analysing heat pumps across the full lifecycle</strong></p><p>The lifecycle map below shows a “cradle to grave” system that outlines the general lifecycle of heat pump technologies across eight phases. The map identifies critical CE intervention points to highlight how materials and information can be re-entered into the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-0LG44RuFG9GmaHnW8sjig.png" /></figure><p>Currently, most heat pumps are manufactured through linear supply chains, with limited adoption of circular practices. During their operational lifespan, the initial material choices, refrigerant systems, and service requirements become “locked-in”, which can lead to inefficient resource use for decades.</p><p><strong>The refrigerant challenge</strong></p><p>Heat pumps’ environmental impact depends heavily on the type of refrigerant they use. Some refrigerants with high global warming potential (GWP) can result in significant F-gas emissions, especially if they leak during operation or when units are disposed of. <a href="https://www.ccacoalition.org/news/time-clear-air-supercharging-climate-finance-combat-super-pollutants">Non-CO₂ greenhouse gases (GHG) make up nearly half of all global warming.</a> This shows how important it is to focus on circular economy strategies throughout the refrigerant life cycle, something the industry has been working on for years.</p><p>GWP and ozone-depleting potential (ODP) of refrigerants have improved significantly due to regulations and international environmental agreements. Natural refrigerants offer a good option with low GWP; however, transitioning to these refrigerants involves efficiency penalties and flammability classification restrictions that increase system complexity.</p><p>Potential strategies, such as refrigerant recovery &amp; reclamation systems, can capture, purify, and reuse refrigerants from end-of-life equipment, preventing atmospheric emissions and reducing the demand for newly produced refrigerants.</p><p>The transition to a CE for heat pumps depends on coordinated action across policy frameworks, standardised metrics, comprehensive lifecycle assessments, and just transition principles. Addressing current circularity gaps can ensure sustainable outcomes, avoid future resource and waste crises, and enable a regenerative approach to heating system transitions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91bc023a2481" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-91bc023a2481">Mapping Heat Pump Innovation via LLMs: A Circular Economy Approach to Sustainable Heating</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mapping Heat Pump Innovation via LLMs: A Circular Economy Approach to Sustainable Heating]]></title>
            <link>https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-36e65f2c6f0d?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/36e65f2c6f0d</guid>
            <category><![CDATA[llm-applications]]></category>
            <category><![CDATA[heat-pumps]]></category>
            <category><![CDATA[circulareconomy]]></category>
            <dc:creator><![CDATA[Diego Pascual Villegas]]></dc:creator>
            <pubDate>Thu, 06 Nov 2025 14:16:35 GMT</pubDate>
            <atom:updated>2025-11-06T14:19:06.115Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part 1: Methodology</em></h4><p><em>Diego Pascual Villegas collaborated with Nesta’s </em><a href="https://www.nesta.org.uk/discovery/"><em>Mission Discovery</em></a><em> team over the Summer 2025 on his Master thesis “Heat Pump Innovation Through a Circular Economy Lens: Analysing Technological Pathways for Sustainable Heating Systems”.</em></p><p>To effectively reduce greenhouse gas emissions, heat pump technology stands out as a proven solution for efficient heating systems. As an MSc in Circular Economy student at the Edinburgh Futures Institute, in collaboration with Nesta’s Discovery Team, we’ve worked on an exciting project to analyse innovations in heat pumps through a circular economy perspective.</p><p>For this purpose, we developed a Large Language Model (LLM) assisted pipeline to scan recent patents and publications on heat pumps (2020–2025), categorise innovations using a practical taxonomy table, and identify readiness signals (TRL), thereby prioritising evidence-based opportunities.</p><p>Working with Nesta’s Discovery team has been a human-centred experience, making data analysis feel collaborative and insightful rather than purely technical. I came into this project with only basic programming knowledge from an industrial engineering career, more than 10 years ago, no Python experience, and LLM knowledge limited to end-user use. At the beginning of the project, when the Head of Data Science asked how I was planning to look for patents and research documents, I replied, “manually, one by one”. Kindly, he didn’t laugh.</p><p>Three things surprised me most about this project. First, the volume of documents these models can process in hours, transforming an unmanageable reading burden into structured evidence we can actually use. Second, the same pattern generalises almost without limit: by swapping the sources and taxonomy, you can scan any domain, from EV batteries to retrofit materials. Third, with reasonable effort and willingness to learn, this approach can be applied without expert knowledge, and its benefits are tremendous, regardless of the field.</p><p><strong>Why did we use this method?</strong></p><p>Homes are directly responsible for creating around 70 million tonnes of CO₂ annually. Changing how we heat our homes is one of the most impactful ways we can cut pollution. Nesta’s “A Sustainable Future” mission goal is that by 2030, the UK will have reduced household carbon emissions by 30%.</p><p>Heat pumps present a highly efficient solution for achieving this goal. They collect heat from the outside air, ground, or water using a refrigerant and use a pump to move the heat into buildings to keep them warm in colder months.</p><p>However, heat pump adoption presents different challenges, highlighting the importance of innovation in areas such as components, cycles, controls, and system design. Yet, addressing the large and varied volume of documents related to heat pump innovation can be overwhelming. Manual reviews are slow and inconsistent, while keyword-based screening often results in too much noise (e.g., generic “pumps”). That is why we built a lightweight, auditable LLM pipeline to (1) retain only items primarily related to Heat Pumps, (2) classify them using a taxonomy, and (3) provide high-level readiness indicators. We ensured the model’s accuracy and reliability by comparing the LLM classifications against human labels on random samples.</p><p><strong>Pipeline overview</strong></p><p>The methodology consisted of 5 steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l1dYAVKSZiXSOXe6yuYZmA.png" /></figure><p><strong>Step 1 — Dataset Inputs:</strong></p><p>To begin our analysis, we gathered about 80,000 research articles and patents dating from 2020 to 2025, using the search terms “heat pump” and “heat pumps”. Research publication data was collected from <a href="https://openalex.org/">OpenAlex</a>, an open-source database of millions of publications. Patents were collected from <a href="https://patents.google.com/">Google Patents</a>, comprising millions of publications from hundreds of patent offices in various languages.</p><p><strong>Step 2 — Structured data extraction:</strong></p><p>Once the dataset was collected, the next step was filtering to identify documents specifically focused on heat pump technology. We used our LLM (GPT-4o-mini) to evaluate abstracts and determine relevance. The model indicated whether a document was primarily about heat pumps (“yes/no”), assigned a confidence level (0–1) and provided a brief rationale.</p><p>We ran the checks in batches (processing many abstracts at once instead of one by one), which speeds things up and keeps costs down without changing the logic. We set the model’s <a href="https://www.ibm.com/think/topics/llm-temperature">temperature to 0</a> so it behaves conservatively and consistently, which is ideal for yes/no classification. And we asked for JSON, so the result is a tiny labelled form (is_heat_pump, confidence, reason) that our scripts can read and combine automatically, and later export to CSV for charts.</p><pre>RELEVANCE_SYSTEM = &quot;&quot;&quot;<br>    You are evaluating if a document is PRIMARILY about Heat Pump technology.<br><br>    Heat Pumps are devices that transfer heat from one place to another using a refrigeration cycle.<br>    They are used for heating, cooling, or both.<br><br>    Return JSON with:<br>    - &quot;is_heat_pump&quot;: &quot;yes&quot; if primarily about Heat Pumps, &quot;no&quot; if just mentions pumps or is unrelated<br>    - &quot;confidence&quot;: 0-1 (how confident you are in your answer)<br>    - &quot;reason&quot;: brief explanation for your answer<br>    &quot;&quot;&quot;</pre><p>The relevance prompt evaluates whether the document is primarily about heat pump technology and returns a JSON containing a categorisation of whether it’s a heat pump, the confidence, and the LLM’s rationale for the categorisation.</p><p><strong>Step 3 — LLM-Powered Tech Classification:</strong></p><p>Next, our LLM processed the filtered subset, prompting it to extract structured categories from a taxonomy table. The goal was automatically scanning thousands of items, tagging each with consistent, human-readable categories without having to manually code every record.</p><pre>SYSTEM_MESSAGE = &quot;&quot;&quot;<br>    You are a sustainable-heating technology analyst evaluating Heat Pump innovations.<br><br><br>    Extract structured information about Heat Pump technologies from the provided text.<br><br><br>    Unless requested otherwise, adhere as precisely as possible to the language and text that is used in the provided text document.<br><br><br>    If the requested information is not described, return N/A. DO NOT make up any false information or false inferences.<br>&quot;&quot;&quot;</pre><pre>    processor = batch_check.LLMProcessor(<br>        model_name=&quot;gpt-4o-mini&quot;,<br>        temperature=0,<br>        output_path=str(INTERIM_OUTPUT_FILE),<br>        system_message=SYSTEM_MESSAGE,<br>        session_name=&quot;classify_heat_pumps&quot;,<br>        output_fields=fields,<br>    )</pre><p>Where LLMProcessor is a custom class for running multiple concurrent calls to the LLM, and the parameter fields defines the taxonomy table to be used for extraction.</p><p>Input example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9z9jDSDAjGS-kS6XBt5_w.png" /><figcaption>Extract of filtered data.</figcaption></figure><p>Taxonomy table example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4y3LL7eD4cNsNv90rjyfeg.png" /><figcaption>Extract of taxonomy table.</figcaption></figure><p>Outcome example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CW8Jf4HUTN1qWj8vpRcH-w.jpeg" /><figcaption>Extract of classified heat pump innovations.</figcaption></figure><p><strong>Step 4 — Manual Testing Protocol:</strong></p><p>To assess how the model performed, we implemented a manual testing protocol based on human labelling. By comparing a random selection of documents with the LLM’s outputs, we could calculate key performance metrics such as accuracy, precision, and recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuBAo5mA9e8JI6eqUllKpw.png" /><figcaption><em>Innovation classification performance shows high accuracy but modest precision and recall (both ~73%) due to class imbalance. Positives are ~11% of cases (33/300).</em></figcaption></figure><p><strong>Step 5 — Quantify and visualise trends</strong>:</p><p>With our data categorised and validated, the next phase involved transforming it into actionable insights. We produced data visualisations that highlighted trends over time, showcasing category shares, the focus on specific components, and application contexts for different heat pump types. For more details, read “Part II: Results”. Through charts and tables, we provided clear insights that could inform policy decisions, funding opportunities, and industry engagement.</p><p><strong>Limitations and Caveats</strong></p><ol><li><strong>Data &amp; coverage</strong></li></ol><ul><li><strong>Search/taxonomy bias:</strong> Tech-heavy queries may underweight interdisciplinary work (academic).</li><li><strong>Research underrepresentation</strong>: Adding other fields, such as “primary_topic”, could improve the outcome.</li><li><strong>Language bias:</strong> While we used English translations of patent summaries, the English-only analysis risks missing innovations in key non-English regions, particularly in academic publications.</li><li><strong>Source bias:</strong> Using one patent database and one research database can underrepresent relevant information.</li><li><strong>Abstract-only parsing:</strong> Skipping full specifications/text may lower precision on niche claims or context.</li></ul><p><strong>2. LLM performance</strong></p><ul><li><strong>TRL reliability: </strong>Low accuracy (~50–55%); fine-grained TRL scores (1–9) were excluded, and only broad phases were referenced in the second test; however, it improved by 5%. TRL outputs were treated as hypotheses. It does not assert actual maturity.</li><li><strong>LLM:</strong> Future model updates could change outputs; reproducibility depends on prompts and models.</li></ul><p><strong>3. Methodology &amp; interpretation</strong></p><ul><li><strong>Time window (2020–2025):</strong> Good for recent signals, limited for long-run baselines.</li><li><strong>Taxonomy limits:</strong> Even a comprehensive schema won’t capture every emerging dimension.</li><li><strong>Counts ≠ impact:</strong> Volume indicates attention, not adoption, performance, or emissions outcomes.</li></ul><p><strong>Reproducibility &amp; reuse</strong></p><p>One of the good things about this method is that the pipeline is a small, repeatable workflow: i) collect data, ii) filter relevant information, iii) label and classify, iv) human evaluation, v) visualise data. It was built for heat pumps, but the same pattern can work for different topics (e.g., Refrigerants, heat networks, retrofit materials, and so on).</p><p><strong>How to reuse this approach for other topics?</strong></p><p>The decision tree below illustrates how this approach could be applied for other topics of interest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiIq-cUwkAtl3BbWAVeWHA.png" /><figcaption><em>Illustrative example of the steps for using the method. For the complete scripts, visit the </em><a href="https://github.com/nestauk/discovery_heat_pump_futures/blob/488ce9e09527e9cc7beeb1a7e54ac7950644b877/discovery_heat_pump_futures/pipeline/classify_innovations.py"><em>GitHub repository</em></a><em>.</em></figcaption></figure><p><strong>Conclusions</strong></p><p>This method shows how lightweight Python scripts, supported by LLM tools, can present new opportunities to decarbonise heating (and other fields) through innovation.</p><p>Transforming numerous documents into clear, actionable evidence, even with limited programming knowledge, can offer new perspectives for tackling specific topic challenges. The pipeline we’ve built is flexible and straightforward to adapt to new applications in sustainable technology.</p><p>The following blog, “Part II: Results”, will present a visualisation of the data, highlighting the most interesting trends through a Circular Economy perspective and providing helpful information for policy and industry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36e65f2c6f0d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/mapping-heat-pump-innovation-via-llms-a-circular-economy-approach-to-sustainable-heating-36e65f2c6f0d">Mapping Heat Pump Innovation via LLMs: A Circular Economy Approach to Sustainable Heating</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerating evidence synthesis: how AI is transforming systematic reviews]]></title>
            <link>https://medium.com/discovery-at-nesta/accelerating-evidence-synthesis-how-ai-is-transforming-systematic-reviews-5ef4a87a3869?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/5ef4a87a3869</guid>
            <category><![CDATA[policy]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[evidence-synthesis]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Karlis Kanders]]></dc:creator>
            <pubDate>Fri, 23 May 2025 15:37:46 GMT</pubDate>
            <atom:updated>2025-05-23T16:02:10.936Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article was co-written by </em><a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/"><em>Luke McNally</em></a><em> and Karlis Kanders.</em></p><p>Systematic reviews are the gold standard for synthesising evidence in fields ranging from healthcare to education to social policy. By rigorously collecting, evaluating, and summarising all relevant research, systematic reviews provide crucial insights that guide interventions, policymaking, and future studies.</p><p>However, the very thoroughness that gives systematic reviews their value also makes them exceptionally time-consuming — often taking months or even years to complete. As the volume of academic literature <a href="https://direct.mit.edu/qss/article/5/4/823/124269/The-strain-on-scientific-publishing">continues to skyrocket</a>, researchers, policymakers, and innovators alike are left sifting through an ever-growing mountain of information.</p><p>The stakes are high: delayed assessment of scientific literature can mean slower progress on important issues. Moreover, policy development and decision making that happens without an up-to-date evidence base will result in sub-optimal outcomes and poorer value for public money.</p><p>At <a href="https://www.nesta.org.uk/">Nesta</a>, we address society’s most pressing challenges around sustainability, healthy life and early years education. As part of this, we are exploring how to enable faster and possibly more thorough ways of synthesising evidence. That’s why we’ve been working on an <strong>AI-powered pipeline</strong> that automates parts of the literature review process <strong>for the purposes of policy interventions</strong>.</p><p>We are keen to work in the open and share our learnings as they emerge. In this article, we’re describing our early work in-progress on how we trialled <strong>OpenAlex,</strong> <strong>large language model-powered data extraction</strong>, and <strong>ontology mapping </strong>during Nesta’s <a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/">data science residency</a><strong>. </strong>We’ll continue building on this work to develop tools <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">for transforming evidence use in policymaking</a>.</p><h3>Why automate systematic reviews?</h3><p>A traditional systematic review involves multiple painstaking steps:</p><ol><li>Formulating a specific research question</li><li>Manually searching databases like PubMed, Scopus, or Google Scholar</li><li>Screening titles and abstracts for relevance</li><li>Extracting essential data (eg, population characteristics, interventions, outcomes, effect sizes) from full-text articles</li><li>Synthesising findings into a coherent summary or meta-analysis</li></ol><p>Each of these steps is time- and labor-intensive, vulnerable to human error, and potentially prone to inconsistencies. By integrating AI at strategic points in the pipeline, we can reduce manual effort, increase consistency, and let researchers focus on interpreting and applying results rather than on data wrangling.</p><h3>The AI-powered literature review pipeline</h3><p>Our prototype pipeline presently automates three core steps in evidence synthesis:</p><ol><li><strong>Searching the literature:</strong> using <a href="https://openalex.org/">OpenAlex</a> to fetch relevant research papers</li><li><strong>Extracting key information:</strong> employing large language models (LLMs) to extract structured data from abstracts and full texts</li><li><strong>Ontology mapping:</strong> Standardising extracted information by using embeddings (numerical representations of text) to group related terms into common categories</li></ol><p>Upcoming work will also develop methods to construct a system network model from the processed evidence data, and use the model to simulate impact from policy interventions acting on components of the system.</p><p>The result will be a streamlined process that shortens the time from a research question to actionable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nFItjX4JsYaRd-w95IQnA.png" /><figcaption>Pipeline for accelerating evidence synthesis with AI. Boxes with dashed lines indicate upcoming future work.</figcaption></figure><p>Now, there are already emerging solutions for conducting literature reviews with AI, such as ChatGPT and Gemini’s Deep Research, <a href="https://www.undermind.ai/">Undermind</a>, <a href="https://elicit.com/">Elicit</a>, <a href="https://www.futurehouse.org/">FutureHouse</a> and others. However, there is value in building tailored pipelines, as tools like Deep Research are not guaranteed to be exhaustive or fit for purpose when it comes to exploring or designing policy interventions. Moreover, users have limited control over the operations and sources of such tools.</p><h3>Step 1: Scalable literature search with OpenAlex</h3><p>Instead of manually scouring databases, our pipeline taps into <strong>OpenAlex</strong>, an open-source academic search tool. This automates the initial search, filtering thousands of potential hits down to the most relevant studies.</p><p>The following prototype code is from our <a href="https://github.com/nestauk/dsp_systems_modelling/tree/mvp_cli">GitHub repo</a>, which is a work in progress.</p><pre>from openalex_utils import fetch_openalex_results<br><br>search_term = &quot;early childhood interventions and learning outcomes&quot;<br>user_email = &quot;researcher@example.com&quot;<br>results_df = fetch_openalex_results(search_term, user_email, min_cites=10, n_works=500)</pre><p><strong>How it works:</strong></p><ul><li>The user provides a <strong>search term</strong> like “early childhood interventions and learning outcomes.”</li><li>The pipeline queries <strong>OpenAlex</strong>, retrieving papers based on relevance and citation count.</li><li>Results are saved as structured metadata, including abstracts and links to full texts.</li></ul><p>This approach reduces the need for manual searching, significantly speeding up a time-consuming step of a systematic review.</p><p>More research, however, is needed to ascertain whether OpenAlex queries retrieve sufficiently accurate references, which are comparable with manual search results.</p><h3>Step 2: AI-powered data extraction</h3><p>After retrieving relevant papers, we need to pull out key insights. Traditionally, researchers would manually comb through abstracts and full texts, noting elements like population characteristics, interventions, outcomes, and results. This is a manual and laborious process.</p><h4>Large language models</h4><p>We leverage large language models developed by OpenAI for automated data extraction. We request <a href="https://python.langchain.com/docs/concepts/structured_outputs/"><strong>structured outputs</strong></a> in a predefined schema, rather than free-text responses, to ensure machine-readability and consistency.</p><pre>from ai_utils import extract_meta_info<br><br>paper_text = &quot;This study investigates the impact of early childhood interventions...&quot;<br>meta_info = extract_meta_info(paper_text)</pre><p>To maintain consistency, we define a strict schema using <strong>Pydantic</strong>:</p><pre>from pydantic import BaseModel<br><br>class MetaInfo(BaseModel):<br> study_title: str<br> population_outcome_measured_in: str<br> population_intervention_affected_or_predictor: str<br> secondary_characteristics: str<br> country: str<br> study_type_letter: str<br> num_main_results: int<br> main_results_list: str</pre><p><strong>How this works:</strong></p><ul><li>LLM is <strong>prompted</strong> with a predefined schema, ensuring outputs match expected fields.</li><li>Data is extracted into key-value pairs, making it easy to analyse and merge with other datasets.</li><li>Missing or unavailable information is explicitly marked as “NA” reducing ambiguity.</li></ul><p><strong>Example output</strong>:</p><pre>{<br>  &quot;study_title&quot;: &quot;Study on Parenting Strategies&quot;,<br>  &quot;population_outcome_measured_in&quot;: &quot;Children aged 2-4&quot;,<br>  &quot;population_intervention_affected_or_predictor&quot;: &quot;Parents of children aged 2-4&quot;,<br>  &quot;secondary_characteristics&quot;: &quot;Families from urban areas&quot;,<br>  &quot;country&quot;: &quot;USA&quot;,<br>  &quot;study_type_letter&quot;: &quot;g&quot;,<br>  &quot;num_main_results&quot;: 3,<br>  &quot;main_results_list&quot;: &quot;Parenting education improved child mental health; Parenting education increased school readiness; Parenting education reduced parental stress&quot;<br>}</pre><h4>Three-pass extraction process</h4><p>Our pipeline often requires extracting different “layers” of information. That’s why we designed a <strong>three-pass</strong> system:</p><pre>from extract_data import run_three_pass_extraction<br><br>pdf_folder = &quot;./downloaded_papers&quot;<br>output_csv = &quot;extraction_results.csv&quot;<br>user_items = [&quot;Did the study control for socio-economic status?&quot;,<br>           &quot;What was the balance between the control and intervention groups?&quot;]<br><br>run_three_pass_extraction(pdf_folder, user_items, output_csv, model=&quot;gpt-4o&quot;)</pre><p><strong>Pass 1: Metadata extraction:</strong> Captures essential fields like study title, population details, country, and descriptions of main results.</p><p><strong>Pass 2: Detailed results extraction:</strong> Identifies numerical outcomes, effect sizes, statistical measures, and any other critical research findings for each main result.</p><p><strong>Pass 3: User-specified variables:</strong> Allows researchers to pose specific follow-up questions (eg, “Did the study control for socio-economic status?”) for a deeper, customised dive.</p><p>We have experimented with different LLM versions and, as expected, the quality of results appears to improve with more advanced models. While gpt-4o-mini might provide a reasonable starting point, models such as o3-mini and gpt-4.1-mini can handle more complex schemas and generate better results.</p><h3>Step 3: Ontology mapping for standardisation</h3><p>Extracted data often includes varying terminologies. One paper might use “early childhood education,” another might say “preschool”. With <strong>ontology mapping</strong>, we transform these disparate terms into standardised concepts, enabling easier comparisons across studies.</p><p>By leveraging text embeddings (numerical representations of text), our pipeline can recognise that “preschool,” “nursery,” and “kindergarten” are related categories, ensuring a more consolidated view of the evidence base.</p><h3>Early results</h3><p>In a test focusing on early childhood interventions, our AI-driven pipeline processed over 2,000 academic full text papers in under 48 hours. This can be further significantly increased through parallelisation. Manually, this could have taken weeks for a small team.</p><p>The process can also be sped up by initially only focussing only on abstracts, where 10,000s of abstracts can be processed in a matter of a few hours. While abstracts contain limited information about the studies, they can be used to screen the search results for relevance, prior to the more detailed data extraction.</p><p>Preliminary spot checks suggest that the model correctly identifies key data points (eg, interventions, outcomes, country) most of the time.</p><h3>Challenges and limitations</h3><h4>Data quality and bias</h4><p>No AI model is perfect and large language models can inherit biases from their training data, and not all abstracts and papers are equally well-structured or detailed. These factors may affect extraction accuracy. We can address this, for example, by:</p><ul><li>Performing <strong>spot checks</strong> of extracted data, and engaging domain experts for <strong>human-in-the-loop</strong> validation.</li><li>Incorporating <strong>validation rules</strong> (like Pydantic schemas) to flag anomalous or incomplete entries.</li></ul><h4>Ethical use of AI in research</h4><p>Accelerating literature reviews doesn’t eliminate ethical considerations. When dealing with sensitive topics, we need transparency in how AI models include or exclude certain literature sources and arrive at certain conclusions. Therefore, it will be important to document the AI system’s decision-making steps as it produces the literature review.</p><p>One should also consider the energy usage of such pipelines and their sustainability. While individual calls to OpenAI API are <a href="https://www.sustainabilitybynumbers.com/p/carbon-footprint-chatgpt">estimated to have a comparatively small energy footprint</a> (~3Wh) this could quickly add up when processing thousands of research papers. This should be contrasted with the energy expended when performing manual reviews (while taking into account the other benefits of accelerated, broader evidence synthesis).</p><h3>Future directions</h3><h4>Building network models</h4><p>We will expand the pipeline to allow users to automatically create <strong>network models</strong> of an area of interest, and perform simple intervention simulations using these models. You can see <a href="https://youtu.be/csiBMtkLvXk?si=AS9Vb3lWACIQMLm1&amp;t=207">here an early prototype in action</a>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcsiBMtkLvXk%3Fstart%3D207%26feature%3Doembed%26start%3D207&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcsiBMtkLvXk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcsiBMtkLvXk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea73a7fc505a732f9e8f2f065463bf17/href">https://medium.com/media/ea73a7fc505a732f9e8f2f065463bf17/href</a></iframe><h4>Expanded data sources</h4><p>While <strong>OpenAlex</strong> is an excellent starting point with <a href="https://openalex.org/stats">260M+ indexed publications</a>, we will need to extend our reach to other databases and grey literature. The more comprehensive our search, the richer our final dataset.</p><h4>Community collaboration</h4><p>We are already testing elements of this pipeline internally at Nesta, and we are keen to work in the open, collaborate with diverse set of partners, and we’ll <strong>open-source</strong> most of this work. By pooling resources and expertise, we can continually refine the approach to tackle diverse research questions.</p><h3>Harnessing AI to improve policy design</h3><p>In a world grappling with complex challenges, using the best available evidence is paramount. As agentic AI research pipelines are already <a href="https://www.futurehouse.org/research-announcements/demonstrating-end-to-end-scientific-discovery-with-robin-a-multi-agent-system">claimed to make scientific discoveries</a>, we also see a huge potential for AI-powered workflows to <a href="https://www.nesta.org.uk/project/policy-atlas-harnessing-ai-to-improve-policy-design/">support policymaking</a> and offer a faster route to evidence-based insights on policy interventions.</p><p>If you wish to collaborate with us on AI-powered evidence synthesis, please reach out <a href="/service/mailto:discoveryhub@nesta.org.uk">on our email</a>.</p><p><a href="https://www.nesta.org.uk/project-updates/nesta-welcomes-data-science-resident/"><em>Luke McNally</em></a><em> is a data scientist based at the University of Edinburgh, and recently a resident at Nesta. He applies artificial intelligence and machine learning to a range of projects, including antimicrobial resistance and AI chatbots. During his residency at Nesta, he was developing methods to improve evidence synthesis in policymaking.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ef4a87a3869" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/accelerating-evidence-synthesis-how-ai-is-transforming-systematic-reviews-5ef4a87a3869">Accelerating evidence synthesis: how AI is transforming systematic reviews</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use AI to summarise decades of political discourse]]></title>
            <link>https://medium.com/discovery-at-nesta/how-to-use-ai-to-summarise-decades-of-political-discourse-fe9faf9554cf?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/fe9faf9554cf</guid>
            <category><![CDATA[discourse-analysis]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[text-summarization]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Rinskejongma]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 12:17:39 GMT</pubDate>
            <atom:updated>2024-11-06T12:17:39.476Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Rinske Jongma collaborated with Nesta’s </em><a href="https://www.nesta.org.uk/discovery/"><em>Discovery Hub</em></a><em> over the Summer 2024 on her Master thesis “Leveraging Large Language Models for Political Discourse Analysis”</em></p><p>Traditional machine learning models for automated multi-document summarisation are highly complex and require extensive fine-tuning to perform effectively. Large language models (LLMs) can offer a more streamlined alternative, as they often don’t require task-specific fine-tuning for summarisation tasks. In this article I evaluate whether LLMs could effectively summarise policy discussions.</p><p>As our test case, I considered debates on heat pumps from UK House of Commons speeches between 2005 and 2024. Heat pumps are a very efficient low-carbon heating technology, and Nesta’s <a href="https://www.nesta.org.uk/sustainable-future/">sustainable future mission</a> is focussed on decarbonising UK homes, including facilitating the adoption of this technology. As shown in Discovery Hub’s earlier research on <a href="https://www.nesta.org.uk/feature/innovation-sweet-spots-green-tech/">green technology innovation trends</a>, it is useful to analyse the political discourse on this and other low-carbon technologies, to understand policymakers’ positions and assess the chances of such innovations to achieve large-scale impact and societal benefit.</p><p>The timeframe of this study spans the previous five parliamentary terms. Over this period, I found a total of 292 speeches related to heat pumps, with the number of speeches in each parliamentary term being 68, 32, 5, 7, and 180, respectively. The substantial number of speeches, particularly for the first and fifth parliamentary terms highlights the appeal for an automated approach to summarising discourse. Manual summarisation would be time-consuming and particularly impractical for exploratory research.</p><p><strong>Summarising parliamentary speeches using a chain-of-thought approach</strong></p><p>The method used here to summarise the political discourse surrounding heat pumps can be considered as a <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG">retrieval-augmented generation (RAG) pipeline</a>. The approach is inspired by the work of <a href="https://link.springer.com/article/10.1007/s13042-024-02225-0">Zhang and colleagues</a> who employed a <a href="https://www.ibm.com/topics/chain-of-thoughts#:~:text=How%20does%20chain%20of%20thought,reasoning%20chains%20for%20novel%20tasks.">chain-of-thought pipeline</a> and tested different versions of <a href="https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-language-models-are-developed">OpenAI</a>’s generative pre-trained transformer (GPT) model to generate summaries of news articles and scientific papers. They found that summaries generated using GPT and the chain-of-thought approach were comparable with more sophisticated, state-of-the-art fine-tuned models for multi-document summarisation. Given the effectiveness of Zhang’s approach, I have adapted it here for summarising political speeches.</p><p>The chain-of-thought pipeline involves two consecutive steps: an attend prompt and a summary prompt.</p><p><strong>Attend prompt: </strong>This step involves prompting the LLM to extract key information from each document in the dataset. For this study that meant instructing the LLM to respond to the following four questions:</p><ul><li>In what context are heat pumps mentioned?</li><li>In what year was the speech held?</li><li>What was the name of the Member of Parliament (MP) who gave the speech?</li><li>To which party belonged the speaker of this speech?</li></ul><p><strong>Summary prompt: </strong>This step prompts the LLM to summarise the key information retrieved across all documents using the attend prompt. For this study I tested two different prompts to generate a summary:</p><ul><li>Prompt 1: “Summarise this information”</li><li>Prompt 2: “Summarise this information and make sure to include everything that is important.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*7fk8qGH6O0kzigpf93Naxw.png" /><figcaption><em>Chain-of-thought pipeline for summarisation</em></figcaption></figure><p><strong>Implementation of the chain-of-thought pipeline</strong></p><p>For the following analyses, I used the GPT-3.5 model (gpt-3.5-turbo), which at the time of conducting this research was among the most cost-efficient LLMs. To implement the above pipeline into Python, the following two functions were applied.</p><p><strong>Attend_prompt(text, questions):</strong> This function takes as input the text you want to retrieve the key information from (in this case the speech) and a list that contains all the questions you want the LLM to answer based on the text. The function then combines the list of questions into one prompt that is presented to the LLM. The function returns a list which contains the answers to all the different questions.</p><pre>def attend_prompt(text, questions):<br> <br>    prompts = questions<br>    <br>    responses = []<br>    for prompt in prompts:<br>       <br>        combined_content = f&quot;context: {text}\n\nUser: {prompt}&quot;<br><br>        simple_prompt = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: combined_content}]<br><br>        chatgpt_output = client.chat.completions.create(<br>            model=&quot;gpt-3.5-turbo&quot;,<br>            messages=simple_prompt,<br>            temperature=0.3,<br>            max_tokens=1000,<br>        )<br>        gpt_output = chatgpt_output.to_dict()<br>        message_content = gpt_output[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;]<br>        responses.append(message_content)<br>    <br>    return responses</pre><p>To retrieve key information from all documents in the dataset, a wrapper function is used that iterates over the entire dataset and calls the attend prompt for every document.</p><p><strong>Summary_prompt(instruction, info_to_summarise):</strong> This function receives the key information from multiple documents, gathered via the attend prompt, along with an instruction to summarise the content. It then produces a concise summary of the key information present in your documents.</p><pre>def summary_prompt(instruction, info_to_summarise):<br><br>    prompt = instruction<br>    combined_content = f&quot;Information: {info_to_summarise}\n\nUser: {prompt}&quot;<br><br>    simple_prompt = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: combined_content}]<br>    <br>    chatgpt_output = client.chat.completions.create(<br>            model=&quot;gpt-3.5-turbo&quot;,<br>            messages=simple_prompt,<br>            temperature=0.3,<br>            max_tokens=1000,<br>        )<br>    gpt_output = chatgpt_output.to_dict()<br>    response = gpt_output[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;]<br><br>    return response</pre><p>For a complete implementation, please refer to <a href="https://github.com/nestauk/discovery_utils/blob/dev/policy_scanning/summarising_discourse/Summary_RAGAS.ipynb">this notebook</a>. The figure below illustrates a summary generated as a result of applying this pipeline on speeches during one parliamentary term.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*-YluSvqJ9AbT4Bt37Qw10g.png" /><figcaption><em>Example of a summary generated with the chain-of-thought pipeline</em></figcaption></figure><p><strong>Effective prompting</strong></p><p>Although LLMs have been widely adopted in various real-world applications, research emphasises the need to use them with caution. This is because LLMs have the tendency to generate inaccurate or hallucinated information. To address this issue, I applied several techniques aimed to reduce the likelihood of such hallucinations. Research of <a href="https://arxiv.org/abs/2309.02772">Zhu and colleagues</a> and <a href="https://arxiv.org/abs/2402.05201">Renze And Guven’s</a> suggests that LLMs operating at lower temperature settings yield more accurate and coherent responses. For further details on the temperature settings of LLMs, please refer to this<a href="https://www.iguazio.com/glossary/llm-temperature/#:~:text=The%20temperature%20is%20a%20numerical,distribution%20of%20the%20next%20word."> link</a>. In my analysis, I tested temperature settings of 0.5 and 0.3.</p><p>Moreover, <a href="https://arxiv.org/abs/2302.11382">researchers</a> found that clearer, more specific prompts that precisely define what the LLM should do and what information it should include in its generated response tend to produce better results. Therefore, I applied the summary prompt twice: first with a generic prompt and then with a more specific one.</p><p><strong>Evaluating the summarisation results</strong></p><p>To ensure that the generated summaries accurately reflect the content of the speeches and did not produce any severely hallucinated results, both the attend prompt and the summary prompt were evaluated.</p><p><em>Attend prompt evaluation</em></p><p>The attend prompt was first tested on a subset of 15 randomly selected speeches before it was applied to the entire dataset. This sample included 5 short speeches, 5 medium-length speeches and 5 long speeches to investigate whether the input length of the speech had an effect on the LMM’s performance. This step was crucial because if the attend prompt produced inaccurate answers, the summary prompt would inevitably generate inaccurate summaries.</p><p>The attend prompt was tested with two different temperature settings of 0.5 and 0.3. This was done to explore whether reducing the temperature value would improve the accuracy of the attend prompt’s results, as suggested by previous research.</p><p>The LLM responses were assessed using both human evaluation and the automated RAGAS (Retrieval Augmented Generation Assessment) framework, developed by <a href="https://arxiv.org/abs/2309.15217">Es and colleagues</a>. The RAGAS framework was specifically chosen because it is designed to evaluate RAG pipelines like the one used here. It provides various metrics to measure the accuracy of LLM-generated responses in an automated way. For this evaluation the following three RAGAS metrics were chosen:</p><ul><li>Faithfulness: assesses whether the LLMs answers are factually consistent by checking the model’s generated response and source documents.</li><li>Answer relevance: measures whether the response is complete and free from redundant information by checking the model’s response and user’s query.</li><li>Context precision: evaluates whether the response prioritises the most relevant or important information by checking the model’s generated response and the context information that was retrieved from the source documents and used to generate the response</li></ul><p>RAGAS employs text embeddings and LLM-based assessment to provide scores between 0 and 1, with higher scores indicating better performance. However, the effectiveness and accuracy of the RAGAS framework have not been thoroughly studied. Moreover, because it uses one LLM to evaluate another, some researchers caution that this approach could reinforce existing biases and inaccuracies in LLMs, potentially misrepresenting the true performance of the model being assessed. Therefore, human evaluation was also conducted to verify whether the RAGAS scores accurately reflect the accuracy of the responses. Human scores were assigned to the LLM generated responses by adhering closely to the definition of the three metrics presented above. This ensured that comparison between the human scores and RAGAS scores were possible. The graphs below show the differences between human scores and RAGAS scores for both temperature values across all three metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KyPk6qjlnYFIEhNq" /><figcaption>Difference between human scores and RAGAS scores for temperature values 0.5 and 0.3 for the RAGAS metrics: faithfulness, Answer Relevance and Context Preciscion</figcaption></figure><p>The human evaluation scores were high across all metrics and speech lengths, indicating that the attend prompt effectively retrieves key information from the speeches. Additionally, there is a reasonable alignment between the RAGAS and human scores for answer relevance and context precision, suggesting that these metrics reliably assess the effectiveness of the attend prompt. However, the RAGAS scores for faithfulness diverge significantly from human evaluations, indicating that RAGAS does not effectively measure this aspect. Notably, evaluation scores were highest at a temperature setting of 0.3 for all three metrics. Consequently, the pipeline was applied to the entire dataset using a temperature value of 0.3.</p><p><em>Summary prompt evaluation</em></p><p>The RAGAS framework also generates a summary score that evaluates how effectively the LLM-generated summary captures key information from the original input. To calculate this score, RAGAS extracts key phrases from the original input and formulates yes/no questions, which should all yield “yes” when answered based on the original text. The summary is then assessed by answering these questions, and the ratio of correctly answered questions is calculated to produce a score between 0 and 1, with higher scores indicating better performance. For a detailed breakdown of the calculation, please refer to <a href="https://docs.ragas.io/en/stable/concepts/metrics/summarization_score.html">RAGAS documentation</a>. In this study, the original input consists of key information retrieved from the attend prompt, which is compared to the generated summary.</p><p>The graphs below show the calculated summary scores for Prompt 1 (the more generic prompt) and Prompt 2 (the more focused and specific prompt).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RzGKvkBrFx6_e8QT" /><figcaption>Two graphs depicting RAGAS summarisation scores by parliamentary term (graph 1) and token length (graph 2)</figcaption></figure><p>The graphs reveal that the RAGAS score is higher for 3 out of the 5 summaries for Prompt 2. For both prompts, the lowest RAGAS scores correspond to Parliamentary Terms 3 and 4 (refer to the figure on the left). These terms have the fewest speeches related to heat pumps. This indicates that GPT-3.5 may struggle to produce accurate summaries when data is limited. However, the relatively high RAGAS scores — particularly for Prompt 2 — suggest that the LLM-generated summaries effectively capture the main points of the original text. The second graph indicates that for both prompts the larger the input text the better the summary scores tend to be.</p><p><strong>Validating summaries with topic modelling</strong></p><p>Due to time constraints, a detailed human evaluation of the summary prompt, similar to that for the attend prompt, was not feasible. As a result, the accuracy of the calculated RAGAS summary scores could not be directly validated. To still gauge the accuracy of the summary scores, I applied BERTopic to the heat pump speeches and compared the generated topics with topics that were manually identified from the five summaries. This manual process involved reading the summaries and colour-coding recurring topics.</p><p>The comparison between BERTopic and the manually identified topics revealed strong alignment, suggesting that the summaries effectively captured the key discussions on heat pumps. Furthermore, the comparison revealed that the summaries were able to provide additional details about the topics identified by BERTopic. However, the summaries frequently overlooked numerical insights that are essential for fully understanding the narrative surrounding each topic. In contrast, the BERTopic model consistently uncovered these numerical facts. To illustrate, the LLM generated summaries identified multiple initiatives promoting heat pumps installations, identifying these initiatives by name. However, they omitted specific numerical targets or grants linked to these initiatives. These details were consistently highlighted by BERTopic. While this could be possibly addressed by further prompt engineering, this example underscores the complementary nature of automated topic modelling and LLM-based summaries in grasping the complete narrative of a discourse.</p><p>The alignment of topics from both models corresponds with the relatively high summary scores calculated by RAGAS. However, this alignment only suggests that the RAGAS summary scores reflect the summaries’ accuracy; it does not serve as validation. To properly validate the RAGAS summary scores, they should be compared with reference scores derived from human evaluations or other computational methods.</p><p><strong>Conclusion</strong></p><p>This research demonstrates the potential of LLMs for automating multi-document summarisation, particularly in the context of political discourse. By employing a RAG pipeline, the study effectively leveraged LLMs to extract key information and generate coherent summaries without the need for extensive fine-tuning. The evaluations revealed that the attend prompt effectively retrieved key information. Furthermore, especially when using a more focused prompt the summary prompt effectively captures this key information.</p><p>Although the RAGAS framework provided valuable metrics for assessing summary quality, the findings highlight the necessity of integrating human evaluation to fully understand the accuracy of LLM-generated outputs.</p><p>Overall, the results underscore the promise of LLMs in facilitating the analysis of large volumes of political discourse, offering a streamlined approach to tracking evolving narratives in a timely and efficient manner. Future work should aim to refine the evaluation methods further and explore additional applications of LLMs in summarising complex datasets.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe9faf9554cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/how-to-use-ai-to-summarise-decades-of-political-discourse-fe9faf9554cf">How to use AI to summarise decades of political discourse</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Catalysing innovation for missions outside government]]></title>
            <link>https://medium.com/discovery-at-nesta/catalysing-innovation-for-missions-outside-government-b168521eb8cf?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/b168521eb8cf</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Laurie Smith]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 15:39:29 GMT</pubDate>
            <atom:updated>2024-09-10T08:54:50.763Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aW-RA4LwIo-ybJnr" /></figure><p>The government’s <a href="https://labour.org.uk/change/mission-driven-government/">five missions</a> need to stretch beyond the public sector. They can only succeed by marshalling ingenuity from across industry and civil society, for example, to unlock a breakthrough technology or to commercialise and scale a new product.</p><p>The state can play a critical role to help this innovation happen in the direction that’s needed, and at pace and scale. However, within public services, innovations are often slow to diffuse. And even in private markets, diffusion can be sluggish. This is where missions come in.</p><p>Yet those beyond government are unlikely to spontaneously rally around missions without some sort of help. A mission-driven government therefore needs a suite of mechanisms to bring about mission-driven innovations in other parts of society and the economy. Many of these are now well-proven, and in the context of the Department for Science, Innovation and Technology (DSIT), four stand out:</p><ol><li>Challenge prizes and pre-procurement</li><li>Venture studios</li><li>Regulatory sandboxes and real-world testbeds</li><li>Opening up data.</li></ol><p>Between them, these four methods provide both supply-side and demand-side support (see figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*LAHuTf30I0JEKQlQ" /></figure><p><strong><em>Figure 1: Orientation of mission-driven innovations methods. </em></strong><em>Venture studios are more on the supply side as they supply businesses and challenge prizes are more on the demand side as they create demand for innovation.</em></p><p>These tools can be effectively used in combination. For example, the 2017 <a href="https://www.nesta.org.uk/blog/introducing-the-open-up-challenge/">Open Up Challenge</a> run by Challenge Works for the Competition and Market Authority (CMA) included a £5 million challenge prize that was timed to coincide with the opening up of banking data and included a sandbox of synthetic data.</p><p>Deploying alternative tools to manage innovation systematically should form part of a <a href="https://media.nesta.org.uk/documents/What_does_a_mission-driven_approach_to_government_mean_and_how_can_it_be_delivered.pdf">wider strategy to harness innovation to achieve missions</a> that includes creating the conditions for public sector innovation and investing in R&amp;D.</p><p>This article draws on Nesta’s work on innovation methods such as the<a href="https://www.nesta.org.uk/report/compendium-innovation-methods/"> Innovation Methods Compendium</a> and <a href="https://www.nesta.org.uk/report/20-tools-innovating-government/">20 tools for Innovating in Government</a>.</p><h3>Policy context</h3><p>Traditionally, the UK government has relied on a narrow range of methods, such as grants, to stimulate innovation. From the 1980s, the state’s focus shifted to supporting basic research, with applied innovation left largely to the private sector. Investment in R&amp;D by most government departments declined, becoming concentrated in the research councils (now UKRI). Policies to support innovation in industry, such as R&amp;D tax credits, were often sector-neutral.</p><p>Over the last 20 years, new innovation methods such as challenge prizes and venture studios have gained traction. The UK government has also more actively encouraged collaboration between the private and public sectors and adopted a more active industrial strategy, choosing specific sectors to focus on rather than remaining neutral.</p><p>A mission-driven government approach offers the chance to build on this shift and would involve redirecting a modest fraction of the <a href="https://researchbriefings.files.parliament.uk/documents/SN04223/SN04223.pdf">UK’s £12.8 billion public sector R&amp;D spend</a> towards mission-driven innovation through these newer methods. While the current government has not committed to concrete research investment targets in its manifesto, it has committed to <a href="https://labour.org.uk/wp-content/uploads/2024/06/Change-Labour-Party-Manifesto-2024-large-print.pdf">sustained ten-year budgets for certain research institutions</a> and subsequently to <a href="https://www.independent.co.uk/news/uk/politics/angela-rayner-wiltshire-britain-conservatives-tories-b2566268.html">sectors that form part of its industrial strategy</a>.</p><h3>Innovation methods for missions</h3><h3>1. Challenge prizes and pre-procurement</h3><p><a href="https://challengeworks.org/about-challenge-prizes/">A challenge prize</a> is a “competition that offers a reward to whoever can solve a problem first or most effectively”. <a href="https://media.nesta.org.uk/documents/Compendium-of-Innovation-Methods-March-2019.pdf">Pre-defined criteria</a> describe success, without indicating how exactly it should be achieved. <a href="https://media.nesta.org.uk/documents/Compendium-of-Innovation-Methods-March-2019.pdf">Prizes</a> “go beyond the usual suspects to reach innovators that other funding mechanisms miss.”</p><p>A similar method is the use of pre-procurement instruments such as Innovate UK’s Contracts for Innovation (previously known as SBRI) where organisations compete for funding to deliver innovative services or products around a particular challenge. Funding is awarded across the programme rather than through a prize at the end.</p><h4>How this method can help achieve missions</h4><p>Challenge prizes are best suited for problems such as missions, which are defined well enough that a clear and unambiguous goal for innovators can be set. They can help unlock progress against measurable milestones, unleash creative ways of seizing promising opportunities and attack known bottlenecks.</p><p><a href="https://media.nesta.org.uk/documents/Compendium-of-Innovation-Methods-March-2019.pdf">Challenge prizes</a> raise the profile of innovators and bring them together with other experts, investors and new customers. They often result in the creation of many commercially successful ventures beyond the prize winner. In addition to the cash prize, challenge prizes often include other support too, such as help with networking, mentoring or access to legal advice.</p><p>Pre-procurement can be used to solve problems in a way that aligns with missions. It can be used to support the prototyping and development of new mission-relevant solutions, often in small and medium businesses.</p><p>Examples include the following.</p><ul><li><a href="https://amr.longitudeprize.org/">The Longitude Prize on Antimicrobial Resistance</a> was launched in 2014 by Challenge Works. It offered £8 million to anyone who could develop a rapid, affordable and easy-to-use diagnostic test to determine whether antibiotics are needed for an infection. The winner, Sysmex Astrego, was announced in 2024, for developing a system that can provide test results in half an hour.</li><li><a href="https://manchesterprize.org/">The Manchester Prize</a>, a UK DSIT initiative “that will award £1 million every year to a team of innovators with the most cutting-edge AI solution for public good”. The inaugural Manchester Prize invites UK-led teams to submit groundbreaking ideas that address challenges in the areas of energy, environment and infrastructure.</li><li><a href="https://www.ukri.org/what-we-do/browse-our-areas-of-investment-and-support/innovate-uk-contracts-for-innovation/">Contracts for Innovation (previously SBRI)</a><strong> i</strong>s an Innovate UK pre-procurement mechanism that<strong> “</strong>helps public sector organisations to solve complex challenges by running a competitive funding opportunity to develop and adopt new solutions and technologies”. Over 100 public sector organisations have invested more than £1.5 billion so far through this mechanism.</li></ul><h3>2. Venture-building</h3><p>Venture studios build new purpose-driven, venture-backable companies and put them on a path to delivering both meaningful social impact and attractive financial returns. Rather than supporting existing companies, venture studios create new companies from scratch that can be directed towards a specific social purpose. They work closely with entrepreneurs to develop each business and put it on an accelerated growth trajectory, drawing on the studio’s institutional support.</p><h4>How this method might help achieve missions</h4><p>Venture studios can steer companies towards a predefined mission in a way that other social investment methods cannot. The initial ideas are generated by the studio in line with defined goals, and a founding team is then recruited to further test and refine the idea. This allows solutions to be directed sharply towards critical social problems and opportunities.</p><p>Instead of a light-touch, one-to-many approach to supporting entrepreneurs found in other mechanisms such as traditional cohort-based accelerators, studios act as active co-founders. They provide intensive one-to-one support and capital to build the business and ensure each solution is properly validated. Similar to some other forms of state business support, venture studios might be run at arm’s length from the government.</p><p>Due to the high-intensity support model, which draws on the experience of successful entrepreneurs, venture studios can help bridge gaps where IP and human capital are getting stuck en route to successful commercial deployment.</p><p>Examples include the following.</p><ul><li>In 2023, <a href="https://www.sanabil.com/en/">Sanabil</a>, part of the Saudi Arabian sovereign wealth fund, launched a <a href="https://www.sanabil.studio/stryber">venture studio</a> dedicated to building startups from scratch across several priority sectors, such as FinTech and tourism. The aim is to establish thriving startup ecosystems and to achieve national missions.</li><li><a href="https://www.nesta.org.uk/project/mission-studio/">Nesta’s Mission Studio</a> creates tech startups that tackle some of the UK’s most pressing social challenges, building four new businesses a year. The studio aims to drive forward Nesta’s three missions to help narrow outcome gaps for young children, create healthier food environments and slash household carbon emissions en route to net zero.</li></ul><h3>3. Regulatory sandboxes and real-world testbeds</h3><p><a href="https://www.nesta.org.uk/blog/innovation-enabling-approaches-regulation/">Regulatory sandboxes</a> “provide regulators and innovators with a controlled and, usually, time-limited opportunity to test ideas under live conditions, often with real customers”. They allow managed waivers to regulation that allow both companies and regulators to learn about new technologies.</p><p><a href="https://media.nesta.org.uk/documents/Compendium-of-Innovation-Methods-March-2019.pdf">Real-world testbeds</a> are a “structured way of testing technologies or services in a real-world setting, to better understand what works. They’re widely used in business and are beginning to be used more in the public sector, particularly for testing out uses of technologies like drones or artificial intelligence.”</p><p>While the <a href="https://www.nesta.org.uk/report/testing-innovation-real-world/">two methods overlap</a>, sandboxes tend to be closer to idea generation rather than implementation stage of innovation and often use more controlled environments.</p><h4>How these methods might help achieve missions</h4><p>Regulatory sandboxes and real-world testbeds could facilitate innovation by allowing ideas to be tried in <a href="https://media.nesta.org.uk/documents/20_Tools_Innovating_Government_RCB1Iuj.pdf">lower-risk environments before being scaled</a>. They can be particularly helpful for exploring fast moving, often digital technologies, for which consequences are uncertain or the current regulatory environment may not be fully appropriate.</p><p>Sandboxes and <a href="https://media.nesta.org.uk/documents/20_Tools_Innovating_Government_RCB1Iuj.pdf">testbeds</a> can “strengthen collaboration and shared learning between the public and private sectors”. They can also focus attention on and attract investment for innovation relevant to missions and improve the government’s understanding of what works, under what circumstances. The current Government has committed to creating a <a href="https://labour.org.uk/wp-content/uploads/2024/06/Change-Labour-Party-Manifesto-2024-large-print.pdf">new Regulatory Innovation Office</a> to help regulators update regulation, speed up approval and coordinate issues that span existing boundaries.</p><p>Examples include the following.</p><ul><li>The <a href="https://www.fca.org.uk/firms/innovation/regulatory-sandbox">Financial Conduct Authorities’ first regulatory sandbox</a> was established in early 2016 to act as “a safe space for businesses to test out innovative ideas with real people.” This model has become the blueprint for similar initiatives elsewhere and sandboxes for the financial sector have since proliferated.</li><li>The Singapore Autonomous Vehicle Initiative (SAVI) was established in 2014 to start research into AV transportation and test-bedding. SAVI includes an open platform where the industry, research institutions and the authorities can jointly conduct self-driving trials.</li></ul><h3>4. Open data</h3><p><a href="https://www.nesta.org.uk/feature/digital-social-innovation/data-big-open-and-linked/">Open data</a> is “the raw data that is gathered by people or organisations, published in an electronic format that machines can read; it is then shared online and allowed to be re-used by others instead of keeping it private”.</p><h4>How these methods might help achieve missions</h4><p>Opening government data relevant to the missions could catalyse innovation around them in industry and civil society. A joined-up and intentional strategy is needed where multiple levers are used in concert to open up data in service of each mission. This includes establishing a keystone institution as the data holder, legislation to require data to be open and funding to facilitate that process. Data could be opened up strategically using a combination of regulatory requirements and <a href="https://www.nesta.org.uk/blog/new-ecosystem-trust/">supporting institutions such as data trusts</a> to govern access.</p><p>Examples include the following:</p><ul><li><a href="https://www.sdruk.ukri.org/">Smart Data Research UK</a> is the national programme for smart data research. It is backed by UK government funding through UKRI and brings together companies, academics, innovators and government to understand our world better and help solve social and economic challenges. This is achieved by providing secure data access, being a trustworthy programme for the public and building capability for cutting-edge research.</li><li>In 2017 the Competition and Markets Authority (CMA) initiated open banking,<strong> </strong>which enables customers to move their data more easily between banks. This has unlocked competition in the UK retail banking sector and provided a boost to the country’s emerging fintech sector. There are now plans to extend this model to other sectors.</li><li><a href="https://labour.org.uk/wp-content/uploads/2024/06/Change-Labour-Party-Manifesto-2024-large-print.pdf">In its manifesto</a>, the Government has committed to creating a National Data Library to “bring together existing research programmes and help deliver data-driven public services, whilst maintaining strong safeguards and ensuring all of the public benefit”.</li></ul><p><em>If you are interested in discussing this further then please e-mail: laurie.smith@nesta.org.uk</em></p><p><em>Thanks to James Plunkett, Celia Hannon, Oli Usher, Matt Seden, Siobhan Chan, Faizal Farook and all the Nesta staff involved in codifying innovations methods. Large Language Models were used in the research and drafting of this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b168521eb8cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/catalysing-innovation-for-missions-outside-government-b168521eb8cf">Catalysing innovation for missions outside government</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing AI for qualitative interviews]]></title>
            <link>https://medium.com/discovery-at-nesta/harnessing-ai-for-qualitative-interviews-76244ed992c6?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/76244ed992c6</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[qualitative-research]]></category>
            <dc:creator><![CDATA[Natalie Lai]]></dc:creator>
            <pubDate>Thu, 01 Aug 2024 10:19:45 GMT</pubDate>
            <atom:updated>2024-08-01T10:19:45.613Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Harnessing AI for qualitative interviews: Key insights from our evidence review</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nthhGGbFlzbwYwOvr43lgg.jpeg" /><figcaption>Source: Generated using Microsoft Designer AI</figcaption></figure><p><em>This article was co-written with </em><a href="https://medium.com/u/574750447653"><em>Camille Stengel</em></a></p><p>Generative Artificial Intelligence (AI) offers considerable opportunities in qualitative research. While much attention has been given to generative AI’s role in <a href="https://link.springer.com/article/10.1007/s40547-024-00143-4">automated text and sentiment analysis</a>, <a href="https://journals.sagepub.com/doi/pdf/10.1177/02683962211048201">literature reviews</a>, and <a href="https://journals.sagepub.com/doi/full/10.1177/16094069231211248">coding</a>, its use in interview settings has been less explored.</p><p>If interviews can be conducted by chatbots rather than people, there is both a huge opportunity for scaling, potentially mechanising what is currently a ‘cottage industry’, and blurring the boundaries between qualitative and quantitative research. Of course this new technology does not come without risk and ethical questions. Chatbots can be unpredictable, exhibit bias, and ‘hallucinate’ by giving incorrect but true-sounding responses. There’s also the question of just because we <em>can, should </em>we have chatbots conducting interviews?</p><p>Rather than replacing human-led interviews, how can this technology complement and enhance existing research methods? Our aim is to understand the potential use cases for impact in qualitative research and in Nesta’s work, as well as identify the pitfalls to avoid when engaging with this tool.</p><p>To better understand how generative AI might be harnessed for interviews as part of qualitative research, Nesta and the Behavioural Insights Team (BIT) have conducted a review of the evidence which is summarised in this blog.</p><p>This is the first in what we aim to be a couple of blogs sharing our reflections and learning as we explore this area further. We have pulled out several key insights for those considering AI-powered chatbots for qualitative interviews:</p><ol><li>AI-powered interviews enable large-scale qualitative data collection</li><li>AI chatbots struggle with depth and flexibility in interviews</li><li>Mixed Participant Preferences for AI vs. Human Interviewers</li><li>Limited social presence in AI interviews impacts participant comfort</li><li>Balancing human control and automation in AI system design to maintain researcher agency</li></ol><h3>What we did</h3><p>As a starting point to assess the potential benefits and risks of generative AI-powered approaches to qualitative data collection, we conducted a rapid evidence scan of the peer reviewed empirical studies published in English. We found six empirical studies that explicitly focused on generative AI and qualitative interviews (selecting only from peer-reviewed journals and reports from reputable research institutions; studies that focused on general use of AI in research were excluded).</p><h3>Key insight 1: AI-powered interviews enable large-scale qualitative data collection</h3><p>One of the most exciting opportunities AI-powered interviews offer is the ability to <a href="https://publi.ludomedia.org/index.php/ntqr/article/download/799/966/2792">scale qualitative research</a>. Scaling AI interviews effectively requires managing both broad topic coverage and deep, nuanced inquiry. Undeniably AI chatbots excel at gathering large volumes of qualitative data quickly. In one of the studies we reviewed, the chatbot collected over<a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4572954"> 7,000 open-ended responses from 395 adults</a> within one month. Some argue that integrating this extensive data with quantitative analysis could<a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4572954"> bridge the gap between qualitative and quantitative methods</a>, leading to a richer and more comprehensive understanding of the issues studied.</p><p>This capability to scale up data collection provides researchers with the opportunity to engage a larger and more diverse participant pool, thereby enriching the insights obtained. The speed and efficiency of AI in processing large volumes of data facilitate a broader exploration of topics, making it possible to collect comprehensive feedback on various issues in a fraction of the time required for traditional methods.</p><h3>Key insight 2: AI chatbots struggle with depth and flexibility in interviews</h3><p>AI-powered chatbots face challenges in maintaining depth and adaptability. Their responses are often limited by human-defined frameworks, which restrict their ability to explore unexpected themes, handle nuanced follow-up questions and conduct more in-depth interactions. For example, the use of pre-defined<a href="https://scholar.archive.org/work/xsfipgtnpjguzp6yjkz5umsmjq/access/wayback/https://publikationen.bibliothek.kit.edu/1000135146/120431975"> reaction guides and strategies led to inappropriate responses</a> in one of the studies we have reviewed. These limitations highlight the need for more sophisticated AI-driven interview designs to improve its depth and flexibility.</p><p>Additionally, if respondents deviate from expected paths, we discovered AI-chatbot <a href="https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/STO-MP-SAS-OCS-ORA-2021/MP-SAS-OCS-ORA-2021-AIML-02-3.pdf">struggle to redirect the conversation</a> effectively, further limiting the depth and adaptability of the interview. Our findings suggest that these constraints impact the consistency and relevance of follow-up questions generated by chatbots, affecting their overall effectiveness in conducting in-depth interviews.</p><h3>Key insight 3: Mixed Participant Preferences for AI vs. Human Interviewers</h3><p>Our review uncovered that participants have mixed preferences when it comes to AI versus human interviewers, which largely depends on the context and nature of the interview questions being asked. Participants preferred AI interviews for sensitive topics, such as<a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4572954"> financial decisions</a>, where the absence of judgement and anonymity reduced concerns about personal disclosure. The perceived neutrality of AI in these scenarios offered a significant benefit</p><p>Conversely, human interviewers were preferred in situations requiring empathy and emotional intelligence, such as <a href="https://academic.oup.com/jcmc/article/28/4/zmad028/7210236?login=false">job interviews</a> or evaluations, due to their ability to convey empathy and understanding. The ability of human interviewers to offer empathetic responses and interpret nuances in emotional context made them more effective in these scenarios. This contrast highlights the ways in which AI and human interviewers are perceived, underscoring the need for careful consideration of context.</p><h3>Key insight 4: Limited social presence in AI interviews impacts participant comfort</h3><p>It was clear that social presence, the extent to which one feels they are in the presence of and interacting with a ‘real person’, is a crucial factor influencing participants’ experiences with AI interviews. In one of the studies participants reported feeling <a href="https://academic.oup.com/jcmc/article/28/4/zmad028/7210236?login=false">greater uncertainty and perceived lower social presence</a> during AI interviews compared to interviews conducted by a human.</p><p>The unfamiliarity and lack of personal interaction with the AI system contributed to these feelings, leading participants to speak more quickly and with fewer pauses. Despite the use of a <a href="https://academic.oup.com/jcmc/article/28/4/zmad028/7210236?login=false">human-like avatar</a>, which somewhat improved the sense of connection, it did not sufficiently ameliorate the issue. This discomfort stemmed from a perceived lack of personal interaction and empathy, highlighting the challenge AI chatbots face in fostering a supportive and engaging interview dynamic.</p><h3>Key insight 5: Balancing human control and automation in AI system design to maintain researcher’s agency</h3><p>Whilst most researchers are keen to make the most of AI’s capabilities, many are worried that <a href="https://www.research-live.com/article/opinion/how-qual-researchers-really-feel-about-generative-ai/id/5116653">automating interviews could undermine their agency</a> and result in a loss of nuance.</p><p>We decided this was worth looking into specifically, so we reviewed the existing studies on the role of <a href="https://books.google.co.uk/books?hl=en&amp;lr=&amp;id=mSRXEAAAQBAJ&amp;oi=fnd&amp;pg=PP1&amp;ots=n24jcekLYb&amp;sig=gEMDpmpaTTvfGhCM9UBiKv_OjqI&amp;redir_esc=y#v=onepage&amp;q&amp;f=false">human researchers’ control</a> versus AI automation and found varying degrees of integration. To make sense of these findings we grouped them using a two-dimensional Human-Centered AI (HCAI) <a href="https://arxiv.org/pdf/2002.04087">framework</a>, suggested by Ben Schneiderman, that plots human control against AI automation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*j5vuYcRnPHtwWN8bWqg7Jg.jpeg" /><figcaption>HCAI Framework extracted from Shneiderman, B. (2020). <a href="https://arxiv.org/abs/2002.04087v1">Human-centered artificial intelligence: Reliable, safe &amp; trustworthy</a>. <em>International Journal of Human–Computer Interaction</em>, <em>36</em>(6), 495–504.</figcaption></figure><ul><li><strong>high human control + high computer automation</strong>: In these instances human researchers created a tight framework within which the AI could operate with a high degree of automation, dynamically adapting its responses based on predefined rules. For instance, one study developed a <a href="https://scholar.archive.org/work/xsfipgtnpjguzp6yjkz5umsmjq/access/wayback/https://publikationen.bibliothek.kit.edu/1000135146/120431975">decision gate process</a> featuring 29 question variations and rule-based probing. Similarly, another study utilised a <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4572954">decision tree</a> that directed the AI to use models that generate follow-up questions, manage topic transitions, and summarise conversations, but that were based on rules established by human researchers.</li><li><strong>high human control + low computer automation</strong>: These involved the overall structure and key interactions being carefully designed and controlled by human researchers. Users can personalise and configure information processes with a low level of automation. For example, one study used a<a href="https://arxiv.org/pdf/2309.10187"> fixed set of primary questions and a set warm-up question</a> designed by human researchers. Follow-up questions were either pre-generated by researchers or by generative AI, depending on the condition.</li><li><strong>low human control + high computer automation: </strong>These examples saw a significant portion of the research process being automated. For example, one of the studies used 5 <a href="https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/STO-MP-SAS-OCS-ORA-2021/MP-SAS-OCS-ORA-2021-AIML-02-3.pdf">seedling questions</a> to form a basic structure for each interview. Generative AI was then allowed to determine whether to ask follow-up questions, and decide what those questions should be, without relying on human-designed guidelines. This flexibility allows AI interviewers to delve deeper into unexpected themes.</li><li><strong>low human control + low computer automation</strong>: This combination would not be a feasible qualitative research scenario and therefore there is no evidence base for this.</li></ul><h3>What’s next?</h3><p>We’re building on our key takeaways from the evidence scan as we test out different AI-powered interview bots within the context of Nesta’s missions. We’re evaluating the bots’ interviewer effectiveness across different platforms and using various forms of criteria. This includes assessing AI-interviewers’ ability to streamline data collection processes, enhance participant engagement, and maintain safeguarding and ethical standards.</p><p>Watch this space for subsequent blogs where we’ll share our reflections of this assessment. If you’re currently using or considering AI-powered interview platforms, we’d love to hear from you — get in touch!</p><p><em>We thank </em><a href="https://medium.com/u/10967b657945"><em>Celia Hannon</em></a><em>, </em><a href="https://medium.com/u/3c34677b39d3"><em>Laurie Smith</em></a><em> and </em><a href="https://medium.com/@faizal.farook"><em>Faizal Farook</em></a><em> for reviewing the article and providing insightful feedback.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76244ed992c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/harnessing-ai-for-qualitative-interviews-76244ed992c6">Harnessing AI for qualitative interviews</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Planning an automated horizon scanning process]]></title>
            <link>https://medium.com/discovery-at-nesta/planning-an-automated-horizon-scanning-process-753360380c44?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/753360380c44</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[horizon-scanning]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vc]]></category>
            <dc:creator><![CDATA[Tom Willcocks]]></dc:creator>
            <pubDate>Tue, 07 May 2024 05:32:07 GMT</pubDate>
            <atom:updated>2024-05-07T10:38:14.443Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Xh-ezfCDYK9BmdSyX5AZw.jpeg" /><figcaption>Image: Blueprint of a space telescope (<a href="https://www.flickr.com/photos/nasawebbtelescope/46826497505">NASA</a>)</figcaption></figure><p><em>This article was co-written with </em><a href="https://medium.com/u/c7cad6c6bfa4"><em>Karlis Kanders</em></a><em>.</em></p><p>In this blog, we introduce our upcoming work on an automated, data-driven horizon scanning approach.</p><p>It is a new project to boost Nesta’s foresight capacity and provide up-to-date insights on innovation related to sustainability, health and education. This should be interesting to anyone doing futures and foresight work.</p><p>We cover the following:</p><ul><li>The advantages of automating horizon scanning</li><li>Prototyping a minimum viable product</li><li>Emerging tech stack for automated scanning</li><li>Conclusion and lessons learned (so far)</li></ul><h3>The advantages of automating horizon scanning</h3><p>The Discovery Hub helps Nesta stay on top of emerging trends, technologies and interventions through the use of established horizon scanning approaches such as desk research as well as through our data-driven projects using data science and machine learning. Over the past few years, we have been developing data-driven approaches for analysing innovation trends in various domains: from <a href="https://www.nesta.org.uk/feature/innovation-sweet-spots-green-tech/">low-carbon heating tech</a> for reducing household carbon emissions, to exploring <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/innovation-sweet-spots-food-innovation/">food technology trends</a> and their impact on food environments, to <a href="https://www.nesta.org.uk/project/innovation-sweet-spots-detecting-and-managing-child-development-needs/#:~:text=We%20will%20uncover%20trends%20in,families%20manage%20these%20development%20needs.">innovations for supporting child development</a>.</p><p>However, innovation is, by definition, not static: new research projects are started, new startups get founded, and new policies are constantly being published. Therefore, it’s critical for Discovery to continuously update its view on the innovation landscape.</p><p>We are thus entering the next phase of our data-driven foresight work, transitioning from one-off ‘snapshots’ to a more automated, continuously updating process.</p><p>This will enable frequent updates to our trends analysis with reduced manual effort, as well as surfacing instances of new innovations, research projects and startups on an ongoing basis — ensuring we are prepared to provide the most up-to-date insights as and when they are needed.</p><h3>An MVP pipeline on investment signals</h3><p>Building an automated analysis pipeline is markedly different from doing a one-off research project and requires a different — data engineering — skill set. As this is new territory for the Discovery Hub, we are taking an iterative approach and starting with a minimum viable product that will be initially tailored to one of our internal stakeholders, <a href="https://www.nesta.org.uk/project/impact-investments/">Nesta’s Impact Investments team</a>.</p><p>We’re <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/funding-future-generations/">building on previous collaborations</a> which have helped to prototype the data-driven analysis process and demonstrate its value for understanding investment trends and originating investment leads. To further understand the user needs, we drew from agile practices and ran a requirements prioritisation workshop, which produced several user stories that are now guiding the development of our minimum viable product.</p><p>The general architecture of our initial prototype pipeline is shown in the figure below. It consists of three main stages: (1) data collection where we update our datasets, (2) processing, where we classify and enrich our data, and synthesise it into information for our stakeholders, and (3) dissemination via various channels to the wider organisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aAFX2ZyXyE25usac8_WI5Q.png" /><figcaption><em>Automated horizon scanning pipeline prototype</em></figcaption></figure><h3>Collection</h3><p>Our minimum viable product begins with an automated daily collection of the most recent venture capital investments data. We determine whether there are any new companies or new venture funding rounds that have been added to the database since the previous data collection. We call these new data points “signals”.</p><p>Going forward, we plan to improve the minimum viable product by adding other valuable innovation data, such as research funding in the UK (e.g., from the UKRI’s <a href="https://gtr.ukri.org/">Gateway to Research</a> database or NIHR’s <a href="https://nihr.opendatasoft.com/pages/homepage/?flg=en-gb">Open Data site</a>), research publications (<a href="https://openalex.org/">OpenAlex</a>), patents (<a href="https://patents.google.com/">Google Patents</a>) and public and policy discourse (e.g., from various news media and public sector websites).</p><p>By approaching this work iteratively, the automated pipeline can start bringing value early on, while we are adding more datasets in parallel.</p><p>Each new dataset, however, adds to the technical complexity and maintenance needs of the pipeline. So it will be important to prioritise them in terms of the value they bring to the organisation.</p><h3>Processing</h3><h4>Classification of new data</h4><p>But before we share the new signals with colleagues in our organisation or incorporate them into our analyses, we need to determine whether they are relevant to Nesta’s three mission areas.</p><p>For this purpose, we have a classification step, where we’ll use methods ranging from simple keyword search to supervised machine learning to classify incoming signals in terms of our three missions and more granular mission area topics. For example, for new companies related to the sustainable future mission, we would also like to determine whether their descriptions mention specific low-carbon heating technologies, renewable energy sources or energy efficiency measures.</p><p>Note that while the datasets we use might often have existing classifications and tags, it has been important for us to establish our own taxonomy that is aligned with Nesta’s mission areas — and that be applied across different datasets to join them up.</p><h4>Enrichment: going beyond the raw data</h4><p>In addition to classification, we can also enrich the data with useful information that can help highlight particularly important signals. For example, by using a company’s financial data and its number of employees, we can determine an initial indication of whether this company is a viable investment opportunity for our investments team.</p><p>By analysing the text content of the incoming signals in greater detail, we could get an indication of whether a company, research project or patent seems particularly novel and innovative — and thus deserves special attention to evaluate its potential impact.</p><p>In the future, we could also link different datasets, and, for example, keep track of a company’s venture funding, patenting activity, and news media profile.</p><h4>Synthesis: from data to information</h4><p>This step takes the classified and enriched signals as input and outputs user-friendly information, to be shared via various channels. This might be a daily message following a specific template detailing new venture funding rounds, a weekly summary of the developments across all three Nesta’s mission areas, or an updated chart for a live trends report.</p><p>As we increase the number of datasets we track, we expect the amount of daily or weekly incoming signals to become overwhelming for us to keep up with (anyone who’s subscribed to more than a few newsletters can attest to that!). Here, we expect to use generative AI and large language models (LLMs) to generate summaries and help us curate the new information more efficiently.</p><h4>Dissemination</h4><p>Finally, the new information needs to be shared with colleagues in our organisation — this is where the new information can be leveraged to create impact.</p><p>For the minimum viable product, we will primarily use Slack for delivering daily messages, as it is our main channel of communication internally. Going forward, we might also produce semi-automated trends reports (in the spirit of our <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/innovation-sweet-spots-food-innovation/">Innovation Sweet Spots</a> work) that are curated further by our foresight analysts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6AfI7mCAeYZb9RnoCwP-g.png" /><figcaption><em>Prototype Slack message alert of new investment signals</em></figcaption></figure><p>More generally, the most appropriate communication channel, format and frequency of updates will depend on the specific stakeholder and their use case — and we expect that getting this right will require some iteration.</p><h3>Emerging tech stack for automated scanning</h3><p>We are still at the beginning of building this pipeline, but some technologies have emerged as potentially useful.</p><p>For the orchestration of the entire pipeline we use <a href="https://airflow.apache.org/">Airflow</a>. It helps to set up the scheduling of data collection, processing and dissemination, and ensures that each analysis step is executed in the correct order. Airflow’s visual interface aids in monitoring the workflows, making it easier to check for completion of analysis runs, inspect logs and identify bottlenecks or failures.</p><p>For the actual data ingestion and processing, we utilise <a href="https://metaflow.org/">Metaflow</a> (where Metaflow scripts are triggered by the Airflow scheduler). Metaflow allows us to break down complex processing into smaller steps, easily manage dependencies for each step, and scale up the required compute resources using AWS Batch for more intensive steps.</p><p>When it comes to classification of the data, we’ll be experimenting with various methods, starting with training simple text classification models based on pre-trained text embeddings or fine-tuning transformer models. We have also found it useful to log our training experiments using the <a href="https://wandb.ai/site">Weights &amp; Biases</a> platform.</p><p>We’re also experimenting with using LLMs to label some of our training data automatically, and to synthesise incoming signals into summaries. Presently, we’re using <a href="https://platform.openai.com/">OpenAI</a>, but we are open to trying out other models.</p><p>We will write more about the technical solutions in the future, and we aim to work as much as possible in the open, and most of our data processing code will be available on <a href="https://github.com/nestauk/discovery_utils">GitHub</a>.</p><h3>Conclusion and lessons learned (so far)</h3><p>We hope this has been a helpful summary of our automated horizon scanning efforts so far.</p><p>It’s important to carry out the development iteratively, with an agile mindset. Perfecting and expanding your automated data processing pipeline with more data sources will take time, but ideally you can start generating value with your minimum viable product as early as possible. For example, for the minimum viable product, it will be enough to tag incoming signals using keyword matching even if this approach is somewhat noisy. This will provide a baseline from which we can build more sophisticated signal classification algorithms using machine learning. Similarly, before producing detailed dashboards of signals and trends, we can start by providing simple daily alerts to our stakeholders about the new signals.</p><p>Through on-going feedback from Nesta’s investments team, we will be able to continually assess whether the companies we are identifying are both mission relevant and at a relevant investment stage to ensure their investment origination efforts are well targeted. Additionally, as new technologies and markets emerge within our Mission areas, based upon market feedback, the investments team can support us in expanding the search to ensure the outputs are responding to the fast pace of technological development.</p><p>While automated collection, processing and synthesis of data opens the door for a more up-to-date and dynamic foresight process, it’s essential to recognize that human input is still crucial. Determining the most valuable information for our decision-makers and converting horizon scanning findings into actionable insights requires human expertise. Therefore, it’s crucial for the automated horizon scanning work to be tightly integrated with the ways of working in the foresight team. We’ll be doing this, for example, by combining the signals from the automated process with manually sourced signals in one repository.</p><p>By automating labour-intensive tasks, we will be able to focus more manual effort on analytical and interpretive aspects rather than data collection and processing. This strategic approach will hopefully allow us to focus our time on deeper analysis of evolving trends and their implications for strategy.</p><p>We will post further updates as we go, so stay tuned and get in touch with the project team if you’re interested to know more.</p><p><em>We thank Solomon Yu, Will Woodward, Celia Hannon, Leo Chandler and Faizal Farook for their feedback on the article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=753360380c44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/planning-an-automated-horizon-scanning-process-753360380c44">Planning an automated horizon scanning process</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Songs of change: data reveals how musicians advocate for the environment]]></title>
            <link>https://medium.com/discovery-at-nesta/songs-of-change-data-reveals-how-musicians-advocate-for-the-environment-d9ea9f4243ad?source=rss----f5a47887598f---4</link>
            <guid isPermaLink="false">https://medium.com/p/d9ea9f4243ad</guid>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[activism]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[climate-change]]></category>
            <dc:creator><![CDATA[Adeola O]]></dc:creator>
            <pubDate>Thu, 25 Apr 2024 10:52:10 GMT</pubDate>
            <atom:updated>2024-04-25T10:52:10.092Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eT-dkiRW4CgbznYwLe4R0g.png" /><figcaption>Original artist: <a href="https://thegreats.co/artworks/earth-has-music-for-those-who-listen">Preeti Singh</a></figcaption></figure><p><em>In this article, we consider how musicians are using their talent and voice to respond to the climate crisis. We have analysed data from Spotify and the Official Singles Charts (UK) and explored trends in music about the environment and climate change across the last six decades.</em></p><p>Throughout history, crises and social challenges have driven artists’ work. Today, climate change has become the defining issue of our time, and musicians have <a href="https://www.nme.com/features/how-musicians-and-fans-have-taken-on-climate-change-2559989">been using their platform</a> to engage in the climate conversation, raise awareness and stir public sentiment.</p><p>Some artists are pledging to <a href="https://www.theguardian.com/music/2021/oct/14/coldplay-pledge-50-lower-co2-emissions-on-2022-world-tour">reduce the carbon emissions</a> of their tours, are <a href="https://earthpercent.org/">raising funds</a> to support projects tackling climate change or using music and arts to highlight the connections between the climate emergency and other issues.</p><p>The music industry is making significant strides toward achieving net-zero emissions, according to a <a href="https://pec.ac.uk/research-reports/creative-industries-and-the-climate-emergency">recent report</a> commissioned by the Creative Industries Policy &amp; Evidence Centre. Its commitment to environmental sustainability can be seen from the <a href="https://www.musicdeclares.net/">Music Declares Emergency</a> campaign, which is endorsed by over 3,000 UK music artists, to initiatives to reduce the carbon footprint of tours and vinyl record production. The sector’s leading artists and businesses have also long been engaged with the cause, with globally recognised musicians such as <a href="https://www.nme.com/news/music/billie-eilish-invites-young-environmental-activists-for-conversation-about-climate-crisis-3374442">Billie Eilish leveraging their platform</a> to foster dialogue about tackling the climate crisis.</p><p>And then there is the long history of singers and songwriters expressing their opinions and emotions directly through song. From one of the first known environmental songs <a href="https://www.youtube.com/watch?v=MjyD5wZjZ-U&amp;ab_channel=candiru63"><em>Woodman! Spare that Tree!</em></a> in the 19th century to Billie Eilish’s <em>All the Good Girls Go To Hell</em> of 2019 referencing wildfires and rising sea levels.</p><p>A question, however, has been raised by <a href="https://www.nme.com/features/music-features/new-study-music-fans-climate-change-billie-eilish-music-declares-emergency-3220850">media</a> and <a href="https://gimletmedia.com/shows/howtosaveaplanet/llhl9xr">researchers</a> about the potential lack of present-day anthems to fight climate change. This is particularly surprising given that <a href="https://www.musicdeclares.net/gb/updates/turn-up-the-volume-survey">a large majority (82%) of music enthusiasts care about climate change</a>.</p><p>In this article, we take <strong>an experimental, data-driven approach to explore the trends </strong>in the representation of environmental themes such as <strong>climate change and general appreciation of the Earth</strong> in popular music over the past few decades. We have analysed the shifting numbers of songs released over the years and ascertained their popularity, as well as less tangible properties — like the mood and energy they convey.</p><h3>Approach</h3><p>To identify songs concerned with environmental themes, we sourced lists of songs published by researchers, music news media and music enthusiasts. We also carried out a small crowd-sourcing exercise at Nesta. This resulted in an initial collection of more than 400 potentially relevant songs.</p><p>We reviewed and narrowed down to <a href="https://open.spotify.com/playlist/4EhBYkl4jtI2POrkUxU6Ul?si=814b037fb0f44164&amp;nd=1">a final playlist of 206 songs</a>, filtering them based on the criteria outlined in the methodology (below), particularly those that are topically and contextually relevant and feature lyrics in the English language.</p><p>While our analysis focuses on songs with English lyrics, it’s essential to recognise that many musicians worldwide are actively addressing climate justice in their work in various languages.</p><p>The aim of this analysis is to uncover possible trends rather than to unearth a complete list of environment-themed songs. Although we were not able to capture every relevant song, we believe that it is representative enough of popular Anglophone music concerning this topic. To give a sense of the songs which make up the playlist, we have picked out a few examples that span the decades covered.</p><p>For example, Louis Armstrong admires the beauty of the world and creation in general in <em>What A Wonderful World</em> (1967).</p><blockquote>I see trees of green<br>Red roses too<br>I see them bloom<br>For me and you<br>And I think to myself<br>What a wonderful world</blockquote><p>Alabama worried about the catastrophic change in the elements in <em>Pass It On Down</em> (1990).</p><blockquote>Well, there’s a change taking place way on the mountains<br>Acid rain is falling on the leaves<br>And down in Brazil, the fires are burning still<br>How we gonna breathe without them trees?</blockquote><p>More recently, King Gizzard And The Lizard Wizard explicitly tell us we have no other planet in <em>Planet B</em> (2019).</p><blockquote>Open your eyes and see<br>There is no Planet B</blockquote><p>In the following sections, we will refer to our final playlist of songs as <strong>‘environment-themed songs’</strong>.</p><h3>Findings</h3><h3>Climate song releases — is a new peak coming?</h3><p>We found that the number of song releases over time suggests a peak in environment-themed songs during the 1990s, with a potential resurgence in the early stages of the 2020s.</p><iframe src="https://flo.uri.sh/story/2296902/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/11bd56b695e57d5ba4dc1a176b1e6ecf/href">https://medium.com/media/11bd56b695e57d5ba4dc1a176b1e6ecf/href</a></iframe><p>The surge of environmentally conscious tracks in the early 1990s aligns with the period when <a href="https://theconversation.com/30-years-ago-global-warming-became-front-page-news-and-both-republicans-and-democrats-took-it-seriously-97658#:~:text=June%2023%2C%201988%20marked%20the,change%20became%20a%20national%20issue.">global warming began to command national attention</a> in the US. This suggests that mainstream artists might have responded to this increased awareness by incorporating environmental themes into their work. That said, this trend also mirrors a more general increase in song releases during the early to mid-1990s, as <a href="https://www.nber.org/papers/w16882">reported by the US-based National Bureau of Economic Research</a>.</p><p>Despite a decrease in environment-themed song releases in the following decades, our analysis reveals an encouraging uptick in recent years. Even though we’re only a few years into the 2020s, we’ve already identified at least 15 environment-themed songs. If this momentum persists, the 2020s could rival, and potentially surpass, the volume of climate-focused music produced in the 90s.</p><h3>The historical events that inspired music about the environment</h3><p>Looking more closely at the yearly data (Figure 2) we notice a distinct peak in climate-related music production during the early 1970s. This period was also marked by significant environmental controversies in the US, including widespread concerns about the ecological risks associated with the pesticide DDT. Joni Mitchell’s <em>Big Yellow Taxi</em> notably referenced this issue, illustrating the direct influence of these environmental discourses on the music of the time.</p><blockquote>Hey, farmer, farmer, put away your DDT<br>I don’t care about spots on my apples<br>Leave me the birds and the bees<br>Please!</blockquote><iframe src="https://flo.uri.sh/story/2295744/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2ce1882cfd93ac46605d1bc0d48605a/href">https://medium.com/media/c2ce1882cfd93ac46605d1bc0d48605a/href</a></iframe><p>Another possible symptom of growing societal concerns is the discovery of a hole in the ozone layer in 1985. A few years later, the <a href="https://www.un.org/en/conferences/environment/rio1992">Earth Summit</a> was held. This period witnessed a noticeable increase in the production of environment-themed songs, reaching a peak in 1991 and 1992.</p><p>Leaping ahead to the 21st century, the early 2000s saw a gradual dip in the release of environment-themed music, which did not go unnoticed. <a href="https://pitchfork.com/features/overtones/climate-change-music/">Pitchfork magazine said that it “almost felt as if culture itself was averting its eyes”.</a></p><p>However, the recent resurgence since 2015 aligns with the rise in climate activism spearheaded by individuals such as Greta Thunberg. Technology may be playing a part here; as the internet and streaming services have increased the ability to share information, climate campaigns may bring attention to songs about these causes, encouraging people to engage with them.</p><p>One standout example from 2018 is Childish Gambino’s <em>Feels Like Summer</em>, a track that addresses the harmful human impacts on politics, race, and climate change.</p><blockquote>Every day gets hotter than the one before<br>Running out of water, it’s about to go down<br>Go down<br>Air that kill the bees that we depend upon<br>Birds were made for singing, wakin’ up to no sound<br>No sound</blockquote><h3>Which climate songs are the most popular?</h3><p>Musicologist Natheniel Sloan said in a <a href="https://grist.org/culture/is-it-possible-to-make-a-good-pop-song-about-climate-change/">2021 interview</a> that there is a unique challenge for pop music that covers topics about the environment, in particular climate change: it is tricky for these songs to connect to the listener on a personal emotional level (compared to a heartbreak song for example).</p><p>A few environment-themed songs and anthems did make an impression on music listeners looking at how they charted in the Official Singles Charts (UK) and how popular they are on Spotify. Out of the 206 songs we collected, around 10% were found in the UK charts.</p><p>Figure 3 shows that the 1980s saw a quarter of these songs charting in the UK Official Charts, followed by those released in the 1960s with a fifth and 1990s with a tenth of songs charting.</p><iframe src="https://flo.uri.sh/visualisation/13883297/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/8624595ae1db3fb72fe792936cc843dc/href">https://medium.com/media/8624595ae1db3fb72fe792936cc843dc/href</a></iframe><p>This could possibly highlight the mainstream anthems that swept the 60s, 80s and 90s pushing for people to identify with other songs covering environmental issues . For example, the 90s saw Michael Jackson sweep the UK charts with <em>Earth Song</em> peaking at #1 and <em>Heal The World</em> peaking at #2.</p><p>Although not as many environment-themed songs are charting now, the concern that audiences have a harder time relating to these songs have a personal level, and the strategy of <a href="https://thelongandshort.org/creativity/top-forty-charts-data-viz">“few songs, big hits”</a> might be contributing to this shift.</p><p>Spotify also measures how popular songs are now on its platform. We find that some songs in our playlist are still being listened to. The Beatles’ <em>Here Comes the Sun</em> is the most popular song on Spotify from our sample. Although the song was created as a general appreciation of the sun, some music fans have reinterpreted the song as <a href="https://catapult.co/stories/here-comes-the-sun-climate-change-the-beatles-abbey-road-in-this-climate-amy-brady">a warning about climate change</a>.</p><iframe src="https://flo.uri.sh/visualisation/13883373/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd18c79396e16a223afe15405303bdc4/href">https://medium.com/media/fd18c79396e16a223afe15405303bdc4/href</a></iframe><h3>Are these songs all doom and gloom or optimistic?</h3><p>We used the song features available on Spotify that measure the positiveness (valence), danceability and energy of songs. In addition, we compared our playlist of environment-themed songs to a baseline, which is the representative mean of the most mainstream songs spanning from 1960s to 2020s (using Spotify official playlists representing each decade with about 150 songs per decade).</p><h4>Positiveness</h4><p>The first measure we’ll look at is the valence score or the measure of positiveness (further explained in methodology). Figure 5 shows that across most decades, the songs span a wide range of the level of positiveness.</p><p>The average valence for the environment-related songs, however, tends to stay below the average valence of the representative mainstream songs (which we will call the baseline).</p><p>Valence of both mainstream as well as environment-themed songs has trended downwards in the past few decades, which suggests that while mainstream songs sound <a href="https://theconversation.com/the-charts-may-be-getting-sadder-but-popular-music-has-never-been-more-varied-96655">less happy</a>, our representative sample of environment-themed songs sound even less happy over time.</p><iframe src="https://flo.uri.sh/visualisation/15239117/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/7988c10cc4d22e96fa489bc5be2de27e/href">https://medium.com/media/7988c10cc4d22e96fa489bc5be2de27e/href</a></iframe><p>To get a sense of the differences between valence, here are a few songs to listen to:</p><ul><li><em>Fallen Fruit</em> (2021) by Lorde has a <strong>low valence score of</strong> <strong>0.10</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F5iUnomffoLeQsdlY2k5TDU%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F5iUnomffoLeQsdlY2k5TDU&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e0236615a0a60523dd62135ab3a&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4ddb02feb74f47d9d2a7338def7c175/href">https://medium.com/media/f4ddb02feb74f47d9d2a7338def7c175/href</a></iframe><ul><li>Whereas <em>Emergency on Planet Earth</em> (1993) by Jamiroquai has a <strong>high valence score of 0.87</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F0mmI8HaXjgzxrtKDmvAN7R%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F0mmI8HaXjgzxrtKDmvAN7R&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e02b6f13e40fbd883767da606bc&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d31affd764331849f9acc37a4e44edd/href">https://medium.com/media/8d31affd764331849f9acc37a4e44edd/href</a></iframe><h4>Danceability</h4><p>Another interesting feature is the danceability score, which can be thought of as how easily you can do a two-step while listening to the song (or three-step if it follows a waltz-like rhythm).</p><p>Over the decades, we see that the danceability of the environment-themed songs stays around the danceability score of 0.5 (Figure 6). Compared to the baseline, mainstream songs over time have increased in danceability.</p><iframe src="https://flo.uri.sh/visualisation/15238891/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b708a1a27497670833970b82e789b56/href">https://medium.com/media/2b708a1a27497670833970b82e789b56/href</a></iframe><p>This may imply that mainstream music has become more danceable over the decades, in contrast to environment-themed songs, which have stayed about the same. The lack of danceability could be one reason these songs haven’t increased in popularity over the last decade.</p><p>Here are some examples of the range of danceability amongst the environment-themed songs:</p><ul><li><em>Hands All Over </em>(1989) by Soundgarden has a <strong>low danceability score of 0.19</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F6HkTiS6SmtO9GcoEQjIDee%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F6HkTiS6SmtO9GcoEQjIDee&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e02ed3ba8ced3d205e80db7726e&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae4736ad5659561889f788762c57f718/href">https://medium.com/media/ae4736ad5659561889f788762c57f718/href</a></iframe><ul><li><em>Fish Ain’t Bitin’</em> (1997) by Corey Harris has a <strong>high danceability score of 0.89</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F3aZcroOlpkzAeZ57Dkrgoj%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F3aZcroOlpkzAeZ57Dkrgoj&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e026b6ecb7fe5dc646b7c52a6d8&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/a736ffd089510fff48c55a1323156b08/href">https://medium.com/media/a736ffd089510fff48c55a1323156b08/href</a></iframe><h4>Energy</h4><p>Our final measure we looked at was energy, where those that score high sound loud, noisy and fast. Here are some examples of songs on the energy scale:</p><ul><li><em>Man Walks Among Us</em> (1990) by Jonathan Richman has a <strong>low energy score of 0.05</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F6sted2HiSwNzt1IWZgjxQw%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F6sted2HiSwNzt1IWZgjxQw&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e025086cbed7efa18e924a7639f&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/44954d4169628c090a76454b8be4b7ae/href">https://medium.com/media/44954d4169628c090a76454b8be4b7ae/href</a></iframe><ul><li>Meanwhile, <em>Planet B </em>(2019) has a <strong>high energy score of 0.90</strong>.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Ftrack%2F6FwC9wgE3oBtM7C0KpOzvm%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Ftrack%2F6FwC9wgE3oBtM7C0KpOzvm&amp;image=https%3A%2F%2Fi.scdn.co%2Fimage%2Fab67616d00001e025fbb93470556087ec976e3df&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="152" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f608ae009961aad6698891b13ed99ef/href">https://medium.com/media/4f608ae009961aad6698891b13ed99ef/href</a></iframe><p>Just like the previous two measures, the mean energy score of environment-themed songs stayed below the mainstream baseline over the decades. Although the energy of environment-themed songs increased to around 0.7 in the 1980s, it steadily decreased from then on (Figure 7). However, the baseline has seen energy increase over the decades followed by a drop in the energy measure in the past two decades.</p><iframe src="https://flo.uri.sh/visualisation/15239098/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/a64cb27239ddbae55e513e6e59d64e3e/href">https://medium.com/media/a64cb27239ddbae55e513e6e59d64e3e/href</a></iframe><p>Compared to danceability and valence, the distribution of the energy scores have more variability. Nonetheless, the downward trend for environment-themed songs is still apparent.</p><h3>Is this the decade for climate-themed music?</h3><p>As showcased in this article, artists have continued to release songs appreciating the environment, pointing out the woes of pollution and climate change and calling for action. However, our sample of environment-themed songs also suggests that these songs might still be quite niche, with only a few making it into the charts over the years.</p><p>Could the difference between environment-themed songs and more mainstream songs in terms of positiveness and other measures of mood reflect the challenge of environment-themed songs relating with listeners on a personal level because of the stylistic choices? Or is this a reflection of how musicians feel about the climate in general: less hopeful and energetic?</p><p>Climate reporter Kendra Pierre-Louis, in a <a href="https://gimletmedia.com/shows/howtosaveaplanet/llhl9xr">discussion on anthems for social movements</a>, pointed out how anthems “basically say that this work [activism] is hard and necessary, and the way we do it is to find joy in each other”. Perhaps this suggests one of the keys to approaching environment-themed music, by reflecting on the joy of coming together to tackle the defining challenge of our time.</p><p>There are, however, encouraging signals that the 2020s might be shaping up to be <em>the</em> decade for climate-themed music and climate activism in the music industry.</p><p>There has been an increasing trend of music artists actively participating in the climate movement, composing for and <em>with</em> nature. In late 2019, Nick Mulvey <a href="https://www.udiscovermusic.com/news/nick-mulvey-new-track-anthropocene/">pressed his single “Anthropocene” onto a record made from recycled plastic</a>, which had been washed up on Cornish beaches. The work explored the themes of human responsibility and freedom affecting the climate and environment, and the proceeds were donated to Surfers Against Sewage.</p><p>Following their 2021 collaboration with Tyndall Centre for Climate Change Research on the <a href="https://tyndall.ac.uk/news/massive-attack-publish-tyndall-centre-climate-change-live-music-roadmap/">Roadmap to Super Low Carbon Live Music</a>, Massive Attack plan to launch a show in 2024 using lower-carbon measures such as powering the side with renewable energy, a meat-free venue and special pre-sale for local neighbourhoods to dissuade people travelling from further afield.</p><p>EarthPercent’s co-founder Brian Eno in 2022 released a <a href="https://pitchfork.com/news/brian-eno-announces-first-solo-album-in-5-years-shares-new-song-listen/">10-song album calling attention to the climate crisis</a> and ironically performing his song “There Were Bells” at the Acropolis in Athens where temperatures were soaring and wildfires happening near the city where he said: “Here we are at the birthplace of Western civilisation, probably witnessing the end of it.”</p><p>Cathy Runciman, the Executive Director of <a href="https://earthpercent.org/">EarthPercent</a> told us: “We notice, at EarthPercent, that many artists are wanting to engage with campaigns like ‘The Earth As you Co-writer’ or our new Featuring Nature campaign as a way to consider how the song can embed the Earth or nature in their work and creative a positive impact.”</p><p>2023 also saw a fair share of climate-focused news in the industry. Just Stop Oil activist Louise Harris achieved the <a href="https://www.creativecarbonscotland.com/news-first-ever-climate-song-to-reach-uk-itunes-no-1/">first ever climate song to reach number no.1</a> in the UK iTunes charts. Yo-Yo Ma, Pattie Gonia and Quinn Christopherson hope to have their song as <a href="https://www.npr.org/2023/11/14/1212505811/climate-change-yo-yo-ma">the next anthem for the climate movement</a> whilst Greenpeace unveils a star-studded cover of Fleetwood Mac’s <em>Don’t Stop</em> <a href="https://www.greenpeace.org.uk/news/greenpeace-collaborates-on-star-studded-cover-of-fleetwood-macs-iconic-dont-stop-with-steve-mcqueen-will-poulter-fraser-t-smith-lammas-park-more/">to inspire young activists</a>. More than 60 artists donated their songs for <a href="https://www.spin.com/2023/04/unreleased-michael-stipe-song-leads-earthpercent-x-earth-day-23-compilation/">an EarthPercent compilation album</a> for Earth Day 2023 where proceeds go to organisations fighting the climate crisis.</p><p>Music industry leaders Universal Music, Sony Music and Warner Music <a href="https://www.billboard.com/business/record-labels/warner-sony-universal-music-industry-climate-collective-micc-1235515464/">recently formed Music Industry Climate Collective (MICC)</a> to amplify environmental stewardship and offer guidance on reducing the industry’s carbon footprint. Following this news, it will be interesting to see if this increases the trend of more climate activism across the industry.</p><p>New and emerging artists like <a href="https://www.theguardian.com/music/2023/apr/08/one-to-watch-louis-vi-earthling">North London-based Louis VI</a> increasingly use music to tackle the climate crisis alongside capitalism and colonialism through a wide range of genres such as hip-hop, Afrobeat, jazz, funk and electronic. In his 2023 single <em>Orange Skies</em>, he is mixing “Amazonian rainstorms, British birdsong and uncomfortable truths”:</p><blockquote>Orange skies all on my feed, yea<br>Use the Thames put more water on the trees, yea<br>Mother nature calling me all week, yea<br>Ain’t humans the puzzle to this piece, yea?</blockquote><p>With all of that said, music is incredibly subjective and personal, as is our relationship with climate change and activism.</p><p>Listen to our <a href="https://open.spotify.com/playlist/4EhBYkl4jtI2POrkUxU6Ul?si=814b037fb0f44164">playlist on Spotify</a> and share your environment-themed songs in the comments below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fopen.spotify.com%2Fembed%2Fplaylist%2F4EhBYkl4jtI2POrkUxU6Ul%3Futm_source%3Doembed&amp;display_name=Spotify&amp;url=https%3A%2F%2Fopen.spotify.com%2Fplaylist%2F4EhBYkl4jtI2POrkUxU6Ul&amp;image=https%3A%2F%2Fmosaic.scdn.co%2F300%2Fab67616d00001e0235f1e8c8f603222e0e5505f9ab67616d00001e024be225a31e2e5bb6cfa921f6ab67616d00001e024cb86870ba4e0626ea07e3aeab67616d00001e02d1a727940d3fb3802a3a49e1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=spotify" width="456" height="352" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f48c48b794ed24b4ac7df8aa2b03d46/href">https://medium.com/media/5f48c48b794ed24b4ac7df8aa2b03d46/href</a></iframe><h3>Methodology</h3><p>All of the code for this analysis is written in <a href="https://www.python.org/">Python</a>. The code lives publicly on this <a href="https://github.com/nestauk/discovery_climate_music_analysis/tree/1_analysis">GitHub page</a>.</p><p><strong>Compiling the environment-themed songs playlist</strong></p><p>For this article, we collated a playlist using the <a href="https://conbio.org/images/content_groups/SSWG/climatechangesongs.pdf">Society for Conservation Biology</a> song list, NPR Music’s “<a href="https://music.apple.com/us/playlist/songs-for-climate-change-a-cop26-playlist/pl.99b023b6adee41eca318400052b1a79c">Songs for Climate Change: A COP26 Playlist</a>”, several Spotify playlists <a href="https://open.spotify.com/playlist/5U2r1PS8r2ot6nNLFI1nsv">created by the NY times</a> and Spotify users (found <a href="https://open.spotify.com/playlist/5Ycj62UDyxNMbP0v22jAaa?si=b91aa9e6645d4c2b&amp;nd=1">here</a> and <a href="https://open.spotify.com/playlist/4aKXYksq0EqWX7KLIQqSoZ?si=643a7b48cd59468a&amp;nd=1">here</a>) and crowdsourced some suggestions from Nesta colleagues.</p><p>We reviewed and selected these songs by filtering them based on the following criteria:</p><ul><li>Songs should mention climate or environmental topics in the lyrics (for example, rising temperatures, rising sea levels, pollution and biodiversity). The meaning of the song shouldn’t be overly vague.</li><li>Songs could also be mentioned in online articles and discussions as relevant to the climate conversation. For example, while <em>Here Comes The Sun</em> (1969) by The Beatles was originally not written about climate change, <a href="https://catapult.co/stories/here-comes-the-sun-climate-change-the-beatles-abbey-road-in-this-climate-amy-brady">some have created that association</a> now.</li><li>Songs should have lyrics. This means that instrumental compositions, such as those by the acclaimed UK-based Fara that have been <a href="https://faramusic.bandcamp.com/album/energy-islands-2">inspired by the renewable energy innovation</a> in Orkney, were out of scope for this analysis.</li><li>Lyrics should be in English. We recognise that many of the communities most affected by climate change are not primarily using English language, but we restricted our analysis to songs most likely listened by Anglophones.</li><li>Songs should be found on Spotify. This allows for easy access to data provided by Spotify such as track features but we understand that there are many songs out there that aren’t found on the platform.</li></ul><p>This list was created into a Spotify playlist and we used <a href="https://spotipy.readthedocs.io/en/2.22.1/">Spotipy</a>, a python library for <a href="https://developer.spotify.com/documentation/web-api">Spotify Web API</a> to extract data and features for the analysis. The final list includes songs which were released from the 1960s to 2022.</p><p><strong>Identifying songs that charted in Official Singles Charts (UK)</strong></p><p>To analyse which songs have charted, we used the <a href="https://www.officialcharts.com/charts/singles-chart/">Official Singles Chart Top 100</a>. To scrape the weekly top 100 charts dating from November 1952 to October 2022, we used and adapted <a href="https://medium.com/@caineosborne/analysing-uk-chart-history-1956-to-2017-6fec0ecc991b">python code written by Caine Osborne</a>.</p><p>To find the environment-themed songs in the chart dataset, we utilised the fuzzy-matching method using <a href="https://github.com/nestauk/jacc-hammer">Nesta’s jacc-hammer library</a> and used the threshold score of 0.7 as we found suitable matches from this score and up.</p><p><strong>Characterising Spotify song features &amp; popularity</strong></p><p>To analyse the features of the songs, we utilised the <a href="https://developer.spotify.com/documentation/web-api/reference/get-audio-features">Spotify API track audio features</a> through Spotipy, and focused on “valence”, “danceability” and “energy”.</p><p>Valence, according to Spotify, measures the musical positiveness conveyed in the track on a scale from 0.0 to 1.0. Where tracks with a high valence sound more positive (e.g. happy, cheerful, euphoric) and tracks with a low valence sound more negative (e.g. sad, depressed, angry)</p><p>Danceability is measured on a scale from 0.0 (least danceable) to 1.0 (most danceable) which describes how suitable a track is for dance based on the musical elements of tempo, rhythm stability, beat strength and overall regularity.</p><p>Energy is scored from 0.0 to 1.0 based on the perceptual measure of intensity and activity in a track. Musical elements such as the dynamic range, perceived loudness, timbre, onset rate and general entropy are included to interpret the energy level of the track. For example, an energetic track that is loud, noisy and fast would have a high energy score according to Spotify.</p><p>Additionally, we looked at the popularity. Popularity is measured on a scale from 0 to 100, least popular to the most popular respectively. This is calculated by Spotify’s algorithm which factors the total number of listens and how recent those listens are.</p><p>The final set of songs used in our analysis <a href="https://open.spotify.com/playlist/4EhBYkl4jtI2POrkUxU6Ul?si=814b037fb0f44164&amp;nd=1">can be found on this playlist</a>.</p><h3>Acknowledgements</h3><p>Thank you to <a href="https://medium.com/u/c7cad6c6bfa4">Karlis Kanders</a> for his guidance, and Deborah Fox and Anna Landreth Strong for their insightful feedback and support for this project. I’m grateful to <a href="https://medium.com/u/6b96e5ac1762">Liz Gallagher</a> and <a href="https://medium.com/u/d84118963631">Chris Williamson</a> from Nesta’s Data Science Practice for their help in reviewing the climate song playlist. Thank you to Alexandra Burns, George Richardson and Siobhan Chan for reviewing the report. Also, thanks to Siobhan Chan and Lauren Orso on the communications front.</p><p>I wish to express my sincere gratitude to EarthPercent and specifically Cathy Runciman for her insightful comments on an earlier version of this report. Please note that their input does not necessarily imply endorsement of the findings or discussion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9ea9f4243ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/discovery-at-nesta/songs-of-change-data-reveals-how-musicians-advocate-for-the-environment-d9ea9f4243ad">Songs of change: data reveals how musicians advocate for the environment</a> was originally published in <a href="https://medium.com/discovery-at-nesta">Discovery at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>