<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Liz Gallagher on Medium]]></title>
        <description><![CDATA[Stories by Liz Gallagher on Medium]]></description>
        <link>https://medium.com/@lizgzil?source=rss-6b96e5ac1762------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZSxFKpdQWhv4UuzQF0lc-Q.jpeg</url>
            <title>Stories by Liz Gallagher on Medium</title>
            <link>https://medium.com/@lizgzil?source=rss-6b96e5ac1762------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:41:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lizgzil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Measuring Workforce Greenness via Job Adverts: Part 3, Skills]]></title>
            <link>https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-3-skills-2015fa18b18f?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/2015fa18b18f</guid>
            <category><![CDATA[skills]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[labour-market]]></category>
            <category><![CDATA[net-zero-emission]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 16:21:47 GMT</pubDate>
            <atom:updated>2024-12-10T11:28:44.427Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c_eDjTzJwQwUfkkKUAPE0g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rgaleriacom?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Ricardo Gomez Angel</a> on <a href="https://unsplash.com/photos/two-men-carrying-a-metal-sheet-MagdWoazARo?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>In our <a href="https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-1-industry-cf75997f2f78">first</a> and <a href="https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-2-occupation-99a1dc12bdef">second</a> instalments in a series of posts about green jobs, we explored how at Nesta we are measuring the greenness of the labour market using job adverts.</p><p>Our greenness measures cut across three components:</p><ol><li>Occupation</li><li>Industry</li><li>Skills</li></ol><p>Our previous posts discussed how we used natural language processing techniques to measure green occupations and industries. In this third instalment, we’ll discuss our methodology for determining what proportion of skills in a job advert were green, how well it worked, and some high level analysis of green skills resulting from our work.</p><p>But first, we’ll discuss the datasets we used to work out what exactly a ‘green skill’ was.</p><h3>Defining green skills</h3><p>Unfortunately, there isn’t a single agreed definition of what constitutes a ‘green skill’.</p><p>The European Centre for the Development of Vocational Training (Cedefop) defines them as:</p><p>“<em>the knowledge, abilities, values and attitudes needed to live in, develop and support a society which reduces the impact of human activity on the environment</em>” [1].</p><p>Alternatively, other definitions don’t detail individual skills, but instead roll them up into the context of green occupations. The International Labour Organisation (ILO) doesn’t define “green skills” specifically, but defines “skills for green jobs” as:</p><p>“<em>Skills … that are necessary to successfully perform tasks for green jobs … and to make any job greener. The term includes both core and technical skills, and covers all types of occupations that contribute to the process of greening products, services and processes, not only in environmental activities but also in other sectors</em>” [2].</p><p>More recently, the European Classification of Occupations, Skills and Competences (ESCO) developed a skills taxonomy for the green transition, adding ‘green’ labels to <a href="https://esco.ec.europa.eu/en/about-esco/what-esco">their taxonomy of skills and knowledge concepts</a>.</p><p>Their approach combined a manual cross referencing of ESCO concepts with Cedefop’s green skill definition and the application of a machine learning model to identify green skills.</p><p>This approach resulted in 571 concepts being labelled as ‘green’, for example “coordinate forestry research” was classified as a green skill.</p><iframe src="https://flo.uri.sh/visualisation/17322555/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef1529515f96f44eb217d27416cc3e8b/href">https://medium.com/media/ef1529515f96f44eb217d27416cc3e8b/href</a></iframe><p>Over in the US, there has also been considerable work defining green tasks and topics.</p><p>In 2011, <a href="https://www.onetonline.org/">O*NET</a> revisited and extended their list of green tasks, expertly curating lists of green enhanced skills and green new and emerging occupations — identifying the key green tasks within them.</p><p>This resulted in a list of 1,371 green tasks across 138 occupations, for example <em>“apply principles of specialized fields of science, such as agronomy, soil science, forestry, or agriculture, to achieve conservation objectives”</em> is a task for ’soil and water conservationists’.</p><p>Building further on this, in 2022 O*NET expertly curated 72 green topics, which were much broader than the green tasks, for example “forestry”.</p><p>In our work on green skills, we didn’t want to (and weren’t best placed to) create a new definition for green skills or manually create a bespoke list of green skills.</p><p>We therefore opted to use ESCO’s green skills taxonomy and O*NET’s list of green topics, and focused on trying to understand which of these green skills were being mentioned at a job advert level.</p><p>We were aware there is a lot of nuance in the way green skills are worded in job adverts — so needed to rely on an automated and sophisticated linguistic approach.</p><h3>Finding green skills</h3><p>Our process of finding green skills in job adverts can be split into three steps:</p><p><strong>Step 1</strong>: Skills are extracted from the job advert.</p><p><strong>Step 2</strong>: Extracted skills are semantically mapped to various green terms.</p><p><strong>Step 3</strong>: Skills are classified as green or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsNUOqM6KC2dVUySOC2--Q.jpeg" /><figcaption><em>Overview of the steps to find green skills.</em></figcaption></figure><h4>Step 1: Extracting skills</h4><p>The first step in identifying proportions of green skills is finding out which skills are being asked for in a job advert.</p><p>Luckily for us, Nesta had already trained a Named Entity Recognition (NER) model to extract skills for job adverts, so we were able to use this! This model works well at identifying parts of the job advert which are likely to be describing skills, and is able to extract skills it hasn’t seen before. The table below gives some examples of job advert texts and the skills extracted from them.</p><p>You can read more about how we did this in <a href="https://www.escoe.ac.uk/the-skills-extractor-library/">our blog article about it</a>, dig into some analysis of common skills in <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/exploring-uk-skills-demand/">our interactive blog</a>, or have a go at using it yourself using <a href="https://github.com/nestauk/ojd_daps_skills">our code</a> or <a href="https://www.nesta.org.uk/data-visualisation-and-interactive/skills-extractor-tool/">our online tool</a>.</p><p>The following table shows two examples of skills extracted from job adverts using our NER model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQQrsZymPJXQ0iE2LUd_Dw.png" /></figure><h4>Step 2: Semantically mapping skills to green terms</h4><p>Once extracted, we then found semantic similarities between our extracted skills and various green terms. As mentioned earlier, we used <a href="https://esco.ec.europa.eu/en/about-esco/publications/publication/green-skills-and-knowledge-concepts-labelling-esco">ESCO’s green skills taxonomy</a> and O*NET’s <a href="https://www.onetonline.org/search/green_topics/">list of green topics</a> for this.</p><p>To do this, we embedded all the green ESCO skills (using both the preferred and alternate skill names) and all the O*NET green topics using the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">“all-MiniLM-L6-v2” Sentence Transformers pretrained model</a>. The cosine similarity scores were then calculated, allowing us to find the closest green ESCO skill, the closest O*NET green topic, and the number of O*NET green topics exactly found in the entity (by using exact word matching). The table below shows these scores for a sample of skills.</p><p>During this process we also broke up any particularly long skills into smaller components using a sliding window — this helped with the matching.</p><p>The following table shows semantic similarities between skills extracted from job adverts and the green ESCO taxonomy and the list of O*NET green topics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SMOVuoka_30KfhLKbGeMjw.png" /></figure><h4>Step 3: Classifying skills as green</h4><p>We trained a random forest classifier to classify whether a skill is green or not using three features:</p><ol><li>The cosine similarity score between the embedded entity and the closest ESCO green skill embedding.</li><li>The cosine similarity score between the embedded entity and the closest O*NET green topics embedding.</li><li>The number of O*NET green topics that were found in the entity (using exact word matching).</li></ol><p>Our training data included 971 skills manually labelled as not-green, and 743 labelled as green.</p><p>The final step in our process of finding which proportion of skills in a job advert were green was applying this classifier to each of our extracted skills. If we classified a skill as green then we used the closest ESCO green skill preferred name as a way to standardise the skill’s name — this was useful for our analysis.</p><p>The following table gives the final green skill results from the running example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5POrjk2dfjt0vYLB1eZjdg.png" /></figure><p>Therefore, the first job advert in our running example had 4 skills, 2 of which were green (50%). The second job advert had 3 skills, 3 of which were green (100%).</p><h3>Evaluation</h3><p>For the green skill classifier our test set results are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iISVdjCb_lAdrIySgV8gzQ.png" /></figure><p>We also evaluated the full pipeline by manually labelling 514 skills from 500 job adverts. We labelled how well we thought the skill was extracted from the job advert, and the appropriateness of the green ESCO skill that was mapped to skills classified as green.</p><p>Our findings showed that:</p><ul><li>92% of the time skills are well extracted from job adverts</li><li>54% of the time green skills are excellently mapped to green ESCO skills</li><li>38% of the time green skills are well mapped to green ESCO skills</li><li>8% of the time green skills are poorly mapped to green ESCO skills</li></ul><p>Some examples of the different qualities of mapping to green ESCO skills are below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jb7Rd9egyn9dwg1jUWtYFw.png" /></figure><h3>Analysis of green skills</h3><p>Below are some of our high level results after applying the skills extraction to over 4 million job adverts.</p><p>We can see that the most common green skill was “health and safety regulations” which featured in almost 160,000 job adverts, followed by “implement environmental protection measures” which was in around 20,000 job adverts.</p><iframe src="https://flo.uri.sh/visualisation/17321581/embed" width="700" height="575" frameborder="0" scrolling="no"><a href="https://medium.com/media/033b7cc714bc6830dfdbb87132ca4896/href">https://medium.com/media/033b7cc714bc6830dfdbb87132ca4896/href</a></iframe><h3>Future steps</h3><p>This blog discussed how we extracted green skills from job adverts. Applying this algorithm to millions of job adverts allows us to say something about the greenness of the labour market from a skills perspective. Teamed with the occupation and industry measures of green jobs discussed in the previous blogs, we will be using all three of these measures in our upcoming <strong>Green Jobs Explorer online tool</strong>. This will be released at the end of April — stay tuned for more on this!</p><h3>References</h3><ol><li>Cedefop (2012). Green skills and environmental awareness in vocational education and training. Luxembourg: Publications Office. <a href="https://www.cedefop.europa.eu/files/5524_en.pdf">https://www.cedefop.europa.eu/files/5524_en.pdf</a> [accessed 11/03/24].</li><li>International Labour Organization. 2019. Skills for a greener future: a global view. <a href="https://www.ilo.org/wcmsp5/groups/public/---ed_emp/documents/publication/wcms_732214.pdf">https://www.ilo.org/wcmsp5/groups/public/---ed_emp/documents/publication/wcms_732214.pdf</a> [accessed 11/03/24].</li><li>European Commission (2022) Green Skills and Knowledge Concepts: Labelling the ESCO classification. <a href="https://esco.ec.europa.eu/en/about-esco/publications/publication/green-skills-and-knowledge-concepts-labelling-esco">https://esco.ec.europa.eu/en/about-esco/publications/publication/green-skills-and-knowledge-concepts-labelling-esco</a> [accessed 11/03/24].</li><li>Dierdorff, E. C., Norton, J. J., Gregory, C. M., Rivkin, D., &amp; Lewis, P. (2011). Greening of the world of work: Revisiting occupational consequences. National Center for O* NET Development. Available at: <a href="https://www.onetcenter.org/reports/Green2.html">https://www.onetcenter.org/reports/Green2.html</a> [accessed 11/03/24].</li><li>National Center for O*NET Development (2022) Green Topics: Identifying Linkages to Occupations and Education Programs Using a Linguistic Approach. Available at: <a href="https://www.onetcenter.org/reports/Green_Topics.html">https://www.onetcenter.org/reports/Green_Topics.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2015fa18b18f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-3-skills-2015fa18b18f">Measuring Workforce Greenness via Job Adverts: Part 3, Skills</a> was originally published in <a href="https://medium.com/data-analytics-at-nesta">Data science at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring Workforce Greenness via Job Adverts: Part 2, Occupation]]></title>
            <link>https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-2-occupation-99a1dc12bdef?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/99a1dc12bdef</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[green-economy]]></category>
            <category><![CDATA[just-transition]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Mon, 15 Jan 2024 09:20:59 GMT</pubDate>
            <atom:updated>2024-12-10T12:13:31.683Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How we link green occupation data to our dataset of job adverts.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEfcy2mw079yDHiXQgRRDw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lukethornton?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Luke Thornton</a> on <a href="https://unsplash.com/photos/green-and-red-tractor-on-brown-field-under-blue-and-white-cloudy-sky-during-daytime-_OoduqcpSjM?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>In <a href="https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-1-industry-cf75997f2f78">our first instalment of a series of posts about green jobs</a>, we explored how at Nesta we are measuring the greenness of the labour market using job adverts. Our greenness measures cut across three components:</p><ol><li>Occupation</li><li>Industry</li><li>Skills</li></ol><p>Our previous post discussed how we used Large Language Models (LLMs), fine-tuning and synthetic data to measure green industries. In this second installment, we will discuss how we calculated the greenness of the <strong>occupation component in each job advert</strong>.</p><p>This process involves <strong>standardising</strong> the job title given in the job advert to a Standardised Occupational Classification (SOC) code, and then <strong>joining</strong> this SOC code to datasets of occupation greenness measures.</p><p>Although this post will focus most on the standardising task (as this is the hardest bit), we will also mention some of the green occupational datasets used, and show some results.</p><h3>Assigning Standardised Occupational Classification codes to job adverts</h3><p>There is noise and spelling mistakes in the way job titles are written in job adverts. For example, “Assistant Nurse”, “Nursing assistant”, “Assistent Nurse — London” could all be describing the same occupation.</p><p>Therefore, before we can join job titles to occupation greenness datasets, we need to assign each of our job adverts a standardised occupation. The UK’s standard way of doing this is using <a href="https://www.ons.gov.uk/methodology/classificationsandstandards/standardoccupationalclassificationsoc">Standardised Occupational Classification (SOC) codes</a>. SOC codes were designed by the ONS, with the most recent version updated in 2020. For each occupation there is the 4-digit SOC or a more granular 6-digit extended SOC.</p><p>The ONS SOC dataset is <a href="https://www.ons.gov.uk/methodology/classificationsandstandards/standardoccupationalclassificationsoc/standardoccupationalclassificationsocextensionproject">downloadable from the ONS website</a> and gives the SOC codes of 29,902 job titles. A simplified sample of this dataset is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wIIlWqHFgkm2qSGgzrXyyw.png" /></figure><p>To standardise each of our job adverts to a SOC code, we use a 5-step process to find the SOC code of the most semantically similar job title from this list of 29,902 job titles. These steps are outlined below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UqUgw05IeOh8Xa_2KaOVQ.jpeg" /><figcaption>Overview of the steps used to map a job title to a Standardised Occupational Classification (SOC) code.</figcaption></figure><p><strong>Step 1:</strong> We clean the inputted job advert. This cleaning involves removing words which describe the job conditions but not the job title; e.g. removing common place names or words such as “part-time”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpBvVmx9fcuGf9h02hyVUw.png" /></figure><p><strong>Step 2:</strong> We process the full ONS SOC dataset to create unique job titles to SOC information. (For those familiar with this dataset — we combine the “INDEXOCC NATURAL WORD ORDER”, “ADD” and “IND” columns of these dataset to create unique job titles).</p><p>A small sample of this is:</p><pre>{<br>  &quot;data scientist&quot;: (&quot;2433/02&quot;, &quot;2433&quot;, &quot;2425&quot;),<br>  &quot;data analyst computing&quot;: (&quot;2134/99&quot;, &quot;2134&quot;, &quot;2136&quot;),<br>  &quot;data analyst&quot;: (&quot;3544/00&quot;, &quot;3544&quot;, &quot;3539&quot;),<br>  &quot;analyst developer&quot;: (&quot;2134/03&quot;, &quot;2134&quot;, &quot;2136&quot;),<br>  &quot;meter assembler&quot;: (&quot;8149/00&quot;, &quot;8149&quot;, &quot;8139&quot;),<br>  &quot;motor assembler electric&quot;: (&quot;8141/00&quot;, &quot;8141&quot;, &quot;8131&quot;),<br>  &quot;motor assembler engineering&quot;: (&quot;8142/02&quot;, &quot;8142&quot;, &quot;8132&quot;)<br>}</pre><p>Where we have the unique job title followed by the SOC 2020 6-digit code, the 2020 4-digit code, and finally the 2010 code (which will come in handy later on!).</p><p><strong>Step 3:</strong> We embed these unique ONS job titles and the input job title using the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">all-MiniLM-L6-v2 Sentence Transformers pre-trained model</a>.</p><p><strong>Step 4:</strong> We then calculate the cosine similarity between the embedded input job title and all the embedded ONS job titles. In our running example, these values are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2-CPQ4k6Tx89I615OGhxA.png" /><figcaption><em>* For the purposes of simplifying this running example this value has been changed to 0.50 from 0.47.</em></figcaption></figure><p><strong>Step 5:</strong> Finally, we find the SOC information for the ONS job title with the highest similarity as long as it is over a certain threshold (our default is 0.67). If there is no single job title with a particularly high similarity, then we use a consensus approach at the SOC 2020 4-digit level (the default to having over 3 matches with over 0.5 similarity score). With the default values, the final matches for each inputted job title would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqjpHkR8C7YkPe__j0lgeg.png" /></figure><p>Thus, this algorithm matches inconsistent or unclean job titles to standardised codes; “Assistant Nurse”, “Nursing assistant”, “Assistent Nurse — London” all get mapped to the SOC code “6131/99 — Nursing auxiliaries and assistants n.e.c.”.</p><h3>Thresholds and evaluation</h3><p>There are over 400,000 unique job titles in our dataset, the most common of which with over 62,000 job adverts is “Care Assistant”, however 84% of the unique job titles only occur in one job advert (e.g. ‘Casual Bilingual Assistant Cantonese’). We want our algorithm to perform particularly well on the most common job titles in our dataset, but acknowledge that these are also likely to be the most generic and standardised job titles. Therefore, to balance evaluating the most common job titles with the long tail of infrequent job titles, we evaluated our SOC matching algorithm on two datasets:</p><ol><li>The most common 300 job titles in our job advert dataset (which account for 22% of the job adverts).</li><li>A randomly selected sample of 200 job titles from our job advert dataset</li></ol><h4>Setting a threshold similarity score</h4><p>As mentioned in the steps to our algorithm, we match to a 6-digit SOC code if the similarity score is over a certain threshold. To find the default value for this threshold, we used the evaluation datasets.</p><p>We found the most likely 6-digit SOC match for all the job titles in the evaluation datasets and manually evaluated whether we thought the match was poor, OK or excellent. We weren’t able to perform this evaluation every time due to lack of domain knowledge, but out of the 459 job titles we could confidently quality check, we thought 72% were excellently matched to a 6-digit SOC, 16% were OK matched and 12% were poorly matched.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Yd3f-0mqNtdRaxLHmDPOw.png" /><figcaption>Match quality and cosine similarity scores of matches to 6-digit SOC codes.</figcaption></figure><p>The similarity scores were generally quite high even if the match quality was poor. However, since most of our matches were OK or excellent, we opted to set a lower threshold value of 0.67. With this threshold value we estimate that 92% of the predicted 6-digit SOC matches are likely to be correct.</p><h4>Evaluation</h4><p>With our similarity score threshold of 0.67 we could then properly evaluate the full SOC mapping pipeline. For this, we also manually coded how well we thought any matches to 4-digit SOC were as well.</p><p>Of the 300 most common job titles:</p><ul><li>90% were mapped to 6-digit SOC;</li><li>4% to 4-digit SOC and;</li><li>7% couldn’t be mapped to any suitable SOC.</li></ul><p>These numbers for the 200 randomly sampled job titles were 56% mapped to 6-digit, 5% to 4-digit and 36% not mapped.</p><p>In the most common job titles we find the quality of the SOC match is excellent or OK 92% of the time for those job titles matched to 6-digit SOC, and 100% when matched to 4-digit SOC. For the random sample of job titles, the quality of the SOC match is excellent or OK 89% of the time for those job titles matched to 6-digit SOC, and 90% when matched to 4-digit SOC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eurCEHLO3ToY8WAHsroAbQ.png" /><figcaption>Evaluation results when mapping job titles to SOC codes.</figcaption></figure><p>Some examples of different quality matches are given in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kmC8KxhCyvkzIdtV1B64lQ.png" /></figure><h3>Occupation greenness data</h3><p>Finally, after being able to estimate SOC codes for most of our job adverts, we were then able to link them to datasets on occupational greenness.</p><p>A core dataset on the greenness of occupations was created by the US’s Occupational Information Network (<a href="https://www.onetcenter.org/overview.html">O*NET</a>). In 2009 O*NET curated a list of green occupations by reviewing publications and other sources on workplace topics relevant to the green economy [1]. Through this research and building on work they’d already completed, 215 occupations were classified into three occupational categories; green increased demand (64 occupations), green enhanced skills (60 occupations), and green new and emerging (91 occupations). Although this work is linked to the US economy (and therefore the US SOC coding system) and quite out of date, it still is widely used in understanding the UK’s green economy. For example, GreenSOC developed by the Warwick Institute for Employment Research is influenced by it [2], and the Greater London Authority (GLA) used it in an analysis of London-based green jobs [3].</p><p>Data provided by the GLA allows us to map UK SOC (2010) to the O*NET US SOC taxonomies, therefore finding green occupational categories for each UK SOC. However, this method is slightly flawed since there isn’t a one to one mapping between SOC 2020 codes and SOC 2010 codes. We also utilise <a href="https://www.ons.gov.uk/economy/environmentalaccounts/articles/developingamethodformeasuringtimespentongreentasks/march2022">a dataset of green timeshares</a> provided by the ONS [4] — this is also based off O*NET, and gives estimates of the fraction of time spent doing green tasks per SOC 2010. Finally, we use <a href="https://www.onetcenter.org/reports/Green_Topics.html">a dataset of green topics</a> provided by more recent work by O*NET [5].</p><p>Thus for the SOC code “2433/02 — Actuaries, economists and statisticians” our green measures include:</p><ul><li>O*NET green category: Green New &amp; Emerging</li><li>O*NET green binary (Green / Not Green): Green</li><li>Fraction of time spent doing green tasks (ONS): 12.5%</li><li>Number of O*NET green topics: 39</li></ul><h3>Analysis of green occupations</h3><p>Our algorithm allows us to investigate the greenness of occupations in the UK when we run it on a sample of 1 million job adverts. Some broad findings include:</p><ul><li>SOC codes could be assigned to 82% of the job adverts.</li><li>The most common occupation in our dataset is “Bookkeepers, payroll managers and wage clerks” (SOC 4122/99) which makes up 2.4% of the jobs in our dataset. The job titles ‘Accounts Assistant’, ‘Management Accountant’, ‘Assistant Accountant’, ‘Assistant Management Accountant’, and ‘Accountant’ make up over 50% of the job titles assigned to this SOC code. This occupation has 0% of time spent on green tasks, and therefore one of the least green occupations.</li><li>The occupation with the highest green timeshare is “Production managers and directors in manufacturing” (SOC 1121/00) with 66.7% of the time spent on green tasks. In our dataset 0.5% of our job adverts are for this occupation, where the job titles ‘Service Manager’, ‘Production Manager’, ‘Product Owner’, ‘Engineering Manager’, and ‘Production Supervisor’ make up around 35% of the job titles assigned to this SOC code.</li><li>The occupation with the second highest green timeshare is “Sustainability officers” (2152/05) with 62.5% of time spent on green tasks. 0.06% of our job adverts are for this occupation, where the job titles ‘Sustainability Consultant’, ‘Sustainability Manager’, ‘Senior Sustainability Consultant’, ‘Graduate Sustainability Consultant’, ‘Head of Sustainability’ make up around 30% of the job titles assigned to this SOC code. There are also 55 O*NET green topics associated with this occupation including ‘Air quality’, ‘Energy efficiency’, ‘Green recreation’, ‘Sustainability’, and ‘Water resources’.</li><li>The region with the highest average green timeshare of 7.3% is Leicester, Rutland and Northamptonshire.</li><li>The region with the lowest average green timeshare of 4.9% is Lancashire.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zd_uKlLcmcQBORtLVL2rpA.png" /><figcaption>The average proportion of time spent on green tasks for job adverts in each region (with higher values being shown in darker green). This is calculated by linking each job advert to a SOC code, and linking this to an <a href="https://www.ons.gov.uk/economy/environmentalaccounts/articles/developingamethodformeasuringtimespentongreentasks/march2022">ONS dataset of green timeshares per SOC</a>.</figcaption></figure><h3>Future steps</h3><p>Watch this space to hear about how we calculate our skill-level greenness measures!</p><p>The full codebase is open and <a href="https://github.com/nestauk/dap_prinz_green_jobs/tree/dev">can be found here.</a> Code relevant to the SOC mapping can be found in <a href="https://github.com/nestauk/dap_prinz_green_jobs/tree/dev/dap_prinz_green_jobs/pipeline/green_measures/occupations">our Github repo</a>.</p><h3>References</h3><p>[1] The National Center for O*NET Development (2009) <a href="https://www.onetcenter.org/dl_files/Green.pdf">Greening of the World of Work: Implications for O*NET® -SOC and New and Emerging Occupations</a>.</p><p>[2] Cardenas Rubio, J., Warhurst, C., &amp; Anderson, P. (2022) <a href="https://www.skillsdevelopmentscotland.co.uk/media/q2lhg1v5/green-jobs-in-scotland-report_final-4.pdf">Green Jobs in Scotland: An Inclusive Approach to Definition, Measurement and Analysis</a>.</p><p>[3] Greater London Authority (2022) <a href="https://www.london.gov.uk/sites/default/files/identifying_green_occupations_-_working_paper_99.pdf">Identifying Green Occupations in London</a>.</p><p>[4] Office for National Statistics (2022) <a href="https://www.ons.gov.uk/economy/environmentalaccounts/datasets/estimatedtimespentongreentasksbyoccupationcode">Estimated time spent on green tasks by occupation code</a>.</p><p>[5] National Center for O*NET Development (2022) <a href="https://www.onetcenter.org/reports/Green_Topics.html">Green Topics: Identifying Linkages to Occupations and Education Programs Using a Linguistic Approach</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99a1dc12bdef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analytics-at-nesta/measuring-workforce-greenness-via-job-adverts-part-2-occupation-99a1dc12bdef">Measuring Workforce Greenness via Job Adverts: Part 2, Occupation</a> was originally published in <a href="https://medium.com/data-analytics-at-nesta">Data science at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agent-Based Modelling for Social Impact]]></title>
            <link>https://medium.com/data-analytics-at-nesta/agent-based-modelling-for-social-impact-9cf98af19939?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cf98af19939</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[modelling-and-simulation]]></category>
            <category><![CDATA[agent-based-modeling]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[social-impact]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Tue, 18 Oct 2022 09:11:14 GMT</pubDate>
            <atom:updated>2022-10-18T12:55:40.778Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.nesta.org.uk/">Nesta</a> has the mission of tackling big societal challenges around <a href="https://www.nesta.org.uk/healthy-life/">increasing the number of healthy lives lived</a>, <a href="https://www.nesta.org.uk/fairer-start/">narrowing the outcome gap for children growing up in disadvantage</a> and <a href="https://www.nesta.org.uk/sustainable-future/">accelerating the UK’s transition to a low-carbon future</a>. In part, these require change at scale through targeted innovations and policy interventions. To investigate how well different interventions might impact social behaviour, simulation modelling can help.</p><p>At Nesta, we organise one or two exploration weeks per year. This allows teams to take a break from their usual projects for a whole week to work on something entirely new. In this article, I will talk you through the reflections and findings that a small team of us (<a href="https://medium.com/u/d84118963631">Chris Williamson</a>, <a href="https://medium.com/u/16a312f50050">juan.mateos-garcia</a>, <a href="https://medium.com/u/475394ab1373">George Richardson</a>, <a href="https://medium.com/u/edb5a788c3a8">Jack Vines</a>, Jack Rasala, <a href="https://twitter.com/Chicco_Stat">Federico Andreis</a> and I) in the Data Analytics Practice (DAP) have gathered during our latest Exploration week. The team traditionally focuses on machine learning methods in our projects. So, taking on the ethos of exploration during our Exploration week, we chose to see<strong> how agent-based modelling (ABM, a type of simulation modelling) could be useful for Nesta</strong>.</p><h3>What is agent-based modelling (and why might it be useful for Nesta)?</h3><p>Agent-based modelling (ABM) is a method for modelling the behaviour of a system by simulating autonomous “agents” that interact with each other and their environment. Complex macro-level behaviours can be simulated this way, even from quite simple individual rules for the agents.</p><p>One of the first ABMs was Thomas Schelling’s 1971 <a href="https://en.wikipedia.org/wiki/Schelling&#39;s_model_of_segregation">model for segregation</a>. This model considers two groups of agents who have a certain level of intolerance for each other. Each agent moves around a grid until a certain percentage of its neighbours are from the same group as the agent. Although segregation of the two groups isn’t coded into the model, this behaviour emerges under certain intolerance levels (ie, macro-level behaviour emerges from simple agent-based rules). What’s interesting about this model is that even with quite low levels of intolerance, segregation will still occur.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*jt-5b4egNWps3jpUdJVhTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*QptCGP9bVw-DvsSiiiP5mw.png" /><figcaption><em>The starting (left) and final (right) states of Schelling’s agent-based model where segregation between red and blue groups has emerged. </em><a href="http://nifty.stanford.edu/2014/mccown-schelling-model-segregation/"><em>Simulation by Frank McCown</em></a><em>.</em></figcaption></figure><p>From Schelling’s model many others have been developed — from the work on <a href="https://ee.stanford.edu/~hellman/Breakthrough/book/pdfs/axelrod.pdf">cooperative behaviours</a> by Robert Axelrod and colleagues (eg, Axelrod, 1997) to Neil Ferguson’s contentious model for <a href="https://www.nature.com/articles/d41586-020-01003-6">controlling the spread of Covid-19</a>. There are many <a href="https://www.researchgate.net/publication/324900247_Agent-Based_Modelling">more examples of ABMs and their history</a>. ABMs lend themselves well to modelling social dynamics, since we often know more about the rules individuals follow than the rules the whole system follows. Because of this, ABMs could be useful for Nesta as a means of understanding the impact of different social policy decisions. Most relevant to Nesta, we could use ABMs to explore food environments and learn more about which interventions might make them easier to navigate. Alternatively, we could use ABMs, heat pump electricity usage and efficiency data to forecast future demand on the electricity grid. Of course, ABMs aren’t the only type of method to model social dynamics. Examples of other methods include modelling <a href="https://mathworld.wolfram.com/PopulationGrowth.html">population growth via differential equations</a>, <a href="https://phys.org/news/2019-01-hydrodynamics-crowds.html">crowd behaviour using fluid dynamics</a> or <a href="https://www.econlib.org/library/Enc/ForecastingandEconometricModels.html">wealth forecasting using econometrics</a>.</p><h3>Our ABM exploration</h3><p>We packed a lot into our ABM exploration week and wanted to both read some of the literature <em>and</em> learn how to code ABMs. Thus, our week consisted of reading about, making and discussing ABMs.</p><h4>Reading about ABMs</h4><p>The week kicked off with a couple of days of reading about ABMs in areas Nesta is interested in — such as <a href="https://www.nesta.org.uk/project/innovation-gaps-in-heat-pump-installation/">heat pump adoption</a>, <a href="https://www.nesta.org.uk/project-updates/improving-the-nations-health-through-tackling-obesity/">tackling obesity</a>, <a href="https://www.nesta.org.uk/blog/why-good-ideas-sometimes-spread-so-badly-and-what-do-about-it/">innovation diffusion</a>, <a href="https://www.nesta.org.uk/blog/collective-action-snowflake-snowball/">collective action</a> and <a href="https://www.nesta.org.uk/feature/six-stories-workplace-automation/">job automation</a>.</p><p>We read the Centre for Net Zero’s <a href="https://www.centrefornetzero.org/wp-content/uploads/2022/05/ABM-Report-Final.pdf">“Hitting the Target” report</a> in which an ABM was used to model the impact that five different scenarios had on the adoption of heat pumps. This allowed them to forecast whether the UK’s current policy interventions, as well as possible interventions, would lead to the 2028 target for heat pump installations. Another heat pump-related <a href="https://dora.dmu.ac.uk/bitstream/handle/2086/11924/JEPO-D-15-00133_en_pol_rhi_paper.pdf?sequence=1&amp;isAllowed=y">report by Snape <em>et al</em> (2015)</a> discussed an ABM for the non-financial barriers to heat pump adoption, where households (the agents) are influenced in their heating system of choice by their neighbours, the cost, the “hassle” and the climate.</p><p><a href="https://ric-colasanti.github.io/Methods/NNM/papers/The%20application%20of%20modeling%20and%20simulation%20to%20public%20health_.pdf">Giabbanelli, Tison and Keith’s 2021 paper</a> on the application of ABMs to public health was also of interest. In this they review and critique the ABMs used to help policy makers understand food behaviour interventions to tackle obesity. Their critique focuses on how much (if at all) these models have been evaluated on their technical quality (eg, parameter sensitivity analysis).</p><p>We also read some papers to do with innovation diffusion and development. A <a href="https://www.sciencedirect.com/science/article/abs/pii/S0048733313000498">paper from Lopolito <em>et al</em> from 2013</a> explored an ABM for the interaction of firms to guide the development and diffusion of new technologies and how policy interventions can affect this. A <a href="https://link.springer.com/article/10.1007/s00191-020-00665-9">2020 paper by He and Lee</a> discussed the effects of social culture (eg, individualism or uncertainty avoidance) on innovation diffusion using an ABM.</p><p>Another paper that caught our eye was that of <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0269976">Chenoweth <em>et al</em> from 2022</a> on nonviolent activist strategies. The ABM developed here looked at the outcomes of three activism strategies, finding which strategies are best at influencing the opponents’ support network.</p><p>We also read a <a href="https://royalsocietypublishing.org/doi/full/10.1098/rsif.2020.0898">paper from del Rio-Chanona <em>et al</em> (2021)</a> who created a model for occupational mobility and used it to test out different responses to job automation scenarios. Another paper we read by <a href="http://jonathancogliano.com/wp-content/uploads/2018/10/Cogliano-Agent-Based-Approach-to-Value-Theory.pdf">Cogliano (2012)</a> explored the emergence and the stability of profit under Karl Marx’s “labour theory of value”, by including agents which exchange commodities and make decisions about their allocation of labour.</p><h4>Making ABMs</h4><p>As well as reading, we wanted to gain experience of creating ABMs and coding them. A few members of the team explored using the <a href="https://mesa.readthedocs.io/en/latest/">Mesa Python package</a> (which was particularly useful for the visualisation of ABMs), and some other members tried coding them from scratch in Python. Although less than a week wasn’t very long to design, code and experiment with a model, <a href="https://chris-williamson.medium.com/">Chris</a>, <a href="https://juan-mateos-garcia.medium.com/">Juan</a> and I did manage to make some first steps with them (although evaluating them and gaining actual insight from them would require more time).</p><p>I was interested in the effect of increased police power on the spread of ideas during a protest — something which may change the effect protests have after the new <a href="https://www.gov.uk/government/publications/police-crime-sentencing-and-courts-bill-2021-factsheets/police-crime-sentencing-and-courts-bill-2021-protest-powers-factsheet">Policing Act</a>. Key components to my model were agents having a propensity for disruption during a protest which, if over a threshold, would lead to arrest by any nearby police. Agents were influenced by their neighbours’ amount of disruption, and the overall success of a protest was proportional to the amount of disruption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*i5b5MR0ttAdEhQNjvGZGlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*x-G7XVVWnssfwBKsGU_phw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*r9yxHw_zfizQnpTx_So5jQ.png" /><figcaption><em>Frames from our agent-based models for protests (left), hungry fish (middle) and research discovery (right).</em></figcaption></figure><p>In order to explore ABMs in continuous space rather than on a grid, <a href="https://chris-williamson.medium.com/">Chris</a> built a “fish tank model” in which fish move around and eat smaller fish which in turn increases their size. Fish can either be risk-averse (prioritising escaping bigger fish) or risk-taking (prioritising eating smaller fish).</p><p>During the week <a href="https://juan-mateos-garcia.medium.com/">Juan</a> refined his <a href="https://twitter.com/JMateosGarcia/status/1549073066101186562">model for research discovery</a> in which agents search for discoveries in a research landscape. This model could be used to see what happens to research discoveries when agents are more risk adverse or interested in generating publications vs. making useful discoveries (these two things are not always the same!).</p><h4>Discussing ABMs</h4><p>The combination of reading about ABMs and designing and coding them lead to some interesting broad discussions. After coding the simulation, each of us felt like the bigger task ahead was the parameter selection and analysis (for example, I explored the effect the number of police, and how often they moved, had on my model outcomes). We talked about how we could generate large datasets of simulation runs with different parameter values and analyse the type of model behaviour (which is also something <a href="https://www.nature.com/articles/s41598-019-42006-2">I explored in relation to forager mobility</a> in my pre-data science life).</p><p>At times our discussions turned to the more theoretical aspects of ABMs — whether all ABMs boil down to the same few templates (eg, cooperation, segregation)? Have all our potential ABM ideas already been published (there are a lot of heat pump adoption ABMs out there)? How can we measure the impact our assumptions have on the results? How much complexity should we add to our ABMs (when does adding complexity start clouding our ability to interpret the model)? How much data do you need to develop an ABM (and what is the interaction between simulation results and the data)? What would it take to get policy makers to utilise the predictions from an ABM and to do so responsibly? What proportion of time should you spend trying to produce nice videos of your ABM (videos are useful for understanding if the model is doing what it should be, but aren’t useful in developing a broad understanding from the model)?</p><p>We also came up with a few areas which ABMs might be of use to Nesta. One such use case would be about <a href="https://www.nesta.org.uk/blog/our-failing-food-system/">food environment mapping</a> — where we could use an ABM to model social networks around food pantries, and find what sort of interventions might make these environments easier to navigate. Another area might be to expand on the Centre for Net Zero’s ABM by using heat pump electricity usage and efficiency data to forecast future demand on the electricity grid. We found that a generic ABM framework that kept coming up in our discussions was how a policy intervention could affect the diffusion of a behaviour and/or information spread and dissipation, as well as generating any unintended consequences. This sort of framework could be built upon to model all sorts of things relevant to Nesta’s missions — for example to promote the uptake of early years government support, to encourage heat pump installation, and to enable people to eat more healthily.</p><h3><strong>Conclusion</strong></h3><p>All in all, we got a lot out of our ABM week and it was great to have some dedicated time to read papers and experiment with ideas. We found agent-based modelling coding and analysis quite a fun challenge (at one point the idea was floated to only use Python’s standard library to code them!), and we felt like some of our other data science skills might come in useful for analysing how model behaviours change as parameters vary. Lots of Nesta’s research questions are based around understanding how policy interventions can affect social behaviour — which we felt seems quite suitable for agent-based modelling.</p><h3><strong>References</strong></h3><p>Axelrod, R. (1997). The complexity of cooperation. In The Complexity of Cooperation. Princeton university press.</p><p>Centre for Net Zero. (2022). Hitting the Target, <a href="https://www.centrefornetzero.org/wp-content/uploads/2022/05/ABM-Report-Final.pdf">https://www.centrefornetzero.org/wp-content/uploads/2022/05/ABM-Report-Final.pdf</a></p><p>Chenoweth, E., Hocking, A., &amp; Marks, Z. (2022). <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0269976">A dynamic model of nonviolent resistance strategy</a>. PloS one, 17(7), e0269976.</p><p>Cogliano, J. F. (2012). <a href="http://jonathancogliano.com/wp-content/uploads/2018/10/Cogliano-Agent-Based-Approach-to-Value-Theory.pdf">An agent-based approach to classical-Marxian value theory and labor mobility</a>.</p><p>Gallagher, E. M., &amp; Bryson, J. J. (2017). <a href="https://jjb.conjugateprior.org/ftp/GallagherBrysonABM-authorsfinal.pdf">Agent-based modelling</a>. Encyclopedia of Animal Cognition and Behavior, 1–9.</p><p>Giabbanelli, P. J., Tison, B., &amp; Keith, J. (2021). <a href="https://ric-colasanti.github.io/Methods/NNM/papers/The%20application%20of%20modeling%20and%20simulation%20to%20public%20health_.pdf">The application of modeling and simulation to public health: Assessing the quality of agent-based models for obesity. </a>Simulation Modelling Practice and Theory, 108, 102268.</p><p>He, M., &amp; Lee, J. (2020). <a href="https://link.springer.com/article/10.1007/s00191-020-00665-9">Social culture and innovation diffusion: a theoretically founded agent-based model</a>. Journal of Evolutionary Economics, 30(4), 1109–1149.</p><p>Lopolito, A., Morone, P., &amp; Taylor, R. (2013). <a href="https://www.sciencedirect.com/science/article/abs/pii/S0048733313000498">Emerging innovation niches: An agent based model</a>. Research Policy, 42(6–7), 1225–1238.</p><p>MRC Centre for Global Infectious Disease Analysis, covid-sim, GitHub repository, <a href="https://github.com/mrc-ide/covid-sim">https://github.com/mrc-ide/covid-sim</a></p><p>del Rio-Chanona, R. M., Mealy, P., Beguerisse-Díaz, M., Lafond, F., &amp; Farmer, J. D. (2021). <a href="https://royalsocietypublishing.org/doi/full/10.1098/rsif.2020.0898">Occupational mobility and automation: a data-driven network model</a>. Journal of the Royal Society Interface, 18(174), 20200898.</p><p>Schelling, T. C. (1971). Dynamic models of segregation. Journal of mathematical sociology, 1(2), 143–186.</p><p>Snape, J. R., Boait, P. J., &amp; Rylatt, R. M. (2015). <a href="https://dora.dmu.ac.uk/bitstream/handle/2086/11924/JEPO-D-15-00133_en_pol_rhi_paper.pdf?sequence=1&amp;isAllowed=y">Will domestic consumers take up the renewable heat incentive? An analysis of the barriers to heat pump adoption using agent-based modelling</a>. Energy Policy, 85, 32–38.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cf98af19939" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-analytics-at-nesta/agent-based-modelling-for-social-impact-9cf98af19939">Agent-Based Modelling for Social Impact</a> was originally published in <a href="https://medium.com/data-analytics-at-nesta">Data science at Nesta</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Much Does Wellcome Fund Technology?]]></title>
            <link>https://medium.com/wellcome-data/how-much-does-wellcome-fund-technology-6084ccbb643?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/6084ccbb643</guid>
            <category><![CDATA[software]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[funding]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Fri, 07 May 2021 15:14:17 GMT</pubDate>
            <atom:updated>2021-05-07T15:14:17.259Z</atom:updated>
            <content:encoded><![CDATA[<p>This piece was co-written by <a href="https://medium.com/u/329e25ce3c58">Becky Knowles</a><em>.</em></p><p>Data and tech are increasingly important to science. At Wellcome, we support data scientists and software engineers who create tools that use data in an innovative way. We put trust into practice by changing how data and software in research are funded, developed and governed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qF5naahA2gVRR5_D1aqpRw.jpeg" /><figcaption><em>Data and tech are increasingly important to science. Photo from </em><a href="https://www.pirbright.ac.uk/news/2019/11/pirbright-enters-bioimaging-collaboration-diamond-light-source"><em>https://www.pirbright.ac.uk/news/2019/11/pirbright-enters-bioimaging-collaboration-diamond-light-source</em></a></figcaption></figure><p>Although we suspected that a significant proportion of Wellcome’s grants have used or created tech in some way, we didn’t have a clear view of Wellcome’s existing tech portfolio. So, we brought teams across Wellcome and University College London to find a reproducible method for identifying and tracking these grants. We combined Wellcome’s data science expertise that is used internally to make better decisions with our externally-focused <a href="https://medium.com/wellcome-data/introducing-data-for-science-and-health-at-wellcome-de49b069e6b">Data for Science and Health</a> team to identify the scale of Wellcome-funded tech.</p><p>But first…</p><h4><strong>What do you mean by ‘tech’?</strong></h4><p>By tech, we’re referring to coding scripts, software or tools, particularly those that have been shared with others. This can include open-source software, packages, coding scripts that perform a certain function (such as data cleaning, linkage, or visualisation, and statistical or data science models).</p><p>In this piece we will discuss how we built a machine learning model to predict ‘tech grants’ in our portfolio.</p><h3>How did we do it?</h3><h4><strong>The Dataset</strong></h4><p>By reading the title and description of grants we managed to tag around 300 grants as having produced some sort of tech and around 300 as not. This process wasn’t easy — and took several people many hours to read through. The dataset of grants and their descriptions is openly available via <a href="https://grantnav.threesixtygiving.org/publisher/360G-wellcome">360Giving</a>.</p><h4><strong>Experimentation</strong></h4><p>We then used the tagged dataset to train a natural language processing model that can take a grant’s description and make a prediction of whether the grant produced tech or not. We trained a variety of models — 12 in total, which included a vectorizer step (turning the words into numbers using methods including <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>, word counts, and <a href="https://huggingface.co/bert-base-uncased">BERT</a> and <a href="https://huggingface.co/allenai/scibert_scivocab_uncased">SciBERT</a> pretrained embeddings) and a classifier step (including logistic regression, Naïve Bayes and SVM). The test metrics for these models are given in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*OQ66OFaU-5k5DgqF3-rFHA.png" /><figcaption>Test metrics for our different models, using a prediction probability threshold of 0.55.</figcaption></figure><p>We experimented with using different combinations of these models in an ensemble, as well as different prediction probability thresholds needed for a grant to be classified as a tech grant. The test precision and recall scores of all these experiments, are shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*HsBFj4kxSTuIFoiTJBXHKg.png" /><figcaption><em>Precision and recall scores for 12 single models (coloured by their vectorizer type, classifier type not shown), and ensemble models (black) with all possible combinations of the 12 models with a range of different probability thresholds.</em></figcaption></figure><h4><strong>Our model</strong></h4><p>As you can see, in general, using an ensemble of models can dramatically increase test metrics (the top right of the plot being the best area)! However, after all this experimentation with ensembles we found our single BERT vectorization + logistic regression classifier model performed really well, especially when using a prediction probability threshold of 0.55. On our test dataset the model achieved a 0.9 precision and recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*0hpb1uiZliO4HDYudkBy6A.png" /><figcaption><em>Test set results, 0 stands for not tech and 1 for tech.</em></figcaption></figure><h4><strong>Fairness</strong></h4><p>We performed several group fairness calculations to see if there were any groups for which the model performed better than others. Something that might be important to check is not using the model to reenforce ideas about the “<a href="https://en.wikipedia.org/wiki/Golden_triangle_(universities)#Members">golden triangle</a>” (Oxbridge and London) universities being <a href="https://theconversation.com/oxbridge-and-london-retain-grip-on-university-research-prowess-35650">more worthy of funding than others</a>. Thus, we grouped the grants by the grant holder’s organisation and calculated the test metrics for each group, the results were as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAQ4JYvdXDr3mUnmo4zQrQ.png" /><figcaption><em>Test results when grouping the test data by whether the grant holder’s organisation is in the golden triangle universities or not.</em></figcaption></figure><p>We have to be careful about how we interpret the model results because it is slightly better at picking up tech from the golden triangle (performing about 0.05 higher on both precision and recall — which applied to the whole dataset would account for a difference in about 1000 grants incorrectly labelled). For universities outside the golden triangle the model is more likely to make errors about whether the grant has tech. Therefore, in future work we do using this model, we need to avoid drawing definite conclusions about how much universities from the golden triangle use tech in comparison to other universities.</p><h4><strong>Evaluation</strong></h4><p>When research produces technology it may be written about in an academic paper. Furthermore, many researchers have to self-report any outputs from their grants, including technology.</p><p>Therefore, using publication data from <a href="https://europepmc.org/">EPMC</a> which was linked to a Wellcome Trust grant number, we labelled 148 publications as having demonstrated producing technology. We then passed the model over these grant descriptions and found that 58% of them were predicted as tech grants. Similarly with self-reported data we have from <a href="https://researchfish.com/">ResearchFish</a> we were able to find 70 grants which reported producing technology, for these only 40% were predicted as tech grants. Although these numbers are small, the model didn’t have this extra information when making its predictions, so we are pleased to recall at least some of the tech grants.</p><h3>What are the Tech Grants?</h3><p>After training the model we used it to make predictions on all 16,854 grants in the open-source dataset — which covers grants from 2005 to 2019. This gave us 3562 predicted tech grants — 21% of all the grants. This accounted for around £2 billion of funding — 27% of the total funding for these grants.</p><p>The largest number of tech grants from any grant type came from the PhD Studentship (Basic) Award, although the tech grants only make up 25% (458 out of 1856) of all grants from this grant type. The Open Access Awards also have a large number of tech grants, and the tech grants make up 99% (424 out of 428) of all grants from this grant type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*mxE5dWUM_PPk6OtSsydz_A.png" /><figcaption><em>The 20 most common grant types for tech grants (blue) with the number in all grants also given (pink).</em></figcaption></figure><p>Wellcome has some well-established schemes to support technological resources that benefit the wider scientific community, such as the Biomedical Resource Grants and Technology Development Grants. The proportion of tech grants in these is quite high — 80% (127 out of 159) and 74% (42 out of 57) respectively, but contrary to what we may have expected there are many other grant types producing tech.</p><h4><strong>Clustering</strong></h4><p>Using the grants’ description, we can visualise semantic similarities in 2D by using TF-IDF vectorization and the <a href="https://umap-learn.readthedocs.io/en/latest/">UMAP</a> dimensionality reduction algorithm. Then the <a href="https://en.wikipedia.org/wiki/DBSCAN">DBSCAN</a> clustering algorithm is then applied to this data. This whole process can be quite easily implemented using the TextClustering function from <a href="https://pypi.org/project/wellcomeml/">WellcomeML</a>.</p><p>The image below shows all the grant data in 2D — each point is a grant. The proportion of tech grants was found for each cluster, and used to colour the points. Hence, we can see that the central grants in yellow/peach are from clusters with a high proportion of tech grants — and grants in dark blue are from clusters with particularly low proportions of tech grants.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GK9POEFj-W7DW6-_h3UR_A.png" /><figcaption><em>All grants (points) in the dataset plotted in 2D using their semantic similarity. Grants are coloured by the proportion of tech grants in the cluster they are grouped into.</em></figcaption></figure><p>Clusters with the 5 highest and 5 lowest proportions of tech grants are in the table below. Keywords about the cluster are generated using those with the highest TF-IDF values. The most tech was found in genomics, neuroscience, urban health, and data resources grants. Areas that had least tech were history of medicine, cellular and molecular biology. Although cellular and molecular biology was identified as an area with the least amount of tech, this could reflect the fact that tech is perhaps more ‘hidden’ in these areas — where researchers didn’t mention the tech in their grant descriptions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRN28ib9K7XAraINnyC0ig.png" /><figcaption>Keywords from clusters with the 5 highest and 5 lowest proportions of tech grants.</figcaption></figure><h3>Conclusion</h3><p>Different models had quite different predictive power — but with experimentation we managed to find one which gave a 0.9 precision and recall. Using this we predicted that 20% of the grants dataset used were tech grants.</p><p>When we applied the model to times where we knew a grant had produced tech based off other sources — publications or self-report, we were only able to recall a small portion of these grants. We know the model performs well when the grant description mentions technology, so in these cases it appears that the grant produced technology but never wrote about it in their grant description. We think this poor performance speaks to an abundance of ‘hidden tech’ in Wellcome’s portfolio: tools and scripts that have been created by researchers during their grant but the grant holders didn’t feel the need to include in the main description.</p><p>A large number of tech grants come from PhD Studentship (Basic) Award and Open Access Awards — combined these account for 882 tech grants. The tech grants were less commonly Technology Development Grants or Biomedical Resources Grants (combined these account for 169 tech grants), contrary to what was anticipated. Instead, the tech grants represented an extensive range of award schemes, reflecting the increasing reliance of science upon data and tech across all research areas.</p><h4><strong>Going forward</strong></h4><p>Given the abundance of hidden tech across Wellcome’s portfolio, we’d like to support researchers better to allow them to share tech with each other and with Wellcome. As a funding body, we need to:</p><ul><li>Demonstrate that tech and code are important outputs of research in their own rights, not only valued by their associations with published papers</li><li>Give credit to people who share their tech with us and other researchers (for example, those who follow open science principles)</li><li>Make it easy for people to tell us about their tech — some people were only too keen to give us examples (see <a href="https://twitter.com/MikhailovDanil/status/1290339790194462720">this Twitter thread</a>), but they may not have a mechanism for doing so.</li></ul><h3>Thanks</h3><p>The grants data used for this analysis is openly accessible <a href="https://wellcome.org/grant-funding/funded-people-and-projects">here</a>. The project’s code can be found on <a href="https://github.com/wellcometrust/nutrition-labels">Github</a>.</p><p>This project was managed by <a href="https://medium.com/u/329e25ce3c58">Becky Knowles</a>. The data science parts of this project were done by <a href="https://twitter.com/NonieAlexander">Nonie Alexander</a> and <a href="https://medium.com/u/6b96e5ac1762">Liz Gallagher</a>. Special thanks to <a href="https://medium.com/u/a24226700f0e">Antonio Campello</a> and <a href="https://medium.com/u/fb9f30dc242a">Nick Sorros</a> for code reviews and their input to <a href="https://pypi.org/project/wellcomeml/">WellcomeML</a> which was used in this project. Thanks also to <a href="https://medium.com/u/cd8adcec3702">Aoife Spengeman</a> on fairness discussions and help tagging training data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6084ccbb643" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wellcome-data/how-much-does-wellcome-fund-technology-6084ccbb643">How Much Does Wellcome Fund Technology?</a> was originally published in <a href="https://medium.com/wellcome-data">Wellcome Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fairness of Finding People’s Names in Text]]></title>
            <link>https://medium.com/wellcome-data/the-fairness-of-finding-peoples-names-in-text-4d5b8d41bc79?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d5b8d41bc79</guid>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Tue, 19 Jan 2021 10:10:18 GMT</pubDate>
            <atom:updated>2021-01-19T10:10:18.676Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Is Our Algorithm Better at Finding John Smith than Other Names?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j4IPWzF7t0n_2Gf8" /><figcaption><em>Figure 1 Is our model equally good at identifying all these people’s names from this COVID-19 </em><a href="https://apps.who.int/iris/bitstream/handle/10665/333074/WHO-2019-nCoV-Policy_Brief-Long-term_Care-2020.1-eng.pdf?sequence=1&amp;isAllowed=y"><em>WHO policy brief document</em></a><em>?</em></figcaption></figure><p>We trained a named entity recognition (NER) model to find people in policy document text — described in <a href="https://medium.com/wellcome-data-labs/searching-for-people-in-text-5017a4526a4">this article</a>. The repercussions of this model performing differently for different names could be quite damaging. For example, what if the model worked the best for finding popular US male names, but is less effective at finding minority ethnic names? We don’t want our tool to contribute to issues of exclusion and under-representativeness in academic research.</p><p>As outlined in previous Data Labs articles (<a href="https://medium.com/wellcome-data-labs/how-we-embed-ethical-thinking-in-our-product-team-a-framework-1ea92c271ac3">here</a> and <a href="https://medium.com/wellcome-data-labs/is-machine-learning-fair-7974ca5d7cf9">here</a>), as a team we are embedding assessing fairness in the tools and models we build, and thus in this blog post I will describe how we did this for our NER model using a synthetic dataset.</p><p>This post has three parts;</p><ol><li>How we created the synthetic dataset,</li><li>Our fairness results (TL;DR the model is unfair, but not too bad),</li><li>Retraining the model using synthetic data (TL;DR the model performs worse, but the model fairness improves).</li></ol><h3><strong>How we did it — creating synthetic data</strong></h3><p>In order to evaluate the fairness of the NER model we need to tag each name in a corpus of texts with metadata about the name. For example, is it a predominantly male, female or neutral name, is the name common to a particular ethnicity? That way we could calculate and compare the performance of the model on various subsets of the data, for example, how does it do on predicting just the male names in comparison to just the female names?</p><p>To get this dataset of text and name metadata we could simply tag the names in the model’s test data with the various metadata about the name. However, tagging the names with this metadata would be time consuming and prone to its own biases (what if you came across a name you hadn’t heard of before), so we opted to create a synthetic dataset.</p><h4>Policy text templates</h4><p>For this we needed to take sections of policy document text, and tag the first names and/or surnames. We then used these tagged text fragments as templates to insert other names into.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mIhldSBYCVA2fvsM" /><figcaption><em>Figure 2 Creating synthetic data from an original policy document text.</em></figcaption></figure><h4><em>Names datasets</em></h4><p>The names we chose to insert into the text templates to create the synthetic data came from a variety of datasets. The ones we included are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1EMatwahldsl8jbzORVzA.png" /><figcaption>Datasets of peoples’ names from <a href="https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/livebirths/datasets/babynamesenglandandwalestop100babynameshistoricaldata">ONS</a>, <a href="https://en.wikipedia.org/wiki/List_of_most_popular_given_names">Wikipedia</a>, <a href="https://www.nltk.org/book/ch02.html">NLTK</a>, <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/TYJKEZ">Harvard Dataverse</a>, and the <a href="https://www.census.gov/topics/population/genealogy/data/2000_surnames.html">US Census</a>.</figcaption></figure><p>Most of these can be further broken down by gender, country, continent, and ethnicity.</p><h4>Synthetic data</h4><p>Thus by inserting the different names datasets into the policy text templates we can create various groups of synthetic data to evaluate fairness between.</p><p>In total, we tagged first names and surnames in 157 text extracts from a random selection of policy documents. In these we tagged 506 surname entities and 177 first name entities. To create the synthetic data each of the 157 text extracts was repeated 50 times with different random names inserted from one of the datasets. Thus, we evaluate the model on a total 8,850 first name entities and 25,300 surname entities.</p><h3>Fairness Results — there are differences in how well different names perform</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*YsZIN5wVJ07wbU8Zp7k7bg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@tingeyinjurylawfirm?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tingey Injury Law Firm</a> on <a href="https://unsplash.com/s/photos/scale-comparison?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>This is how well our model performed on some of the different groups of synthetic data (with the model’s test data performance at the top), more results are given in the <a href="https://docs.google.com/document/d/1pLFtA9k2iz6p6HJH21gP6qqE5Ao8e8piNGi4_kLa3Lg/edit?usp=sharing">Appendix 1</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gHF7npG1geaDozPWURDDjA.png" /><figcaption>How well the model performs on different groups of names.</figcaption></figure><p>We debated as a team about how to interpret the results of the different performance metrics. It’s common just to look at accuracy rate or F1, but we know that it is better to look at a range of metrics as they each tell you something different about how the algorithm is performing. For example F1 can be misleading in cases where the opposite types of error have occurred in different groups, i.e. male names may be frequently mistakenly assigned (false positives) and female names may be frequently not found (false negatives), but the overall F1 for both groups may still be the same.</p><h4>Significance and differences in groups’ results — how do we know what is “too” different?</h4><p>Thus, from the results above we could say certain pairs of results have a X% difference in the metric, but is this difference significant? We repeated the evaluation of each dataset 20 times with different independent subsets of the synthetic data — allowing us to perform a two-sample t-test to see if we can reject the null hypothesis that the means of these sets of results are the same.</p><p>To do this, first we identified datasets that it would be interesting to see the difference between and performed the t-test. When the p-value is less than 0.05 we can say there are significant differences in the average of these 20 evaluation results between the 2 datasets. Some of the pairs which we found to be significant in at least one of the three metrics are given below, with the full results in the <a href="https://docs.google.com/document/d/1pLFtA9k2iz6p6HJH21gP6qqE5Ao8e8piNGi4_kLa3Lg/edit?usp=sharing">Appendix 2</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ttyIu8oSJNw0Prq8H_AUow.png" /><figcaption>The difference in model results between two groups of names. *The p-value for the t-test is less than 0.05</figcaption></figure><p>We didn’t find significant differences in how the model performed in any of the metrics between male and female names when using the NLTK and the Wikipedia data. We also didn’t see statistically significant differences in the white US surnames and Asian pacific islander and hispanic surnames, or the white US first names with black US first names datasets. Thus it seems like we are generally only seeing gender and ethnicity biases when using the UK first name and Wikipedia datasets respectively.</p><h4>Conclusions — Asian first names perform worst, US first names perform best</h4><p>Thus, we have found some biases with our NER model. It performs significantly better for US surnames than it does with double barrelled US surnames (+10% on all metrics). Another difference in results is between United States Wikipedia first names and Asia Wikipedia first names — the latter perform about 5% worse on the F1 metric, 7% on the recall metric, and 2% worse on the precision metric. In general, comparing US and UK Wikipedia first names with non-UK and non-US first names, we see the latter performs 4% worse on the F1 score, 2% on precision and 6% worse on recall. Comparing gendered names performance using the NLTK or the Wikipedia data shows no significant difference in results, however comparing the UK male first names vs UK female first names datasets do show significant differences for the recall metric but only of 2% worse.</p><p>To conclude, we see the worst performance when popular first names in Asia are used and the best when popular first names in the US are used.</p><p>We note that it is not straightforward to tell how much of a difference between groups is a noteworthy difference, even if the difference is statistically significant. We decided to take a user-centred approach and we thought about the impact of a 10 point difference between groups (which was the highest difference found). If the model is used 100 times, then that means that in 10 of those times it will perform worse for one of the groups. Is this noteworthy? Considering that our tool currently has very few users, we decided that this was not something that would significantly impact users, but it is something we will need to review in the future.</p><h3>Retraining using synthetic data — a trade-off of our model’s general performance and being less unfair</h3><p>After creating this synthetic data we thought we’d see what effect it had on the training of the model. Does it perform better on real test data when trained on these extra names?</p><p>We decided to just add synthetic data from the 4 worst performing names datasets; ‘Wikipedia first names — Not US or UK’, ‘US surnames — double barrelled’, ‘UK first names — female’, and ‘Wikipedia first names — Asia’. We added one random synthetic data point from every text template in the training data for each of the 4 datasets, therefore multiplying the size of the training set by 4. To avoid over-fitting it was important to not use the same text templates in the training and test sets. In comparison to our previous model where no synthetic data was added to the training data, we see a slight increase in the precision metric but a big decrease in the recall metric. Since our training data has a different distribution to our test data this result might be expected — the problem has become harder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P57Hto6DlfGQsypKXhMA-w.png" /><figcaption>Model results when adding synthetic data to the training data.</figcaption></figure><p>We can also evaluate the fairness results again on this newly trained model. The results show an improvement (I’ve given the previous results in brackets for ease of comparison):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UC8m0z1WzFWHCW7nExwVfw.png" /><figcaption>The difference in model results between two groups of names, using a model trained with synthetic data. For ease in comparison, the previous results from the model with no synthetic training data are given in brackets. *The p-value for t-test is less than 0.05</figcaption></figure><h3>Final thoughts</h3><p>As may be expected, our analysis of fairness in our NER model showed that our model performs slightly better for US/UK first names. Furthermore, the largest difference comes between US first names and Asian first names. We have also shown how by adding synthetic training data we don’t necessarily see an improvement in the model performance, but we do see an improvement in the fairness of the model.</p><p>We hope we have highlighted one way in which you can uncover model unfairness in a relatively simple way by creating synthetic data, our approach is similar to [1]. We’ve also shown how in this analysis you may be faced with a decision of improving this fairness at the expense of decreasing the performance of the model overall. A nice review of some tools for evaluating model fairness and bias is covered in this <a href="https://towardsdatascience.com/evaluating-machine-learning-models-fairness-and-bias-4ec82512f7c3">post</a>.</p><h3>References</h3><p>[1] <a href="https://arxiv.org/pdf/2007.05124.pdf">DiCiccio, C., Vasudevan, S., Basu, K., Kenthapadi, K., &amp; Agarwal, D. (2020, August). Evaluating fairness using permutation tests. In <em>Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining</em> (pp. 1467–1477).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d5b8d41bc79" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wellcome-data/the-fairness-of-finding-peoples-names-in-text-4d5b8d41bc79">The Fairness of Finding People’s Names in Text</a> was originally published in <a href="https://medium.com/wellcome-data">Wellcome Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Searching for People (in text)]]></title>
            <link>https://medium.com/wellcome-data/searching-for-people-in-text-5017a4526a4?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/5017a4526a4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[entity-linking]]></category>
            <category><![CDATA[policy-making]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Tue, 18 Aug 2020 14:34:17 GMT</pubDate>
            <atom:updated>2020-08-18T14:34:17.046Z</atom:updated>
            <content:encoded><![CDATA[<p>Part of Data Labs’ newly released product <a href="https://reach.wellcomedatalabs.org/">Reach</a> is the ability to search for research being mentioned by global health organisations — you can read more about Reach <a href="https://medium.com/wellcome-data-labs/introducing-reach-find-and-track-research-being-put-into-action-dec2a2fca93b">here</a>. For example, you might want to see all the policy documents which have something to do with malaria. You also might want to use the tool to search for the times a particular person, e.g Chris Whitty, has been mentioned in policy documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvuAasRvYgwBiWy8evIl-w.jpeg" /><figcaption>St Bernard dogs effectively search for people in the Alps, we want to effectively search for people in policy text. Head of a St. Bernard dog. Reproduction of an etching by F. Lüdecke.. Credit: <a href="https://wellcomecollection.org/works/f2qcgqz9">Wellcome Collection</a>. <a href="https://creativecommons.org/share-your-work/public-domain/pdm/">Public Domain Mark</a></figcaption></figure><p>We are now working towards optimising searching for people. In a simple approach to searching text we can just search for someone’s name and see if there is an exact match for it somewhere. If we do this for ‘Whitty’ we currently find 176 mentions, most of which are to do with a person called Whitty. However, there are cases where the name you are searching for is a bit more ambiguous — if we search for the name ‘Wood’ we find 4136 mentions, but a large number of these are false positives (i.e. not discussing a person, but rather places like ‘Wood Green’ and ‘Old Wood Farm). To solve this problem we use <a href="https://en.wikipedia.org/wiki/Named-entity_recognition">named entity recognition</a> (NER) and <a href="https://en.wikipedia.org/wiki/Entity_linking">entity linking</a>.</p><h4>Finding people using named entity recognition</h4><p>Luckily for us we can use a pre-trained named entity recognition (NER) model from <a href="https://spacy.io/">spaCy</a> to find people entities within the text of policy documents. SpaCy’s model uses <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">convolutional neural networks</a> trained on a <a href="https://catalog.ldc.upenn.edu/LDC2013T19">large bank</a> of English language text including telephone conversations, news articles and web-blogs. Some things that this model recognised as people’s names were ‘Prison Finder’, ‘Nine Elms’, ‘Bandstand Marathon’, and ‘Foodborne Diseases’. The reason this doesn’t perform too well is because the text the model was trained on wasn’t from the same domain as what we are applying the model to — i.e. policy document text. But all is not lost, we can tag some people’s names in policy text and give the model this data to retrain on.</p><h4>Creating training data — tagging names</h4><p>We used a tool called <a href="https://prodi.gy/">Prodigy</a> to tag all the names of people within chunks of samples of policy text. So far we have tagged 1,033 people in 316 chunks of policy text. This process involves being presented with a few sentences of text and having to highlight all the people you think there are in it, an example of this in action is shown below.</p><figure><img alt="A screenshot of using Prodigy to tag people entities" src="https://cdn-images-1.medium.com/max/1024/0*Ei65V8uBR4W3D9um" /><figcaption>Tagging people in policy text using Prodigy</figcaption></figure><h4>Named entity recognition results</h4><p>SpaCy’s pretrained <em>en_core_web_sm </em>model achieves a performance metric (we use the <a href="https://en.wikipedia.org/wiki/F1_score">F1 Score</a>, where a score of 1 would be perfect predictions, you can read more about this metric <a href="https://trustnet.wellcome.ac.uk/news/linking-fairness-metrics-and-machine-learning-our-approach">here</a>) of 0.58 for person entities on our tagged data, and retraining it with this extra data we tagged gives us a performance metric of 0.83 for person entities. These scores can be broken down by where in the document the person entity was found:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*aQA8gzWOAK9fqfOSKSrk9A.png" /><figcaption>How well the model performs when the name is in different parts of a policy document. We see it does particularly well for names in a references section, but less well for names in citations within the text.</figcaption></figure><p>Two examples of finding people entities tagged in policy documents using our NER model are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-i2TqhXy_NKq_mc1" /></figure><p>Here you can see an example of a person being correctly tagged as a person (‘Manisha Shridhar’), but an address in New Delhi (‘Mahatma Gandhi Marg)’ being incorrectly tagged as a person (‘Marg’ being the Hindi and Punjabi word for ‘road’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QlOcsESbiDDAyN1e" /></figure><p>Here the citation to a reference by ‘Maitland’ is correctly identified as a person, but ‘Kenya’ has been misidentified as a person.</p><h4>Disambiguating people’s names</h4><p>The next (and slightly more ambitious) thing we can do is to link the people found with who they are, this is called entity linking/disambiguation. For example, searching for ‘Farrar’ gives us sentences which are to do with a few different people called ‘Farrar’ such as:</p><blockquote>… Trends Microbiol. 2015;23(7):429–36. doi:10.1016/j.tim.2015.02.006.69. <strong>Farrar</strong> D, Duley L, Medley N, Lawlor DA. Different strategies for diagnosing gestational diabetes to improve maternal and infant health. Cochrane Database Syst Rev. 2015;(1):CD007122. Review. 70. Fioretti BT, Re…</blockquote><p>and</p><blockquote>… School of Nursing, Federal University of Minas Gerais, Belo Horizonte, MG, Brazil Dr Jeremy <strong>Farrar</strong>, Director, Oxford University Clinical Research Unit in Viet Nam, The Hospital for Tropical Diseases, Ho Chi Minh City, Viet Nam Professor Maria Guzman, Head, Virology Department…</blockquote><p>So how do we know that the first of these examples is about the Bradford-based maternal health researcher Diane Farrar, and the second is about the director of the Wellcome Trust, ex-director of the Oxford University Clinical Research Unit in Vietnam and infectious disease researcher, Jeremy Farrar?</p><p>To disambiguate them we need a ‘knowledge base’ — a collection of text about each person. For this we use the information publicly available on <a href="https://orcid.org/">ORCID’s</a> website (an organisation dedicated to providing unique identifiers for researchers). The way we’ve approached this so far is to download all the ORCID text available for certain names — in this example ‘Farrar’. Then we convert these texts to numbers using TF-IDF (term frequency — inverse document frequency) in order to be able to compare them quantitatively.</p><h4>Linking people’s names to their ORCID</h4><p>TF-IDF vectorisation gives weights for each of the words in the ORCID texts based on how frequently they turn up and how unique they are to the document, e.g. the word ‘the’ comes up a lot in Jeremy Farrar’s ORCID profile, but it also comes up a lot in every other ORCID profile, so this word is given a low weighting. The highest weighted words in <a href="https://orcid.org/0000-0002-2700-623X">Jeremy Farrar’s ORCID</a> TF-IDF vector are ‘dengue’, ‘Vietnam’, ‘Vietnamese’, ‘influenza’, ‘meningitis’; and those in <a href="https://orcid.org/0000-0002-5625-761X">Diane Farrar’s ORCID</a> are ‘diabetes’, ‘pregnancy’, ‘gestational’, ‘women’ and ‘Bradford’.</p><p>We compute the vectorisation for the sentence containing the person we want to disambiguate, and then find the closest TF-IDF vector to this from all the ORCID profiles using the <a href="https://en.wikipedia.org/wiki/Cosine_similarity">cosine distance</a> between the two vectors. If the cosine distance to the closest TF-IDF vector is less than a threshold then we will classify this person entity as not having an associated ORCID.</p><h4>Entity linking results</h4><p>As a proof of concept we looked at how well we could disambiguate people with the name ‘Farrar’. Of the 168 mentions of a ‘Farrar’ in the policy text we found 81 were about Jeremy Farrar), 9 were about Diane Farrar, 1 was about <a href="http://xn--0000000186565157-fg6iee/">John Farrar</a> and 77 were about people who don’t have an ORCID. Our model achieved a performance metric of 0.62.</p><h4>Next Steps</h4><p>More training data — we could always do with more training data in order to make these models better. In particular it’d be good to have more training data from people found in parts of policy documents outside of references, since these currently perform less well.</p><p>We also want to test how well the entity linking does on people other than those with the name ‘Farrar’, so need to expand this feature.</p><p>We need other sources for the knowledge base. Since not everyone has an ORCID we could also try to get text from other sources, one option is Wikipedia.</p><p>We are still learning about how to improve this feature. If you have any thoughts or you are interested in testing out this feature, then please get in touch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5017a4526a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wellcome-data/searching-for-people-in-text-5017a4526a4">Searching for People (in text)</a> was originally published in <a href="https://medium.com/wellcome-data">Wellcome Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using deep learning to find references in policy documents]]></title>
            <link>https://medium.com/wellcome-data/using-deep-learning-to-find-references-in-policy-documents-cd3b3f1099e3?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd3b3f1099e3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[funding]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Thu, 25 Jun 2020 15:58:33 GMT</pubDate>
            <atom:updated>2020-06-25T15:58:33.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7n9MCERyHsLXxZvu_3VFmw.jpeg" /><figcaption>Neural networks’ namesake. Neurons derived from neural stem cells. Credit: <a href="https://wellcomecollection.org/works/ru3ktp9z">Yirui Sun</a>. <a href="https://creativecommons.org/licenses/by/4.0/">Attribution 4.0 International (CC BY 4.0)</a></figcaption></figure><p><em>This blog was written by Matt Upson from </em><a href="https://www.muanalytics.co.uk/"><em>Mu Analytics</em></a><em> whilst contracting with Wellcome Data Labs.</em></p><p>One of the key concerns for any organisation that does or funds research is: where does that research go, who does it reach, and what impact does it have. At the Wellcome <a href="https://wellcome.ac.uk/about-us/teams/wellcome-data-labs">Data Labs</a>, we’re working on an open source tool called <a href="https://reach.wellcomedatalabs.org/">Reach </a>which helps to answer this question by looking for references to academic publications published in global policy documents, for example those produced by policy organizations such as the World Health Organisation, Médecins Sans Frontières, and the UK government.</p><p>A key functionality of Reach is the ability to recognise academic references within these ‘grey literature’ policy documents which are not captured by the traditional academic publishing metrics.</p><h3>Finding references is disappointingly hard</h3><p>The main source of difficulty is the great variety of typographic formats in which policy documents are published, whilst almost universally these documents are published in portable document format (pdf) which is difficult to read automatically (there are some great examples of this from <a href="https://www.filingdb.com/pdf-text-extraction">fillingdb</a>). In addition, there is a huge variety in the possible ways to present bibliographic information; the <a href="https://github.com/citation-style-language/styles">Citation Style Language</a> project records over 2000 possible reference formats, presenting a challenge to software automatically trying to extract references. Moreover some policy documents do not adhere to any known citation style at all; sometimes all you get is a web link.</p><p>All these factors come together to make the task of <em>reference extraction</em> (finding a reference within a text) and <em>reference parsing</em> (finding the individual components like author, title, etc.) quite tricky. In the current implementation of Reach, we use a combination of deterministic and probabilistic methods to deal with this problem. The process looks like this:</p><ol><li>A pdf is read by Reach, and a simple heuristic determines which pages contain reference sections</li><li>The reference sections are passed to a splitter model which uses machine learning to determine where a reference starts and ends, returning a list of all the found references</li><li>Each found reference is processed by another machine learning algorithm to classify whether a token (word) is an author, title, year, volume, etc.</li><li>The title is then fuzzy matched against a database of known publications to return a unique document ID (for example a <a href="https://en.wikipedia.org/wiki/PubMed_Central">PMCID</a> or <a href="https://www.sciencemag.org/site/misc/doi.xhtml">DOI</a>)</li></ol><h3>Deep Learning</h3><p>We wanted to see if we could improve on our relatively <a href="https://medium.com/wellcome-data-labs/structuring-references-from-text-using-machine-learning-19bdcc3bf655">naive machine learning models</a> (used in steps 2 and 3) with a deep learning approach that would perform better and be simpler to evaluate and maintain. Deep Learning is a subset of machine learning that uses artificial neural networks (ANNs) to learn progressively simpler representations of a problem, allowing a computer to build complex concepts out of simpler ones.</p><p>In their 2018 paper, Rodrigues et al. used a deep learning model called a Bi-Directional Long-Term Short-Term (BiLSTM) Recurrent Neural Network (RNN) to solve a similar problem to Reach but in the arts and humanities. This type of model excels at sequence data like text documents. We planned to replicate their model but specifically for references to medical references in policy documents.</p><h3>No bricks without clay</h3><p>With this new approach, we needed data from policy documents that had been labelled at the token (word) level. This means that every single token in the document needs to be manually annotated, for example the reference:</p><p><em>Dambisya YM, Kadama P, Matinhure S, Malema N, Dulo C (2013). Literature review on codes of practice on recruitment of health professionals in global health diplomacy…</em></p><p>becomes:</p><pre>(Dambisya, author) (YM, author) (,, author) (Kadama, author) (P, author) (,, author) (Matinhure, author) (S, author) (,, author) (Malema, author) (N, author) (,, author) (Dulo, author) (C, author) ((, o) (2013, year) (), o) (., o) (Literature, title) (review, title) (on, title) (codes, title) (of, title) (practice, title) (on, title) (recruitment, title) (of, title) (health, title) (professionals, title) (in, title) (global, title) (health, title) (diplomacy, title)</pre><p>This annotation process is manual and laborious. We used a tool called <a href="http://prodi.gy/">prodigy</a> and developed a number of our <a href="https://github.com/wellcometrust/deep_reference_parser/tree/master/deep_reference_parser/prodigy">own tools</a> to make it significantly faster, but even so the manual annotation task took a few weeks.</p><h3>Open source for the win</h3><p>Rodrigues et al. (2018) published their model openly, providing us with a great headstart to building and training our own model. Nonetheless we needed to make quite a few changes to prepare the model for use in a production setting — details of which, and results, can be found <a href="https://github.com/wellcometrust/deep_reference_parser/blob/master/README.md#the-model">here</a>. After a few attempts we were able to achieve good success on the individual reference splitting and parsing tasks using a combination of Rodrigues’ and our own data. Like Rodrigues et al. we also combined our splitting and parsing models together into a single model which reduces the resource requirements for training and using the model in Reach. This multi-task model also achieved good results.</p><p>These models are quite time consuming to create, and take between 11 and 16 hours to run on a high performance machine in the cloud with a cost of between about $10 and $14 each time we train it.</p><p>We have published these models openly on G<a href="https://github.com/wellcometrust/deep_reference_parser">itHub</a>. If you have some basic knowledge of python you can get started very quickly with just two lines of code.</p><h3>Next steps</h3><p>It has taken a number of months to go from the idea of using deep learning to a workable implementation. Along the way we’ve learnt some valuable lessons, not least having to rethink the backend architecture of Reach in order to accommodate the more computationally intensive deep learning models, and developing a way for data scientists to get access to high power machines for training deep learning models.</p><p>The obvious next step is to annotate more data and retrain the model as we expand Reach to take in more sources of policy documents, but for now the results are quite encouraging.</p><h3>References</h3><p>Rodrigues Alves, D., Colavizza, G., &amp; Kaplan, F. (2018). Deep Reference Mining From Scholarly Literature in the Arts and Humanities. Frontiers in Research Metrics and Analytics, 3(July), 1–13. <a href="https://www.frontiersin.org/articles/10.3389/frma.2018.00021/full">https://doi.org/10.3389/frma.2018.00021</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd3b3f1099e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wellcome-data/using-deep-learning-to-find-references-in-policy-documents-cd3b3f1099e3">Using deep learning to find references in policy documents</a> was originally published in <a href="https://medium.com/wellcome-data">Wellcome Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structuring References from Text Using Machine Learning]]></title>
            <link>https://medium.com/wellcome-data/structuring-references-from-text-using-machine-learning-19bdcc3bf655?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/19bdcc3bf655</guid>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 09:59:10 GMT</pubDate>
            <atom:updated>2019-10-29T09:59:10.026Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Using a naïve Bayes classifier in </em><a href="https://scikit-learn.org/stable/"><em>scikit-learn</em></a><em> to predict whether a sentence from a references section is an author, title, journal, etc</em></p><p>In Wellcome Data Labs we have been developing a tool which flags any Wellcome Trust funded publications found in the reference sections of policy documents (e.g. <a href="https://apps.who.int/iris/">WHO </a>and <a href="https://www.nice.org.uk/guidance">NICE</a> guidelines). This relies on us being able to categorise the reference titles as such from a large amount of unstructured text. An example of this unstructured text is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwUuL7la6-Y770kJxv5m_A.png" /><figcaption>What unstructured text from a policy document looks like. How can we extract all the titles from millions of lines of this type of text?</figcaption></figure><p>How do we know which parts of this text are the titles (or journal names or authors for that matter)? To explain this we will use a much simpler example.</p><p><strong>Is a sentence from a magazine about cats or a magazine about dogs?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ge54Kkp_py4EyrU2bHjwyQ.png" /><figcaption><a href="https://www.behance.net/gallery/29697633/The-Cat-magazine"><em>“The Cat magazine”</em></a><em> by Ryan O’Hara is licensed under </em><a href="https://creativecommons.org/licenses/by-nc-nd/4.0/?ref=ccsearch&amp;atype=rich"><em>CC BY-NC-ND 4.0</em></a></figcaption></figure><p>Imagine we have some sentences and a tag about whether they are from a magazine about cats or a magazine about dogs. Using this data, we can use supervised machine learning to build a model to predict whether a new sentence is likely to be from a cat magazine or a dog magazine.</p><p>The first step is to vectorise the tagged sentences. One way to do this is taking every single word (columns) in all the sentences and calculate for each sentence (rows) how many of each word it uses. With an example of 6 basic sentences this looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jNnF-yfxyx2W2C7PAXUTQw.png" /></figure><p>You may also end up only using the top 1000 words, or making everything lower case, or removing common words (e.g. ‘and’).</p><p>Next you can build a model from this vectorised data — one type of model (and the one we use) is a multinomial <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">naïve Bayes classifier</a>. This model uses some quite simple maths to work out how likely each word in a sentence is to be in a cat or dog magazine. Then by multiplying all these probabilities we can predict whether a sentence is more of a dog or a cat magazine sentence (a simple example of this maths can be seen <a href="https://monkeylearn.com/blog/practical-explanation-naive-bayes-classifier/">here</a>).</p><p>I trained a model with these 6 sentences and then used it to predict whether 5 new sentences were from a dog or cat magazine, the results were:</p><pre>+ — — — — — - - - - - - - - + — — — — — - - + — — — — — — - +<br>| Sentence to predict       | Prediction    | Probability   |<br>+ — — — — — - - - - - - - - + — — — — — - - + — — — — — — - +<br>| dog                       | dog           | 0.82          |<br>| cat                       | cat           | 0.66          |<br>| this is a cat sentence    | dog           | 0.63          |<br>| dog or cat dog            | dog           | 0.72          |<br>| dog or cat                | dog           | 0.53          |<br>+ — — — — — - - - - - - - - + — — — — — - - + — — — — — — - +</pre><p>All this was done using CountVectorizer and MultinomialNB from the Python package <a href="https://scikit-learn.org/stable/">scikit-learn</a>. With an input of the training sentences and classifications, in only a few lines of code we can build a model and predict the categories of 5 new sentences:</p><pre>&gt;&gt;&gt; training_sentences = [&#39;the cat sat on the mat&#39;, &#39;dogs are better than cats&#39;, &#39;i love dogs&#39;, &#39;jake is a dog&#39;, &#39;the cat&#39;, &#39;the dog is nice&#39;]</pre><pre>&gt;&gt;&gt; training_classifications = [&#39;cat&#39;, &#39;dog&#39;, &#39;dog&#39;, &#39;dog&#39;, &#39;cat&#39;, &#39;dog&#39;]</pre><pre>&gt;&gt;&gt; vectorizer = CountVectorizer(analyzer=’word’, token_pattern=r’(?u)\b\w+\b’)</pre><pre>&gt;&gt;&gt; count_words_sentences = vectorizer.fit_transform(training_sentences)</pre><pre>&gt;&gt;&gt; mnb = MultinomialNB()</pre><pre>&gt;&gt;&gt; mnb.fit(count_words_sentences, training_classifications)</pre><pre>&gt;&gt;&gt; predict_vec_list = vectorizer.transform([&#39;dog&#39;, &#39;cat&#39;, &#39;this is a cat sentence&#39;, &#39;dog or cat dog&#39;, &#39;dog or cat&#39;]).toarray()</pre><pre>&gt;&gt;&gt; mnb.predict(predict_vec_list)</pre><p>So, going back to our reference example …</p><p><strong>How do we categorise titles/authors/journals in sentences?</strong></p><p>We have lots of training data available from <a href="https://europepmc.org/">EMPC</a>, where we can download thousands of titles, authors and journal names (amongst other reference attributes), e.g.:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*Bkx6dT2yBIdlv0lUlWMWbg.png" /></figure><p>This data is then vectorised and a multinomial naïve Bayes classifier is trained using it. We can then split up the unstructured policy document text up — where we split by full stop, exclamation mark and question mark — and then use the trained model to predict what type of sentence (or ‘reference component’) it is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEH-XE2Ata6KIhkhPgd9QA.png" /></figure><p>This data shows the most likely category for each reference component as predicted by our model, and how probable the model thought this category was.</p><p>As you can see, it doesn’t always predict the categories correctly (I don’t think there is a Dr ‘J Acquire Immune Defic Syndr’ out there), but we can estimate how much we can trust our model. One metric for this is the accuracy score. Using a new dataset (i.e. not the dataset we trained the model on), the ‘test’ data, we calculate how many times a category was correctly predicted divided by the total size of the test data. In our case the accuracy score for the model predicting titles is 87%. You can see (<a href="https://towardsdatascience.com/metrics-to-evaluate-your-machine-learning-algorithm-f10ba6e38234">this blog post</a>) about other ways to evaluate your model.</p><p>And voilà, we’ve gone from lots of text to a neat list of the titles (or authors etc) within it, which can then be matched to a database of titles from Wellcome Trust acknowledged publications!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19bdcc3bf655" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wellcome-data/structuring-references-from-text-using-machine-learning-19bdcc3bf655">Structuring References from Text Using Machine Learning</a> was originally published in <a href="https://medium.com/wellcome-data">Wellcome Data</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding image pathways]]></title>
            <link>https://stacks.wellcomecollection.org/finding-image-pathways-12d31ae347f9?source=rss-6b96e5ac1762------2</link>
            <guid isPermaLink="false">https://medium.com/p/12d31ae347f9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[museums]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Liz Gallagher]]></dc:creator>
            <pubDate>Thu, 05 Sep 2019 15:57:52 GMT</pubDate>
            <atom:updated>2019-09-05T15:57:52.250Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Using images from the Wellcome Collection archive (which you can search through <a href="https://wellcomecollection.org/works">here</a>), we made pathways of ‘connecting’ images. This helps us expose the collection to audiences who might want to look at what we have and also produces some quite nice plots. In this article, I will explain how we found these pathways.</p><p>Our collection is very varied and contains a lot of medical-related images from the past.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4sXxlZhziROZioXSIuFh5A.png" /><figcaption>Examples of the types of images in our collection</figcaption></figure><p>Inspired by Google’s <a href="https://artsexperiments.withgoogle.com/xdegrees/8gHu5Z5RF4BsNg/BgHD_Fxb-V_K3A">X Degrees of Separation</a> we wanted to take 2 images from our collection and find a path between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qLttKLegdrP4QjYZplrfqQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*To5-WUOcV6ch6U2_ffEPfQ.jpeg" /><figcaption>How can we link a scanning electron micrograph of a pollen grain with a photograph of Florence Nightingale?</figcaption></figure><h3>Feature vectors</h3><p>We can use a pretrained model from torch vision (<a href="https://pytorch.org/docs/stable/torchvision/models.html">VGG16</a>) to classify what the subject of an image is. The model outputs a single prediction (e.g. cat, truck, person) using a 16-layer convolutional neural network (read about how this was trained in <a href="https://arxiv.org/pdf/1409.1556.pdf">this</a> article). For the image of Florence Nightingale, the model predicted this to depict a “kimono” (suggesting we might like to retrain the model with some of our own images!).</p><p>Rather than taking the final prediction (i.e. “kimono”), if we take the weights from the layer before the output layer, we have a rich 1x1x4096 feature vector of information about the image. The feature vector for the photograph of Florence Nightingale looks something like this:</p><pre>[[-2.9406, -1.8962, -0.6647,  ..., -2.5079, -0.2727, -1.0538]]</pre><p>Feature vectors for each of the Collection’s 120,576 images were found in this way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*lBdjiF4jiMKycs-jMh4gWw.png" /><figcaption>The structure of the 16-layer VGG16 neural network. If we output the weights from the layer before the output layer then we have a 1x1x4096 feature vector. Image taken from <a href="https://neurohive.io/en/popular-networks/vgg16/">https://neurohive.io/en/popular-networks/vgg16/</a></figcaption></figure><p>If we reduce the dimensionality of all those feature vectors to 2D (for the sake of visualisation) we get a galaxy of the images, where you’ll see similar ones close together. On the top right we have the oval framed black &amp; white portraits, we have some electron microscopy images on the bottom left, and botanical paintings just above the centre.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OpUSFSJuh_Zf7z1vcoeWqQ.png" /></figure><h3>Image similarity</h3><p>To find the relatedness between two images we can measure the cosine distance (the angle between two vectors) between their feature vectors.</p><p>These two images have very similar feature vectors:</p><figure><img alt="Images ‘L0078444’ and ‘L0078481’" src="https://cdn-images-1.medium.com/max/1024/1*S4SD1UKOLl1Ny3hClX5JiQ.png" /></figure><p>The two are quite similar:</p><figure><img alt="Images ‘V0003760’ and ‘V0006594’" src="https://cdn-images-1.medium.com/max/1024/1*RXkBbcIjC8V26GBYglWonA.png" /></figure><p>And these two are very different:</p><figure><img alt="Images ‘B0000663’ and ’V0014173&#39;" src="https://cdn-images-1.medium.com/max/1024/1*2wHdpE69BF3Sc_3dOygbfA.png" /></figure><h3>How to find a path</h3><h4>Graph paths</h4><p>Our first approach to finding paths between images using their feature vectors was to use graphs.</p><p>First of all, we found the cosine distances between the feature vectors for every pair of images. Then for each image (a node in the graph), we connected it (an edge in the graph) to the 3 images which had the smallest cosine distances with it. We then used <a href="https://networkx.github.io/">NetworkX</a> to create a graph where the weights for each edge were the cosine distances between image neighbours.</p><p>We can then apply popular pathway finding algorithms on this graph, for example, <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">Dijkstra’s algorithm</a> or the <a href="https://en.wikipedia.org/wiki/A*_search_algorithm">A* search algorithm</a> (which is what Google’s <a href="https://artsexperiments.withgoogle.com/xdegrees/8gHu5Z5RF4BsNg/BgHD_Fxb-V_K3A">X Degrees of Separation</a> uses). Although these work well, they don’t allow you to specify how many nodes you want in the path. Thus, we also experimented with a graph based algorithm to find <em>n</em> node pathways which had minimum variance in the distances between nodes.</p><p>Building a graph using the top 3 distances of a sample of 10,000 images took about 100 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5OH-Wnbyi6iKjhFZ43m9zw.png" /><figcaption>A path between two images using Dijkstra’s algorithm.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9AGEZ50p20ZlKdkAOPCAg.png" /><figcaption>The pathway found using Dijkstra’s algorithm</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKME9kfq2nVCRAkHyOvlEw.png" /><figcaption>The pathway found using our graph-based algorithm</figcaption></figure><h4>How we found the path</h4><p>We then decided that we might not need to use graphs, and instead came up with our own algorithm to find paths just using the feature vectors. We were happier with the results of this algorithm <em>and</em> it was quicker to run.</p><p>This is our algorithm for a path of size <em>n</em>:</p><p><strong>Step1:</strong> Create a line between the feature vectors for your start and end images, and find the coordinates of (<em>n</em>-2) equally spaced points between the start and end of the line (the ‘ideal’ vectors).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*jhgJ_1dygdGTLYksuOeXlg.png" /></figure><p><strong>Step 2:</strong> Find the cosine distances between the first ideal vector and the feature vectors for all of the images (not including the start and end image feature vectors). Select the closest one as the first node of the pathway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*H4fUqEmDSmOTny7X5yir-Q.png" /></figure><p><strong>Step 3:</strong> Repeat step 2 for each ideal vector in order, but don’t include pathway nodes from the previous ideal vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*xwOa45LhVR9gXo6FMj0mkQ.png" /></figure><p>And thus we have found a pathway between a pollen grain and Florence Nightingale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*te93--Q1mwNFjAZK-PsmuA.png" /><figcaption>A pathway between a pollen grain and Florence Nightingale.</figcaption></figure><p>It’s hard to fully understand why each step in this pathway was chosen as being close together. Sometimes there are some obvious similarities, but sometimes the model may have picked up on some small detail of the texture of the image which might not be obvious to a human.</p><p>We see that two images of Florence Nightingale have come up in this pathway which are practically the same (we have a lot of very similar images in our collection!). To a human these look very similar and it is surprising they’ve both come up in a supposedly equally spaced pathway, but in feature space details like the change from sepia to black and white, and the slight difference in zoom might have meant that the feature space is different enough.</p><p>We can visualise this path in the 2D space of all the images. Note that although some images are plotted in very similar locations, the 2D data that went into making this plot has been dramatically reduced from the original 4096D of data that was used to decide the pathway. Thus in 4096D this pathway would look a lot more equally spaced (if we could visualise that).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJvNu0WgTl2mJEh7YhvXRQ.png" /></figure><h3>More paths!</h3><p>Here are some more fun paths we found:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AKmNRjPzXmiRm5-KsN4pQ.png" /><figcaption>The pathway from image ‘A0000001’ to image ‘B0006893’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJ_nyl61S9ByG_SxijssCA.png" /><figcaption>The pathway from image ‘V0044783’ to image ‘V0046793’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNM5K6QHY9rZe6zHsMaXrw.png" /><figcaption>The pathway from image ‘V0044783’ to image ‘V0046793’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWPvR0v7HgyKlpFnGPMYew.png" /><figcaption>The pathway from image ‘L0045857’ to image ‘V0035629ER’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s921u6g6ZCD2DJlWtbdcHg.png" /><figcaption>The pathway from image ‘L0032287’ to image ‘A0000632’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gapm_r_bFSiUk2ynobchXw.png" /><figcaption>The pathway from image ‘A0000785’ to image ‘V0040933’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xBZgwsKt3YxPYYjmbFrV8w.png" /><figcaption>The pathway from image ‘V0001893’ to image ‘V0047369EL’</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12d31ae347f9" width="1" height="1" alt=""><hr><p><a href="https://stacks.wellcomecollection.org/finding-image-pathways-12d31ae347f9">Finding image pathways</a> was originally published in <a href="https://stacks.wellcomecollection.org/">Stacks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>