We’re looking for a supplier to help us scope out what we might do in this space — this blog post explains why.

Image by Nick Turner on Unsplash.

Recruitment and retention in mental health

Clinical trials are essential for developing new, safe, and effective tools in healthcare. However, finding eligible participants and keeping them engaged throughout a trial is hard.

At Wellcome, we’re working to build a world where no one is held back by mental health problems. Clinical trials are critical for finding new ways to intervene, but challenges with recruitment and retention in mental health trials remain a significant blocker.

In mental health clinical trials, recruitment is slow and inadequate. For example, around half of all trials funded by two large UK funding agencies between 2002 and 2008 needed an extension to recruit enough participants. But the challenge goes beyond finding enough people to take part; it’s also difficult to recruit diverse and representative participants. A review of clinical trials for depression over a 36-year period (1981 to 2016) found that demographic information on race, for example, wasn’t even reported in many trials. That means we don’t know how representative those trials are of broad populations.

Once participants are recruited, retention becomes an issue. Many mental health clinical trials struggle to keep their participants engaged, and so they suffer from high drop-out rates. For example, a review of schizophrenia drug trials found an average dropout rate of 66%.

Challenges recruiting and retaining participants in mental health clinical trials lead to many problems. Not enough people engaging in trials leads to underpowered results. This means we cannot be sure that the intervention tested works as effectively as it seems. Furthermore, the lack of diversity in participant samples makes the results less generalisable. Less generalisable results mean that we don’t know if the intervention tested will work in the specific patient populations who need it. Ultimately, these challenges delay (and even prevent) safe and effective mental health interventions reaching those who need them.

Innovative (digital) tools

We need to innovate. We need better ways to recruit diverse participants into trials, and we need participant-centred methods to keep people motivated and engaged. Digital tools have potential here for a few reasons.

First, digital tools make it easier for some participants to access trials. Not everyone can attend in-person trial visits, not least because many participants live far away from trial centres. Digital tools allow participants to take part in trials no matter where they are. For example, researchers can use videoconference software to engage with participants remotely.

Second, digital tools offer more ways to tailor to participants’ needs. Trial designs often fail to account for the lived experiences of those with mental health conditions. For example, some symptoms of mental health conditions, such as low mood and low motivation, might limit patients’ ability to regularly engage with trial activities. Digital platforms like Nuvoteq lower the burden of participating in a trial by allowing participants to report symptoms and other trial outcomes from home.

Third, digital tools open the door to using AI to boost recruitment and retention rates. For example, Cogstack, a natural language processing model that analyses patients’ electronic records, reduced the need for manual screening of potentially eligible participants by 85%. For another example, machine learning algorithms were used in the mMitra programme to identify participants at risk of disengaging from a free maternal health service call. Researchers allocated resources to keep at-risk participants engaged, successfully reducing the number of dropouts by 32%.

The bottom line here is that we think digital tools could enable more efficient and participant-centred clinical trials.

Trustworthy tools

It’s not enough for a digital (or other) tool to be effective, it also needs to be trustworthy. These tools need to handle sensitive participant data securely and they need safeguarding measures in place to protect participant safety.

We must also ensure that these tools are as beneficial as possible for communities with limited access to or experience with digital technologies, for example, the elderly, or people living in rural or low-resource settings.

Finally, we must work with trial participants and broader communities to tailor recruitment and retention tools so that they support positive experiences for people involved in clinical trials.

Next steps

We’re starting by commissioning landscaping research to explore the current state of the art, and where there are opportunities for more effective, safe, and inclusive solutions in the space.

Check out our request for proposals for further information on the landscaping research. The deadline to submit an expression of interest and questions on the work is the 25th of March, 2024. The deadline for submitting the full proposal is the 22nd of April, 2024.

Investigating how to innovate mental health clinical trials was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Wellcome, we’re exploring how we can advance the utility of digital sensors for mental health research.

Read on to find out about a new opportunity to work with us on this, and the potential we see for digital sensors in health research more broadly.

Digital sensors are devices that can measure your behaviour, physiology, and environment as you go about your daily life. The GPS tracker in your phone can measure where you go, how you get there, and how long it takes you. The heart rate sensors and accelerometers in a smartwatch can measure how well you sleep. And thermostats can measure the air temperature in your home. Long story short: sensors can measure many different things, and this can be useful in many different ways.

Sensors are already proving useful in many different fields of health research. For example, sensors have been used to detect and monitor symptoms of Alzheimer’s disease, to measure cognitive impairment in people diagnosed with type 2 diabetes, and to measure post-operative physical activity levels in patients undergoing cancer surgery.

The opportunity for mental health research

The ability to measure and monitor mood, activity levels, sleep patterns and more provides clinicians and researchers with objective real-time data relevant to mental health symptoms. These data could be used to detect mental health conditions early on in their progression, to stratify populations according to their condition, and to deliver more effective and timely treatments. For example, see this review of using sensors to monitor depression.

At Wellcome, we want to support effective and trustworthy sensor use in mental health research, which we know requires looking at the technical and ethical challenges that this kind of data collection presents. And we’re not alone — others in the field are also considering this. In March 2023, we attended a workshop which brought together researchers, device manufacturers, and regulators to discuss our shared vision for sensor use in mental health and the challenges blocking that vision from coming to life.

Our first step to support work in this space is to commission research to understand how we might help solve the technical challenges facing researchers who use sensors in their work (click here to find out more). We already have an idea of what the challenges are — now, we want to know how we can solve them.

The challenges

First, the technical bar to using sensors in mental health research is high. Interested researchers must learn to interact with many different sensor devices — some people have Fitbits, some have Apple Watches, and others have no smartwatch but one of many different mobile phones. To collect data from all possible devices, researchers must wrestle with different operating systems, APIs, and standards for what data is available to them. Any solution they build must be updated regularly to keep up with the latest hardware and software. Researchers must do all of this without much support; there are some high-quality open-source platforms for sensor data collection (e.g., Beiwe, RADAR-Base), but support to make full use of these platforms is limited. In short, unless researchers happen to have access to a team of trained software engineers, collecting sensor data is difficult.

Even if researchers manage to collect any sensor data, the sensors available to them may not be the ideal sensors for robust mental health research. Many sensors used in mental health today were not developed for their value in mental health research and clinical practice. They were developed to sell phones and smartwatches. An Apple Watch, for example, comes with heart rate monitors, an accelerometer, a gyroscope, a barometer, and a blood oxygen monitor. How many of these sensors were developed primarily to track and help improve our mental health? The answer: likely none. Which begs the question: what would a purpose-built mental health sensor look like?

Second, there are also a bunch of thorny ethical issues to consider. The sheer amount of data sensors can collect about us is unlike many other data collection methods. Sensors can, in principle, measure where you go, who you meet, how long you stay there, and what you say. This level of data collection may well give researchers and clinicians the data they need to understand and improve our health, but at what cost? Despite their value, sensors may never reach their full potential unless we seriously consider how to minimise that cost and protect privacy.

These are not simple issues, but some solutions do already exist. Apple, for example, provides only abstracted navigation data centred around “landmarks” — I could find out how far you travelled away from home (the landmark) without actually knowing where your home is nor where you went. The bottom line is that acceptability from the participant’s point of view is vital for any attempt to advance sensor use in mental health.

Beyond mental health

At Wellcome, we’re focussed on driving progress in three areas: mental health, climate and health, and infectious diseases. While our initial focus is on mental health, it’s clear that sensors have the potential to advance our aims in all three.

In the climate and health field, researchers are often dependent on coarse population-level data on who has been exposed to the most recent climate-related event (e.g., a heatwave) and how their health has suffered as a result. For example, most studies investigating the health effects of extreme heat use morbidity and mortality data aggregated over an entire population — individual level data is rare (Campbell et al., 2018). Sensors, by contrast, can collect granular individual-level data at scale, providing detailed information on the experiences and health of a large number of individual people. We could then understand exactly how the health of individual people is affected by exposure to extreme heat, instead of making broad and imprecise statements about average trends in the entire population. These insights could demonstrate the effectiveness of adaptation and mitigation interventions, which could in turn inform action and protect health.

When it comes to infectious diseases, sensors can be used to support surveillance systems by monitoring factors that drive new and emerging diseases, like close human-animal interaction, and physiological and behavioural data. For example, some scientists believe that the origin of the COVID-19 pandemic can be traced back to the Huanan animal market. If true, a system collecting navigation and physiological data may have been able to identify that some people who visited the market in late 2019 went on to develop a common set of symptoms a few days later, alerting us to the outbreak of an emerging threat (COVID-19) much earlier.

So while we’re starting with mental health, we’re keeping an eye out for opportunities to advance the utility of sensors in research more broadly.

What next?

Wellcome is commissioning research to understand how we can solve the challenges faced by researchers using sensors in mental health. Click here to find out more.

If you’d like to hear more about this or other work we’ll be doing in the sensor space in the future, follow our Medium, or follow me on Twitter/X: @GwydionTW.

Digital sensors for (mental) health research was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

‘Equity’, ‘inclusive research’, ‘promise for mental health’.

When we asked Wellcome’s mental health lived experience advisers why they work with us, these are some of the comments we heard.

Mental health is a strategic priority for Wellcome — and we work with a team of advisers to ensure lived experience is at the heart of the decisions we make. Our goal is to create a step change in early intervention for anxiety, depression and psychosis and we fund work that will deepen our understanding of how these conditions develop and resolve.

Longitudinal data is a valuable resource to unlock this understanding. Wellcome recently commissioned a landscaping report which catalogued longitudinal datasets from around the world. It found that there are over 3000 existing longitudinal datasets worldwide and highlighted several that offer most promise when it comes to mental health research. But — as the insights from our lived experience advisers indicate — as well as supporting the creation and use of databanks, we want to ensure the data is collected and used in a transparent and fair way, with the interests of people and communities at its core.

Identifying and managing risk

To develop the thinking in this area we commissioned Sage Bionetworks in the US, who worked in collaboration with the Aapti Institute in India, to produce a framework for ethical data governance for mental health databanks. It is now openly available on Wellcome’s website.

The framework provides a structure to understand and manage ethical risks for people setting up and running databanks, and researchers using the data. It:

• Maps out ethical considerations when databanks are being built and once the data is being used
• Identifies risks at both these stages
• Suggests risk mitigations from a platform hygiene and participatory research perspective.

The framework is comprehensive, with an extensive catalogue of risks and mitigations. It is also thoughtfully laid out, so that users can navigate quickly through the sections. Plus there is a useful glossary of frequently used and technical terms.

Who is it for?

We hope the framework will be used by:

• Databank builders and researchers analysing the data, to embed an equitable approach to their planning and execution
• Funders supporting databanks, to guide evaluation of research proposals
• Civil society organisations, to advocate for trustworthy practice in this field
• Communities and people with lived experience, to help them in evaluating opportunities to engage with research in this area.

Our ambition is that the framework supports the development and implementation of ethical practices, clearly indicates where lived experience and community participation is of mutual value, and contributes to the broader debate on ethical standards for longitudinal data in mental health research.

How we got here

Sage developed the framework in three phases (summarised in the documents we link to here):

• A review of the existing literature on data governance for health data and its risks
• Analysis to consolidate the key considerations and mitigations for each of the risk areas identified
• Engaging lived experience advisers and subject matter experts to review the framework, ensuring that critical nuance was not lost through the synthesis process.

A crucial element of this work was the co-design workshops hosted by Sage with our lived experience advisers and Wellcome staff. This enabled in-depth review and iteration of the framework. Thank you to the advisers who participated in these lively and constructive workshops and in particular to Chantelle Booysen and Margaret Osolo Odhiambo who worked with us closely on this project.

We will advocate for the use of the framework within Wellcome, among our grantees and in our engagement with other funders and civil society organisations.

Tell us what you think

As Sage mentioned in their record of the co-design workshops…

‘Participants grappled with the essential truth that the risks and benefits of building and using a databank are dynamic and often difficult to quantify definitively.’

This framework is a robust and well-considered attempt to do just that, and we are very grateful to Sage and Aapti for their expertise and commitment.

The framework is available under a Creative Commons licence, so please do share and adapt it for your work — and let us know whether it measures up to the tough task of capturing this ‘dynamic’ landscape. We’d love to hear from you.

Rebecca Asher and Emily Jesper-Mir, Partnerships and Engagement Managers (job share) in Wellcome’s Data for Science and Health team.

emilyandrebecca-dsh@wellcome.org

Navigating ethical risks for mental health research data: a dynamic framework was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ll start by being honest with you… the title to our blog is a little misleading. This is not a ‘How to’ guide and we don’t have all the answers about how to fund health data science on the African continent.

What we do know, working in the Data for Science and Health team at Wellcome, is that as a global health funder we will need to consider both what and how we fund, to support data science for better health. We also know that there won’t be just one approach — but many, and that they will need to be codesigned with researchers, data scientists and broader community stakeholders.

So, we went to the DS-I Africa community’s annual meeting and ran a workshop to discuss what funders need to consider when supporting health data science on the continent. DS-I Africa is a grant program funded by the National Institutes of Health, to create and support a pan-continental network of data scientists and technologies. We’re sharing what we heard from the DS-I Africa community for other global health funders also working in this space.

Understand what exists already: datasets, data scientists and data platforms

The first step is to understand what exists and where the gaps are. What datasets and data platforms are in use already? What data science training is already provided, and for what career stages? Getting an accurate picture will require funders to go beyond their existing networks, and listen to researchers, policy makers and public health professionals who work in and understand the context. And it will vary hugely across different countries, regions, cities, universities, research institutes, academic disciplines and so on. With a reasonable understanding of the context, funders can then tailor their approach. We heard some specific ideas from members of the DS-I Africa community about how to fund data science in a way that is more responsive to the needs of research communities. These include:

Support the art of ‘ideation’: data scientists on the continent face a number of barriers to getting the funding they need for new or early ideas. Co-producing grants, funding smaller scale projects and testing ‘experimental’ funding processes for data science could help support early ideas. Impact isn’t always proportional to the amount of funding — small catalytic grants, or funding a few skilled and driven teams, can have a big impact further down the line.

Be intentional in expanding beyond existing ‘hubs’ of funding for data science: specific institutions and cities tend to develop as hubs for data science skills and infrastructure. These places then become ‘centres of gravity’ for funding, which means that data scientists, datasets and data infrastructure tend to be concentrated in a few fields or locations. Funders need to intentionally reach out beyond those hubs, for example by funding work that pairs recognised data science institutions with lesser-known institutions or individuals with potential.

Encourage flexibility and collaboration across existing initiatives: mandating that there’s an African lead institution during a funding call can help improve representation, but can also provide barriers. Pairing an African institution with a high-income based institution can help with administrative burden but must ensure that benefits are not solely felt by the HIC institution. Events like data hackathons or data prizes can facilitate partnerships and “matchmaking” between different sectors, geographies and experience levels. This DS-I Africa project is a nice example of pairing institutions to build research capacity.

Recognise how fundamental foundational infrastructure is

Funding models often focus on answering specific research questions, related to the organisation’s strategy and priorities. But funders also need to consider contributing to the creation and maintenance of the foundational data infrastructure that research depends on. The impact of this kind of work may be longer term, and less easy to track back to a specific grant or individual funder. But it can be crucial to future of research. Specifically, the DS-I Africa community mentioned funders should look to:

Improve infrastructure for data collection and processing: this must consider the needs of the community the data is coming from, especially for data involving marginalised or historically excluded groups. Funding for digital tools, computing infrastructure, diverse data collection and the dissemination of outcomes can all make significant impact. eLwazi is a brilliant example of this from the DS-I Africa consortium itself: it’s an open data science platform for teams to find and access data, select tools and workflows and run analyses on a variety of computing environments.

Support data harmonisation and standardisation: non-standardized data within and across countries, and the lack of standards or common frameworks for interoperability and reuse of data, are a barrier to robust data science. Initiatives that support longitudinal, population-based, harmonized and standardized data, which can be linked to other relevant sources, can have important long-term impacts. INSPIRE network was mentioned a few times as a great example of this.

Facilitate robust governance and ethical oversight: The regulatory environment around data sharing and data protection across different countries varies. This can be a significant blocker when supporting the development of robust governance structures for ethical sharing and oversight across countries. Getting this right is critical in order to deliver trustworthy systems for impactful results. To give an example from DS-I Africa, this project will develop evidence-based, context specific guidance for the governance of data science initiatives in sub-Saharan Africa.

Put communities at the centre of research design

The data scientists and researchers we heard from at DS-I Africa cared deeply about making sure their work made a difference to the communities they work in. They want funders to support them to do so. The first step is being clear on who ‘the community’ is, which means asking questions like: is it population level or specific groups? Who is most affected by the collection and use of data? Who’s most at risk? Who will actually benefit? Once these kinds of questions are clear then funders and researchers can start to make sure their needs are prioritised through the research process. Within this context, the community at DS-I Africa suggested that funders should:

Support engagement to understand concerns and hesitations that researchers may be facing: health data can feel sensitive and personal, and people may understandably be hesitant about data being used for research, particularly if there are concerns about commercial and governmental agendas. Funding to support comprehensive public engagement work in the data and health space, and in a region-specific manner, is crucial to understanding anxieties and concerns and to know how to address them. Researchers need time and resources to plan and deliver public engagement work, which funders can help with. A number of the DS-I Africa projects have a strong focus on engagement — this project, for example, will explore public attitudes to big data and genomic medicine and develop frameworks to support future engagement.

Reach out beyond existing contacts and networks: working in collaboration with local leaders and civil society representatives to co-develop study protocols or interventions can help reach new audiences — including those reached less frequently. Connecting with new audiences may also require using a range of communication channels and innovative methods — which requires funding to design and implement. For example, in another session at DS-I Africa we heard of a team who had run a fashion show that facilitated engagement with data on gender-based violence.

Ensure there is mutual benefit: there’s a role for funders to incentivise dissemination of the impact of research back to the community. This must go well beyond just publishing papers — and needs to be built into the research process from the outset. An great example from DS-I Africa, is the UZIMA Data Science Hub — at the centre of the Hub’s strategy engaging with key stakeholders to ‘identify pathways for dissemination and sustainability’ of the AI models the team develops.

Final thoughts

It was brilliant to join the DS-I Africa community and talk about these complex and knotty issues — huge thanks to everyone who attended our workshop. Take a look at the DS-I Africa website to learn more about the great work the teams are doing.

We’re already using what we heard to inform the design of a new initiative for data science in mental health on the African continent — more on that in the summer. More broadly, Wellcome is also trying different approaches to support more researchers from low- and middle-income countries apply for funding.

If you’re a funder working through these ideas too or a researcher with ideas related to what we’ve covered in this blog — continue the conversation with the Data for Science and Health team at Wellcome by writing to us at ContactDataForScienceAndHealth@wellcome.org.

Funding equitable data science for better health in Africa was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the years, the Wellcome Trust has received a huge number of grant applications and funded thousands of research projects. As a result, millions of files have accumulated documenting Wellcome’s communication with its grant applicants and grant holders. These documents range from application forms to award letters and reports, and they come in a variety of file formats. As part of a wider NLP task to build a classifier to tag these documents according to their categories, we were faced with the challenge of having to extract all text contained within these documents. This mainly required dealing with PDFs, which are notorious for their varying levels of machine-readability. Nowadays there are several tools available to data scientists who need to extract text from PDFs, so how did we decide which one to use for our project? We needed a solution that was:

(a) scalable to millions of documents

(b) able to deal with a mixture of different file formats in addition to PDF

(c) able to apply optical character recognition (OCR) to scanned documents

(d) straightforward to set up

We decided to use Apache Tika, which covers most of our requirements… perhaps apart from (d), but this is what I attempt to solve by writing this blog post.

Introducing Apache Tika

Apache Tika is an open source Java framework for file type detection and parsing, with an impressive collection of ~75 parsers (see here for more information on the available parsers). Using these parsers, Tika can extract text and metadata from >100 file formats (the full list of supported formats can be found here). In addition, Tesseract is integrated into Tika, making it possible to apply OCR to documents containing scans and other images.

Tika already comes with API bindings in Java, and many developers are actively creating bindings in other languages. In addition, Tika can be accessed as a RESTful API via the tika-server module. Pre-built docker images for tika-server are available on Dockerhub, making it easy to use the standard server endpoints for document parsing.

Introducing tika-pipes

It’s relatively straightforward to set up Tika Server to parse contents and metadata of individual files (made even easier for data scientists thanks to the tika-python library), but what if you have thousands of files to parse? Sequential parsing can be tedious especially if you have lots of PDFs containing images and don’t want to wait around for Tesseract to finish processing before moving on to the next file.

This is where tika-pipes comes in. First added in Tika 2.0 (released in 2021) and aside from other benefits, this module allows you to parse files asynchronously. For a more detailed overview of tika-pipes, check out this talk by Tika developer Tim Allison at ApacheCon 2021:

https://www.youtube.com/watch?v=BeYVRpWbCfQ

There is some technical documentation on tika-pipes available here, but it can be tricky to figure out how to set everything up properly as it is lacking in some places.

Setting up tika-pipes

If you look at the official documentation for tika-pipes, the first thing you’ll notice is a security warning about potential access vulnerabilities. The developers therefore strongly suggest running tika-pipes modules for tika-server in tightly controlled networks. However, in order to use tika-pipes (and access the server’s /async endpoint) with Docker, some tweaks are required, which I will outline below. In short, you’ll need a custom Dockerfile and run your docker container with a custom tika config.

Firstly, in your Dockerfile, you’ll need to fetch additional artifacts from maven central (repository for Java packages) as required for tika-pipes to work. In my case, the raw documents were stored on S3, so I added the S3 fetcher and emitter. If you want to parse files in your local filesystem, just change these extra dependencies to the filesystem emitter (tika-emitter-fs). Note that the filesystem fetcher is already included as standard, so you’ll only need to download the emitter in this case. As suggested in the official documentation, I used the Dockerfile based on https://github.com/LogicalSpark/docker-tikaserver/tree/2.0.0 and added my modifications from there. This is my modified Dockerfile:

FROM ubuntu:focal as base
RUN apt-get update

ENV TIKA_VERSION 2.6.0
ENV TIKA_SERVER_JAR tika-server-standard

FROM base as dependencies

RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get -y install gdal-bin tesseract-ocr \
        tesseract-ocr-eng curl gnupg

# Set this environment variable if you need to run OCR
ENV OMP_THREAD_LIMIT=1

RUN echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | debconf-set-selections \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y xfonts-utils fonts-freefont-ttf fonts-liberation ttf-mscorefonts-installer wget cabextract

RUN wget -O adoptium-public.key https://packages.adoptium.net/artifactory/api/gpg/key/public && \
    apt-key add adoptium-public.key && \
    echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" > /etc/apt/sources.list.d/adoptium.list && \ 
    apt-get update && apt-get -y install temurin-17-jdk


FROM dependencies as fetch_tika

ENV NEAREST_TIKA_SERVER_URL="https://www.apache.org/dyn/closer.cgi/tika/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar?filename=tika/${TIKA_VERSION}/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar&action=download" \
    ARCHIVE_TIKA_SERVER_URL="https://archive.apache.org/dist/tika/${TIKA_VERSION}/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar" \
    DEFAULT_TIKA_SERVER_ASC_URL="https://downloads.apache.org/tika/${TIKA_VERSION}/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc" \
    ARCHIVE_TIKA_SERVER_ASC_URL="https://archive.apache.org/dist/tika/${TIKA_VERSION}/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc" \
    TIKA_VERSION=$TIKA_VERSION

RUN DEBIAN_FRONTEND=noninteractive apt-get -y install gnupg2 \
    && wget -t 10 --max-redirect 1 --retry-connrefused -qO- https://downloads.apache.org/tika/KEYS | gpg --import \
    && wget -t 10 --max-redirect 1 --retry-connrefused $NEAREST_TIKA_SERVER_URL -O /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar || rm /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar \
    && sh -c "[ -f /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar ]" || wget $ARCHIVE_TIKA_SERVER_URL -O /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar || rm /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar \
    && sh -c "[ -f /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar ]" || exit 1 \
    && wget -t 10 --max-redirect 1 --retry-connrefused $DEFAULT_TIKA_SERVER_ASC_URL -O /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc  || rm /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc \
    && sh -c "[ -f /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc ]" || wget $ARCHIVE_TIKA_SERVER_ASC_URL -O /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc || rm /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc \
    && sh -c "[ -f /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc ]" || exit 1 \
    && gpg --verify /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar.asc /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar

# This is where we get the extra dependencies
RUN wget -t 10 --max-redirect 1 --retry-connrefused https://repo1.maven.org/maven2/org/apache/tika/tika-fetcher-s3/${TIKA_VERSION}/tika-fetcher-s3-${TIKA_VERSION}.jar -O /tika-fetcher-s3-${TIKA_VERSION}.jar \
    && wget -t 10 --max-redirect 1 --retry-connrefused https://repo1.maven.org/maven2/org/apache/tika/tika-fetcher-s3/${TIKA_VERSION}/tika-fetcher-s3-${TIKA_VERSION}.jar.asc -O /tika-fetcher-s3-${TIKA_VERSION}.jar.asc \
    && gpg --verify /tika-fetcher-s3-${TIKA_VERSION}.jar.asc /tika-fetcher-s3-${TIKA_VERSION}.jar \
    && wget -t 10 --max-redirect 1 --retry-connrefused https://repo1.maven.org/maven2/org/apache/tika/tika-emitter-s3/${TIKA_VERSION}/tika-emitter-s3-${TIKA_VERSION}.jar -O /tika-emitter-s3-${TIKA_VERSION}.jar \
    && wget -t 10 --max-redirect 1 --retry-connrefused https://repo1.maven.org/maven2/org/apache/tika/tika-emitter-s3/${TIKA_VERSION}/tika-emitter-s3-${TIKA_VERSION}.jar.asc -O /tika-emitter-s3-${TIKA_VERSION}.jar.asc \
    && gpg --verify /tika-emitter-s3-${TIKA_VERSION}.jar.asc /tika-emitter-s3-${TIKA_VERSION}.jar

FROM dependencies as runtime
RUN apt-get clean -y && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
ENV TIKA_VERSION=$TIKA_VERSION
RUN mkdir /tika-bin
COPY --from=fetch_tika /${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar /tika-bin/${TIKA_SERVER_JAR}-${TIKA_VERSION}.jar
# The extra dependencies need to be added into tika-bin together with the tika-server jar
COPY --from=fetch_tika /tika-fetcher-s3-${TIKA_VERSION}.jar /tika-bin/tika-fetcher-s3-${TIKA_VERSION}.jar
COPY --from=fetch_tika /tika-emitter-s3-${TIKA_VERSION}.jar /tika-bin/tika-emitter-s3-${TIKA_VERSION}.jar

EXPOSE 9998
ENTRYPOINT [ "/bin/sh", "-c", "exec java -cp \"/tika-bin/*\" org.apache.tika.server.core.TikaServerCli -h 0.0.0.0 $0 $@"]

You’ll also need to create a custom tika-config.xml file to set up your fetcher and emitter. Each fetcher and emitter needs to include a name parameter. This can be anything you like, as long as it is unique for each fetcher and emitter. In addition, S3 fetcher and emitter parameters need to include the S3 bucket name and region, as well as an AWS credentialsProvider. If the environment you’re running the docker container in (in my case, an EC2 instance) is on the same network as the S3 bucket(s) you are fetching from/emitting to, you can set this to instance and everything should work from here. If you need to provide your AWS credentials to access the bucket, you can set the credentials provider to ‘profile’ and provide the profile name in the profile parameter (more on this later).

Note that you also need to set the server parameter enableUnsecureFeatures to true in order for tika-pipes endpoints to work.

This covers all the minimum requirements for tika-pipes to work, but it is worth keeping in mind that there are lots of ways in which Tika can be further customised via the config. For example, I’ve included a timeout parameter in my config, but check out the official documentation on configuring Tika or the server config overview for more options. For anything more specific you may also need to consult the API documentation.

Here is a sample config file (just delete the AWS credentials provider you aren’t using):

<?xml version="1.0" encoding="UTF-8" ?>
<properties>
    <parsers>
        <parser class="org.apache.tika.parser.DefaultParser">
        </parser>
    </parsers>
    <fetchers>
        <fetcher class="org.apache.tika.pipes.fetcher.s3.S3Fetcher">
            <params>
                <name>s3f</name> <!-- this can be any name you like -->
                <region>eu-west-1</region>
                <bucket>my-bucket</bucket> <!-- bucket containing raw data -->
                <credentialsProvider>instance</credentialsProvider>
                <!-- or use a profile -->
                <credentialsProvider>profile</credentialsProvider>
                <profile>default</profile>
            </params>
        </fetcher>
    </fetchers>
    <emitters>
        <emitter class="org.apache.tika.pipes.emitter.s3.S3Emitter">
            <params>
                <name>s3e</name> <!-- this can be any name you like -->
                <region>eu-west-1</region>
                <bucket>my-bucket</bucket> <!-- destination bucket -->
                <credentialsProvider>instance</credentialsProvider>
                <!-- or use a profile -->
                <credentialsProvider>profile</credentialsProvider>
                <profile>default</profile>
            </params>
        </emitter>
    </emitters>
    <server>
        <params>
            <enableUnsecureFeatures>true</enableUnsecureFeatures>
        </params>
    </server>
    <pipes>
        <params>
            <tikaConfig>./config/tika-config.xml</tikaConfig>
        </params>
    </pipes>
    <async>
        <params>
            <timeoutMillis>1000000</timeoutMillis>
        </params>
    </async>
</properties>

Now I’ll explain how to try this out locally. Let’s say you have both these files in a directory tika_dir. First you need to build your docker container:

docker build -t tikapipes tika_dir

You can then start the server in a docker container with your custom config file. If you’ve set the credentials provider for your S3 fetcher/emitter to ‘profile’, you’ll also need to add your AWS credentials to the container. I did this by attaching them as a read-only volume:

docker run -d \
    --name tika_container \
    -v ~/.aws/:/root/.aws:ro \ # Remove this line if credentials provider is set to 'instance'
    -v tika_dir:/config \
    -p 9998:9998 tikapipes:latest \
    -c ./config/tika-config.xml

You can now parse your files by submitting a request containing a list of FetchEmitTuples to the server’s /async endpoint (see here for the official documentation on FetchEmitTuples). In essence, FetchEmitTuples look like dictionaries which need to include: (1) a fetcher, in my case the name of the S3 fetcher as specified in my custom tika config (s3f), (2) the path to the file you want to parse (fetchKey), (3) the name of the emitter, and (4) the path including name you want the emitted JSON file to have (emitKey). Note that this works in the same way if you’re using a filesystem fetcher to parse files in your local filesystem. Here is an example, where the source documents my_input_file1.pdf and my_input_file2.pdf are located in my-bucket/path1, and the output files my_output_file1.json and my_output_file2.json will be written to my-bucket/path2:

curl -X POST -H "Content-Type: application/json" \
-d '[{"fetcher":"s3f","fetchKey":"path1/my_input_file1.pdf","emitter":"s3e","emitKey":"path2/my_output_file1"}, 
{"fetcher":"s3f","fetchKey":"path1/my_input_file2.pdf","emitter":"s3e","emitKey":"path2/my_output_file2"}]' \
http://localhost:9998/async

If all is well, you should end up with JSON files in your output path containing the parsed contents of the input PDFs.

So far, so good.

If you’ve looked at the Dockerfile in more detail, you may have noticed a curious new environment variable which warrants a brief explanation: OMP_THREAD_LIMIT=1. If you are new to Tesseract you may not be aware that, by default, a single Tesseract process will actually start multiple threads. This means that, if you are trying to apply OCR to several documents using tika-pipes in async mode, you will end up with multiple Tesseract instances running in parallel, each of which will actually be starting multiple threads. As a result, parse times will be significantly slower (see here for a more detailed discussion of this issue). In my case, it slowed down processing to the extent that async was much slower than parsing each document sequentially! However, simply disabling Tesseract multithreading in your environment (as above) will do the trick.

How fast is it?

To get a better estimate of expected parse times, I ran some experiments comparing async with sequential parsing. I used a sample of PDF files where each document was about 10–20 pages long and ~20% of documents required OCR. I wrote my parsers in Python using the requests module and calculated parse times by reading S3 object metadata with boto3.

For reference, here are some code snippets from my async parser:

As described earlier, the filenames need to be passed as a string of FetchEmitTuples, which can be created like this:

def format_data(docs, output_dir):
    data = []
    for doc in docs:
        fetchkey = str(doc)
        emitkey = str(Path(output_dir) / Path(doc).stem)
        data.append(f'{{"fetcher":"s3f","fetchKey":"{fetchkey}","emitter":"s3e","emitKey":"{emitkey}"}}')
    return f'[{",".join(data)}]'

And here is a function to parse several files with the /async endpoint:

def parse(docs, output_dir):
    res = requests.post(
        url=f'http://localhost:9998/async', 
        data=format_data(docs, output_dir),
        headers={'Accept':'application/json'}
        )
    if res.status_code == 200:
        print('status: ok')

Finally, these are the results of my parse time experiments:

Actual parse times will of course depend on the properties of your own mix of raw documents, but my results should give you an idea of what to expect in relative terms. If you only need to parse a few documents at a time, you won’t gain much from using async tika-pipes and might as well process the documents sequentially using one of the convenience Docker images published by the Tika Dev team. However, async mode will pay off as you get towards large numbers of documents. In addition, when it’s necessary to apply OCR and run Tesseract, choosing an instance with more compute is likely to speed things up even more (just make sure to switch off Tesseract’s multithreading!).

How to Parse Millions of PDF Documents Asynchronously with Apache Tika was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

I love cohort studies. I did my PhD using a British-based pregnancy cohort (ALSPAC) and have more recently conducted cross-cohort comparison studies using ALSPAC and another cohort, the Millennium Cohort Study (MCS). When I was a researcher interested in understanding the links between cannabis use and psychosis, longitudinal datasets were the bread and butter for exploring this relationship. They’re a hugely important way to explore patterns in health and how these are related to potential risk or protective factors, even if they’re not the best way to get at causality.

But. These studies are often designed to explore many and varied health behaviours, illnesses and other outcomes. While mental health outcomes are often included in these studies, they are usually very brief. And that’s not surprising. All the researchers involved in designing a cohort are fighting for space — every measure added to investigate one outcome means another one can’t be. Participants only have so much time they’re willing or able to devote to answering surveys or attending a clinic, so difficult priority decisions have to be made.

Mental health is harder to measure than something like blood pressure, or a diagnosis of cancer or heart disease. There are no biomarkers (yet) so we rely on asking someone about their symptoms and the impact they have on them. Often this takes the form of a self-report questionnaire. These take up valuable participant time and space, and so they are often kept short and perfunctory.

What if we could do it differently?

In February this year I started working for Wellcome — a global charitable foundation, with a goal to help everyone benefit from science’s potential to improve health and save lives. Within Wellcome, there are three key challenge areas, one of which is mental health.

As part of my work I’ve been tasked with thinking about how longitudinal data (not just limited to cohort studies) can help us achieve our mental health strategy — namely to gain a better understanding of how the brain, body and environment interact in depression, anxiety and psychosis, so we can spot potential points for early intervention, find better ways of identifying groups for intervention, and find new and improved ways of intervening.

So I’ve been thinking about what a perfect mental health dataset would look like. If we weren’t constrained at all, what variables would we collect, and how often? What would our sample size look like, and who would be our underlying population? Would a perfect dataset for depression look the same as a perfect dataset for psychosis research?

And how do we go about doing it? There are a couple of options for helping to make a perfect (caveat, nothing is perfect) dataset — do we start from scratch (which would mean waiting for years before longitudinal data would be available), or do we enrich existing studies? As an aside, we are currently scoping existing longitudinal datasets as well, and more information about that project can be found here, including how to inform the team about existing datasets.

I’m really keen to hear from mental health researchers who work with longitudinal data — what are the variables you wish were in the datasets you use? Are there occasions where you wish concepts had been measured using different techniques, or at different frequencies? What would your perfect dataset look like?

I’d love to hear from you — you can fill out this survey, or you can send me an email and tell me your thoughts s.gage@wellcome.ac.uk.

What would the perfect mental health dataset look like? was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Research Sustainability (ReSus) Project

Key details

The Data for Science and Health team at Wellcome has commissioned a report on how the environmental impacts of research are measured and reduced.

RAND Europe, who are conducting the research, will identify the tools and methods currently being used, map out how Wellcome could integrate them into our work, and highlight key gaps in the space.

We’re looking for interested folks to share insights with us! We want to hear from you if:

📨 You’re working in this space and would be happy to chat

🧫 You’re a Wellcome grant holder and would be interested in taking part in a focus group held by RAND

Please feel free to reach out to t.caplan@wellcome.org for more information.

What is the ReSus project?

The ReSus project focuses on the environmental impact of the research we fund. Unfortunately, we don’t have gold standard, accessible ways of measuring the emissions for many types of research, let alone their wider environmental impact. Evidenced practices on how to reduce these negative impacts are also hard to find. This is a problem not just for Wellcome, but for the research sector as a whole.

Why now?

Our focus on Climate and Health increases the need for Wellcome to be an informed leader in this area. Unless we can find ways to make the science we fund more sustainable, we’ll continue to be part of the problem.

We’ve already taken some action as an organisation to combat the emissions of our funded work:

• Training on sustainable research practices is an allowable cost Under Wellcome’s Continued Professional Development policy.
• As a part of our carbon offset travel policy, we expect grantees to minimise travel, choose the lowest carbon option when feasible and offset the emissions for the journeys they do make. Also, all organisations applying for Wellcome funding must have or be developing policies and procedures to help employees measure the carbon impact of the travel they need to take.

The environmental impact of science

Science doesn’t exist in a bubble. The health research we fund has an impact on the environment and can contribute to climate change. So, a degraded ecosystem and a climate in crisis are not environments in which good science, or healthy people, can thrive.

Different types of research will affect the environment in different ways. Wet labs tend to be energy hungry, producing large amounts of plastic and toxic waste. And the social sciences can have large footprints from fieldwork related travel. In big data driven health research, lots of energy is used to store data and train models. Put simply, unsustainable mining practices and e-waste disposal impact the environment throughout the lifecycle of research hardware used daily.

Here are some quick stats to help contextualise the type of impacts we’re talking about:

• In 2021, clinical trials globally were estimated to equal around half the national emissions of Denmark, a country of 5.85 million people. That’s 27.5 million tons of carbon dioxide equivalent to be exact.
• The natural language processing model GPT-3 developed by OpenAI produced 552 tons of Co2e to train* alone. That’s the equivalent of roughly 308 direct flights from London to Sydney. This type of large computation is becoming increasingly common in health research, see the Crick’s most recent collaboration with DeepMind for more information on this.

*training a model is the process through which, by using existing data, it learns how to do what it was built for. Here’s an example video to help get your head around this concept.

We also know from engagement work led by our Research Funding team that many scientific researchers are deeply concerned about the environmental impact of their work.

ReSus to the rescue!

The Research Sustainability (ReSus) project was designed as an attempt to fill these knowledge gaps. So, we’ve commissioned RAND Europe to conduct a knowledge gathering exercise.

We are hoping to highlight and share tools and methods that can be used across the research sector and contribute to the conversation in this area. And we are also looking to answer questions around how we can best support our researchers to conduct more sustainable research.

A note on scope

A decision was made early on to focus not just on CO2 emissions, but on the wider environmental impacts of health research, like harmful waste disposal practices. Doing this allows us to target more of the negative impacts harming marginalised communities. For example, the detrimental health consequences that result from unsustainable waste disposal particularly effect individuals in low- and middle-income countries, where waste from the global north is typically exported to.

It’s an exciting time ahead

RAND have just completed the inception phase of the project. They collaborated with a group of twelve experts across the academic, public and private research sector to help guide their work.

As a next step, RAND is conducting a crowdsourcing exercise to collect examples of practice that reduce or measure the environmental impact of health research. If you are interested in participating, please email sustainablehealthresearch@randeurope.org

The evidence you provide will be used to help us map out the landscape and understand what is already being done, what can be learnt from and built on, and where the gaps are. This will inform the development of a set of case studies, and further consultation to see what opportunities there may be to take ideas forward or address any gaps in knowledge that might exist, by Wellcome and others. The information that is provided will be used by RAND Europe to prepare a report for Wellcome about environmentally sustainable research.

We’re looking forward to sharing insights along this journey, and connecting with interested researchers, grantees, and individuals working in this space for the next phase!

Help us advance environmentally sustainable research was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

A part of Wellcome Trust’s vision is a world where no one is held back by mental health problems. So we have been working on how to give researchers the tools they need to find better ways of intervening in depression, anxiety and psychosis so that people experiencing these mental health challenges can thrive.

Wellcome commissioned Sage Bionetworks and a consortia of researchers, technologists and young people with lived experience of mental health challenges in India, UK and South Africa to explore the technical, ethical and regulatory challenges of collecting new forms of data in new ways.

Using a prototype mobile app and a series of deliberative democracy engagements, the team have collected a rich and nuanced set of evidence about how young people feel about different approaches to the collection and use of data about their lives and mental health.

The team was committed to foregrounding the perspectives of young people with lived experience of depression and anxiety in every aspect of the work. This piece gives a short summary of lived experience involvement in the MindKind study, followed by three key lived experience learnings taken from the project as a whole.

What is lived experience involvement?

Traditionally, mental health research has been conducted without the input of those who experience mental health challenges. Arguably, this means that studies are less reflective of researched groups and can risk strengthening exclusions which enforce the divide between academics and communities. Ultimately, this can lead to less than effective research, dissemination and implementation.

Wellcome are committed to meaningfully involving those with lived experiences of mental health challenges in their work. Such involvement recognises that those with lived experiences of mental health challenges have a unique form of knowledge and can apply this to positively shape decisions across the research cycle.

What did the MindKind team do to embed lived experience perspectives in the study?

In the MindKind study, Sage Bionetworks (who led the project), grounded their work in a youth-adult participatory research approach to involve young people with lived experiences of mental health challenges as equal partners throughout the study.

The team not only included academics, researchers and data scientists, but also engagement specialists working alongside youth with lived expertise of anxiety and depression. Youth with lived expertise were embedded throughout MindKind as colleagues and equals, from strategic decision-making to leading grassroots engagement and co-authoring academic papers.

To lead lived experience involvement, the team hired Professional Youth Advisors (PYA) in India, South Africa and the United Kingdom, who could dedicate a significant amount of time to planning and delivering youth engagement. These PYAs were responsible for developing and facilitating sub-groups of young people in each country, also known as Young People’s Advisory Groups (YPAG). These YPAGs were engaged throughout MindKind to shape discussion and decision making around key areas such as data governance, data usability, and data collection in the study. Young people were also involved in co-authoring academic papers resulting from the study.

Finally, each YPAG contributed members to another youth engagement group — a Global Youth Panel — which was also supplemented by an ad hoc International Youth Panel convened by the University of Washington. You can read a more in-depth article on this infrastructure here.

What we learnt about lived experience involvement in complex programmes

What worked well?

Confronting digital inequities

The MindKind Study was conducted across three geographic sites which brought their unique geographies, languages, and levels of accessibility. Lived experience involvement in MindKind was largely facilitated digitally. Whilst this provided some opportunities for improved access, it also meant that some youth found it harder to engage.

Online meetings require access to a digital device with stable internet connection. Within some global contexts this was particularly difficult as some young people did not have ready access to the internet and required extra financial support for data to participate. Furthermore, internet availability would disappear due to ‘load shedding’ which restricted electrical power across the country. All of these challenges meant that synchronous online engagement wasn’t possible in all contexts.

To overcome this, platforms like WhatsApp were used to facilitate asynchronous participation and some young people received financial support to purchase data. Finally, some meetings were undertaken in-person. Overall, latent digital inequalities manifested in the study which required adaptable and agile thinking to ensure many young people were able to be involved in MindKind.

What could have been improved?

Having a central point of contact

Complex global studies always require effective project management and support. Similarly, it is important that lived experience involvement is effectively coordinated so it has the best chance to improve research decisions and direction.

Throughout the MindKind study, we realised that lived experience involvement requires clear channels of communication as well as opportunities to identify and share key learnings across research sites. Ultimately, this would have helped to develop a community of practice, enabling PYAs to be more confident, comfortable and coordinated in their approaches to involvement.

What remains a challenge for future programmes?

Closing the feedback loop

In some cases, the insights of those with lived experiences of mental health challenges are extracted and never acted upon, whilst a research team can technically say they enabled lived experience involvement. This dishonesty risks causing harm and widening existing exclusions.

Sage recognised that it was integral to ensure that feedback loops between lived experience insights and decision making were properly closed. PYAs used platforms like AirTable to record YPAG insights which were then sent to the relevant project leads. From this, project leads provided feedback about how insights were considered and whether they had impacted decision making. PYAs then relayed this to their groups and project leads also personally reconnected with groups to share how their knowledge had been used to shape the study.

Challenging extractive dynamics within lived experience involvement is essential to moving beyond tokenisation. Through small steps, the MindKind study attempted to close this feedback loop to demonstrate the importance and weight of lived experience expertise in the study as a whole. However, this is an issue that remains to be developed further to ensure that mental health research cultures can move towards mutual and equitable decision-making.

Conclusion

Lived experience involvement is a key discipline within mental health science and it aims to improve research relevance, reflection, and outcomes. In a complex project like The MindKind Study, it is essential that lived experience involvement is embedded at the core.

Whilst this is truly reflected in MindKind, it is important to recognise how involvement can evolve and improve with ongoing learning. It is only through open and honest reflection that we can celebrate our wins and also look forward to a better future in which mental health science is truly impactful, participatory and equitable for all.

You can read about Wellcome’s focus in mental health here as well as how we meaningfully embed lived experience involvement in our programmes.

3 key learnings about lived experience involvement was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Wellcome Trust (Wellcome) is a charitable foundation with a focus on funding Health Research. Over time, Wellcome has accumulated a database of some 130 thousand grant applications from a wide range of projects, focusing on (amongst others) Infectious Disease, Mental Health and more recently in Climate Change.

Each grant application is represented in a database as an array of information including whether an application was successful, the committed amount, the recipient, the related organisation to the recipient, etc. Whilst the application forms submitted by an applicant contains much useful categorical information — information that allows us to categorise a grant — there are also a number of free text fields that are collected, for instance the application title and project synopsis.

This blog post is about some experiments we conducted to extract useful information from these free-text fields using an approach called Named Entity Recognition (NER). Typically NER can be done in a range of ways: from simply searching verbatim for words of interest (think country names, or other geographical locations) to using sophisticated machine learning models that can learn the entities of interest.

This post follows the project timeline: first a wide exploration of all types of entities we could extract, followed by a more focused exploration of extracting location related entities, with the goal of answering two questions:

1. Are there location entities in free text fields that aren’t captured elsewhere? I.e. can the location information in our database be enhanced?
2. How can we interpret the newly found entities?

Casting the net wide

Applying NER to Wellcome grant information had not been tried in the past. Since many types of entities can be extracted, we started off by extracting as many entities as possible in order for Wellcome stakeholders to determine where to focus the project. We used an off-the-shelf deep learning model from the spaCy framework to extract all the traditional named entities that are generally collected such as PERSON, LOCATION, AMOUNT, etc (see the addendum for a complete list).

We also used four specialised models for processing biomedical and scientific text (Sci spaCy). These models extract such specific entities like SIMPLE_CHEMICAL, CELLULAR_COMPONENT and IMMATERIAL_ANATOMICAL_ENTITY. Some of these entities are very specific to the biomedical field that it can be difficult for a lay person to decipher exactly what the entities are. For this we sought help from scientists at Wellcome and have included in the addendum a full list of all these entities and their meanings.

To showcase what was possible, we built a small web-based application to allow stakeholders to see what kind of information they could extract from grants and from other arbitrary text.

As you’ll see in the screenshot, since we used off-the-shelf models for the demo without training them specifically on Wellcome’s grants, they don’t always perform perfectly. Cardiomyopathy is indeed a DISEASE and Africa a LOCATION but Left Ventricular Systolic Dysfunction (LVSD) is not an ORGANISATION! Overall, we can see that the models have done a good job of detecting diseases and locations.

Narrowing down our focus

After some conversations to narrow the scope of the project, we found that Wellcome stakeholders were particularly interested in entities that had to do with location. As you might expect, Wellcome already collects a large amount of location information about its grants, however it gets complicated quite quickly. It’s easy to know where an organisation that is receiving funding is based, but very often that organisation will use the grant to conduct research elsewhere.

As an example, Wellcome has co-funded the World Mosquito Program (https://www.worldmosquitoprogram.org/). This program aims to reduce transmission of mosquito borne diseases such as Dengue, Zika, Yellow Fever and Chikungunya by introducing a Wolbachia spp. bacteria into mosquitoes which prevents the diseases’ spread. The grant is registered to Monash University, based in Australia. Both the organisation name and location are captured directly from the application form in Wellcome’s database.

However, the World Mosquito Program’s trial sites are situated in Brazil, Columbia, Sri Lanka, Fiji and others (reaching over 6M people). These locations are not directly captured in Wellcome’s database but are mentioned in the grant’s summary or synopsis. If we could extract these locations, it would help to answer the question of where the ultimate beneficiaries of the funding are, rather than the people and organisations who receive the funding.

Finding a country relevant to a grant that is not already recorded in Wellcome’s database is thus a three step process:

• Extracting the locations with NER
• Resolving the locations to countries
• Comparing to existing Wellcome records

Extracting Locations

Extracting locations is in many ways the easy part, as it is a very generic task. Many people have tackled this problem before and made their solutions available online for us to use. Helpfully, we also didn’t need help from scientists with expert knowledge to determine what is, and what is not a location.

We experimented with a number of approaches for extracting location entities:

• Creating a spaCy PhraseMatcher, which uses a simple list of countries that we curated from a list published by the World Bank. Note that this is not as easy as it may sound, as there are many countries that have more than one name by which they are referred: for example Czechia and the Czech Republic.
• SpaCy’s standard approach to NER, using the en_core_web_md model, which is a medium sized variant of spaCy’s English language model based on an artificial neural network (this is one of the same models we used before we decided to focus on locations).
• An alternative, experimental approach using spaCy, called the span categorizer. This approach is similar to the standard spaCy approach, but consists of a function which suggests likely entities, and a model which selects the most promising (or none) from the suggestions.

Resolving locations to countries

Once a true location is found, it still needs to be resolved. In our case, we want to resolve a location back to a country. e.g. if the entity “Paris” is found, we probably want to resolve this back to “Paris, France”. One needs to be careful though, as there is also a “Paris” in Denmark, Kiribati, Panama, Texas and Ontario.

There are a couple of strategies to solve this problem, more generally this is known as entity resolution, and it is a common task when dealing with entities that do not have unique names: for example locations and people.

One approach is to train a model to recognise the difference between entities with the same name based on the context where the entity was found. This approach is relatively time consuming however, so in this project we opted to use a third party Geo-coding service OpenCage to handle the resolution problem.

A Geo-coding service simply allows us to send the extracted location entity, and receive back a best guess resolution including metadata about the location. This approach does not take into account the context of the word, so it would not know the difference between “Paris, a town in Kiribati”, and “Paris, a city in France”: both instances of Paris will resolve to the most well known, which is Paris, France.

Comparison with existing records

Finally, once we have resolved locations to countries, they can be compared to the information already available in Wellcome’s database. This is the simplest step because we can simply compare the three digit ISO country codes (e.g. GBR for the United Kingdom) for the countries that we have found in the free text fields, against those already recorded for a given grant.

What we found

We set out at the start of this project with two questions:

1. Are there location entities in free text fields that aren’t captured elsewhere? I.e. can the location information in our database be enhanced?
2. How can we interpret the newly found entities?

Are there location entities in free text fields that aren’t captured elsewhere?

To answer this question, we manually annotated 651 grants using a tool called prodigy. This entailed using an NER model to suggest entities, and then manually confirming, correcting, or rejecting them. This created a gold standard ‘evaluation set’ against which we could compare the results of the workflow we built.

We evaluated all three of the NER approaches, but for this proof of concept we stuck with the simplest: the PhraseMatcher. Whilst this approach found fewer location entities than the others (low recall), it was much less likely to create false positives (high precision), which was preferable in this case.

From these 651 grants, the PhraseMatcher identified 1417 locations, of which 1047 corresponded with human annotations, the remainder were false positives, i.e. the model identified a spurious location.

Of these 1047 locations, 891 were resolved correctly to the country by the OpenCage Geo-coding API, and of these 283 were unique occurrences (at the grant level). Once we removed the countries already associated with a grant in the Wellcome database (209), we were left with 74 new countries. As a result, we increased the available location data in the sample by 35%.

This is an impressive result, and demonstrates that this approach has potential to augment the location data that Wellcome collects. Additionally, whilst in this project, we resolved entities to country, there is scope to resolve to finer scale geographies such as region, city, or town.

How can we interpret the newly found entities?

Now we know more location entities can be captured, the question turns to how these newly extracted entities can be interpreted. As mentioned before, location can be difficult to interpret when it comes to grants as Wellcome funding doesn’t always remain with the recipient institutions, it can flow through to clinical trial sites, other research locations, collaborators, etc.

To evaluate the meaning of the newly found entities, we manually annotated them to be in one of three categories.

• “Not relevant”, which are locations that are correctly found but are actually not relevant to the grant. e.g. a grant states in their synopsis that they are inspired by work in Tanzania and now want to do similar work in Kenya. Both Kenya and Tanzania will be picked up, but only Kenya is relevant. We found 24% of newly found locations to be not relevant.
• “Related to applicant, organisation or institution related to applicant or collaborator”, which are locations that are traditionally captured well in our systems as they refer to the more administrative locations of the grant. They often don’t reflect where the actual research occurs. We found 16% of newly found locations to fall in this category.
• “Beneficiary location”, which are the locations that relate to where the research actually occurs, where clinical trials or field tests are set up. These are closest to the location of impact that we really would like to capture. We found that 59% of grants fell in this category.

From the above we concluded that a large proportion of the newly found grants relate to beneficiary location, which is an exciting prospect. However, human moderation of the newly extracted entities is advisable at this stage. Given the importance of this information, this is something that can be considered in the future.

Next steps

To wrap this project up, we set up an automated pipeline that runs NER extraction periodically on all Wellcome grants. This pipeline, which currently runs in the development environment, is the first step towards implementing NER as a routine task on Wellcome’s grants, and making this information available to all users.

In this project we focused on using off the shelf models, and we only annotated a small amount of data sufficient to create an evaluation set to measure performance. In a future iteration, we’d like to train our own models using annotated grants to improve the performance of the machine learning approaches. This will enable the pipeline to identify locations it has not seen before, and obviate the need to manually curate a list of entities we would like to recognise. Finally we would like to explore if we can automatically categorise which of the interpretations is applicable to the newly found locations and how we can involve grant experts in this process.

Both approaches will help us to better answer the question: where is impact occurring for the grants we are funding?

Addendum

Below is a list of all the entities that can be extracted by Spacy NER reproduced from https://spacy.io/models/en.

SciSpacy extracts biomedical and scientific entities. The below descriptions and examples have been sourced by subject experts at Wellcome.

Finding location entities in Wellcome grants was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why did we build these models and why are they open?

We built these models to enable Wellcome to better evaluate their scientific funding portfolio and to enable fine-grained search through grants and applications. Medical Subject Headings are a very rich hierarchical vocabulary, with descriptors ranging from general terms such as “Infections” and “Anatomy” to very specific terms such as “Sleep Disorders, Circadian Rhythm”.

We have released our models and code publicly because we believe that it could be of interest to other organisations in the sector that want to automatically classify their portfolio. Tagging applications with such a rich vocabulary manually is not practical, even for small research portfolios, but can greatly improve the way organisations interrogate their data.

What are the models based on?

As with most systems to process natural language nowadays, models are hardly trained from scratch. In our case, we investigated two types of models:

• General “cheap” models that perform well for a huge number of labels. A breakthrough in the field happened in 2020, when a research group from Amazon published the first draft for PECOS (Prediction for enormous and correlated output spaces). “Enormous” and “Correlated” output spaces sounded appropriate to our problem. PECOS works by clustering the label space and recursively applying linear models to identify the correct cluster (see picture below). We will refer to the model built based on this framework as XLinear.

• Models that performed well for similar benchmarks, such as the Biomedical Language Understanding and Reasoning Benchmark (BLURB). Most recent machine learning frameworks trained for such benchmarks are based on neural networks, and allow for “transfer learning”, i.e., re-purposing for a slightly different task. We have decided to follow on the BertMeSH architecture proposed in a recent academic article and fine-tune PubBertMesh, a large neural language model that performs well on BLURB . We will refer to this model as WellcomeBertMeSH.

The BertMeSH model was trained using the excellent framework developed by Huggingface (yes, that’s the real name of the company!) and released on their hub, allowing for frictionless usage with a couple of lines of code.

Training and evaluation

The models use training data from BioASQ (http://bioasq.org), an annual challenge for biomedical semantic indexing. The training data consist of about 2.5 million academic publications from 2016 to 2019, whereas the evaluation data consists of 220,000 publications from 2020 to 2021.

We have also evaluated the production system (based on XLinear) on data manually annotated by in-house domain experts. Interestingly, the model performs about 5% better on grants. There might be a couple of reasons for that: grant text is often simpler than publications, our grants only cover a small subset of MeSH, and, for our applications, the annotators are more forgiving to small errors.

We felt that the 6% difference in performance did not justify deploying such a large model, therefore the current production model is XLinear.

How to get involved?

Despite many advances in the area, automatically tagging academic text data with huge vocabularies is still an open problem. We believe that this model can be of relevance for searching through biomedical research in general.

Contributions to the hugging-face model are sought, in particular in three fronts:

• For the XLinear model, investigate whether using the MeSH inherent hierarchy as “seeds” for the clustering step improves performance.
• For the neural model, in principle, the literature [2] tells us that we can achieve about 70% on the benchmark, however we weren’t able to reproduce these claims. Any contributions towards this direction are welcome.
• Improve production-level inference time of WellcomeBertMesh so we can replace the linear model.

And if you have any questions about the models, code, or how to get started, don’t hesitate to get in touch directly or to raise a github ticket.

References

[1] — Presentation by Nick Sorros at PyData London 2022 Extreme Multilabel Classification in the NLP Domain
[2] — Bertmesh model — https://pubmed.ncbi.nlm.nih.gov/32976559/
[3] — Blurb benchmark — https://microsoft.github.io/BLURB/
[4] — PECOS model — https://arxiv.org/pdf/2010.05878.pdf

Open-source machine learning models for Medical Subject Headings was originally published in Wellcome Data on Medium, where people are continuing the conversation by highlighting and responding to this story.