In many organizations, there is a unique vocabulary that maps names to known entities within that domain. At the United Nations, for instance, we have many specific entities which it is useful to identify in documents, including specific named committees and assemblies, important topics like the Sustainable Development Goals (SGDs), and many different countries and cultural groups that must be identified correctly. Exhaustively naming each and every important topic that may appear in a document, however, is not reasonable considering the shear number that may be important, especially considering the context of a document or sentence in which this entity is present. Instead, we want to be able to automatically identify and predict named entities using Named Entity Recognition (NER) to avoid creating a Regex parser with thousands or hundreds of thousands of possible named entities to match.

In this article I will show you how to use the Hugging Face library to fine-tune a BERT model on a new dataset to achieve better results on a domain specific NER task. In this case, we want to predict United Nations named entities in transcripts of meetings of the General Assembly. Key phrases such as “the general assembly”, “second plenary meeting”, “the rohingya muslims”, and more will need to be identified and extracted as named entities within these transcripts in a consistent and automated fashion.

The Dataset

First we need to retrieve a dataset that is set up with text and it’s associated entity labels. Because we want to fine-tune a BERT NER model on the United Nations domain, we will need labelled text from United Nations documents. We will use Leslie Huang’s generously open sourced UN NER dataset as training and test data.

git clone https://github.com/leslie-huang/UN-named-entity-recognition

Notice how this data is structured in the tagged-training folder. Each text file is formatted with text from sentences and a label for each word which describes it as either not an entity (O) or as a specific type of entity (I-LOC, I-MISC, I-ORG, etc.). To get this into training format, we will read all of these files and, for each sentence, create a list of words and a list of these words entity labels. We will store this in a dataframe by reading the files in the UN NER dataset and reading each sentence as a row:

which yields the dataframe:

Now that we have the data in a workable format, we will use the Hugging Face library to fine-tune a BERT NER model to this new domain.

Using the BERT Tokenizer

A tokenizer is responsible for preparing the text inputs as inputs to the transformer model (in this case, BERT). Each of these tokens have a mapping to a number that will be used as the transformer input. In order to get value from the fine-tuning process, we will need to use the same tokenizer on our UN NER dataset that was used during the original training. This will ensure that what was learned during upstream training will be carried over into this downstream task.

First, we need to get the Hugging Face transformer and datasets libraries.

pip install transformers
pip install datasets
pip install seqeval

Next, we will tokenize our inputs and match the labels in the UN NER dataset (some of which have data quality issues which must be handled) to those labels used in upstream BERT training.

Which yields us the datasets for both training and testing. Next, we need to tokenize these datasets to convert the tokens and labels into numeric form for our BERT model.

Each token has now been transformed into a numeric representation and each label has been mapped to those used in upstream training, all in Hugging Face data format.

Fine-Tuning the BERT Model

Now that we have our dataset in the proper format, we will train our BERT model using the pre-trained model weights as our initial checkpoint. During this process, we will evaluate the model on the testing data by reporting F1 score and accuracy and finally save the model in the “un-ner.model” directory.

After running this pipeline with these model parameters, we were able to achieve an F1 score of 0.85 and an accuracy of 0.98 on unseen data. Now we can load these saved model weights and predict on sentences in our current General Assembly documents to qualitatively assess the results!

Predicting on New Sentences

Now that we have our saved fine-tuned model weights, we can load them into a Hugging Face transformer object and predict entities on new sentences! We will run the entity recognition on the following paragraph,

Before proceeding further, I should like to inform members that action on draft resolution iv, entitled situation of human rights of Rohingya Muslims and other minorities in Myanmar is postponed to a later date to allow time for the review of its programme budget implications by the fifth committee. The assembly will take action on draft resolution iv as soon as the report of the fifth committee on the programme budget implications is available. I now give the floor to delegations wishing to deliver explanations of vote or position before voting or adoption.

and extract the associated entities.

Using this fine-tuned model, we were able to extract many useful concepts related the the decisions made in this General Assembly meeting. We were able to correctly identify the Rohingya Muslims, the Fifth Committee, the General Assembly, Myanmar, and other entities within this paragraph. By fine-tuning our BERT model on this UN specific dataset we were able to achieve excellent quantitative and qualitative results.

The full codebase can be found on GitHub, and the results for this prediction task on this paragraph can be found in un_ner.csv.

Wrap-up

This information can be critical in extracting useful information from the staggering number of documents created within the United Nations. For instance, by extracting these entities from documents we can create knowledge graphs connecting otherwise disparate documents together. This can be used for document topic inference, relating useful documents to eachother, and question and answering tasks. This NER work is just a small part of the work we are doing at the UN Emerging Technology Lab (ETL) to accelerate the UN’s mission.

The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) manages a multibillion-dollar fund intended for projects related to humanitarian emergencies. Donors provide these funds to OCHA with a list of strategic priorities that they‘d like their donations to be aligned with, including initiatives centered around education in crisis, women and girls, protection, and disability. There’s a challenge in prioritizing where these funds land, however. Non-governmental organizations (NGOs) request funding from this pool by submitting lengthy project proposals that then must be aligned to donor funds. Once projects are selected and funded, the OCHA team must also justify their selection decisions by aligning key outcomes to the goals of the donor fund.

Currently, the entire process of project proposal ingestion, alignment determination, and reporting is manual. The Emerging Technologies Lab (ETL) at the UN partnered with Slalom to find ways that innovative technology like AI/ML and knowledge graphs can:

· Reduce the time required to process these project proposals

· Increase the rate at which these proposals successfully align with the key strategic initiatives

· Explain how and why these decisions were made

The end result was a highly explainable natural language processing pipeline that leveraged keyword entity extraction and an open source knowledge graph to achieve these goals.

Learn more about this project in our story on slalom.com.

How we work

For every project Slalom takes on, we follow a standard framework to solve the problem meaningfully and responsibly. We:

i) Determine metrics that constitute a successful outcome

ii) Identify ethical considerations to address during data profiling and modeling

iii) Explore the dataset and prototype approaches

iv) Implement a solution using metrics as a benchmark

Measure what matters

Ensuring that funds are well spent is paramount for OCHA, its donors, and ultimately the people in need. A manual process obviously imposes limitations on the speed of proposal assessments and the variety and granularity in reporting on finished projects. An automated tool to help humans in this process would need to be accurate, fast, and flexible. We defined success as achieving a > 85% accuracy in classification.

An important component of every solution we propose is model explainability. Any funding that OCHA provides must have an associated, discernible reason behind that decision that can be reported back to the donors. In contrast with many modern black box models, the text extraction and knowledge graph pipeline we implemented embeds a high level of transparency into the inference process, allowing for a robust reporting structure that ties these model outcomes to key terms and phrases that had a disproportionate impact on the assigned label.

Do what’s right

A key question to ask during any machine learning engagement is, what kind of outcomes are we driving? Is it possible that our dataset is inherently biased, and if so, do we have the tools to mitigate this bias that could lead to the perpetuation of unjust outcomes for a subset of people? As machine learning practitioners on the front lines of innovation, we have a responsibility to ask these questions to prevent machine learning models from perpetuating biased learned from historical data. For this engagement, we investigated the ethical considerations relating to the project’s purpose, the dataset, and the machine learning models.

The core of this collaboration was to help the United Nations coordinate assistance for people in need more efficiently. When this pipeline we built is deployed into production, it will have a significant positive impact on people in need of this OCHA funding. Money designated for the strategic initiatives will be more efficiently allocated, allowing NGOs to leverage this funding to help people faster. Taking these points into consideration, we concluded that this was an ethical engagement that will make a positive difference as long as we implement it correctly.

For the dataset and machine learning models, we prepared a set of ethical considerations to investigate while exploring and prototyping approaches. We knew we would be working with NGO proposal documents and detailed descriptions of the strategic initiatives. These are already heavily vetted by humanitarian experts, so we assume that these are mostly free of any glaring inherent biases. However, the machine learning approach we settled upon was an open source knowledge graph trained on Wiktionary called ConceptNet. These models trained on open source data are notorious for containing bias extracted from users using discriminatory language on these platforms. One of the most striking examples of this is Microsoft Tay, which quickly became racist, homophobic, and antisemitic after being exposed to Twitter users for a few hours. During our prototyping phase, we committed to eliminating bias like this in ConceptNet before using the model for our purposes. Read more about our findings in the model prototyping section.

Begin with the end in mind

The OCHA dataset

The solution required data from NGO project proposals and formal definitions for the four strategic initiatives we targeted. Each NGO that applies for funding must submit an extensive (30+ pages) proposal which includes a statement of purpose, detailed descriptions of its mission, metrics that show why it needs funding, and a rich set of diagrams and images that help visualize the information. These proposals have no associated labels linking them to the strategic initiatives: education in crisis, women and girls, protection, and disability.

Every project proposal that the UN receives needs to be tagged with one of these strategic initiative labels by our NLP pipeline. Each of these topics is accompanied by a descriptive supporting document that outlines the definition of that strategic initiative and what OCHA’s responsibility is in addressing it. For example, see the definition of protection.

Linking strategic initiatives to project proposals

So how do you create an explainable document classification algorithm with no explicitly assigned labels? The idea is fairly simple: use commonsense word relationships derived from an open source knowledge graph to link key words extracted from project proposal documents to words related to the strategic initiatives ‘education in crisis,’ ‘protection,’ ‘women and girls,’ and ‘disability.’

The first step is to extract key words from the project proposals that are representative of the documents. Once we have these terms, we can start building a pipeline that labels which strategic initiative these terms are closest to. There are many well-documented methods to find words or phrases that are important to a document. Methods like TF-IDF, LDA, and other variants are commonly used for this purpose. In this case, the ETL had already developed a robust implementation of YAKE (Yet Another Keyword Extractor) to gather these phrases from its documents. In addition to this extraction method, Slalom implemented LDA to gather topics from these documents and the keywords related to those topics. Below is an extract of terms found in one of the UN project proposals:

keywords = ['wfp', 'provides', 'dispatch', 'rations', 'idps', 'delivery', 'assistance', 'severely', 'situation', 'percent', 'programme', 'partners', 'hudaydah', 'insecurity', 'security', 'insecure', 'million', 'humanitarian', 'food', 'emergency', 'general', 'world', 'response']

The million-dollar question is: how do we map these key terms to the strategic initiatives the UN cares about? Our proposed solution was to create a knowledge graph centered around those humanitarian concepts (education in crisis, women and girls, protection, and disability) and implement a distance metric that finds the closest initiative given a set of key words.

For example, if we extracted these keywords from a UN proposal:

keywords = ['assist', 'safe', 'defend', 'guard']

We’d want to label that document as ‘protection.’ Alternatively, if the keywords we extracted were:

keywords = ['learning', 'traumatic', 'schooling', 'unstable']

Our mechanism for labeling these documents would ideally label the document as ‘education in crisis.’

We used an open source knowledge graph trained on Wiktionary called ConceptNet to represent the logical similarities between words and phrases. However, this knowledge graph contains many words and relationships that are not useful to us because of the vast number of topics it was trained on through Wiktionary. To narrow down the scope of what was represented in this graph, we used the ConceptNet API to gather words related to our strategic initiatives. Below is an extract of the first-depth graph centered around the strategic initiative ‘protection.’

As you can see, each one of these terms is closely associated with the protection initiative that OCHA is invested in. These relationships are best thought of in terms of sentences found in these NGO documents. For example, in the Yemen humanitarian needs funding proposal, the organization explicitly calls out its need for protection:

“The escalation of the conflict since March 2015 has dramatically aggravated the protection crisis in which millions face risks to their safety and basic rights.”

Similarly, the South Sudan humanitarian needs proposal shows the need for protection, but doesn’t call it out specifically:

“Five years of the most recent conflict has forced almost 4.2 million people to flee their homes in search of safety, nearly 2 million of them within and nearly 2.2 million outside the country.”

In both of these situations, funding for protection is paramount. However, notice how in this South Sudan excerpt, ‘protection’ isn’t mentioned, but a related word, ‘safety’, is. This knowledge graph is key for linking the implicit need for protection highlighted in, “flee their homes in search of safety,” to the initiative ‘protection’ that will help provide them with funding. We expanded on this simple 8 node knowledge graph to encompass 4,538 concepts by including each initiative in the knowledge graph and running a depth 3 breadth first search.

The last piece is to implement a function that takes keywords and scores extracted from documents as an input and outputs an initiative label based on concepts embedded in the knowledge graph. To do this, we use a simple softmax of keyword scores divided by the distance of each keyword to the initiative in the knowledge graph.

For example, given a set of terms and their associated scores:

entity_scores = {'learning': .93, 'traumatic': .87, 'schooling': .80, 'unstable': .23, 'money': .77, 'poverty': .52}

We can calculate the shortest path between each of these terms and an associated initiative in the knowledge graph. The following shows the distances between these terms and ‘education in crisis’:

entity_distances = {'learning': 1, 'traumatic': 1, 'schooling': 1, 'unstable': 1, 'money': 2, 'poverty': 3}

Based on the entity scores and distances, we get the following relevance scores r for each initiative:

initiative_relevance_scores = {'women_and_girls': 1.17, 'education_in_crisis': 2.58, 'protection': 1.26, 'disability': 0.99}

We then apply softmax for each initiative i:

initiative_softmax_scores = {'women_and_girls': 0.142, 'education_in_crisis': 0.583, 'protection': 0.156, 'disability': 0.119}

Which gives an upper-bounded score that maps the relevance of the entities extracted from a given document to the strategic initiatives that the documents will be labeled with. In this case, the document would be labeled ‘education in crisis’ due to its high score of 0.583. The graph below depicts the holistic pipeline:

Addressing open source model bias

We quickly identified discriminatory language and bias while using ConceptNet to find terms related to the key strategic initiatives. We found the most prominent biases when we looked for terms related to ‘women and girls.’ Here are some of the terms we found while using the ConceptNet API: ‘woman is a gold digger,’ ‘woman is a nymphet,’ ‘woman is a baggage,’ ‘girl is a lolicon,’ ‘woman is an attendant,’ etc. We then looked at terms like ‘refugee’ and ‘disability,’ which are related to the initiatives, and found some disturbing results: ‘refugee related to refujew’ and ‘disability related to useless eater.’ Note that these are specific examples taken out of many valuable results, but regardless, it’s inexcusable to allow these associations to remain.

To address these issues, the first step we took was to remove all verbiage not found in the project proposal vocabulary. This removed much of the specific slang and discriminatory words that would never be found in an official document unless it was specifically relevant. The second was to remove associations that we deemed biased from the associations to women, including ‘baggage,’ ‘servant,’ and ‘attendant.’ We completed a manual review of all concepts directly related to the strategic initiatives and ensured this bias would not persist into any future iterations of the knowledge graph we built. That being said, we believe the ideal future process includes a group of subject matter experts of various backgrounds reviewing this terminology to promote inter-rater reliability and ensure all model bias is removed.

ConceptNet has worked to address many of the issues we outlined here and has specifically highlighted its ability to reduce inherent gender bias in its model in a 2017 blog post. But despite its best efforts, there’s still clear and present bias that we were fortunately able to mitigate with our small knowledge graph subset.

There’s quite a bit of work left to do in the machine learning field to automatically remove bias from tainted datasets. An example of recent work aimed at reducing inherent bias outlines a method using neutralizing edits in Wikipedia articles to find instances of demographic bias, like using the term ‘mankind’ instead of ‘humankind’ or ‘his career’ as opposed to ‘their careers’ when describing a group of people (Pryzant et al., 2020). Although this work did not directly serve our use case, it’s encouraging to see the research community start to address this problem plaguing many of the most used machine learning datasets.

Tying it all together

Our implementation was incredibly fast and inexpensive. With the current limited-size knowledge graph, we were able to run it on a micro-instance virtual machine far cheaper than our $150,000 benchmark. As the solution is scaled to accommodate real-time document classification, this cost will increase but not above the target amount. Also, through our initial discovery, we saw very promising results for meeting the 85% required classification accuracy necessary to justify the scaling of this model. However, there’s still a significant amount of expert review required to verify these results.

There are many future refinements that can be implemented to improve this current pipeline. The UN team is interested in leveraging expertise in its organization to embed domain specific knowledge into the existing knowledge graph. This would seamlessly integrate into the current pipeline by allowing the distance metric to better link these extracted key words to the strategic initiatives that are most relevant to a given document. The distance metric itself is fairly simplistic and could be modified to better represent how graph distance between keywords and initiatives maps to how relevant that term is. The current solution is powerful and is truly just the first step in what will become a scalable, explainable natural language processing pipeline that will help the UN more efficiently allocate donor funds to the humanitarian crises that need it.

Learn about us

This work was delivered by Slalom’s AI Center of Purpose which is dedicated to improving the world with artificial intelligence. Learn more about this team and the associated Innovation for Good initiative.

Acknowledgements

Big thanks to Lambert Hogenhout, Kevin Bradley, Mulin Tang, and Jiayue Wu at the United Nations Emerging Technology Lab for partnering with our team at Slalom. Also, to Raki Rahman, Benjamin Chan, Adrien Galamez, Michelle Yi, Tony Ko, Zahra Zahid, Parul Patel, and many others at Slalom for making this solution come to life.