diff --git a/6_information_extraction_answers.ipynb b/6_information_extraction_answers.ipynb new file mode 100644 index 0000000..ad57837 --- /dev/null +++ b/6_information_extraction_answers.ipynb @@ -0,0 +1,6214 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Lab 6: Information Extraction\n", + "\n", + "In this lab, you'll learn how to carry out the two main steps of information extraction: named entity recognition, and relation extraction.\n", + "\n", + "\n", + "### Aims\n", + "* Know how to train an NER sequence tagger using a CRF.\n", + "* Learn how to construct a relation classifier for extracting relations between named entities.\n", + "* Understand how syntactic features can be used in NER and relation extraction (RE).\n", + "\n", + "### Outline\n", + "\n", + "* Loading the re3d corpus.\n", + "* Training and testing a CRF NER tagger.\n", + "* Experimenting with PoS speech tags and dependency parse features.\n", + "* Training a naive Bayes relation classifier.\n", + "\n", + "### How To Complete This Lab\n", + "\n", + "Read the text and the code then look for 'TODOs' that instruct you to complete some missing code or answer a question. You don't have to stick rigidly to the lab -- feel free to explore other methods and data to help you understand what's going on or to go beyond this lab. \n", + "\n", + "Aim to work through the lab during the scheduled lab hour. You can also contact TAs with questions at the scheduled times throughout the week, or post your questions to our Teams conversation.\n", + "\n", + "The labs *will not be marked*. However, they will prepare you for the coursework, so try to keep up with the weekly labs and have fun with the exercises!\n", + "\n", + "### More Information\n", + "\n", + "This lab relates to [Chapter 17 on Information Extraction from Jurafsky and Martin's 2020 draft of Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/17.pdf).\n", + "\n", + "For more on the NLTK library related to this lab, please see [chapters 5 (tagging words)](https://www.nltk.org/book/ch05.html) and [7 (extracting information from text)](https://www.nltk.org/book/ch07.html) of the NLTK book. For details of the sequence tagger implementations, refer to [the NLTK documentation](http://www.nltk.org/api/nltk.tag.html?highlight=hmm).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, run the cell below to import various bits of NLTK and Sklearn." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting package metadata (current_repodata.json): done\n", + "Solving environment: done\n", + "\n", + "# All requested packages already installed.\n", + "\n" + ] + } + ], + "source": [ + "!conda install -y python-crfsuite" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# these magic lines make the notebook reload any module from an external file that changes.\n", + "# They're useful if you are working on an external script as well as the notebook\n", + "%load_ext autoreload\n", + "%autoreload 2\n", + "\n", + "import numpy as np\n", + "\n", + "import matplotlib.pyplot as plt\n", + "from IPython import display\n", + "\n", + "import nltk\n", + "from nltk.tag import CRFTagger\n", + "\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn.metrics import f1_score\n", + "\n", + "import os" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We also need to install python-crfsuite for named entity recognition. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 1 Named Entity Recognition (NER)\n", + "\n", + "How can we extract information from text? The first step is usually to identify the entities involved, i.e., the people, places, organisations, times and other subjects discussed in the text. NER is the task of tagging the entities in a piece of text, which is usually modelled as a BIO sequence labelling task. In a BIO task, we use different tags to differentiate the beginning ('B' tag) and inside ('I' tag) of a named entity span as well as tokens outside ('O' tag) any span. This allows us to extract named entities that span multiple tokens.\n", + "\n", + "Let's start by loading some data with NER tags.\n", + "\n", + "In this lab, we'll work with part of the [re3d dataset](https://github.com/dstl/re3d), which contains articles from government and news websites." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 98 documents\n", + "Loaded 2433 non-overlapping entities in total.\n", + "We have loaded a dataset with 953 sentences.\n" + ] + } + ], + "source": [ + "from lab6_data.load_re3d import load_re3d_data\n", + "\n", + "sentences, tags, tag_names_to_idx, tag_idx_to_names, relation_pairs = load_re3d_data()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we can look at an example of some text and the kind of entity tags we would like to apply. \n", + "\n", + "(note if performance is poor, we can change to a simpler BIO task without entity types)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Hibhib\t\t\tB-Location\n", + "(\t\t\tO\n", + "Arabic\t\t\tB-Nationality\n", + ":\t\t\tO\n", + "ناحية\t\t\tO\n", + "هبهب‎‎\t\t\tO\n", + ",\t\t\tO\n", + "Hibhib\t\t\tB-Location\n", + "Village\t\t\tI-Location\n", + ")\t\t\tO\n", + "is\t\t\tO\n", + "a\t\t\tO\n", + "village\t\t\tO\n", + "in\t\t\tO\n", + "northern\t\t\tB-Location\n", + "Iraq\t\t\tI-Location\n", + ",\t\t\tO\n", + "located\t\t\tO\n", + "8\t\t\tB-Quantity\n", + "km\t\t\tI-Quantity\n", + "(\t\t\tO\n", + "5.0\t\t\tB-Quantity\n", + "mi\t\t\tI-Quantity\n", + ")\t\t\tO\n", + "northwest\t\t\tO\n", + "of\t\t\tO\n", + "Baquba\t\t\tB-Location\n", + ".\t\t\tO\n" + ] + } + ], + "source": [ + "# Print an example of the text aligned with the tags (use tag names)\n", + "tagged_words = zip(sentences[0], tag_idx_to_names[tags[0]])\n", + "for word, tag in tagged_words:\n", + " print(f'{word}\\t\\t\\t{tag}')\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's create a training and test split, then show some basic counts of the data:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of training sentences: 762\n", + "Number of test sentences: 191\n" + ] + } + ], + "source": [ + "# Use the commented out code below if you want to remove non-ascii characters.\n", + "\n", + "# def is_english(s):\n", + "# try:\n", + "# s.encode(encoding='utf-8').decode('ascii')\n", + "# except UnicodeDecodeError:\n", + "# return False\n", + "# else:\n", + "# return True\n", + " \n", + "# for i, sent in enumerate(sentences):\n", + "# for j, tok in enumerate(sent):\n", + "# if not is_english(tok):\n", + "# sentences[i][j] = 'BLANK'\n", + "\n", + "\n", + "# *** split dataset into train and test\n", + "data = []\n", + "for i in range(len(sentences)):\n", + " tagged_words = zip(sentences[i], tag_idx_to_names[tags[i]])\n", + " data.append(list(tagged_words))\n", + "\n", + "tagged_sents_with_rels = list(zip(data, relation_pairs))\n", + "\n", + "train_set_with_rels, test_set_with_rels = train_test_split(\n", + " tagged_sents_with_rels,\n", + " train_size=0.80,\n", + " test_size=0.20,\n", + " random_state=101\n", + ")\n", + "\n", + "# separate out the relations data from the sentences and tags\n", + "train_set = [tagged_sent for tagged_sent, _ in train_set_with_rels]\n", + "test_set = [tagged_sent for tagged_sent, _ in test_set_with_rels]\n", + "\n", + "train_rels = [rels for _, rels in train_set_with_rels]\n", + "test_rels = [rels for _, rels in test_set_with_rels]\n", + "\n", + "\n", + "print(f'Number of training sentences: {len(train_set)}')\n", + "print(f'Number of test sentences: {len(test_set)}')" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of tagged words in the training set: 19506\n", + "Number of tagged words in the test set: 4799\n" + ] + } + ], + "source": [ + "train_tagged_words = [ tup for sent in train_set for tup in sent ]\n", + "test_tagged_words = [ tup for sent in test_set for tup in sent ]\n", + "\n", + "print(f'Number of tagged words in the training set: {len(train_tagged_words)}')\n", + "print(f'Number of tagged words in the test set: {len(test_tagged_words)}')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, print out the different named entity tags. Notice that each entity type has both a 'B-' and an 'I-' tag. The labels include a lot of specialised types of entities for extracting information from security and defence news." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of possible tags: 27\n", + "Possible tags: ['I-Person', 'B-DocumentRefere', 'B-Money', 'I-CommsIdentifie', 'O', 'B-MilitaryPlatfo', 'B-Nationality', 'I-Quantity', 'B-CommsIdentifie', 'B-Temporal', 'B-Weapon', 'B-Frequency', 'I-Money', 'I-Organisation', 'B-Quantity', 'I-Weapon', 'B-Organisation', 'I-DocumentRefere', 'I-MilitaryPlatfo', 'B-Location', 'I-Nationality', 'I-Vehicle', 'I-Location', 'B-Person', 'B-Vehicle', 'I-Frequency', 'I-Temporal']\n" + ] + } + ], + "source": [ + "ne_tags = {tag for word, tag in train_tagged_words} # a set of possible tags\n", + "ne_tags = list(ne_tags) # let's turn the set into a list as it is more useful in future steps.\n", + "print(f'Number of possible tags: {len(ne_tags)}')\n", + "print(f'Possible tags: {ne_tags}')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see an example sentence from the training set:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Sentence example: [('Last', 'B-Temporal'), ('year', 'I-Temporal'), (\"'s\", 'O'), ('influx', 'O'), ('of', 'O'), ('hundreds', 'B-Quantity'), ('of', 'I-Quantity'), ('thousands', 'I-Quantity'), ('to', 'O'), ('Europe', 'O'), ('partly', 'O'), ('resulted', 'O'), ('from', 'O'), ('cuts', 'O'), ('to', 'O'), ('food', 'O'), ('aid', 'O'), ('and', 'O'), ('cash', 'O'), ('payments', 'O'), ('.', 'O')]\n" + ] + } + ], + "source": [ + "print('Sentence example: {}'.format(train_set[0]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, let's train a CRF tagger on our training set. The method you need to use from NLTK is the [train method of the conditional random field (CRF)](https://www.nltk.org/_modules/nltk/tag/crf.html). The interface differs from that of the HMM tagger, so you may need to refer to the documentation. Briefly, you need to call the constructor with default arguments, then the train() function.\n", + "\n", + "**TODO 1.1: Write a function to train and return a CRF named entity recogniser.**" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Train a CRF NER tagger\n", + "def train_CRF_NER_tagger(train_set):\n", + " ### WRITE YOUR OWN CODE HERE\n", + " tagger = nltk.tag.CRFTagger()\n", + " tagger.train(train_set, 'model.crf.tagger')\n", + " return tagger # return the trained model\n", + "\n", + "tagger = train_CRF_NER_tagger(train_set)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, let's see how well it performs. We do prediction in the same way as with the HMM in lab 5. First, we need to get the test data into the right format as a list of sentences, where each sentence is a list of tokens. \n", + "\n", + "**TODO 1.2: Complete the function below to convert the test data to the correct format and predict the tags on the test set.**" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[('The', 'B-Organisation'), ('SDF', 'I-Organisation'), (',', 'O'), ('made', 'O'), ('up', 'O'), ('in', 'O'), ('part', 'O'), ('by', 'O'), ('local', 'B-Organisation'), ('Arabs', 'I-Organisation'), ('and', 'I-Organisation'), ('its', 'I-Organisation'), ('Coalition', 'I-Organisation'), ('trained', 'O'), ('and', 'O'), ('equipped', 'O'), ('Arab', 'O'), ('component', 'O'), (',', 'O'), ('the', 'B-Organisation'), ('Syrian', 'I-Organisation'), ('Arab', 'I-Organisation'), ('Coalition', 'I-Organisation'), (',', 'O'), ('and', 'O'), ('supported', 'O'), ('by', 'O'), ('Coalition', 'B-Organisation'), ('advisers', 'I-Organisation'), ('and', 'O'), ('air', 'O'), ('strikes', 'O'), ('began', 'O'), ('the', 'O'), ('operation', 'O'), ('to', 'O'), ('isolate', 'O'), ('Raqqah', 'O'), ('on', 'O'), ('Nov.', 'B-Temporal'), ('5', 'I-Temporal'), ('.', 'O')], [('``', 'O'), ('``', 'O'), (\"''\", 'O'), ('Now', 'O'), ('these', 'O'), ('people', 'O'), ('live', 'O'), ('in', 'O'), ('dignity', 'O'), (',', 'O'), (\"''\", 'O'), (\"''\", 'O'), ('he', 'O'), ('says', 'O'), ('.', 'O')]]\n" + ] + } + ], + "source": [ + "# Test\n", + "def tag_test_set(test_set, tagger):\n", + " ### WRITE YOUR OWN CODE HERE\n", + " test_sents = [[token for token,tag in sent] for sent in test_set]\n", + " predicted_tags = tagger.tag_sents(test_sents)\n", + " ###\n", + " return predicted_tags\n", + "test_sents_with_predicted_tags = tag_test_set(test_set,tagger)\n", + "print(test_sents_with_predicted_tags[:2]) # Print two tagged sentences" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see how well the tagger is performing. In NER, we evaluate performance by finding correctly matched entities, rather than correctly tagged tokens. Only an exact entity match counts as correct. Therefore, we need to compute precision, recall and F1 score by computing true positives, false positives and false negatives by looking for the predicted entity spans and the gold-labelled entity spans in the test set.\n", + "\n", + "The code below contains a function that extract a list of spans from the tagged sentences. The next function calls extract_spans() and computes the precision, recall and f1 scores. However, the function is incomplete.\n", + "\n", + "**TODO 1.3: Complete the cal_span_level_F1() function below to compute span-level F1 scores for the predictions.** \n", + "\n", + "Have a look at the results. Which types of entity are being recognised well and which are very poor?" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "F1 score for class Person = 0.7350427350427351\n", + "F1 score for class DocumentRefere = 0\n", + "F1 score for class Money = 0\n", + "F1 score for class CommsIdentifie = 0\n", + "F1 score for class MilitaryPlatfo = 0.16666666666666669\n", + "F1 score for class Nationality = 0.5\n", + "F1 score for class Quantity = 0\n", + "F1 score for class Temporal = 0.5714285714285714\n", + "F1 score for class Weapon = 0.22222222222222224\n", + "F1 score for class Frequency = 0\n", + "F1 score for class Organisation = 0.7017543859649124\n", + "F1 score for class Location = 0.45454545454545453\n", + "F1 score for class Vehicle = 0\n", + "Macro-average f1 score = 0.257820002759274\n" + ] + } + ], + "source": [ + "def extract_spans(tagged_sents, ne_tags):\n", + " \"\"\"\n", + " Extract a list of tagged spans for each named entity type, \n", + " where each span is represented by a tuple containing the \n", + " start token and end token indexes.\n", + " \n", + " returns: a dictionary containing a list of spans for each entity type.\n", + " \"\"\"\n", + " spans = {}\n", + " for ne_tag in ne_tags:\n", + " if ne_tag == 'O':\n", + " continue\n", + "\n", + " spans[ne_tag[2:]] = [] # create an empty list to store the spans of each type\n", + " \n", + " for sent in tagged_sents:\n", + " start = -1\n", + " entity_type = None\n", + " for i, (tok, lab) in enumerate(sent):\n", + " if 'B-' in lab:\n", + " start = i\n", + " end = i + 1\n", + " entity_type = lab[2:]\n", + " elif 'I-' in lab:\n", + " end = i + 1\n", + " elif lab == 'O' and start >= 0:\n", + " spans[entity_type].append((start, end))\n", + " start = -1\n", + " \n", + " return spans\n", + "\n", + "\n", + "def cal_span_level_f1(test_sents, test_sents_with_pred, ne_tags):\n", + " # get a list of spans from the test set labels\n", + " gold_spans = extract_spans(test_sents, ne_tags)\n", + "\n", + " # get a list of spans predicted by our tagger\n", + " pred_spans = extract_spans(test_sents_with_pred, ne_tags)\n", + " \n", + " # compute the metrics for each class:\n", + " f1_per_class = []\n", + " \n", + " ne_types = gold_spans.keys() # get the list of named entity types (not the tags)\n", + " \n", + " for ne_type in ne_types:\n", + " # compute the confusion matrix\n", + " true_pos = 0\n", + " false_pos = 0\n", + " \n", + " ### WRITE YOUR OWN CODE HERE TO COUNT TRUE POSITIVES, FALSE POSITIVES, AND FALSE NEGATIVES.\n", + " for span in pred_spans[ne_type]:\n", + " if span in gold_spans[ne_type]:\n", + " true_pos += 1\n", + " else:\n", + " false_pos += 1\n", + " \n", + " false_neg = 0\n", + " for span in gold_spans[ne_type]:\n", + " if span not in pred_spans[ne_type]:\n", + " false_neg += 1\n", + " \n", + " ### \n", + " \n", + " if true_pos + false_pos == 0:\n", + " precision = 0\n", + " else:\n", + " precision = true_pos / float(true_pos + false_pos)\n", + " \n", + " if true_pos + false_neg == 0:\n", + " recall = 0\n", + " else:\n", + " recall = true_pos / float(true_pos + false_neg)\n", + " \n", + " if precision + recall == 0:\n", + " f1 = 0\n", + " else:\n", + " f1 = 2 * precision * recall / (precision + recall)\n", + " \n", + " f1_per_class.append(f1)\n", + " print(f'F1 score for class {ne_type} = {f1}')\n", + " \n", + " print(f'Macro-average f1 score = {np.mean(f1_per_class)}')\n", + "\n", + "cal_span_level_f1(test_set, test_sents_with_predicted_tags, ne_tags)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The code below prints out a sample of a sentence where there is an error. \n", + "\n", + "**TODO 1.4: Look at the errors below. Can you see any common patterns that you think might be causing some of the errors? Post your suggestions to the lab bubble chat.**" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error found:\n", + " Text: ['by', 'local', 'Arabs', 'and', 'its', 'Coalition']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['local', 'Arabs', 'and', 'its', 'Coalition', 'trained']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Arabs', 'and', 'its', 'Coalition', 'trained', 'and']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['trained', 'and', 'equipped', 'Arab', 'component', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'equipped', 'Arab', 'component', ',', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Coalition', 'advisers', 'and', 'air', 'strikes', 'began']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['advisers', 'and', 'air', 'strikes', 'began', 'the']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Weapon', 'I-Weapon', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['``', \"''\", 'Now', 'these', 'people', 'live']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: [\"''\", 'Now', 'these', 'people', 'live', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Al', 'Assad', 'University', 'Hospital', 'in']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Al', 'Assad', 'University', 'Hospital', 'in', 'Damascus']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['University', 'Hospital', 'in', 'Damascus', 'is', 'one']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'O', 'B-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['teaching', 'hospitals', 'in', 'Syria', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Weeks', 'earlier', ',', 'the', 'so-called', 'Islamic']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['earlier', ',', 'the', 'so-called', 'Islamic', 'State']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: [',', 'the', 'so-called', 'Islamic', 'State', 'had']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['made', 'it', 'to', 'this', 'street', '-']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['it', 'to', 'this', 'street', '-', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Gen.', 'Richard', '``', 'Tex', \"''\"]\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Gen.', 'Richard', '``', 'Tex', \"''\", 'Coe']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'UK', 'stands', 'by', 'Iraq']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'UK', 'stands', 'by', 'Iraq', 'to']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['UK', 'stands', 'by', 'Iraq', 'to', 'defeat']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['used', 'to', 'hire', 'Egyptians', 'at', 'two']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Money']\n", + "\n", + "Error found:\n", + " Text: ['to', 'hire', 'Egyptians', 'at', 'two', 'Jordanian']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['hire', 'Egyptians', 'at', 'two', 'Jordanian', 'dinars']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['Egyptians', 'at', 'two', 'Jordanian', 'dinars', '(']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['at', 'two', 'Jordanian', 'dinars', '(', '£2.10']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'B-Money', 'I-Money', 'I-Money', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['(', '£2.10', ';', '$', '2.80', ')']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['£2.10', ';', '$', '2.80', ')', 'an']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'construction', 'of', 'simple', 'chemical', 'weapons']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Weapon', 'I-Weapon']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['construction', 'of', 'simple', 'chemical', 'weapons', ',']\n", + " Prediction: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['how', 'to', 'build', 'a', 'chemical', 'munition']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['to', 'build', 'a', 'chemical', 'munition', 'from']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['build', 'a', 'chemical', 'munition', 'from', 'an']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'O', 'B-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['chemical', 'munition', 'from', 'an', 'ordinary', 'artillery']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Weapon', 'I-Weapon', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['munition', 'from', 'an', 'ordinary', 'artillery', 'round']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Weapon', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['from', 'an', 'ordinary', 'artillery', 'round', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['an', 'ordinary', 'artillery', 'round', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['In', 'December', ',', 'Syrian', 'state', 'media']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['December', ',', 'Syrian', 'state', 'media', 'said']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'Syrian', 'state', 'media', 'said', 'an']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['strike', 'had', 'targeted', 'the', 'airport', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['had', 'targeted', 'the', 'airport', 'in', 'Mezzeh']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['one', 'strike', 'suppressed', 'an', 'ISIL', 'tactical']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['strike', 'suppressed', 'an', 'ISIL', 'tactical', 'unit']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['is', 'based', 'on', \"'\", 'Z', \"'\"]\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['based', 'on', \"'\", 'Z', \"'\", 'or']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['on', \"'\", 'Z', \"'\", 'or', 'Greenwich']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'B-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['Z', \"'\", 'or', 'Greenwich', 'Mean', 'Time']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: [\"'\", 'or', 'Greenwich', 'Mean', 'Time', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['or', 'Greenwich', 'Mean', 'Time', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Ziad', 'Khalaf', 'al-Karbouly', '(', 'زياد']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'native', 'of', \"Al-Qa'im\", ',', 'was']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['1', ']', 'former', 'Iraqi', 'officer', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Nationality', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'son', 'of', 'an', 'Iraqi', 'tribal']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['son', 'of', 'an', 'Iraqi', 'tribal', 'sheikh']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['of', 'an', 'Iraqi', 'tribal', 'sheikh', 'of']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['an', 'Iraqi', 'tribal', 'sheikh', 'of', 'the']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Iraqi', 'tribal', 'sheikh', 'of', 'the', 'Al-Karabla']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['tribal', 'sheikh', 'of', 'the', 'Al-Karabla', 'clan']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'Al-Karabla', 'clan', 'of', 'the']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Al-Karabla', 'clan', 'of', 'the', 'Dulaim', '.']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['clan', 'of', 'the', 'Dulaim', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['forward', 'to', 'resolving', 'the', 'Syrian', 'crisis']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'resolving', 'the', 'Syrian', 'crisis', '.']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['utterly', 'dependent', 'on', 'foreign', 'militias', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['dependent', 'on', 'foreign', 'militias', 'and', 'Russian']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['foreign', 'militias', 'and', 'Russian', 'air', 'support']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['militias', 'and', 'Russian', 'air', 'support', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'Russian', 'air', 'support', ',', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['further', 'atrocities', 'against', 'the', 'Syrian', 'people']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['atrocities', 'against', 'the', 'Syrian', 'people', '.']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['against', 'the', 'Syrian', 'people', '.']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['back', 'and', 'destroying', 'ISIL', 'heavy', 'weapons']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['and', 'destroying', 'ISIL', 'heavy', 'weapons', ',']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['destroying', 'ISIL', 'heavy', 'weapons', ',', 'vehicles']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'fortifications', ',', 'IED', 'facilities', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['fortifications', ',', 'IED', 'facilities', ',', 'VBIEDs']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'VBIEDs', ',', 'armored', 'vehicles', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['VBIEDs', ',', 'armored', 'vehicles', ',', 'technical']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'B-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['armored', 'vehicles', ',', 'technical', 'vehicles', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['vehicles', ',', 'technical', 'vehicles', ',', 'bridges']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-MilitaryPlatfo', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['About', '350', 'Syrians', 'work', 'for']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Syrians', 'work', 'for', 'al-Rahman', 'Farms', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['work', 'for', 'al-Rahman', 'Farms', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Boris', 'Johnson', 'met', 'the', 'Iraqi']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Person', 'I-Person', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Boris', 'Johnson', 'met', 'the', 'Iraqi', 'Foreign']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['al-', 'Jaafari', 'in', 'London', 'today', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Location', 'B-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Jaafari', 'in', 'London', 'today', '.']\n", + " Prediction: ['I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'O', 'B-Location', 'B-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['welcomes', 'new', 'Syria', 'sanctions', 'listings', 'that']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['new', 'Syria', 'sanctions', 'listings', 'that', 'apply']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['support', 'from', '14', 'donor', 'countries', 'over']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['from', '14', 'donor', 'countries', 'over', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['projects', 'that', 'reach', '2', 'million', 'Syrians']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['that', 'reach', '2', 'million', 'Syrians', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Quantity', 'I-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['million', 'Syrians', 'in', 'the', 'sectors', 'of']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Quantity', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Syrians', 'in', 'the', 'sectors', 'of', 'health']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['military', 'forces', 'conducted', '26', 'strikes', 'against']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Quantity', 'B-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'conducted', '26', 'strikes', 'against', 'ISIL']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Quantity', 'B-Weapon', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['ISIL', 'terrorists', 'in', 'Syria', 'and', 'Iraq']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Location', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Syria', 'and', 'Iraq', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'O', 'B-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['will', 'meet', 'with', 'national', 'and', 'provincial']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['meet', 'with', 'national', 'and', 'provincial', 'Iraqi']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['with', 'national', 'and', 'provincial', 'Iraqi', 'officials']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['national', 'and', 'provincial', 'Iraqi', 'officials', 'to']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'our', 'allies', 'and', 'friends', '.']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['our', 'allies', 'and', 'friends', '.']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['on', 'Syria', 'for', 'international', 'donors', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Syria', 'for', 'international', 'donors', 'in', 'London']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['one', 'case', ',', 'three', 'women', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['case', ',', 'three', 'women', 'and', 'three']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: [',', 'three', 'women', 'and', 'three', 'children']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['three', 'women', 'and', 'three', 'children', 'were']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['women', 'and', 'three', 'children', 'were', 'allegedly']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'use', 'of', 'suicide', 'vehicles', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['use', 'of', 'suicide', 'vehicles', ',', 'IEDs']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['He', 'survived', 'the', 'night', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['He', 'survived', 'the', 'night', 'of', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['survived', 'the', 'night', 'of', 'the', 'attack']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['the', 'night', 'of', 'the', 'attack', 'by']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['night', 'of', 'the', 'attack', 'by', 'jumping']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['for', 'Lebanon', \"'s\", 'Hezbollah', 'movement', 'several']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Lebanon', \"'s\", 'Hezbollah', 'movement', 'several', 'times']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'support', 'of', 'Turkey', '’', 's']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['support', 'of', 'Turkey', '’', 's', 'democratically']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'Turkey', '’', 's', 'democratically', 'elected']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Turkey', '’', 's', 'democratically', 'elected', 'government']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['’', 's', 'democratically', 'elected', 'government', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['s', 'democratically', 'elected', 'government', 'and', 'institutions—institutions']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['institutions—institutions', 'enshrined', 'in', 'Turkey', '’', 's']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['enshrined', 'in', 'Turkey', '’', 's', 'constitution']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Turkey', '’', 's', 'constitution', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Turkey', '’', 's', 'constitution', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['militants', 'shot', 'dead', 'three', 'women', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['shot', 'dead', 'three', 'women', 'and', 'three']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['dead', 'three', 'women', 'and', 'three', 'children']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['three', 'women', 'and', 'three', 'children', 'from']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['women', 'and', 'three', 'children', 'from', 'Rufaila']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['trailing', 'behind', 'a', 'group', 'of', 'other']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['behind', 'a', 'group', 'of', 'other', 'people']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['a', 'group', 'of', 'other', 'people', 'from']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['group', 'of', 'other', 'people', 'from', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['other', 'people', 'from', 'the', 'same', 'village']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['people', 'from', 'the', 'same', 'village', 'who']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['from', 'the', 'same', 'village', 'who', 'were']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'relocate', 'to', 'another', 'area', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['relocate', 'to', 'another', 'area', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['his', 'desire', 'that', 'the', 'command', 'attracts']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['desire', 'that', 'the', 'command', 'attracts', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['his', 'time', 'with', 'a', 'rifle', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['time', 'with', 'a', 'rifle', 'in', 'one']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['benefit', 'from', 'this', 'U.S.', 'sovereign', 'loan']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['from', 'this', 'U.S.', 'sovereign', 'loan', 'guarantee']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['this', 'U.S.', 'sovereign', 'loan', 'guarantee', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Money', 'I-Money', 'I-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['U.S.', 'sovereign', 'loan', 'guarantee', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Money', 'I-Money', 'I-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['fighter', ',', 'and', 'remotely', 'piloted', 'aircraft']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: [',', 'and', 'remotely', 'piloted', 'aircraft', 'as']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'remotely', 'piloted', 'aircraft', 'as', 'well']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['as', 'well', 'as', 'rocket', 'artillery', 'against']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['well', 'as', 'rocket', 'artillery', 'against', 'ISIL']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['rocket', 'artillery', 'against', 'ISIL', 'targets', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Weapon', 'I-Weapon', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['artillery', 'against', 'ISIL', 'targets', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Weapon', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['one', 'strike', 'destroyed', 'a', 'repeater', 'station']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['strike', 'destroyed', 'a', 'repeater', 'station', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['destroyed', 'a', 'repeater', 'station', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['at', 'the', 'hospital', 'use', 'the', 'library']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'O', 'B-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['the', 'hospital', 'use', 'the', 'library', \"'s\"]\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['hospital', 'use', 'the', 'library', \"'s\", 'books']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['use', 'the', 'library', \"'s\", 'books', 'to']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'library', \"'s\", 'books', 'to', 'advise']\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['treat', 'patients', ';', 'untrained', 'teachers', 'use']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['patients', ';', 'untrained', 'teachers', 'use', 'them']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['classes', ';', 'and', 'aspiring', 'dentists', 'raid']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: [';', 'and', 'aspiring', 'dentists', 'raid', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['One', 'man', 'I', 'spoke', 'to']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['B-Person', 'I-Person', 'B-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['predicted', 'that', 'after', 'nearly', 'four', 'long']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['that', 'after', 'nearly', 'four', 'long', 'years']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['after', 'nearly', 'four', 'long', 'years', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['nearly', 'four', 'long', 'years', 'of', 'siege']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['bodies', 'thrown', 'into', 'a', 'river', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['thrown', 'into', 'a', 'river', 'in', 'an']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['liberate', 'strategically', 'valuable', 'terrain', 'surrounding', 'Raqqah']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['strategically', 'valuable', 'terrain', 'surrounding', 'Raqqah', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['valuable', 'terrain', 'surrounding', 'Raqqah', 'and', 'enable']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['than', '40', 'members', 'of', 'the', 'Coalition']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['40', 'members', 'of', 'the', 'Coalition', 'will']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'the', 'regime', 'until', 'it', 'reassesses']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['cooperative', 'venture', 'of', 'the', 'Future', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['venture', 'of', 'the', 'Future', 'of', 'Babylon']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'Future', 'of', 'Babylon', 'project']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Future', 'of', 'Babylon', 'project', '–']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Future', 'of', 'Babylon', 'project', '–', 'a']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'United', 'States', 'and', 'WMF', 'to']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['United', 'States', 'and', 'WMF', 'to', 'support']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'defense', 'of', 'the', 'island', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['defense', 'of', 'the', 'island', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Karim', 'symbolises', 'a', 'country', 'once']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Karim', 'symbolises', 'a', 'country', 'once', 'famed']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['once', 'famed', 'in', 'the', 'region', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['famed', 'in', 'the', 'region', 'and', 'beyond']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'news', 'on', 'this', 'Thanksgiving', 'Day']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['news', 'on', 'this', 'Thanksgiving', 'Day', 'that']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['on', 'this', 'Thanksgiving', 'Day', 'that', 'one']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['Thanksgiving', 'Day', 'that', 'one', 'of', 'our']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Day', 'that', 'one', 'of', 'our', 'brave']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['that', 'one', 'of', 'our', 'brave', 'service']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['one', 'of', 'our', 'brave', 'service', 'members']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['of', 'our', 'brave', 'service', 'members', 'has']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['our', 'brave', 'service', 'members', 'has', 'been']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Carter', 'said', 'in', 'a', 'Nov.', '24']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Temporal', 'I-Temporal']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['said', 'in', 'a', 'Nov.', '24', 'statement']\n", + " Prediction: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['in', 'a', 'Nov.', '24', 'statement', '.']\n", + " Prediction: ['O', 'O', 'B-Temporal', 'I-Temporal', 'O', 'O']\n", + "Gold labels: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'Nov.', '24', 'statement', '.']\n", + " Prediction: ['O', 'B-Temporal', 'I-Temporal', 'O', 'O']\n", + "Gold labels: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Europe', 'and', 'the', 'Middle', 'East']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Europe', 'and', 'the', 'Middle', 'East', 'aim']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'the', 'Middle', 'East', 'aim', 'to']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['“', 'The', 'entire', 'counter-ISIL', 'coalition']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['“', 'The', 'entire', 'counter-ISIL', 'coalition', 'sends']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['entire', 'counter-ISIL', 'coalition', 'sends', 'our', 'condolences']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['our', 'condolences', 'to', 'this', 'hero', \"'s\"]\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error found:\n", + " Text: ['condolences', 'to', 'this', 'hero', \"'s\", 'family']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['commander', 'of', 'Combined', 'Joint', 'Task', 'Force']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Joint', 'Task', 'Force', '-', 'Operation', 'Inherent']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Task', 'Force', '-', 'Operation', 'Inherent', 'Resolve']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Force', '-', 'Operation', 'Inherent', 'Resolve', ',']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['-', 'Operation', 'Inherent', 'Resolve', ',', 'said']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Today', 'at', 'the', 'Iraqi', 'Museum']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Today', 'at', 'the', 'Iraqi', 'Museum', 'in']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['at', 'the', 'Iraqi', 'Museum', 'in', 'Baghdad']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['Iraqi', 'Museum', 'in', 'Baghdad', ',', 'the']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['I-Location', 'I-Location', 'O', 'B-Location', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Baghdad', ',', 'the', 'U.S.', 'Embassy']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Location', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Baghdad', ',', 'the', 'U.S.', 'Embassy', 'Chargé']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Location', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [',', 'the', 'U.S.', 'Embassy', 'Chargé', 'd']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['the', 'U.S.', 'Embassy', 'Chargé', 'd', '’']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['U.S.', 'Embassy', 'Chargé', 'd', '’', 'Affaires']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Embassy', 'Chargé', 'd', '’', 'Affaires', ',']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Chargé', 'd', '’', 'Affaires', ',', 'a.i.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'a.i.', ',', 'Jonathan', 'Cohen', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a.i.', ',', 'Jonathan', 'Cohen', 'and', 'Dr.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Person']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['Ahmed', 'Kamil', 'Murad', ',', 'Acting', 'General']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Kamil', 'Murad', ',', 'Acting', 'General', 'Director']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Acting', 'General', 'Director', 'of', 'the', 'Office']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['General', 'Director', 'of', 'the', 'Office', 'of']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Director', 'of', 'the', 'Office', 'of', 'Museums']\n", + " Prediction: ['I-Person', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'Office', 'of', 'Museums', 'at']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Office', 'of', 'Museums', 'at', 'the']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['of', 'Museums', 'at', 'the', 'Iraqi', 'Ministry']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Museums', 'at', 'the', 'Iraqi', 'Ministry', 'of']\n", + " Prediction: ['I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Iraqi', 'Ministry', 'of', 'Culture', ',']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Iraqi', 'Ministry', 'of', 'Culture', ',', 'State']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Board', 'of', 'Antiquities', 'and', 'Heritage', '(']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'Antiquities', 'and', 'Heritage', '(', 'SBAH']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Antiquities', 'and', 'Heritage', '(', 'SBAH', ')']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['and', 'Heritage', '(', 'SBAH', ')', 'announced']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Heritage', '(', 'SBAH', ')', 'announced', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['of', 'Babylon', 'in', 'Iraq', '.']\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'O', 'B-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'regime', 'and', 'its', 'backers']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['The', 'regime', 'and', 'its', 'backers', 'must']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'recently', 'adopted', 'UN', 'Security', 'Council']\n", + " Prediction: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['recently', 'adopted', 'UN', 'Security', 'Council', 'Resolution']\n", + " Prediction: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['adopted', 'UN', 'Security', 'Council', 'Resolution', '2328']\n", + " Prediction: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['UN', 'Security', 'Council', 'Resolution', '2328', ',']\n", + " Prediction: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Security', 'Council', 'Resolution', '2328', ',', 'to']\n", + " Prediction: ['I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'to', 'allow', 'the', 'UN', 'unconditional']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'allow', 'the', 'UN', 'unconditional', ',']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['allow', 'the', 'UN', 'unconditional', ',', 'safe']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['State', 'has', 'contributed', 'more', 'than', '$']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'B-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['has', 'contributed', 'more', 'than', '$', '4']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['contributed', 'more', 'than', '$', '4', 'million']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Money', 'I-Money', 'I-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['more', 'than', '$', '4', 'million', 'to']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Money', 'I-Money', 'I-Money', 'I-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['than', '$', '4', 'million', 'to', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Money', 'I-Money', 'I-Money', 'I-Money', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['4', 'million', 'to', 'the', 'Future', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Money', 'I-Money', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['million', 'to', 'the', 'Future', 'of', 'Babylon']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Money', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['to', 'the', 'Future', 'of', 'Babylon', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Future', 'of', 'Babylon', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['HRW', 'said', 'the', 'woman', \"'s\"]\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['HRW', 'said', 'the', 'woman', \"'s\", 'section']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['said', 'the', 'woman', \"'s\", 'section', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['the', 'woman', \"'s\", 'section', 'of', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['woman', \"'s\", 'section', 'of', 'the', 'al-Khani']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: [\"'s\", 'section', 'of', 'the', 'al-Khani', 'mosque']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['section', 'of', 'the', 'al-Khani', 'mosque', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'al-Khani', 'mosque', 'in', 'Daquq']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'Kurdish-controlled', 'town', 'about', '30km', '(']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Kurdish-controlled', 'town', 'about', '30km', '(', '19']\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O', 'B-Quantity']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['town', 'about', '30km', '(', '19', 'miles']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'B-Quantity', 'I-Quantity']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['about', '30km', '(', '19', 'miles', ')']\n", + " Prediction: ['O', 'O', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['30km', '(', '19', 'miles', ')', 'south']\n", + " Prediction: ['O', 'O', 'B-Quantity', 'I-Quantity', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['(', '19', 'miles', ')', 'south', 'of']\n", + " Prediction: ['O', 'B-Quantity', 'I-Quantity', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['19', 'miles', ')', 'south', 'of', 'Kirkuk']\n", + " Prediction: ['B-Quantity', 'I-Quantity', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['miles', ')', 'south', 'of', 'Kirkuk', ',']\n", + " Prediction: ['I-Quantity', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: [')', 'south', 'of', 'Kirkuk', ',', 'had']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Security', 'Council', 'urge', 'all', 'political']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Security', 'Council', 'urge', 'all', 'political', 'parties']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['political', 'parties', 'and', 'their', 'supporters', 'to']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['parties', 'and', 'their', 'supporters', 'to', 'remain']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['process', 'governed', 'by', 'the', 'Constitution', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['governed', 'by', 'the', 'Constitution', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['May', '2013', ',', '700', 'domains', 'registered']\n", + " Prediction: ['B-Temporal', 'I-Temporal', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['2013', ',', '700', 'domains', 'registered', 'by']\n", + " Prediction: ['I-Temporal', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'O', 'B-Quantity', 'I-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['seized', 'by', 'the', 'U.S.', 'DNS', 'infrastructure']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['by', 'the', 'U.S.', 'DNS', 'infrastructure', 'operator']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'U.S.', 'DNS', 'infrastructure', 'operator', 'Network']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['U.S.', 'DNS', 'infrastructure', 'operator', 'Network', 'Solutions']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['DNS', 'infrastructure', 'operator', 'Network', 'Solutions', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['infrastructure', 'operator', 'Network', 'Solutions', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['two', 'strikes', 'disabled', 'two', 'bridges', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['strikes', 'disabled', 'two', 'bridges', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['which', 'include', 'three', 'tactical', 'units', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['include', 'three', 'tactical', 'units', ',', 'a']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['tactical', 'units', ',', 'a', 'command-and-control', 'node']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['units', ',', 'a', 'command-and-control', 'node', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'a', 'command-and-control', 'node', 'and', 'five']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['node', 'and', 'five', 'supply', 'routes', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'five', 'supply', 'routes', 'in', 'support']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['currently', 'led', 'by', 'Commodore', 'Andrew', 'Burns']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['led', 'by', 'Commodore', 'Andrew', 'Burns', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['by', 'Commodore', 'Andrew', 'Burns', ',', 'commander']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['Andrew', 'Burns', ',', 'commander', 'of', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Burns', ',', 'commander', 'of', 'the', 'Amphibious']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Person', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [',', 'commander', 'of', 'the', 'Amphibious', 'Task']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['commander', 'of', 'the', 'Amphibious', 'Task', 'Group']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'Amphibious', 'Task', 'Group', ',']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Amphibious', 'Task', 'Group', ',', 'embarked']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'embarked', 'on', 'the', 'U.K.', \"'s\"]\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['embarked', 'on', 'the', 'U.K.', \"'s\", 'large-deck']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['on', 'the', 'U.K.', \"'s\", 'large-deck', 'amphibious']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['the', 'U.K.', \"'s\", 'large-deck', 'amphibious', 'assault']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['U.K.', \"'s\", 'large-deck', 'amphibious', 'assault', 'ship']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: [\"'s\", 'large-deck', 'amphibious', 'assault', 'ship', 'HMS']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-MilitaryPlatfo']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'B-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['large-deck', 'amphibious', 'assault', 'ship', 'HMS', 'Ocean']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error found:\n", + " Text: ['the', 'first', 'time', 'the', 'U.S.', 'Secretary']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['first', 'time', 'the', 'U.S.', 'Secretary', 'of']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['behalf', 'of', ',', 'a', 'terrorist', 'organization']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', ',', 'a', 'terrorist', 'organization', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'a', 'terrorist', 'organization', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['scheduled', 'to', 'last', '45', 'minutes', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'last', '45', 'minutes', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['swear', 'the', 'library', 'holds', 'a', 'special']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'library', 'holds', 'a', 'special', 'place']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['library', 'holds', 'a', 'special', 'place', 'in']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['holds', 'a', 'special', 'place', 'in', 'all']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'about', '60', 'regional', 'and', 'international']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['about', '60', 'regional', 'and', 'international', 'nations']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['60', 'regional', 'and', 'international', 'nations', 'have']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['regional', 'and', 'international', 'nations', 'have', 'joined']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Men', 'and', 'Women', 'of', 'the', '407th']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['and', 'Women', 'of', 'the', '407th', 'AEG']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Women', 'of', 'the', '407th', 'AEG', ',']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['together', 'with', 'our', 'Joint', 'and', 'Coalition']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['with', 'our', 'Joint', 'and', 'Coalition', 'partners']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['our', 'Joint', 'and', 'Coalition', 'partners', ',']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['water', 'stored', 'from', 'the', 'Samarra', 'Barrage']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['stored', 'from', 'the', 'Samarra', 'Barrage', 'and']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: [';', 'destroyed', 'four', 'fighting', 'positions', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['destroyed', 'four', 'fighting', 'positions', ',', 'two']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['two', 'vehicles', ',', 'an', 'ISIL-held', 'building']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['vehicles', ',', 'an', 'ISIL-held', 'building', ',']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['building', ',', 'and', 'a', 'rocket-propelled', 'grenade']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: [',', 'and', 'a', 'rocket-propelled', 'grenade', 'launcher']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['and', 'a', 'rocket-propelled', 'grenade', 'launcher', ';']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'rocket-propelled', 'grenade', 'launcher', ';', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Weapon', 'I-Weapon', 'I-Weapon', 'I-Weapon', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['tunnels', ',', 'and', 'a', 'front-end', 'loader']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Vehicle', 'I-Vehicle', 'I-Vehicle']\n", + "\n", + "Error found:\n", + " Text: [',', 'and', 'a', 'front-end', 'loader', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Vehicle', 'I-Vehicle', 'I-Vehicle', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'a', 'front-end', 'loader', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Vehicle', 'I-Vehicle', 'I-Vehicle', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'hospital', 'contains', '645', 'beds']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['The', 'hospital', 'contains', '645', 'beds', ',']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['hospital', 'contains', '645', 'beds', ',', '36']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'B-Quantity', 'I-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['hunger', 'stalks', 'the', 'streets', ',', 'you']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['might', 'have', 'thought', 'people', 'would', 'have']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['little', 'interest', 'in', 'books', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['been', 'liberated', 'from', 'villages', 'around', 'Raqqa']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['liberated', 'from', 'villages', 'around', 'Raqqa', 'are']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['from', 'villages', 'around', 'Raqqa', 'are', 'seeking']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'French', 'anti-air', 'frigate', 'FS']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['The', 'French', 'anti-air', 'frigate', 'FS', 'Forbin']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['French', 'anti-air', 'frigate', 'FS', 'Forbin', 'joined']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['anti-air', 'frigate', 'FS', 'Forbin', 'joined', 'U.S']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['FS', 'Forbin', 'joined', 'U.S', '.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'B-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'U.S.', 'military', 'rejected', 'Maliki', \"'s\"]\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['U.S.', 'military', 'rejected', 'Maliki', \"'s\", 'first']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['first', 'choice', ',', 'Mohan', 'al-Freiji', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['choice', ',', 'Mohan', 'al-Freiji', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['have', 'what', 'I', \"'d\", 'call', 'a']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Person', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['what', 'I', \"'d\", 'call', 'a', 'mini']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'B-Person', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['I', \"'d\", 'call', 'a', 'mini', 'library']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Person', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [\"'d\", 'call', 'a', 'mini', 'library', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['call', 'a', 'mini', 'library', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['with', 'the', 'government', 'and', 'people', 'of']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'government', 'and', 'people', 'of', 'Iraq']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['The', 'head', 'of', 'the', 'country']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['The', 'head', 'of', 'the', 'country', \"'s\"]\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['head', 'of', 'the', 'country', \"'s\", 'counter-terrorism']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'country', \"'s\", 'counter-terrorism', 'operations']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['the', 'country', \"'s\", 'counter-terrorism', 'operations', 'said']\n", + " Prediction: ['B-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['country', \"'s\", 'counter-terrorism', 'operations', 'said', 'retaking']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Established', 'by', 'a', 'group', 'of']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Established', 'by', 'a', 'group', 'of', 'professional']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['by', 'a', 'group', 'of', 'professional', 'businessmen']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['a', 'group', 'of', 'professional', 'businessmen', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['group', 'of', 'professional', 'businessmen', 'in', '2004']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'it', 'has', '700', 'employees', 'across']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['it', 'has', '700', 'employees', 'across', 'Iraq']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'Lebanon', ',', 'the', 'United', 'Arab']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Lebanon', ',', 'the', 'United', 'Arab', 'Emirates']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: [',', 'the', 'United', 'Arab', 'Emirates', 'and']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'United', 'Arab', 'Emirates', 'and', 'Jordan']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['United', 'Arab', 'Emirates', 'and', 'Jordan', '.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Arab', 'Emirates', 'and', 'Jordan', '.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [\"''\", \"''\", 'says', 'the', 'farms', \"'\"]\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: [\"''\", 'says', 'the', 'farms', \"'\", 'owner']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'fortified', 'city', 'fell', 'to']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['coalition', 'forces', 'after', 'seven', 'days', 'of']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'after', 'seven', 'days', 'of', 'fighting']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Temporal', 'I-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['an', 'attack', 'on', 'a', 'mosque', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['attack', 'on', 'a', 'mosque', 'in', 'northern']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Location']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['northern', 'Iraq', 'on', 'Friday', 'that', 'killed']\n", + " Prediction: ['B-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Friday', 'that', 'killed', 'at', 'least', '13']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['that', 'killed', 'at', 'least', '13', 'women']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['killed', 'at', 'least', '13', 'women', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['at', 'least', '13', 'women', 'and', 'children']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['least', '13', 'women', 'and', 'children', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['13', 'women', 'and', 'children', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Royal', 'Air', 'Force', 'and', 'other', 'Coalition']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Air', 'Force', 'and', 'other', 'Coalition', 'forces']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['air', 'support', 'for', 'local', ',', 'legitimate']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['support', 'for', 'local', ',', 'legitimate', 'ground']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['for', 'local', ',', 'legitimate', 'ground', 'forces']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['local', ',', 'legitimate', 'ground', 'forces', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Text: [',', 'legitimate', 'ground', 'forces', ',', 'focused']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'construction', 'of', 'the', 'Mosul', 'Dam']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['construction', 'of', 'the', 'Mosul', 'Dam', 'upstream']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Mosul', 'Dam', 'upstream', 'and', 'several']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['several', 'other', 'large', 'dams', 'in', 'Turkey']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['other', 'large', 'dams', 'in', 'Turkey', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['large', 'dams', 'in', 'Turkey', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'following', 'day', ',', 'Iraqi']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'reportedly', 'discovered', 'the', 'bodies', 'of']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['reportedly', 'discovered', 'the', 'bodies', 'of', '70']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['discovered', 'the', 'bodies', 'of', '70', 'civilians']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'bodies', 'of', '70', 'civilians', 'inside']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Naser', ',', 'about', '35km', 'south', 'of']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: [',', 'about', '35km', 'south', 'of', 'Mosul']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['about', '35km', 'south', 'of', 'Mosul', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['35km', 'south', 'of', 'Mosul', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Forces', 'announced', 'that', 'they', 'began', 'their']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['advance', 'to', 'isolate', 'Raqqah', ',', 'Syria']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['isolate', 'Raqqah', ',', 'Syria', ',', 'Nov.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Temporal']\n", + "Gold labels: ['O', 'B-Location', 'O', 'B-Location', 'O', 'B-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['airstrikes', ',', 'delivering', '101', 'munitions', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'delivering', '101', 'munitions', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Minister', 'Abadi', 'and', 'his', 'government', 'in']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Abadi', 'and', 'his', 'government', 'in', 'recent']\n", + " Prediction: ['I-Person', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['this', ',', 'including', 'today', '’', 's']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: [',', 'including', 'today', '’', 's', 'announcement']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['including', 'today', '’', 's', 'announcement', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['today', '’', 's', 'announcement', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['with', 'Security', 'Council', 'Resolution', '2328', 'to']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Security', 'Council', 'Resolution', '2328', 'to', 'ensure']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'B-DocumentRefere', 'I-DocumentRefere', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['In', '2004', 'an', 'Arabic-language', 'manual', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['2004', 'an', 'Arabic-language', 'manual', ',', 'which']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Buried', 'beneath', 'a', 'bomb-damaged', 'building']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Buried', 'beneath', 'a', 'bomb-damaged', 'building', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['beneath', 'a', 'bomb-damaged', 'building', ',', 'it']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'the', 'besieged', 'Damascus', 'suburb', 'of']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['the', 'besieged', 'Damascus', 'suburb', 'of', 'Darayya']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['is', 'affiliated', 'with', 'Damascus', 'University', '.']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['affiliated', 'with', 'Damascus', 'University', '.']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['fully', 'and', 'by', 'all', 'the', 'parties']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['and', 'by', 'all', 'the', 'parties', 'to']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['committed', 'to', 'defeating', 'these', 'terrorists', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'defeating', 'these', 'terrorists', 'and', 'have']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'of', 'over', '25', '%', 'of']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['of', 'over', '25', '%', 'of', 'Iraqi']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Location']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['over', '25', '%', 'of', 'Iraqi', 'territory']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['25', '%', 'of', 'Iraqi', 'territory', ',']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['A', 'scorching', 'Baghdad', 'heat', 'clung']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['sun', 'had', 'set', 'a', 'few', 'hours']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['had', 'set', 'a', 'few', 'hours', 'before']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['set', 'a', 'few', 'hours', 'before', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'few', 'hours', 'before', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['operation', 'to', 'eliminate', 'the', 'ISIL', 'terrorist']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'eliminate', 'the', 'ISIL', 'terrorist', 'group']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['eliminate', 'the', 'ISIL', 'terrorist', 'group', 'and']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'ISIL', 'terrorist', 'group', 'and', 'the']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Questions', 'from', 'the', 'Pentagon', 'Press']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Questions', 'from', 'the', 'Pentagon', 'Press', 'to']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Pentagon', 'Press', 'to', 'the', 'investigating', 'officer']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Organisation', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Press', 'to', 'the', 'investigating', 'officer', 'will']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['to', 'the', 'investigating', 'officer', 'will', 'follow']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['the', 'investigating', 'officer', 'will', 'follow', 'Brig']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['investigating', 'officer', 'will', 'follow', 'Brig', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['officer', 'will', 'follow', 'Brig', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['will', 'follow', 'Brig', '.']\n", + " Prediction: ['O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [',', '2006', ',', 'the', 'Swedish', 'citizen']\n", + " Prediction: ['I-Temporal', 'I-Temporal', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['2006', ',', 'the', 'Swedish', 'citizen', 'Mohamed']\n", + " Prediction: ['I-Temporal', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Temporal', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [',', 'the', 'Swedish', 'citizen', 'Mohamed', 'Moumou']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Swedish', 'citizen', 'Mohamed', 'Moumou', ',']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Swedish', 'citizen', 'Mohamed', 'Moumou', ',', 'who']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['uncontested', 'leader', 'of', 'an', 'extremist', 'group']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['leader', 'of', 'an', 'extremist', 'group', 'centered']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'an', 'extremist', 'group', 'centered', 'around']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['an', 'extremist', 'group', 'centered', 'around', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Location']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['extremist', 'group', 'centered', 'around', 'the', 'Brandbergen']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Location', 'I-Location']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['group', 'centered', 'around', 'the', 'Brandbergen', 'Mosque']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['centered', 'around', 'the', 'Brandbergen', 'Mosque', 'in']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['around', 'the', 'Brandbergen', 'Mosque', 'in', 'Stockholm']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'Brandbergen', 'Mosque', 'in', 'Stockholm', \"''\"]\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Brandbergen', 'Mosque', 'in', 'Stockholm', \"''\", \"''\"]\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['was', 'put', 'on', 'the', 'United', 'Nations']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['put', 'on', 'the', 'United', 'Nations', 'Security']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['on', 'the', 'United', 'Nations', 'Security', 'Council']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['the', 'United', 'Nations', 'Security', 'Council', 'Committee']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['United', 'Nations', 'Security', 'Council', 'Committee', '1267']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['Nations', 'Security', 'Council', 'Committee', '1267', 'list']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['Security', 'Council', 'Committee', '1267', 'list', 'of']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['Council', 'Committee', '1267', 'list', 'of', 'foreign']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['Committee', '1267', 'list', 'of', 'foreign', 'terrorists']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['1267', 'list', 'of', 'foreign', 'terrorists', '.']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['list', 'of', 'foreign', 'terrorists', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'I-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'lives', 'of', 'people', 'from', 'all']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['lives', 'of', 'people', 'from', 'all', 'communities']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'people', 'from', 'all', 'communities', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['people', 'from', 'all', 'communities', 'in', 'both']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['from', 'all', 'communities', 'in', 'both', 'Iraq']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['all', 'communities', 'in', 'both', 'Iraq', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['communities', 'in', 'both', 'Iraq', 'and', 'Syria']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['in', 'both', 'Iraq', 'and', 'Syria', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['both', 'Iraq', 'and', 'Syria', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Ministers', 'discussed', 'the', 'Syria', 'crisis', 'and']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'O', 'O', 'B-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'conflict', 'in', 'Ukraine', 'Boris', 'Johnson']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['conflict', 'in', 'Ukraine', 'Boris', 'Johnson', 'A']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Location', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['The', 'Foreign', 'Secretary', 'met', 'Russian', 'Foreign']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Foreign', 'Secretary', 'met', 'Russian', 'Foreign', 'Minister']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Minister', 'Sergei', 'Lavrov', 'earlier', 'today', 'on']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'I-Person', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'margins', 'of', 'the', 'UNGA', 'meeting']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['margins', 'of', 'the', 'UNGA', 'meeting', 'in']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['UNGA', 'meeting', 'in', 'New', 'York', ',']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['meeting', 'in', 'New', 'York', ',', 'their']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['face-to-face', 'meeting', 'since', 'Boris', 'Johnson', 'was']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['meeting', 'since', 'Boris', 'Johnson', 'was', 'appointed']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['We', 'hid', 'from', 'the', 'police', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['hid', 'from', 'the', 'police', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['of', 'the', 'city', \"'s\", 'eastern', 'half']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['the', 'city', \"'s\", 'eastern', 'half', '.']\n", + " Prediction: ['B-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['city', \"'s\", 'eastern', 'half', '.']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['aircraft', 'employed', 'in', 'a', 'strike', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['impact', 'points', 'against', 'a', 'target', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['campaign', 'to', 'liberate', 'the', 'Iraqi', 'city']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Location', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['to', 'liberate', 'the', 'Iraqi', 'city', 'of']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'B-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['liberate', 'the', 'Iraqi', 'city', 'of', 'Mosul']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'B-Location', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['and', 'in', 'clearing', 'areas', 'already', 'liberated']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['in', 'clearing', 'areas', 'already', 'liberated', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['clearing', 'areas', 'already', 'liberated', ',', 'he']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error found:\n", + " Text: ['injuries', 'and', 'I', 'ask', 'myself', ',']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['s', 'tenure', 'with', 'the', 'command', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['tenure', 'with', 'the', 'command', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'United', 'States', 'and', 'Iraq', 'signed']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['United', 'States', 'and', 'Iraq', 'signed', 'a']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'will', 'enable', 'Iraq', 'to', 'access']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['access', 'up', 'to', '$', '1', 'billion']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'I-Money']\n", + "\n", + "Error found:\n", + " Text: ['up', 'to', '$', '1', 'billion', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', '$', '1', 'billion', 'in', 'low-cost']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Money', 'I-Money', 'I-Money', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['(', 'Arabic', ':', 'الحجاب', 'الفولاذي', 'Al']\n", + " Prediction: ['O', 'B-Nationality', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'B-Nationality', 'O', 'O', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['Arabic', ':', 'الحجاب', 'الفولاذي', 'Al', 'Hejab']\n", + " Prediction: ['B-Nationality', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Nationality', 'O', 'O', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [':', 'الحجاب', 'الفولاذي', 'Al', 'Hejab', 'Elfulathi']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['coalition', 'forces', 'in', 'early', 'November', '2005']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'O', 'O', 'B-Temporal', 'I-Temporal']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'in', 'early', 'November', '2005', 'to']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'B-Temporal', 'I-Temporal', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'flow', 'of', 'foreign', 'insurgents', 'crossing']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['flow', 'of', 'foreign', 'insurgents', 'crossing', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['border', 'and', 'joining', 'the', 'Iraqi', 'insurgency']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Nationality', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'joining', 'the', 'Iraqi', 'insurgency', '.']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Nationality', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['joining', 'the', 'Iraqi', 'insurgency', '.']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'B-Nationality', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'Iraq', 'by', '£20', 'million', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Iraq', 'by', '£20', 'million', ',', 'bringing']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['total', 'contribution', 'to', '£59.5', 'million', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Money', 'I-Money', 'O']\n", + "\n", + "Error found:\n", + " Text: ['contribution', 'to', '£59.5', 'million', ',', 'supporting']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Money', 'I-Money', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['tanks', ',', 'three', 'artillery', 'systems', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'three', 'artillery', 'systems', ',', 'two']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['buildings', ',', 'two', 'tactical', 'vehicles', ',']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'two', 'tactical', 'vehicles', ',', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'and', 'an', 'air', 'defense', 'artillery']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['and', 'an', 'air', 'defense', 'artillery', 'system']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo']\n", + "\n", + "Error found:\n", + " Text: ['an', 'air', 'defense', 'artillery', 'system', 'were']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['air', 'defense', 'artillery', 'system', 'were', 'also']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['U.S.', 'airstrikes', 'against', 'al-Qaida', 'operatives', 'this']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'B-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['airstrikes', 'against', 'al-Qaida', 'operatives', 'this', 'week']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'B-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['against', 'al-Qaida', 'operatives', 'this', 'week', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'B-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['al-Qaida', 'operatives', 'this', 'week', 'in', 'northwestern']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Location']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'B-Temporal', 'I-Temporal', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['in', 'northwestern', 'Syria', '’', 's', 'Idlib']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'O', 'O', 'B-Location']\n", + "\n", + "Error found:\n", + " Text: ['northwestern', 'Syria', '’', 's', 'Idlib', 'province']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'O', 'B-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['Syria', '’', 's', 'Idlib', 'province', 'killed']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['I-Location', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Idlib', 'province', 'killed', 'more', 'than', '20']\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-Quantity', 'I-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['province', 'killed', 'more', 'than', '20', 'militants']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['killed', 'more', 'than', '20', 'militants', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'B-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['more', 'than', '20', 'militants', 'and', 'destroyed']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Quantity', 'I-Quantity', 'I-Quantity', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['militants', 'and', 'destroyed', 'eight', 'vehicles', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'O', 'O', 'B-Quantity', 'B-Vehicle', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'destroyed', 'eight', 'vehicles', 'and', 'nine']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Quantity', 'B-Vehicle', 'O', 'B-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['eight', 'vehicles', 'and', 'nine', 'structures', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Quantity', 'B-Vehicle', 'O', 'B-Quantity', 'B-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['vehicles', 'and', 'nine', 'structures', ',', 'Pentagon']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Person']\n", + "Gold labels: ['B-Vehicle', 'O', 'B-Quantity', 'B-Location', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['Peter', 'Cook', 'told', 'reporters', 'today', '.']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Organisation', 'B-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Cook', 'told', 'reporters', 'today', '.']\n", + " Prediction: ['I-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'O', 'B-Organisation', 'B-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['who', 'was', 'supporting', 'Operation', 'Inherent', 'Resolve']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['was', 'supporting', 'Operation', 'Inherent', 'Resolve', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['supporting', 'Operation', 'Inherent', 'Resolve', ',', 'was']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['is', 'based', 'in', 'Virginia', 'Beach', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['based', 'in', 'Virginia', 'Beach', ',', 'Virginia']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Virginia', 'Beach', ',', 'Virginia', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Virginia', 'Beach', ',', 'Virginia', ',', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'I', 'meet', 'a', 'middle-aged', 'carpenter']\n", + " Prediction: ['O', 'B-Person', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Person', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['I', 'meet', 'a', 'middle-aged', 'carpenter', 'who']\n", + " Prediction: ['B-Person', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Person', 'O', 'B-Person', 'I-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['meet', 'a', 'middle-aged', 'carpenter', 'who', 'asks']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'Northern', 'Iraq', 'including', 'Rabiyah', 'and']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Northern', 'Iraq', 'including', 'Rabiyah', 'and', 'Zumar']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['as', 'defined', 'in', 'the', 'CJTF', 'releases']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['defined', 'in', 'the', 'CJTF', 'releases', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['location', 'to', 'produce', 'a', 'single', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['UN', 'Security', 'Council', 'adopted', 'Resolution', '2328']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'B-DocumentRefere', 'I-DocumentRefere']\n", + "\n", + "Error found:\n", + " Text: ['Security', 'Council', 'adopted', 'Resolution', '2328', '.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Council', 'adopted', 'Resolution', '2328', '.']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['permission', 'to', 'build', 'a', 'Muslim', 'Cultural']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['to', 'build', 'a', 'Muslim', 'Cultural', 'Center']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['build', 'a', 'Muslim', 'Cultural', 'Center', 'next']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['a', 'Muslim', 'Cultural', 'Center', 'next', 'to']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'the', 'city', 'center', '.']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['On', 'Thanksgiving', 'Day', ',', 'Defense']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Person']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['his', 'condolences', 'to', 'Dayton', '’', 's']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['condolences', 'to', 'Dayton', '’', 's', 'family']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['to', 'Dayton', '’', 's', 'family', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Dayton', '’', 's', 'family', ',', 'friends']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['’', 's', 'family', ',', 'friends', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['s', 'family', ',', 'friends', 'and', 'other']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['family', ',', 'friends', 'and', 'other', 'loved']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: [',', 'friends', 'and', 'other', 'loved', 'ones']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['friends', 'and', 'other', 'loved', 'ones', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'other', 'loved', 'ones', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', 'agreement', 'reached', 'yesterday', 'between', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['Accompanying', 'the', 'aircraft', 'were', '300']\n", + " Prediction: ['O', 'O', 'O', 'O', 'B-Organisation']\n", + "Gold labels: ['O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: [',', 'deployed', 'from', 'the', '158th', 'Fighter']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['deployed', 'from', 'the', '158th', 'Fighter', 'Wing']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['from', 'the', '158th', 'Fighter', 'Wing', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', '158th', 'Fighter', 'Wing', ',', 'who']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['who', 'will', 'comprise', 'the', '134th', 'Expeditionary']\n", + " Prediction: ['O', 'O', 'O', 'B-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['will', 'comprise', 'the', '134th', 'Expeditionary', 'Fighter']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['comprise', 'the', '134th', 'Expeditionary', 'Fighter', 'Squadron']\n", + " Prediction: ['O', 'B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', '134th', 'Expeditionary', 'Fighter', 'Squadron', ',']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['134th', 'Expeditionary', 'Fighter', 'Squadron', ',', 'which']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['and', 'ultimately', 'eliminate', 'the', 'terrorists', 'associated']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['ultimately', 'eliminate', 'the', 'terrorists', 'associated', 'with']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['eliminate', 'the', 'terrorists', 'associated', 'with', 'ISIL']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'terrorists', 'associated', 'with', 'ISIL', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['terrorists', 'associated', 'with', 'ISIL', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['are', 'they', 'bombing', 'this', 'place', '?']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['they', 'bombing', 'this', 'place', '?', \"'\"]\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['far', 'some', '20,000', 'work', 'permits', 'have']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O']\n", + "\n", + "Error found:\n", + " Text: ['some', '20,000', 'work', 'permits', 'have', 'been']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['was', 'killed', 'told', 'the', 'campaign', 'group']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['killed', 'told', 'the', 'campaign', 'group', 'that']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['told', 'the', 'campaign', 'group', 'that', 'he']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['plane', 'flying', 'over', 'his', 'home', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['flying', 'over', 'his', 'home', ',', '300m']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['984ft', ')', 'from', 'the', 'mosque', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: [')', 'from', 'the', 'mosque', ',', 'shortly']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Near', 'Tal', 'Afar', ',', 'one']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['one', 'strike', 'destroyed', 'a', 'VBIED', 'factory']\n", + " Prediction: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error found:\n", + " Text: ['strike', 'destroyed', 'a', 'VBIED', 'factory', '.']\n", + " Prediction: ['O', 'O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['destroyed', 'a', 'VBIED', 'factory', '.']\n", + " Prediction: ['O', 'B-Weapon', 'I-Weapon', 'I-Weapon', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['transition', 'away', 'from', 'the', 'murderous', 'regime']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['away', 'from', 'the', 'murderous', 'regime', 'of']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['from', 'the', 'murderous', 'regime', 'of', 'Assad']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['But', ',', 'I', 'ask', 'him', ',']\n", + " Prediction: ['O', 'O', 'B-Person', 'I-Person', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['him', ',', 'in', 'a', 'besieged', 'town']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: [',', 'in', 'a', 'besieged', 'town', 'that']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'a', 'besieged', 'town', 'that', 'has']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['aid', 'convoys', 'in', 'nearly', 'four', 'years']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['convoys', 'in', 'nearly', 'four', 'years', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['in', 'nearly', 'four', 'years', ',', 'would']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Secretary', 'statement', 'on', 'UN', 'chemical', 'weapons']\n", + " Prediction: ['I-Person', 'O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "Gold labels: ['I-Person', 'O', 'O', 'B-Organisation', 'B-Weapon', 'I-Weapon']\n", + "\n", + "Error found:\n", + " Text: ['statement', 'on', 'UN', 'chemical', 'weapons', 'report']\n", + " Prediction: ['O', 'O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['on', 'UN', 'chemical', 'weapons', 'report', 'The']\n", + " Prediction: ['O', 'B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'B-Person']\n", + "Gold labels: ['O', 'B-Organisation', 'B-Weapon', 'I-Weapon', 'O', 'B-Person']\n", + "\n", + "Error found:\n", + " Text: ['UN', 'chemical', 'weapons', 'report', 'The', 'Foreign']\n", + " Prediction: ['B-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'I-DocumentRefere', 'B-Person', 'I-Person']\n", + "Gold labels: ['B-Organisation', 'B-Weapon', 'I-Weapon', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Foreign', 'Secretary', ',', 'Boris', 'Johnson', ',']\n", + " Prediction: ['I-Person', 'I-Person', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Secretary', ',', 'Boris', 'Johnson', ',', 'said']\n", + " Prediction: ['I-Person', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "Gold labels: ['I-Person', 'O', 'B-Person', 'I-Person', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Syrian', 'people', 'on', 'at', 'least', 'three']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Quantity', 'I-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['people', 'on', 'at', 'least', 'three', 'occasions']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['on', 'at', 'least', 'three', 'occasions', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['at', 'least', 'three', 'occasions', ',', 'and']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Quantity', 'I-Quantity', 'I-Quantity', 'I-Quantity', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', 'and', 'that', 'Daesh', 'has', 'used']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Daesh', 'has', 'used', 'such', 'weapons', 'at']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'O', 'O', 'B-Weapon', 'I-Weapon', 'B-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['has', 'used', 'such', 'weapons', 'at', 'least']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'B-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['used', 'such', 'weapons', 'at', 'least', 'once']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Weapon', 'I-Weapon', 'B-Quantity', 'I-Quantity', 'I-Quantity']\n", + "\n", + "Error found:\n", + " Text: ['such', 'weapons', 'at', 'least', 'once', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Weapon', 'I-Weapon', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['weapons', 'at', 'least', 'once', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Weapon', 'B-Quantity', 'I-Quantity', 'I-Quantity', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'siege', 'of', 'Darayya', 'by', 'government']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['siege', 'of', 'Darayya', 'by', 'government', 'and']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['of', 'Darayya', 'by', 'government', 'and', 'pro-Assad']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['pro-Assad', 'forces', 'began', 'nearly', 'four', 'years']\n", + " Prediction: ['I-Organisation', 'I-Organisation', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'I-Organisation', 'O', 'B-Temporal', 'B-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['forces', 'began', 'nearly', 'four', 'years', 'ago']\n", + " Prediction: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Organisation', 'O', 'B-Temporal', 'B-Temporal', 'I-Temporal', 'I-Temporal']\n", + "\n", + "Error found:\n", + " Text: ['began', 'nearly', 'four', 'years', 'ago', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Temporal', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['nearly', 'four', 'years', 'ago', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Temporal', 'B-Temporal', 'I-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['59.173616°N', '18.167988°E', '/', '59.173616', ';', '18.167988']\n", + " Prediction: ['B-Location', 'I-Location', 'O', 'O', 'O', 'B-Quantity']\n", + "Gold labels: ['B-Location', 'I-Location', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['18.167988°E', '/', '59.173616', ';', '18.167988']\n", + " Prediction: ['I-Location', 'O', 'O', 'O', 'B-Quantity']\n", + "Gold labels: ['I-Location', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['/', '59.173616', ';', '18.167988']\n", + " Prediction: ['O', 'O', 'O', 'B-Quantity']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['1', 'hit', 'two', 'al-Qaida', 'vehicles', 'that']\n", + " Prediction: ['I-Temporal', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O']\n", + "\n", + "Error found:\n", + " Text: ['hit', 'two', 'al-Qaida', 'vehicles', 'that', 'had']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-MilitaryPlatfo', 'I-MilitaryPlatfo', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['large', 'al-Qaida', 'headquarters', 'near', 'Sarmada', ',']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'O', 'B-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['al-Qaida', 'headquarters', 'near', 'Sarmada', ',', 'Syria']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['headquarters', 'near', 'Sarmada', ',', 'Syria', ',']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['near', 'Sarmada', ',', 'Syria', ',', 'Cook']\n", + " Prediction: ['I-Location', 'I-Location', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['to', 'go', 'through', 'bombed-out', 'buildings', 'to']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['go', 'through', 'bombed-out', 'buildings', 'to', 'hide']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['targeted', 'by', 'coalition', 'air', 'strikes', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Weapon', 'I-Weapon', 'O']\n", + "\n", + "Error found:\n", + " Text: ['by', 'coalition', 'air', 'strikes', 'in', 'October']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Weapon', 'I-Weapon', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['resupply', 'or', 'reinforce', 'their', 'positions', 'in']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['or', 'reinforce', 'their', 'positions', 'in', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['reinforce', 'their', 'positions', 'in', 'the', 'east']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['their', 'positions', 'in', 'the', 'east', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['positions', 'in', 'the', 'east', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['The', 'Prime', 'Minister', 'and', 'President', 'Hollande']\n", + " Prediction: ['B-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person']\n", + "Gold labels: ['B-Person', 'I-Person', 'I-Person', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: ['Prime', 'Minister', 'and', 'President', 'Hollande', 'met']\n", + " Prediction: ['I-Person', 'I-Person', 'I-Person', 'I-Person', 'I-Person', 'O']\n", + "Gold labels: ['I-Person', 'I-Person', 'O', 'B-Person', 'I-Person', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Hollande', 'met', 'on', '3', 'March', 'for']\n", + " Prediction: ['I-Person', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['I-Person', 'O', 'O', 'B-Temporal', 'I-Temporal', 'O']\n", + "\n", + "Error found:\n", + " Text: ['met', 'on', '3', 'March', 'for', 'the']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'B-Temporal']\n", + "Gold labels: ['O', 'O', 'B-Temporal', 'I-Temporal', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['3', 'March', 'for', 'the', '2016', 'UK-France']\n", + " Prediction: ['O', 'O', 'O', 'B-Temporal', 'I-Temporal', 'O']\n", + "Gold labels: ['B-Temporal', 'I-Temporal', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['March', 'for', 'the', '2016', 'UK-France', 'Summit']\n", + " Prediction: ['O', 'O', 'B-Temporal', 'I-Temporal', 'O', 'O']\n", + "Gold labels: ['I-Temporal', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['for', 'the', '2016', 'UK-France', 'Summit', 'at']\n", + " Prediction: ['O', 'B-Temporal', 'I-Temporal', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['the', '2016', 'UK-France', 'Summit', 'at', 'Amiens']\n", + " Prediction: ['B-Temporal', 'I-Temporal', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['``', 'Many', 'of', 'his', 'comrades']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['``', 'Many', 'of', 'his', 'comrades', 'also']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: ['Many', 'of', 'his', 'comrades', 'also', 'have']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['comrades', 'also', 'have', 'their', 'own', 'mini']\n", + " Prediction: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['I-Organisation', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['also', 'have', 'their', 'own', 'mini', 'front']\n", + " Prediction: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['have', 'their', 'own', 'mini', 'front', 'line']\n", + " Prediction: ['O', 'B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['their', 'own', 'mini', 'front', 'line', 'libraries']\n", + " Prediction: ['B-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['own', 'mini', 'front', 'line', 'libraries', ',']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'I-Location', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['mini', 'front', 'line', 'libraries', ',', 'he']\n", + " Prediction: ['I-Location', 'I-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['at', 'just', 'about', 'every', 'defence', 'point']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Location', 'I-Location', 'I-Location']\n", + "\n", + "Error found:\n", + " Text: ['just', 'about', 'every', 'defence', 'point', ',']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Location', 'I-Location', 'I-Location', 'O']\n", + "\n", + "Error found:\n", + " Text: ['about', 'every', 'defence', 'point', ',', 'which']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'B-Location', 'I-Location', 'I-Location', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['working', 'together', 'with', 'the', 'U.S.', 'and']\n", + " Prediction: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'O']\n", + "\n", + "Error found:\n", + " Text: ['together', 'with', 'the', 'U.S.', 'and', 'Royal']\n", + " Prediction: ['O', 'O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'O', 'B-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['with', 'the', 'U.S.', 'and', 'Royal', 'Navy']\n", + " Prediction: ['O', 'B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation']\n", + "Gold labels: ['O', 'O', 'O', 'O', 'B-Organisation', 'I-Organisation']\n", + "\n", + "Error found:\n", + " Text: ['the', 'U.S.', 'and', 'Royal', 'Navy', ',']\n", + " Prediction: ['B-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'I-Organisation', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Organisation', 'I-Organisation', 'O']\n", + "\n", + "Error found:\n", + " Text: [',', \"''\", 'said', 'Capt', '.']\n", + " Prediction: ['O', 'O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'O', 'B-Person', 'I-Person']\n", + "\n", + "Error found:\n", + " Text: [\"''\", 'said', 'Capt', '.']\n", + " Prediction: ['O', 'O', 'O', 'O']\n", + "Gold labels: ['O', 'O', 'B-Person', 'I-Person']\n", + "\n" + ] + } + ], + "source": [ + "# some code to print out errors made by one of the sequence taggers. \n", + "window_size = 3\n", + "for i, sent in enumerate(test_set):\n", + " token_shown = -1\n", + " \n", + " for j, (tok, label) in enumerate(sent):\n", + " predicted_label = test_sents_with_predicted_tags[i][j][1]\n", + "\n", + " if j < token_shown + window_size:\n", + " continue\n", + " \n", + " token_shown = -1\n", + " \n", + " if label != predicted_label:\n", + " start = j - window_size\n", + " if start < 0:\n", + " start = 0\n", + " \n", + " end = j + window_size\n", + " if end > len(sent):\n", + " end = len(sent)\n", + " \n", + " print('Error found:')\n", + " \n", + " text = [tok for tok, lab in test_sents_with_predicted_tags[i][start:end]]\n", + " preds = [lab for tok, lab in test_sents_with_predicted_tags[i][start:end]]\n", + " gold = [lab for tok, lab in sent[start:end]]\n", + " print(f' Text: {text}')\n", + " print(f' Prediction: {preds}')\n", + " print(f'Gold labels: {gold}')\n", + " print()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can try to help the CRF tagger by adding some more features. Part-of-speech tags often provide useful information for identifying entites. The code below includes a modified CRFTagger class that adds a PoS feature to the feature vector for each word. " + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "# *** Improve the CRF NER tagger using parts of speech (see lab 5) as additional features.\n", + "class CRFTaggerWithPOS(nltk.tag.CRFTagger):\n", + " _current_tokens = None\n", + " \n", + " def _get_features(self, tokens, index):\n", + " \"\"\"\n", + " Extract the features for a token and append the POS tag as an additional feature.\n", + " \"\"\"\n", + " basic_features = super()._get_features(tokens, index)\n", + " \n", + " if tokens != self._current_tokens:\n", + " self._pos_tagged_tokens = nltk.pos_tag(tokens)\n", + " self._current_tokens = tokens\n", + " \n", + " basic_features.append(self._pos_tagged_tokens[index][1])\n", + " \n", + " return basic_features" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**TODO 1.5: Complete the training function below to use the new tagger class with PoS features. Then use the training function to train a tagger, predict the tags for the test set and compute the span-level F1 scores.**" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "<__main__.CRFTaggerWithPOS object at 0x7fed1c648e10>\n", + "[('The', 'B-Organisation'), ('SDF', 'I-Organisation'), (',', 'O'), ('made', 'O'), ('up', 'O'), ('in', 'O'), ('part', 'O'), ('by', 'O'), ('local', 'O'), ('Arabs', 'O'), ('and', 'O'), ('its', 'O'), ('Coalition', 'O'), ('trained', 'O'), ('and', 'O'), ('equipped', 'O'), ('Arab', 'O'), ('component', 'O'), (',', 'O'), ('the', 'B-Organisation'), ('Syrian', 'I-Organisation'), ('Arab', 'I-Organisation'), ('Coalition', 'I-Organisation'), (',', 'O'), ('and', 'O'), ('supported', 'O'), ('by', 'O'), ('Coalition', 'B-Organisation'), ('advisers', 'I-Organisation'), ('and', 'O'), ('air', 'O'), ('strikes', 'O'), ('began', 'O'), ('the', 'O'), ('operation', 'O'), ('to', 'O'), ('isolate', 'O'), ('Raqqah', 'O'), ('on', 'O'), ('Nov.', 'B-Temporal'), ('5', 'I-Temporal'), ('.', 'O')]\n", + "F1 score for class Person = 0.746031746031746\n", + "F1 score for class DocumentRefere = 0\n", + "F1 score for class Money = 0.22222222222222224\n", + "F1 score for class CommsIdentifie = 0\n", + "F1 score for class MilitaryPlatfo = 0.16666666666666669\n", + "F1 score for class Nationality = 0.5\n", + "F1 score for class Quantity = 0\n", + "F1 score for class Temporal = 0.6274509803921569\n", + "F1 score for class Weapon = 0.4\n", + "F1 score for class Frequency = 0\n", + "F1 score for class Organisation = 0.7112676056338029\n", + "F1 score for class Location = 0.44776119402985076\n", + "F1 score for class Vehicle = 0\n", + "Macro-average f1 score = 0.2939538780751112\n" + ] + } + ], + "source": [ + "# Train\n", + "def train_CRF_NER_tagger_POS(train_set):\n", + " ### WRITE YOUR OWN CODE HERE\n", + " tagger = CRFTaggerWithPOS()\n", + " tagger.train(train_set, 'model.crf.tagger')\n", + " return tagger # return the trained model\n", + "\n", + "### WRITE YOUR OWN CODE HERE\n", + "tagger_with_POS = train_CRF_NER_tagger_POS(train_set)\n", + "print(tagger_with_POS)\n", + "\n", + "\n", + "# Test\n", + "predicted_tags_with_POS = tag_test_set(test_set, tagger_with_POS)\n", + "print(predicted_tags_with_POS[0])\n", + "\n", + "\n", + "cal_span_level_f1(test_set, predicted_tags_with_POS, ne_tags)\n", + "###" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**OPTIONAL TODO 1.6: use the code from above to print out some labelling errors for the CRF tagger with PoS tags. Have the types of errors that are present changed? How do the different features affect the NER results?**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, we can try adding in some more features to improve the tagger further. The code below starts the CoreNLP server,\n", + "which we will use to provide dependency parse features.\n", + "\n", + "Hint: if the server does not restart after it was previously running, the old version may still be running. You can kill off any Java processes to allow the server to start again." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "#Corenlp will need you have java added to path\n", + "# java_path = \"C:/Program Files/Java/jre1.8.0_281/bin/java.exe\"# You may need to replace it with the path on your PC\n", + "# os.environ['JAVAHOME'] = java_path\n", + "\n", + "#Add GhostScripts to path to illustrate the trees\n", + "# gs_path = \"C:/Program Files/gs/gs9.53.3/bin\"# You may need to replace it with the path on your PC\n", + "# os.environ['PATH'] = gs_path" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "ename": "CoreNLPServerError", + "evalue": "Could not connect to the server.", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mCoreNLPServerError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0;31m# Start the server in the background\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0mserver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(self, stdout, stderr)\u001b[0m\n\u001b[1;32m 151\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 152\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 153\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mCoreNLPServerError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Could not connect to the server.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 154\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 155\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mCoreNLPServerError\u001b[0m: Could not connect to the server." + ] + } + ], + "source": [ + "from nltk.parse.corenlp import CoreNLPServer, CoreNLPParser\n", + "from nltk.parse.corenlp import CoreNLPDependencyParser\n", + "import os\n", + "\n", + "# Stanford Core NLP runs as a server on the local machine. \n", + "# The NLTK wrapper will make calls to this server to parse our text.\n", + "STANFORD = \"./stanford-corenlp-4.2.0\"\n", + "# You may need to replace it with the path on your PC or use realtive path\n", + "server = CoreNLPServer(\n", + " os.path.join(STANFORD, \"stanford-corenlp-4.2.0.jar\"),\n", + " os.path.join(STANFORD, \"stanford-corenlp-4.2.0-models.jar\"), \n", + ")\n", + "# server.stop() # in case it's already running\n", + "\n", + "# Start the server in the background\n", + "server.start()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Below we have a new CRF class that includes some dependency parse tree features as well as the PoS tags. \n", + "For each token, we add a feature for the dependency relation type that connects the token to its head. We also add\n", + "the tag of the head of that word (i.e., the parent node in the dependency parse tree).\n", + "\n", + "**TODO 1.6: Complete the training function and use the new class to train and test a tagger with dependency features.**" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "ename": "KeyboardInterrupt", + "evalue": "", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 41\u001b[0m \u001b[0;31m### WRITE YOUR OWN CODE HERE\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 42\u001b[0;31m \u001b[0mtagger_with_deps\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_CRF_NER_tagger_deps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_set\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 43\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagger_with_POS\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36mtrain_CRF_NER_tagger_deps\u001b[0;34m(train_set)\u001b[0m\n\u001b[1;32m 36\u001b[0m \u001b[0;31m### WRITE YOUR OWN CODE HERE\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 37\u001b[0m \u001b[0mtagger\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCRFTaggerWithDeps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m \u001b[0mtagger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'model.crf.tagger'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 39\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtagger\u001b[0m \u001b[0;31m# return the trained model\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/tag/crf.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, train_data, model_file)\u001b[0m\n\u001b[1;32m 183\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtrain_data\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 184\u001b[0m \u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m \u001b[0mfeatures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_feature_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 186\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/tag/crf.py\u001b[0m in \u001b[0;36m\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m 183\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtrain_data\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 184\u001b[0m \u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m \u001b[0mfeatures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_feature_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 186\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36m_get_features\u001b[0;34m(self, tokens, index)\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0msent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\" \"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 16\u001b[0m \u001b[0msent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreplace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'%'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'percent'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# it breaks on the percent symbol for some reason\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parse_tree\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtree\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtree\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dep_parser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_parse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 18\u001b[0m \u001b[0mparsed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mraw_parse\u001b[0;34m(self, sentence, properties, *args, **kwargs)\u001b[0m\n\u001b[1;32m 227\u001b[0m return next(\n\u001b[1;32m 228\u001b[0m self.raw_parse_sents(\n\u001b[0;32m--> 229\u001b[0;31m \u001b[0;34m[\u001b[0m\u001b[0msentence\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproperties\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdefault_properties\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 230\u001b[0m )\n\u001b[1;32m 231\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mraw_parse_sents\u001b[0;34m(self, sentences, verbose, properties, *args, **kwargs)\u001b[0m\n\u001b[1;32m 282\u001b[0m \u001b[0;32myield\u001b[0m \u001b[0miter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtree\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 283\u001b[0m \"\"\"\n\u001b[0;32m--> 284\u001b[0;31m \u001b[0mparsed_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapi_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproperties\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdefault_properties\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 285\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mparsed_sent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mparsed_data\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"sentences\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 286\u001b[0m \u001b[0mtree\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_tree\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_sent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mapi_call\u001b[0;34m(self, data, properties, timeout)\u001b[0m\n\u001b[1;32m 245\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"properties\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdumps\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdefault_properties\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 246\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 247\u001b[0;31m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 248\u001b[0m )\n\u001b[1;32m 249\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mpost\u001b[0;34m(self, url, data, json, **kwargs)\u001b[0m\n\u001b[1;32m 588\u001b[0m \"\"\"\n\u001b[1;32m 589\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 590\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'POST'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 591\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 592\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mput\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m 540\u001b[0m }\n\u001b[1;32m 541\u001b[0m \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 542\u001b[0;31m \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 543\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 544\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m 653\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 654\u001b[0m \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 655\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 656\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 657\u001b[0m \u001b[0;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m 447\u001b[0m \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 448\u001b[0m \u001b[0mretries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 449\u001b[0;31m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 450\u001b[0m )\n\u001b[1;32m 451\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m 704\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 705\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 706\u001b[0;31m \u001b[0mchunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchunked\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 707\u001b[0m )\n\u001b[1;32m 708\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m 443\u001b[0m \u001b[0;31m# Python 3 (including for exceptions like SystemExit).\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 444\u001b[0m \u001b[0;31m# Otherwise it looks like a bug in the code.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 445\u001b[0;31m \u001b[0msix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 446\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSocketTimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mBaseSSLError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mSocketError\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 447\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_raise_timeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout_value\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mread_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/packages/six.py\u001b[0m in \u001b[0;36mraise_from\u001b[0;34m(value, from_value)\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m 438\u001b[0m \u001b[0;31m# Python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 439\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 440\u001b[0;31m \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 441\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mBaseException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 442\u001b[0m \u001b[0;31m# Remove the TypeError from the exception chain in\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1342\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1343\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1344\u001b[0;31m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1345\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1346\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 304\u001b[0m \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 305\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 306\u001b[0;31m \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 307\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 308\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 265\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 266\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 267\u001b[0;31m \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 268\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 269\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m 587\u001b[0m \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 588\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 590\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 591\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mKeyboardInterrupt\u001b[0m: " + ] + } + ], + "source": [ + "# *** Improve the CRF NER tagger using dependency features (see lab 5).\n", + "# Useful explanation: https://www.analyticsvidhya.com/blog/2020/07/part-of-speechpos-tagging-dependency-parsing-and-constituency-parsing-in-nlp/\n", + "\n", + "from time import sleep \n", + "\n", + "class CRFTaggerWithDeps(CRFTaggerWithPOS):\n", + " _dep_parser = CoreNLPDependencyParser()\n", + " \n", + " def _get_features(self, tokens, index):\n", + " \"\"\"\n", + " Extract the features for a token and append the POS tag as an additional feature.\n", + " \"\"\"\n", + " if tokens != self._current_tokens: # if we haven't seen this sentence yet\n", + " parsed = False\n", + " sent = \" \".join(tokens)\n", + " sent = sent.replace('%', 'percent') # it breaks on the percent symbol for some reason\n", + " self._parse_tree = [tree for tree in self._dep_parser.raw_parse(sent)][0]\n", + " parsed = True\n", + " \n", + " basic_features = super()._get_features(tokens, index)\n", + " \n", + " # we include the relation label\n", + " basic_features.append(self._parse_tree.nodes[index+1]['rel'])\n", + " \n", + " # we include the tag of the parent node of this tag\n", + " basic_features.append(self._parse_tree.nodes[self._parse_tree.nodes[index+1]['head']]['tag'])\n", + " if basic_features[-1] is None:\n", + " basic_features[-1] = 'None'\n", + " if basic_features[-2] is None:\n", + " basic_features[-2] = 'None'\n", + " \n", + " return basic_features\n", + " \n", + "# Train\n", + "def train_CRF_NER_tagger_deps(train_set):\n", + " ### WRITE YOUR OWN CODE HERE\n", + " tagger = CRFTaggerWithDeps()\n", + " tagger.train(train_set, 'model.crf.tagger')\n", + " return tagger # return the trained model\n", + "\n", + "### WRITE YOUR OWN CODE HERE\n", + "tagger_with_deps = train_CRF_NER_tagger_deps(train_set)\n", + "print(tagger_with_POS)\n", + "\n", + "# Test\n", + "predicted_tags_with_deps = tag_test_set(test_set, tagger_with_deps)\n", + "print(predicted_tags_with_deps[0])\n", + "\n", + "cal_span_level_f1(test_set, predicted_tags_with_deps, ne_tags)\n", + "###" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Relation Extraction (RE)\n", + "\n", + "We can extract semantic information from text by extracting relations between entities. The code below prints out the relations in the training dataset. Have a look at the output and see if you can identify any factual information from the relations." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "tags": [ + "outputPrepend" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "['Jordanian', 'intelligence', 'officers'] --> AlliesOf --> ['Jordanian', '71st', 'Counter', 'Terrorism', 'Battalion']\n", + "['Daesh'] --> FightingAgainst --> ['mourners']\n", + "['mourners'] --> CoLocated --> ['a', 'funeral', 'tent']\n", + "['a', 'funeral', 'tent'] --> none --> ['today']\n", + "['a', 'funeral', 'tent'] --> CoLocated --> ['Baghdad']\n", + "['Baghdad'] --> CoLocated --> ['Iraq']\n", + "['Baghdad'] --> none --> ['today']\n", + "['About', '30,000', 'Iraqi', 'security', 'force', 'personnel'] --> AlliesOf --> ['US-led', 'coalition']\n", + "['About', '30,000', 'Iraqi', 'security', 'force', 'personnel'] --> AlliesOf --> ['Kurdish', 'fighters']\n", + "['About', '30,000', 'Iraqi', 'security', 'force', 'personnel'] --> AlliesOf --> ['Sunni', 'Arab', 'tribesmen']\n", + "['About', '30,000', 'Iraqi', 'security', 'force', 'personnel'] --> AlliesOf --> ['Shia', 'militiamen']\n", + "['US-led', 'coalition'] --> AlliesOf --> ['About', '30,000', 'Iraqi', 'security', 'force', 'personnel']\n", + "['US-led', 'coalition'] --> AlliesOf --> ['Kurdish', 'fighters']\n", + "['US-led', 'coalition'] --> AlliesOf --> ['Sunni', 'Arab', 'tribesmen']\n", + "['US-led', 'coalition'] --> AlliesOf --> ['Shia', 'militiamen']\n", + "['Kurdish', 'fighters'] --> AlliesOf --> ['About', '30,000', 'Iraqi', 'security', 'force', 'personnel']\n", + "['Kurdish', 'fighters'] --> AlliesOf --> ['US-led', 'coalition']\n", + "['Kurdish', 'fighters'] --> AlliesOf --> ['Sunni', 'Arab', 'tribesmen']\n", + "['Kurdish', 'fighters'] --> AlliesOf --> ['Shia', 'militiamen']\n", + "['Sunni', 'Arab', 'tribesmen'] --> AlliesOf --> ['About', '30,000', 'Iraqi', 'security', 'force', 'personnel']\n", + "['Sunni', 'Arab', 'tribesmen'] --> AlliesOf --> ['US-led', 'coalition']\n", + "['Sunni', 'Arab', 'tribesmen'] --> AlliesOf --> ['Kurdish', 'fighters']\n", + "['Sunni', 'Arab', 'tribesmen'] --> AlliesOf --> ['Shia', 'militiamen']\n", + "['Shia', 'militiamen'] --> AlliesOf --> ['About', '30,000', 'Iraqi', 'security', 'force', 'personnel']\n", + "['Shia', 'militiamen'] --> AlliesOf --> ['US-led', 'coalition']\n", + "['Shia', 'militiamen'] --> none --> ['eight', 'days', 'ago']\n", + "['Shia', 'militiamen'] --> AlliesOf --> ['Kurdish', 'fighters']\n", + "['Shia', 'militiamen'] --> AlliesOf --> ['Sunni', 'Arab', 'tribesmen']\n", + "['ISIL', 'tactical', 'units'] --> CoLocated --> ['Ayn', 'Isa']\n", + "['fighting', 'positions'] --> none --> ['a', 'VBIED']\n", + "['fighting', 'positions'] --> CoLocated --> ['Ayn', 'Isa']\n", + "['command', 'and', 'control', 'node'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['command', 'and', 'control', 'node'] --> CoLocated --> ['Ayn', 'Isa']\n", + "['a', 'heavy', 'machine', 'gun'] --> CoLocated --> ['Ayn', 'Isa']\n", + "['Ayn', 'Isa'] --> none --> ['a', 'VBIED']\n", + "['Ayn', 'Isa'] --> none --> ['a', 'heavy', 'machine', 'gun']\n", + "['Defense', 'Secretary', 'Ash', 'Carter'] --> BelongsTo --> ['the', 'two', 'leaders']\n", + "['Defense', 'Secretary', 'Ash', 'Carter'] --> CommWith --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['Defense', 'Secretary', 'Ash', 'Carter'] --> CoLocated --> ['Baghdad']\n", + "['Defense'] --> none --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists']\n", + "['Defense'] --> none --> ['Defense']\n", + "['Defense'] --> none --> ['Defense']\n", + "['Defense'] --> none --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> none --> ['Defense']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> none --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> none --> ['Defense', 'Secretary', 'Ash', 'Carter']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> InChargeOf --> ['the', 'Iraqi', 'city', 'of', 'Mosul']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> none --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'terrorists'] --> none --> ['Defense']\n", + "['Defense'] --> none --> ['today']\n", + "['Defense'] --> none --> ['the', 'Iraqi', 'city', 'of', 'Mosul']\n", + "['Defense'] --> none --> ['today']\n", + "['today'] --> none --> ['Defense', 'Secretary', 'Ash', 'Carter']\n", + "['today'] --> none --> ['the', 'Iraqi', 'city', 'of', 'Mosul']\n", + "['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi'] --> none --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi'] --> CommWith --> ['Defense', 'Secretary', 'Ash', 'Carter']\n", + "['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi'] --> BelongsTo --> ['the', 'two', 'leaders']\n", + "['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi'] --> none --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi'] --> CoLocated --> ['Baghdad']\n", + "['Baghdad'] --> none --> ['the', 'two', 'leaders']\n", + "['Baghdad'] --> none --> ['Iraqi', 'Prime', 'Minister\\u200e', 'Haider', 'al-Abadi']\n", + "['the', 'highway', 'from', 'Lebanon', 'to', 'Damascus'] --> CoLocated --> ['the', 'town']\n", + "['the', 'city', 'of', 'Raqqah'] --> none --> ['the', 'SDF']\n", + "['the', 'city', 'of', 'Raqqah'] --> none --> ['Inherent', 'Resolve', 'Commander']\n", + "['The', 'Combined', 'Joint', 'Task', 'Force'] --> none --> ['the', 'city', 'of', 'Raqqah']\n", + "['the', 'city', 'of', 'Raqqah'] --> none --> ['Inherent', 'Resolve', 'Commander']\n", + "['territory', 'in', 'Syria'] --> none --> ['the', 'city', 'of', 'Raqqah']\n", + "['territory', 'in', 'Syria'] --> none --> ['The', 'Combined', 'Joint', 'Task', 'Force']\n", + "['territory', 'in', 'Syria'] --> none --> ['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend']\n", + "['Inherent', 'Resolve', 'Commander'] --> none --> ['Inherent', 'Resolve', 'Commander']\n", + "['Inherent', 'Resolve', 'Commander'] --> none --> ['the', 'city', 'of', 'Raqqah']\n", + "['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend'] --> InChargeOf --> ['Inherent', 'Resolve', 'Commander']\n", + "['the', 'global', 'coalition'] --> AlliesOf --> ['Iraqi', 'forces']\n", + "['Syrian', 'democratic', 'forces'] --> none --> ['24', 'hours']\n", + "['the', 'ISIL', \"'s\", 'War', 'Committee'] --> InChargeOf --> ['Palmyra', ',', 'Syria']\n", + "['Palmyra', ',', 'Syria'] --> none --> ['the', 'ISIL', \"'s\", 'War', 'Committee']\n", + "['nine', 'members'] --> BelongsTo --> ['his', 'family']\n", + "['the', 'American', 'service', 'member'] --> CoLocated --> ['Syria']\n", + "['this', 'Thanksgiving'] --> none --> ['service', 'members']\n", + "['ISIL-held', 'terrain'] --> HasAttrOf --> ['700', 'square', 'miles']\n", + "['Minister', 'Ellwood'] --> CommWith --> ['the', 'Syrian', 'National', 'Coalition']\n", + "['Minister', 'Ellwood'] --> CommWith --> ['Turkish', 'and', 'UK', 'organisations']\n", + "['Turkish', 'and', 'UK', 'organisations'] --> none --> ['Minister', 'Ellwood']\n", + "['subject', 'matter', 'experts'] --> none --> ['the', 'Jordanians']\n", + "['At', 'least', '10'] --> none --> ['U.S.', 'Marines']\n", + "['U.S.', 'Marines'] --> HasAttrOf --> ['At', 'least', '10']\n", + "['Iraqi', 'soldiers'] --> none --> ['U.S.', 'Marines']\n", + "['Coalition', 'military', 'forces'] --> AlliesOf --> ['the', 'government', 'of', 'Iraq']\n", + "['Coalition', 'military', 'forces'] --> InChargeOf --> ['attack', ',', 'bomber', ',', 'fighter', ',', 'and', 'remotely', 'piloted', 'aircraft']\n", + "['Coalition', 'military', 'forces'] --> InChargeOf --> ['rocket', 'artillery']\n", + "['the', 'government', 'of', 'Iraq'] --> AlliesOf --> ['Coalition', 'military', 'forces']\n", + "['a', 'VBIED', 'storage', 'facility'] --> CoLocated --> ['Al', 'Huwayjah']\n", + "['The', 'United', 'States'] --> none --> ['Jabla']\n", + "['Latakia'] --> none --> ['the', 'port', 'city', 'of', 'Tartus']\n", + "['Latakia'] --> none --> ['Jabla']\n", + "['Latakia'] --> none --> ['the', 'northwest', 'province', 'of', 'Latakia']\n", + "['the', 'port', 'city', 'of', 'Tartus'] --> none --> ['a', 'hospital']\n", + "['Da', '’', 'esh'] --> CoLocated --> ['the', 'port', 'city', 'of', 'Tartus']\n", + "['Da', '’', 'esh'] --> FightingAgainst --> ['60', 'civilians']\n", + "['Da', '’', 'esh'] --> FightingAgainst --> ['bus', 'stations']\n", + "['Da', '’', 'esh'] --> FightingAgainst --> ['a', 'hospital']\n", + "['Da', '’', 'esh'] --> CoLocated --> ['Jabla']\n", + "['60', 'civilians'] --> CoLocated --> ['the', 'port', 'city', 'of', 'Tartus']\n", + "['60', 'civilians'] --> CoLocated --> ['bus', 'stations']\n", + "['60', 'civilians'] --> CoLocated --> ['a', 'hospital']\n", + "['60', 'civilians'] --> CoLocated --> ['Jabla']\n", + "['bus', 'stations'] --> none --> ['Jabla']\n", + "['bus', 'stations'] --> none --> ['the', 'northwest', 'province', 'of', 'Latakia']\n", + "['bus', 'stations'] --> none --> ['today']\n", + "['a', 'hospital'] --> none --> ['Latakia']\n", + "['Jabla'] --> CoLocated --> ['the', 'northwest', 'province', 'of', 'Latakia']\n", + "['today'] --> none --> ['The', 'United', 'States']\n", + "['the', 'global', 'coalition'] --> FightingAgainst --> ['Da', '’', 'esh']\n", + "['Gen.', 'Joseph', 'L.', 'Votel'] --> IsSynonymOf --> ['U.S.', 'Central', 'Command', 'commander']\n", + "['Gen.', 'Joseph', 'L.', 'Votel'] --> none --> ['today']\n", + "['Navy', 'Special', 'Warfare', 'Operator', '1st', 'Class', 'Charles', 'H.', 'Keating', 'IV'] --> none --> ['today']\n", + "['Navy', 'Special', 'Warfare', 'Operator', '1st', 'Class', 'Charles', 'H.', 'Keating', 'IV'] --> CoLocated --> ['northern', 'Iraq']\n", + "['Navy', 'Special', 'Warfare', 'Operator', '1st', 'Class', 'Charles', 'H.', 'Keating', 'IV'] --> BelongsTo --> ['We']\n", + "['U.S.', 'Central', 'Command', 'commander'] --> none --> ['yesterday']\n", + "['U.S.', 'Central', 'Command', 'commander'] --> none --> ['northern', 'Iraq']\n", + "['U.S.', 'Central', 'Command', 'commander'] --> IsSynonymOf --> ['Gen.', 'Joseph', 'L.', 'Votel']\n", + "['We'] --> none --> ['yesterday']\n", + "['War', 'Council'] --> none --> ['the', 'West']\n", + "['a', 'school', 'for', 'girls'] --> CoLocated --> ['a', 'southern', 'district', 'of', 'the', 'city']\n", + "['Coalition', 'military', 'forces'] --> AlliesOf --> ['the', 'government', 'of', 'Iraq']\n", + "['A', 'spokesman'] --> none --> ['Rupert', 'Colville']\n", + "['the', 'UN', 'High', 'Commissioner', 'for', 'Human', 'Rights'] --> InChargeOf --> ['A', 'spokesman']\n", + "['the', 'UN', 'High', 'Commissioner', 'for', 'Human', 'Rights'] --> IsSynonymOf --> ['Rupert', 'Colville']\n", + "['Rupert', 'Colville'] --> CommWith --> ['a', 'news', 'conference']\n", + "['last', 'year'] --> none --> ['10', 'per', 'cent']\n", + "['besieged', 'areas'] --> none --> ['10', 'per', 'cent']\n", + "['U.S.', '5th', 'Fleet', 'area', 'of', 'operations'] --> none --> ['the', 'U.S.', '5th', 'Fleet', 'area', 'of', 'operations']\n", + "['the', 'U.S.', '5th', 'Fleet', 'area', 'of', 'operations'] --> none --> ['Royal', 'Navy']\n", + "['the', 'U.S.', '5th', 'Fleet', 'area', 'of', 'operations'] --> none --> ['U.S.', '5th', 'Fleet', 'area', 'of', 'operations']\n", + "['I'] --> none --> ['all', 'Iraqi', 'leaders']\n", + "['all', 'Iraqi', 'leaders'] --> none --> ['a', 'common', 'enemy']\n", + "['a', 'common', 'enemy'] --> none --> ['I']\n", + "['innocent', 'people'] --> HasAttrOf --> ['tens', 'of', 'thousands']\n", + "['its'] --> none --> ['The', 'group']\n", + "['the', 'town'] --> none --> ['The', 'group']\n", + "['the', 'Coalition'] --> AlliesOf --> ['its', 'partners', 'on', 'the', 'ground']\n", + "['the', 'Government', 'of', 'Iraq'] --> CommWith --> ['the', 'Kurdistan', 'Regional', 'Government']\n", + "['the', 'Kurdistan', 'Regional', 'Government'] --> CommWith --> ['the', 'Government', 'of', 'Iraq']\n", + "['defense', 'ministers'] --> CoLocated --> ['Brussels', ',', 'Belgium']\n", + "['defense', 'ministers'] --> none --> ['February', '11', ',', '2016']\n", + "['Brussels', ',', 'Belgium'] --> none --> ['February', '2', ',', '2016']\n", + "['February', '11', ',', '2016'] --> none --> ['Coalition', 'foreign', 'ministers']\n", + "['Coalition', 'foreign', 'ministers'] --> CoLocated --> ['Rome', ',', 'Italy']\n", + "['Rome', ',', 'Italy'] --> none --> ['Coalition', 'foreign', 'ministers']\n", + "['a', 'sun-baked', 'field'] --> CoLocated --> ['the', 'Jordanian', 'city', 'of', 'Ramtha']\n", + "['a', 'young', 'group', 'of', 'Syrian', 'men', 'and', 'women'] --> CoLocated --> ['a', 'sun-baked', 'field']\n", + "['Omar', 'Abu', 'Anas'] --> CoLocated --> ['his', 'home', 'town']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['the', 'Al', 'Bu', 'Nimr', 'tribe'] --> CoLocated --> ['Anbar', 'province']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Minister', 'for', 'the', 'Middle', 'East']\n", + "['I'] --> none --> ['Anbar', 'province']\n", + "['I'] --> none --> ['Tobias', 'Ellwood']\n", + "['The', 'Coalition'] --> AlliesOf --> ['the', 'SDF']\n", + "['the', 'Syrian', 'opposition'] --> CoLocated --> ['Riyadh']\n", + "['The', 'area', 'of', 'Mosul', 'district', ',', 'Ninewa', 'province'] --> CoLocated --> ['Iraq']\n", + "['Criminal'] --> none --> ['The', 'area', 'of', 'Mosul', 'district', ',', 'Ninewa', 'province']\n", + "['3rd', 'Battalion', ',', '321st', 'Field', 'Artillery', 'Regiment'] --> none --> ['the', 'Jordan', 'Armed', 'Forces', '–', 'Arab', 'Army']\n", + "['the', 'Jordan', 'Armed', 'Forces', '–', 'Arab', 'Army'] --> Likes --> ['Soldiers']\n", + "['Soldiers'] --> BelongsTo --> ['3rd', 'Battalion', ',', '321st', 'Field', 'Artillery', 'Regiment']\n", + "['Soldiers'] --> BelongsTo --> ['3rd', 'Battalion', ',', '321st', 'Field', 'Artillery', 'Regiment']\n", + "['Soldiers'] --> none --> ['the', 'Jordan', 'Armed', 'Forces', '–', 'Arab', 'Army']\n", + "['Soldiers'] --> CoLocated --> ['the', 'Jordan', 'Armed', 'Forces', '–', 'Arab', 'Army']\n", + "['The', 'Hamrin', 'Mountains'] --> none --> ['a', 'small', 'mountain', 'ridge']\n", + "['The', 'Hamrin', 'Mountains'] --> CoLocated --> ['northeast', 'Iraq']\n", + "['northeast', 'Iraq'] --> none --> ['The', 'Hamrin', 'Mountains']\n", + "['northeast', 'Iraq'] --> none --> ['Arabic']\n", + "['Knox', 'Thames'] --> IsSynonymOf --> ['the', 'Special', 'Advisor', 'to', 'Ambassador', 'Saperstein', 'for', 'Religious', 'Minorities', 'in', 'the', 'Near', 'East', 'and', 'South', 'and', 'Central', 'Asia']\n", + "['Knox', 'Thames'] --> none --> ['Lebanon']\n", + "['Central', 'Asia'] --> none --> ['Turkey']\n", + "['November', '16', '-', '17'] --> none --> ['Knox', 'Thames']\n", + "['November', '16', '-', '17'] --> none --> ['the', 'Special', 'Advisor', 'to', 'Ambassador', 'Saperstein', 'for', 'Religious', 'Minorities', 'in', 'the', 'Near', 'East', 'and', 'South', 'and', 'Central', 'Asia']\n", + "['November', '18', '-19'] --> none --> ['Turkey']\n", + "['November', '18', '-19'] --> none --> ['November', '16', '-', '17']\n", + "['November', '18', '-19'] --> none --> ['Turkey']\n", + "['I'] --> Likes --> ['Turkey']\n", + "['ally'] --> none --> ['I']\n", + "['ally'] --> none --> ['Syrian', 'opposition', 'forces']\n", + "['Turkey'] --> none --> ['our', 'international', 'coalition']\n", + "['Turkey'] --> AlliesOf --> ['Syrian', 'opposition', 'forces']\n", + "['our', 'international', 'coalition'] --> AlliesOf --> ['Syrian', 'opposition', 'forces']\n", + "['Syrian', 'opposition', 'forces'] --> FightingAgainst --> ['ISIL']\n", + "['the', 'Syrian', 'town', 'of', 'Dabiq'] --> none --> ['Syrian', 'opposition', 'forces']\n", + "['ISIL'] --> FightingAgainst --> ['Syrian', 'opposition', 'forces']\n", + "['ISIL'] --> none --> ['the', 'Syrian', 'town', 'of', 'Dabiq']\n", + "['chemical', 'weapons'] --> BelongsTo --> ['the', 'Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant']\n", + "['the', 'international', 'coalition'] --> AlliesOf --> ['the', 'Iraqi', 'government']\n", + "['Alsumaria', 'News'] --> HasAttrOf --> ['Iraqi']\n", + "['Nilesat'] --> none --> ['Arabic']\n", + "['Arabic'] --> none --> ['Nilesat']\n", + "['Arabic'] --> none --> ['Iraqi']\n", + "['Iraqi'] --> none --> ['Nilesat']\n", + "['U.S.', 'Air', 'Forces', 'Central', 'Command'] --> none --> ['Nov.', '29', ',', '2016']\n", + "['September', '17'] --> none --> ['Dayr', 'az', 'Zawr', ',', 'Syria']\n", + "['Dayr', 'az', 'Zawr', ',', 'Syria'] --> none --> ['U.S.', 'Air', 'Forces', 'Central', 'Command']\n", + "['the', 'United', 'States'] --> Likes --> ['the', 'people', 'of', 'Syria']\n", + "['a'] --> none --> ['dump', 'truck']\n", + "['a'] --> none --> ['Ar', 'Raqqah']\n", + "['a'] --> none --> ['two', 'bridges']\n", + "['a'] --> none --> ['pontoon', 'bridge']\n", + "['a'] --> none --> ['six', 'oil', 'refinement', 'stills']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['two', 'bridges']\n", + "['a', 'VBIED'] --> none --> ['a', 'VBIED']\n", + "['a', 'VBIED'] --> none --> ['a']\n", + "['two', 'bridges'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['two', 'bridges'] --> none --> ['pontoon', 'bridge']\n", + "['artillery', 'systems'] --> none --> ['a', 'VBIED']\n", + "['artillery', 'systems'] --> none --> ['a']\n", + "['artillery', 'systems'] --> none --> ['three', 'supply', 'caches']\n", + "['dump', 'truck'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['dump', 'truck'] --> none --> ['a', 'supply', 'route']\n", + "['armored', 'vehicle'] --> none --> ['a']\n", + "['armored', 'vehicle'] --> none --> ['six', 'oil', 'refinement', 'stills']\n", + "['a', 'supply', 'route'] --> none --> ['two', 'bridges']\n", + "['a', 'supply', 'route'] --> none --> ['five', 'ISIL', 'tactical', 'units']\n", + "['five', 'ISIL', 'tactical', 'units'] --> none --> ['a']\n", + "['five', 'ISIL', 'tactical', 'units'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['five', 'ISIL', 'tactical', 'units'] --> none --> ['armored', 'vehicle']\n", + "['five', 'ISIL', 'tactical', 'units'] --> none --> ['six', 'oil', 'refinement', 'stills']\n", + "['six', 'oil', 'refinement', 'stills'] --> none --> ['two', 'bridges']\n", + "['six', 'oil', 'refinement', 'stills'] --> none --> ['dump', 'truck']\n", + "['six', 'oil', 'refinement', 'stills'] --> none --> ['armored', 'vehicle']\n", + "['six', 'oil', 'refinement', 'stills'] --> none --> ['three', 'supply', 'caches']\n", + "['three', 'supply', 'caches'] --> none --> ['two', 'bridges']\n", + "['The', 'former', 'leader', 'of', 'Al', 'Qaeda', 'in', 'Iraq'] --> none --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['The', 'former', 'leader', 'of', 'Al', 'Qaeda', 'in', 'Iraq'] --> IsSynonymOf --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['Abu', 'Musab', 'al-Zarqawi'] --> IsSynonymOf --> ['The', 'former', 'leader', 'of', 'Al', 'Qaeda', 'in', 'Iraq']\n", + "['Abu', 'Musab', 'al-Zarqawi'] --> none --> ['the', 'village']\n", + "['Abu', 'Musab', 'al-Zarqawi'] --> InChargeOf --> ['a', 'safehouse']\n", + "['a', 'safehouse'] --> CoLocated --> ['the', 'village']\n", + "['a', 'safehouse'] --> none --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['the', 'scenes', 'of', 'bombings'] --> none --> ['this', 'city']\n", + "['Saudi', 'Arabia', ',', 'Emirates', ',', 'Turkey', ',', 'Lebanon', ',', 'Egypt', ',', 'Jordan', ',', 'Qatar'] --> none --> ['the', 'end', 'of', 'the', 'year']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> none --> ['Baghdad']\n", + "['Foreign', 'Office', 'Minister'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Foreign', 'Office', 'Minister'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Minister', 'for', 'the', 'Middle', 'East']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Foreign', 'Office', 'Minister']\n", + "['Tobias', 'Ellwood'] --> none --> ['Baghdad']\n", + "['Karrada', 'district', 'of', 'Baghdad'] --> CoLocated --> ['Baghdad']\n", + "['Karrada', 'district', 'of', 'Baghdad'] --> none --> ['the', 'Karrada', 'district', 'of', 'Baghdad']\n", + "['Karrada', 'district', 'of', 'Baghdad'] --> CoLocated --> ['Baghdad']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Minister', 'for', 'the', 'Middle', 'East']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Foreign', 'Office', 'Minister']\n", + "['Tobias', 'Ellwood'] --> none --> ['the', 'Karrada', 'district', 'of', 'Baghdad']\n", + "['the', 'Turks'] --> BelongsTo --> ['our', 'Coalition', 'partners']\n", + "['FS', 'Forbin'] --> BelongsTo --> ['CTF', '50']\n", + "['FS', 'Forbin'] --> CoLocated --> ['the', 'Gulf']\n", + "['CTF', '50'] --> none --> ['FS', 'Forbin']\n", + "['CTF', '50'] --> CoLocated --> ['the', 'Gulf']\n", + "['U.S.', ',', 'U.K.', 'and', 'French', 'maritime', 'security', 'operations'] --> CoLocated --> ['the', 'Gulf']\n", + "['the', 'Gulf'] --> none --> ['CTF', '50']\n", + "['Army', 'of', 'the', 'Victorious', 'Sect'] --> none --> ['al-Ahwal', 'Brigades']\n", + "['Army', 'of', 'the', 'Victorious', 'Sect'] --> none --> ['al-Qaeda', 'in', 'Iraq']\n", + "['Saray', 'al-Jihad', 'Group'] --> none --> ['al-Ahwal', 'Brigades']\n", + "['al-Ghuraba', 'Brigades'] --> none --> ['Army', 'of', 'the', 'Victorious', 'Sect']\n", + "['government', 'officials'] --> none --> ['November', '23']\n", + "['Deputy', 'Secretary', 'Blinken'] --> CommWith --> ['senior', 'regional', 'government', 'officials']\n", + "['senior', 'regional', 'government', 'officials'] --> none --> ['Deputy', 'Secretary', 'Blinken']\n", + "['November', '23'] --> none --> ['the', 'Iraqi', 'Kurdistan', 'Region']\n", + "['Mosul'] --> none --> ['Iraqi', 'forces']\n", + "['Pentagon', 'press', 'operations', 'director', 'Navy', 'Capt', '.'] --> none --> ['24', 'hours']\n", + "['Iraqi', 'forces'] --> InChargeOf --> ['defensive', 'positions']\n", + "['Iraqi', 'forces'] --> FightingAgainst --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters'] --> none --> ['24', 'hours']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters'] --> none --> ['the', 'past', '24', 'hours']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters'] --> FightingAgainst --> ['Iraqi', 'forces']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters'] --> none --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters']\n", + "['three'] --> none --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant', 'fighters']\n", + "['the', 'past', '24', 'hours'] --> none --> ['Pentagon', 'press', 'operations', 'director', 'Navy', 'Capt', '.']\n", + "['The', 'UK'] --> none --> ['the', 'regime']\n", + "['the', 'international', 'community'] --> none --> ['the', 'regime']\n", + "['attack', ',', 'bomber', ',', 'fighter', 'and', 'remotely', 'piloted', 'aircraft'] --> BelongsTo --> ['Coalition', 'military', 'forces']\n", + "['Some', 'members'] --> BelongsTo --> ['the', 'Syrian', 'Computer', 'Society']\n", + "['Some', 'members'] --> BelongsTo --> ['the', 'Syrian', 'Electronic', 'Army']\n", + "['a', 'single', 'weapon'] --> none --> ['a', 'compound']\n", + "['a', 'lone', 'ISIL', 'vehicle'] --> none --> ['a', 'compound']\n", + "['a', 'group', 'of', 'buildings'] --> none --> ['a', 'single', 'aircraft']\n", + "['a', 'single', 'aircraft'] --> none --> ['a', 'group', 'of', 'buildings']\n", + "['a', 'group', 'of', 'buildings'] --> CoLocated --> ['a', 'compound']\n", + "['a', 'group', 'of', 'buildings'] --> none --> ['a', 'single', 'weapon']\n", + "['weapon', 'systems'] --> CoLocated --> ['a', 'compound']\n", + "['a', 'compound'] --> none --> ['a', 'lone', 'ISIL', 'vehicle']\n", + "['a', 'compound'] --> none --> ['weapon', 'systems']\n", + "['a', 'single', 'weapon'] --> none --> ['a', 'single', 'aircraft']\n", + "['a', 'lone', 'ISIL', 'vehicle'] --> none --> ['a', 'group', 'of', 'buildings']\n", + "['a', 'lone', 'ISIL', 'vehicle'] --> none --> ['a', 'compound']\n", + "['33°47′N', '44°30′E'] --> IsSynonymOf --> ['33.783°N', '44.500°E']\n", + "['33°47′N', '44°30′E'] --> IsSynonymOf --> ['33.783', ';', '44.500']\n", + "['33.783°N', '44.500°E'] --> IsSynonymOf --> ['33°47′N', '44°30′E']\n", + "['33.783°N', '44.500°E'] --> IsSynonymOf --> ['33.783', ';', '44.500']\n", + "['33.783', ';', '44.500'] --> IsSynonymOf --> ['33°47′N', '44°30′E']\n", + "['33.783', ';', '44.500'] --> IsSynonymOf --> ['33.783°N', '44.500°E']\n", + "['December', '2014'] --> none --> ['Al-Raqqa', 'province']\n", + "['Samarra', 'Barrage'] --> CoLocated --> ['the', 'Tigris', 'River']\n", + "['Samarra', 'Barrage'] --> none --> ['Tharthar', 'Barrage']\n", + "['Samarra'] --> none --> ['Arabic']\n", + "['Arabic'] --> none --> ['the', 'Tigris', 'River']\n", + "['Baghdad', ',', 'Iraq'] --> none --> ['the', 'Tigris', 'River']\n", + "['Tharthar', 'Barrage'] --> CoLocated --> ['the', 'Tigris', 'River']\n", + "['Da', '’', 'esh'] --> FightingAgainst --> ['scores', 'of', 'others']\n", + "['Da', '’', 'esh'] --> FightingAgainst --> ['at', 'least', '35', 'more', 'innocent', 'civilians']\n", + "['lieutenant', 'Muath', 'al-Kasasbeh'] --> BelongsTo --> ['Royal', 'Jordanian', 'Air', 'Force']\n", + "['lieutenant', 'Muath', 'al-Kasasbeh'] --> none --> ['Sajida', 'Mubarak', 'Atrous', 'al-Rishawi']\n", + "['the', 'Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant'] --> FightingAgainst --> ['Royal', 'Jordanian', 'Air', 'Force']\n", + "['the', 'Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant'] --> none --> ['Sajida', 'Mubarak', 'Atrous', 'al-Rishawi']\n", + "['Sajida', 'Mubarak', 'Atrous', 'al-Rishawi'] --> none --> ['4', 'February', '2015']\n", + "['The', 'UK'] --> CommWith --> ['member', 'states']\n", + "['I'] --> none --> ['service', 'members']\n", + "['the', 'city', 'of', 'Husaybah'] --> none --> ['the', 'city', 'of', 'Husaybah']\n", + "['the', 'coalition', 'forces'] --> FightingAgainst --> ['the', 'city', 'of', 'Husaybah']\n", + "['the', 'city', 'of', 'Husaybah'] --> none --> ['the', 'city', 'of', 'Husaybah']\n", + "['the', 'city', 'of', 'Husaybah'] --> none --> ['10', 'November']\n", + "['Coalition', 'military', 'forces'] --> FightingAgainst --> ['ISIL', 'targets']\n", + "['Coalition', 'military', 'forces'] --> InChargeOf --> ['remotely', 'piloted', 'aircraft']\n", + "['ISIL', 'targets'] --> none --> ['remotely', 'piloted', 'aircraft']\n", + "['remotely', 'piloted', 'aircraft'] --> none --> ['ISIL', 'targets']\n", + "['The', 'U.S.', 'Department', 'of', 'State', \"'s\", 'Rewards', 'for', 'Justice', 'program'] --> Dislikes --> ['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant'] --> IsSynonymOf --> ['ISIL']\n", + "['the', 'Jordanian', 'army', 'and', 'intelligence'] --> none --> ['the', 'prosecution']\n", + "['Popular', 'Mobilization', 'Forces'] --> AlliesOf --> ['the', 'coalition']\n", + "['the', 'Iraqi', 'Armed', 'Forces'] --> AlliesOf --> ['the', 'coalition']\n", + "['the', 'city'] --> none --> ['the', 'insurgency']\n", + "['his', 'nephews'] --> BelongsTo --> ['The', 'victims']\n", + "['his', 'nephews'] --> none --> ['the', 'family', \"'s\", 'clothing', 'business']\n", + "['his', 'nephews'] --> BelongsTo --> ['the', 'family', \"'s\", 'clothing', 'business']\n", + "['the', 'family', \"'s\", 'clothing', 'business'] --> none --> ['his', 'nephews']\n", + "['their', 'wives', 'and', 'children'] --> BelongsTo --> ['The', 'victims']\n", + "['their', 'wives', 'and', 'children'] --> BelongsTo --> ['the', 'family', \"'s\", 'clothing', 'business']\n", + "['a', 'Russian', 'military', 'spokesman'] --> Dislikes --> ['the', 'US-led', 'coalition']\n", + "['60', 'civilians'] --> none --> ['a', 'Russian', 'military', 'spokesman']\n", + "['60', 'civilians'] --> none --> ['the', 'past', 'three', 'days']\n", + "['Jaish', \"al-Ta'ifa\", 'al-Mansurah'] --> BelongsTo --> ['the', 'Iraqi', 'insurgency']\n", + "['Jaish', \"al-Ta'ifa\", 'al-Mansurah'] --> IsSynonymOf --> ['Army', 'of', 'the', 'Victorious', 'Sect']\n", + "['Arabic'] --> none --> ['Jaish', \"al-Ta'ifa\", 'al-Mansurah']\n", + "['Arabic'] --> none --> ['a', 'militant', 'Sunni', 'group']\n", + "['Arabic'] --> none --> ['the', 'Iraqi', 'insurgency']\n", + "['Arabic'] --> none --> ['Army', 'of', 'the', 'Victorious', 'Sect']\n", + "['Army', 'of', 'the', 'Victorious', 'Sect'] --> IsSynonymOf --> ['Jaish', \"al-Ta'ifa\", 'al-Mansurah']\n", + "['the', 'town'] --> none --> ['IS-controlled', 'territory']\n", + "['Kurdish', 'Peshmerga', 'fighters'] --> none --> ['the', 'town']\n", + "['Kurdish', 'Peshmerga', 'fighters'] --> none --> ['IS-controlled', 'territory']\n", + "['Kurdish', 'Peshmerga', 'fighters'] --> InChargeOf --> ['a', 'base']\n", + "['Kurmanji'] --> none --> ['Kitab']\n", + "['Kurdish'] --> none --> ['The']\n", + "['Kurdish'] --> none --> ['Kurmanji']\n", + "['the', 'Netherlands'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'Kingdom'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'States'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['U.S.', 'Army', 'General', 'Joseph', 'Votel'] --> none --> ['U.S.', 'Central', 'Command']\n", + "['U.S.', 'Army', 'General', 'Joseph', 'Votel'] --> InChargeOf --> ['U.S.', 'Central', 'Command']\n", + "['U.S.', 'Army', 'General', 'Joseph', 'Votel'] --> CommWith --> ['U.S.', 'Air', 'Force', 'civilian', 'employees']\n", + "['coalition', 'counterparts'] --> CommWith --> ['the', 'US', 'Defense', 'Secretary', 'Ash', 'Carter']\n", + "['the', 'US', 'Defense', 'Secretary', 'Ash', 'Carter'] --> CommWith --> ['coalition', 'counterparts']\n", + "['senior', 'Iraqi', 'officials'] --> BelongsTo --> ['the', 'Government', 'of', 'Iraq']\n", + "['a', 'UAV', 'launch', 'site'] --> none --> ['a', 'VBIED']\n", + "['the', 'mountains'] --> CoLocated --> ['the', 'frontier', 'region']\n", + "['Pentagon', 'Press'] --> CoLocated --> ['the', 'Press', 'Briefing', 'Room']\n", + "['Navy', 'Capt', '.'] --> none --> ['the', 'Press', 'Briefing', 'Room']\n", + "['Albert', 'Connelly'] --> none --> ['an', 'artillery', 'crew', 'chief', 'in', 'Abel', 'Battery', ',', '3rd', 'Battalion']\n", + "['the', 'territory'] --> none --> ['30', 'percent']\n", + "['US', 'Special', 'Operations'] --> CoLocated --> ['Syrian', 'territory']\n", + "['Syrian', 'territory'] --> none --> ['US', 'Special', 'Operations']\n", + "['The', 'US-led', 'coalition'] --> Dislikes --> ['Islamic', 'State', 'in', 'Iraq']\n", + "['Syria'] --> Dislikes --> ['Israel']\n", + "['Syria'] --> none --> ['a', 'military', 'airport']\n", + "['Mr', 'Ellwood'] --> none --> ['the', 'UK', '’', 's', 'brave', 'Armed', 'Forces']\n", + "['the', 'UK', '’', 's', 'brave', 'Armed', 'Forces'] --> none --> ['Mr', 'Ellwood']\n", + "['aid', 'workers'] --> none --> ['Mr', 'Ellwood']\n", + "['aid', 'workers'] --> none --> ['the', 'UK', '’', 's', 'brave', 'Armed', 'Forces']\n", + "['The', 'Prime', 'Minister'] --> CommWith --> ['President', 'Hollande']\n", + "['President', 'Hollande'] --> CommWith --> ['The', 'Prime', 'Minister']\n", + "['over', '9', 'billion', 'euros'] --> BelongsTo --> ['The', 'EU']\n", + "['the', 'neighbouring', 'countries'] --> none --> ['The', 'EU']\n", + "['Turkey'] --> none --> ['Russia']\n", + "['29', 'December'] --> none --> ['Syria']\n", + "['Syria'] --> none --> ['Russia']\n", + "['Sheikh', 'Abu', 'Omar', 'al-Ansari'] --> InChargeOf --> ['The', 'group']\n", + "['the', 'library'] --> none --> ['next', 'door']\n", + "['one', 'child'] --> CoLocated --> ['the', 'library']\n", + "['one', 'child'] --> CoLocated --> ['next', 'door']\n", + "['our', 'men', 'and', 'women', 'in', 'uniform'] --> none --> ['Combined']\n", + "['this', 'Thanksgiving'] --> none --> ['the', 'vicinity', 'of', 'Ayn', 'Issa']\n", + "['the', 'vicinity', 'of', 'Ayn', 'Issa'] --> CoLocated --> ['northern', 'Syria']\n", + "['Combined'] --> none --> ['the', 'vicinity', 'of', 'Ayn', 'Issa']\n", + "['Joint', 'Task', 'Force', '-', 'Operation', 'Inherent', 'Resolve'] --> none --> ['Operation', 'Inherent', 'Resolve']\n", + "['a', 'front-end', 'loader'] --> CoLocated --> ['Ar', 'Raqqah']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a', 'fighting', 'position']\n", + "['ISIL', 'tactical', 'units'] --> CoLocated --> ['Ar', 'Raqqah']\n", + "['a', 'VBIED'] --> CoLocated --> ['Ar', 'Raqqah']\n", + "['a', 'fighting', 'position'] --> none --> ['a', 'front-end', 'loader']\n", + "['a', 'fighting', 'position'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['a', 'fighting', 'position'] --> CoLocated --> ['Ar', 'Raqqah']\n", + "['a', 'tunnel', 'entrance'] --> none --> ['a', 'VBIED']\n", + "['a', 'tunnel', 'entrance'] --> CoLocated --> ['Ar', 'Raqqah']\n", + "['Ar', 'Raqqah'] --> none --> ['a', 'tunnel', 'entrance']\n", + "['The', 'United', 'States'] --> Dislikes --> ['ISIL']\n", + "['Syrian', 'civilians'] --> none --> ['The', 'United', 'States']\n", + "['a', 'hospital'] --> CoLocated --> ['marketplace']\n", + "['ISIL'] --> none --> ['The', 'United', 'States']\n", + "['ISIL'] --> FightingAgainst --> ['Syrian', 'civilians']\n", + "['ISIL'] --> InChargeOf --> ['suicide', 'attacks']\n", + "['suicide', 'attacks'] --> CoLocated --> ['the', 'town', 'of', 'Tel', 'Tamer']\n", + "['the', 'town', 'of', 'Tel', 'Tamer'] --> IsSynonymOf --> ['Hasaka', 'Province']\n", + "['The', 'Foreign', 'Secretary', 'Boris', 'Johnson'] --> CommWith --> ['international', 'partners']\n", + "['The', 'Foreign', 'Secretary', 'Boris', 'Johnson'] --> none --> ['The', 'Foreign', 'Secretary', 'Boris', 'Johnson']\n", + "['Doura', 'district', 'of', 'Baghdad'] --> none --> ['Interior', 'Ministry', 'officers']\n", + "['official', 'ministry', 'vehicles'] --> none --> [\"Shi'ite\", 'civilians']\n", + "['the', 'city', 'of', 'Mosul'] --> CoLocated --> ['Iraq']\n", + "['The', 'Foreign', ',', 'Home', 'and', 'Defence', 'Secretaries'] --> CommWith --> ['their', 'French', 'counterparts']\n", + "['a'] --> BelongsTo --> ['The', 'UN', 'commission', 'of', 'Inquiry', 'on', 'Syria']\n", + "['UN', 'human', 'rights', 'staff'] --> none --> ['Iraqi', 'government', 'forces']\n", + "['UN', 'human', 'rights', 'staff'] --> none --> ['Iraqi', 'government', 'forces']\n", + "['Iraqi', 'government', 'forces'] --> none --> ['Iraqi', 'government', 'forces']\n", + "['Iraqi', 'government', 'forces'] --> none --> ['UN', 'human', 'rights', 'staff']\n", + "['Iraqi', 'government', 'forces'] --> CoLocated --> ['Mosul']\n", + "['Islamic', 'State', 'militants'] --> CoLocated --> ['Mosul']\n", + "['The', 'Foreign', 'Secretary'] --> CommWith --> ['the', 'US', 'Secretary', 'of', 'State']\n", + "['The', 'Foreign', 'Secretary'] --> CommWith --> ['international', 'partners']\n", + "['Syria'] --> none --> ['the', 'US', 'Secretary', 'of', 'State']\n", + "['the', 'US', 'Secretary', 'of', 'State'] --> CommWith --> ['international', 'partners']\n", + "['Government', 'of', 'Iraq'] --> none --> ['Minister', 'Ellwood']\n", + "['Minister', 'Ellwood'] --> Likes --> ['the', 'Government', 'of', 'Iraq']\n", + "['an', 'excavator'] --> none --> ['oil', 'tanker', 'trucks']\n", + "['eastern', 'Aleppo'] --> none --> ['Security', 'Council', 'members']\n", + "['eastern', 'Aleppo'] --> none --> ['23', 'December']\n", + "['The', 'UN'] --> Apart --> ['eastern', 'Aleppo']\n", + "['The', 'UN'] --> CommWith --> ['Security', 'Council', 'members']\n", + "['Security', 'Council', 'members'] --> none --> ['eastern', 'Aleppo']\n", + "['Security', 'Council', 'members'] --> none --> ['23', 'December']\n", + "['both', 'countries'] --> BelongsTo --> ['the', 'European', 'Union']\n", + "['Baquba'] --> none --> ['Hibhib', 'Village']\n", + "['Hibhib', 'Village'] --> CoLocated --> ['northern', 'Iraq']\n", + "['Hibhib', 'Village'] --> none --> ['8', 'km']\n", + "['8', 'km'] --> none --> ['Hibhib']\n", + "['8', 'km'] --> none --> ['Hibhib', 'Village']\n", + "['8', 'km'] --> IsSynonymOf --> ['5.0', 'mi']\n", + "['Elite', 'troops'] --> CoLocated --> ['the', 'compound']\n", + "['enemy', 'fighters'] --> FightingAgainst --> ['Iraqi', 'forces']\n", + "['enemy', 'fighters'] --> none --> ['Coalition', 'aircraft']\n", + "['enemy', 'fighters'] --> CoLocated --> ['the', 'location']\n", + "['Coalition', 'aircraft'] --> AlliesOf --> ['the', 'Iraqi', 'Security', 'Forces']\n", + "['The', 'regime'] --> none --> ['other', 'towns', 'and', 'areas']\n", + "['February', '2007'] --> none --> ['Iraqi', 'Prime', 'Minister', 'Nouri', 'al-Maliki']\n", + "['the', 'Arabian', 'Gulf'] --> none --> ['North', 'Arabian', 'Sea']\n", + "['Special', 'Advisor', 'Thames'] --> CoLocated --> ['the', 'G-20', 'Interfaith', 'Conference']\n", + "['the', 'Department'] --> none --> ['religious', 'minorities']\n", + "['those', 'who', 'have', 'been', 'killed', 'and', 'injured'] --> none --> ['the', 'family', 'and', 'friends']\n", + "['Tobias', 'Ellwood'] --> none --> ['12', 'months']\n", + "['Tobias', 'Ellwood'] --> none --> ['ISIL', 'terrorists']\n", + "['Tobias', 'Ellwood'] --> none --> ['ISIL', 'terrorists']\n", + "['the', 'city', 'of', 'Mosul'] --> CoLocated --> ['Northern', 'Iraq']\n", + "['Northern', 'Iraq'] --> none --> ['the', 'city', 'of', 'Mosul']\n", + "['Northern', 'Iraq'] --> none --> ['the', 'people', 'of', 'Mosul']\n", + "['12', 'months'] --> none --> ['the', 'city', 'of', 'Mosul']\n", + "['ISIL', 'terrorists'] --> InChargeOf --> ['the', 'city', 'of', 'Mosul']\n", + "['ISIL', 'terrorists'] --> Dislikes --> ['the', 'people', 'of', 'Mosul']\n", + "['the', 'people', 'of', 'Mosul'] --> none --> ['the', 'city', 'of', 'Mosul']\n", + "['the', 'people', 'of', 'Mosul'] --> none --> ['Northern', 'Iraq']\n", + "['the', 'people', 'of', 'Mosul'] --> none --> ['12', 'months']\n", + "['ISIL', 'terrorists'] --> InChargeOf --> ['the', 'city', 'of', 'Mosul']\n", + "['ISIL', 'terrorists'] --> Dislikes --> ['the', 'people', 'of', 'Mosul']\n", + "['The', 'Members', 'of', 'the', 'Security', 'Council'] --> none --> ['the', 'Iraq', 'people']\n", + "['the', 'Government', 'of', 'Iraq'] --> none --> ['the', 'international', 'community']\n", + "['this', 'terrorist', 'organization'] --> none --> ['the', 'international', 'community']\n", + "['this', 'terrorist', 'organization'] --> none --> ['our', 'partners']\n", + "['Prime', 'Minister', 'Al-Abadi'] --> none --> ['this', 'terrorist', 'organization']\n", + "['Prime', 'Minister', 'Al-Abadi'] --> none --> ['our', 'partners']\n", + "['the', 'Iraqi', 'Security', 'Forces'] --> none --> ['the', 'international', 'community']\n", + "['the', 'Iraqi', 'Security', 'Forces'] --> none --> ['our', 'partners']\n", + "['our', 'partners'] --> BelongsTo --> ['the', 'international', 'community']\n", + "['the', 'Iraqi', 'government'] --> Likes --> ['the', 'Kurdistan', 'regional', 'government']\n", + "['the', 'Kurdistan', 'regional', 'government'] --> Likes --> ['the', 'Iraqi', 'government']\n", + "['terrorist', 'organisation'] --> none --> ['Tobias', 'Ellwood']\n", + "['ISIL'] --> none --> ['residents', 'of', 'Mosul']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['civilians']\n", + "['civilians'] --> CoLocated --> ['Madaya']\n", + "['the', 'IS', 'group'] --> CoLocated --> ['the', 'university', 'complex']\n", + "['the', 'IS', 'group'] --> none --> ['chemical', 'weapons']\n", + "['chemical', 'weapons'] --> BelongsTo --> ['the', 'IS', 'group']\n", + "['40,000', 'civilians'] --> CoLocated --> ['the', 'city']\n", + "['The', 'EU'] --> none --> ['European', 'shores']\n", + "['Philip', 'Hammond'] --> none --> ['Kurdistan', 'Regional', 'governments']\n", + "['Iraqi'] --> CommWith --> ['Kurdistan', 'Regional', 'governments']\n", + "['United', 'States'] --> none --> ['all']\n", + "['Camp', 'Hurriya', 'residents'] --> none --> ['Camp', 'Hurriya', 'residents']\n", + "['all'] --> none --> ['a', 'permanent', 'and', 'safe', 'location']\n", + "['a', 'VBIED', 'facility'] --> none --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['a', 'VBIED', 'facility'] --> none --> ['a', 'heavy', 'machine', 'gun']\n", + "['tactical', 'vehicles'] --> none --> ['a', 'VBIED', 'facility']\n", + "['a', 'tunnel', 'entrance'] --> none --> ['a', 'mortar']\n", + "['a', 'tunnel', 'entrance'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['a', 'tunnel', 'entrance'] --> none --> ['a', 'fighting', 'position']\n", + "['a', 'mortar'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['mortar', 'teams'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a', 'VBIED', 'facility']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['mortar', 'teams']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['a', 'fighting', 'position'] --> none --> ['mortar', 'teams']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['Minister', 'for', 'the', 'Middle', 'East']\n", + "['Tobias', 'Ellwood'] --> none --> ['Sinjar']\n", + "['the', 'Government', 'of', 'Iraq'] --> FightingAgainst --> ['ISIL']\n", + "['ISIL'] --> none --> ['Tobias', 'Ellwood']\n", + "['ISIL'] --> none --> ['Sinjar']\n", + "['Minister', 'Ellwood'] --> none --> ['Tobias', 'Ellwood']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Sinjar'] --> none --> ['the', 'Government', 'of', 'Iraq']\n", + "['Sinjar'] --> none --> ['ISIL']\n", + "['Philip', 'Hammond'] --> none --> ['I']\n", + "['Kurdistan', 'Regional', 'Government'] --> none --> ['Philip', 'Hammond']\n", + "['the', 'Federal', 'Government', 'of', 'Iraq'] --> CommWith --> ['the', 'Kurdistan', 'Regional', 'Government']\n", + "['Prime', 'Minister', 'Haider', 'Al', 'Abadi'] --> none --> ['Philip', 'Hammond']\n", + "['Prime', 'Minister', 'Haider', 'Al', 'Abadi'] --> BelongsTo --> ['the', 'Federal', 'Government', 'of', 'Iraq']\n", + "['the', 'Kurdistan', 'Regional', 'Government'] --> CommWith --> ['the', 'Federal', 'Government', 'of', 'Iraq']\n", + "['Prime', 'Minister', 'Nechirvan', 'Barzani'] --> BelongsTo --> ['the', 'Kurdistan', 'Regional', 'Government']\n", + "['2', 'December'] --> none --> ['Philip', 'Hammond']\n", + "['2', 'December'] --> none --> ['I']\n", + "['government', 'forces'] --> AlliesOf --> ['their', 'allies']\n", + "['government', 'forces'] --> none --> ['their', 'fighters']\n", + "['government', 'forces'] --> none --> ['areas', 'under', 'IS', 'control']\n", + "['their', 'allies'] --> AlliesOf --> ['government', 'forces']\n", + "['the', 'civilians', 'who', 'escape'] --> none --> ['suspected', 'IS', 'fighters']\n", + "['suspected', 'IS', 'fighters'] --> none --> ['Mr', 'Colville']\n", + "['The', 'Secretary', 'of', 'State'] --> none --> ['$', '5', 'million']\n", + "['a', 'VBIED'] --> none --> ['ISIL-held', 'building']\n", + "['a', 'VBIED'] --> none --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['a', 'VBIED'] --> none --> ['ISIL-held', 'building']\n", + "['Jeff', 'Davis'] --> CoLocated --> ['the', 'room']\n", + "['the', 'room'] --> CoLocated --> ['the', 'Pentagon']\n", + "['the', 'United', 'States'] --> Likes --> ['the', 'people', 'of', 'Syria']\n", + "['a'] --> none --> ['ISIL-held', 'buildings']\n", + "['rocket', 'propelled', 'grenade', 'launcher'] --> HasAttrOf --> ['a']\n", + "['rocket', 'propelled', 'grenade', 'launcher'] --> HasAttrOf --> ['a']\n", + "['rocket', 'propelled', 'grenade', 'launcher'] --> HasAttrOf --> ['a', 'VBIED']\n", + "['rocket', 'propelled', 'grenade', 'launcher'] --> HasAttrOf --> ['a']\n", + "['a'] --> none --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['VBIED', 'staging', 'facility'] --> none --> ['a']\n", + "['a', 'VBIED'] --> none --> ['ISIL-held', 'buildings']\n", + "['a'] --> none --> ['rocket', 'propelled', 'grenade', 'launcher']\n", + "['mortar', 'team'] --> HasAttrOf --> ['a']\n", + "['mortar', 'team'] --> HasAttrOf --> ['a']\n", + "['mortar', 'team'] --> HasAttrOf --> ['a', 'VBIED']\n", + "['mortar', 'team'] --> HasAttrOf --> ['a']\n", + "['mortar', 'team'] --> none --> ['ISIL-held', 'buildings']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a', 'VBIED']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['a']\n", + "['ISIL-held', 'buildings'] --> none --> ['mortar', 'team']\n", + "['a', 'VBIED'] --> none --> ['a']\n", + "['a', 'VBIED'] --> none --> ['rocket', 'propelled', 'grenade', 'launcher']\n", + "['The', 'Syrian', 'Democratic', 'Forces'] --> none --> ['Dec.', '10']\n", + "['The', 'Syrian', 'Democratic', 'Forces'] --> none --> ['ISIL']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['Minister', 'Tobias', 'Ellwood']\n", + "['al-Hilla'] --> CoLocated --> ['Iraq']\n", + "['al-Hilla'] --> none --> ['restaurant']\n", + "['Iraq'] --> none --> ['Minister', 'Tobias', 'Ellwood']\n", + "['Minister', 'Tobias', 'Ellwood'] --> none --> ['Iraq']\n", + "['I'] --> none --> ['a', 'petrol', 'station']\n", + "['a', 'petrol', 'station'] --> CoLocated --> ['al-Hilla']\n", + "['a', 'petrol', 'station'] --> none --> ['I']\n", + "['a', 'petrol', 'station'] --> CoLocated --> ['al-Hilla']\n", + "['restaurant'] --> CoLocated --> ['al-Hilla']\n", + "['restaurant'] --> none --> ['I']\n", + "['restaurant'] --> CoLocated --> ['al-Hilla']\n", + "['al-Hilla'] --> CoLocated --> ['Iraq']\n", + "['Daesh'] --> none --> ['restaurant']\n", + "['Daesh'] --> none --> ['al-Hilla']\n", + "['a', 'petrol', 'station'] --> CoLocated --> ['al-Hilla']\n", + "['a', 'petrol', 'station'] --> CoLocated --> ['al-Hilla']\n", + "['restaurant'] --> CoLocated --> ['al-Hilla']\n", + "['restaurant'] --> none --> ['Iraq']\n", + "['restaurant'] --> CoLocated --> ['al-Hilla']\n", + "['the', 'UK'] --> BelongsTo --> ['the', 'Global', 'Coalition']\n", + "['the', 'UK'] --> BelongsTo --> ['the', 'Global', 'Coalition']\n", + "['the', 'Global', 'Coalition'] --> none --> ['the', 'UK']\n", + "['Free', 'Syrian', 'Army', 'fighters'] --> CoLocated --> ['the', 'town']\n", + "['one', 'of', 'the', 'SNCOs'] --> Likes --> ['Pittsburgh', 'Steelers']\n", + "['Isiah', 'L.', 'Booker'] --> none --> ['Jan.', '7']\n", + "['Minister', 'Tobias', 'Ellwood'] --> none --> ['UK']\n", + "['Iraq'] --> none --> ['Tikrit']\n", + "['Iraq'] --> none --> ['the', 'Government', 'of', 'Iraq']\n", + "['Government', 'of', 'Iraq'] --> none --> ['Minister', 'Tobias', 'Ellwood']\n", + "['Government', 'of', 'Iraq'] --> FightingAgainst --> ['ISIL']\n", + "['UK'] --> none --> ['Iraq']\n", + "['UK'] --> Likes --> ['Government', 'of', 'Iraq']\n", + "['the', 'Government', 'of', 'Iraq'] --> none --> ['Minister', 'Tobias', 'Ellwood']\n", + "['the', 'Syrian', 'opposition'] --> none --> ['the', 'United', 'States']\n", + "['Syrian', 'opposition'] --> none --> ['2012']\n", + "['Syrian', 'opposition'] --> none --> ['$', '500', 'million']\n", + "['Deputy', 'Secretary', 'of', 'State', 'Antony', 'Blinken'] --> none --> ['$', '100', 'million']\n", + "['Deputy', 'Secretary', 'of', 'State', 'Antony', 'Blinken'] --> none --> ['$', '500', 'million']\n", + "['Deputy', 'Secretary', 'of', 'State', 'Antony', 'Blinken'] --> CoLocated --> ['the', 'Manama', 'Dialogue']\n", + "['the', 'United', 'States'] --> Likes --> ['the', 'Syrian', 'opposition']\n", + "['the', 'United', 'States'] --> Likes --> ['the', 'Syrian', 'opposition']\n", + "['the', 'United', 'States'] --> none --> ['the', 'Manama', 'Dialogue']\n", + "['the', 'fund'] --> BelongsTo --> ['the', 'United', 'Nations']\n", + "['Ninety-Four', 'Citizen', 'Airmen'] --> BelongsTo --> ['the', '507th', 'Air', 'Refueling', 'Wing']\n", + "['Ninety-Four', 'Citizen', 'Airmen'] --> CoLocated --> ['Incirlik', 'Air', 'Base', ',', 'Turkey']\n", + "['Turkey'] --> none --> ['the', '507th', 'Air', 'Refueling', 'Wing']\n", + "['Turkey'] --> none --> ['here']\n", + "['KC-135R', 'Stratotanker'] --> none --> ['Incirlik', 'Air', 'Base', ',', 'Turkey']\n", + "['KC-135R', 'Stratotanker'] --> none --> ['Incirlik', 'Air', 'Base', ',', 'Turkey']\n", + "['the', '507th', 'Air', 'Refueling', 'Wing'] --> none --> ['Turkey']\n", + "['Incirlik', 'Air', 'Base', ',', 'Turkey'] --> none --> ['Turkey']\n", + "['Incirlik', 'Air', 'Base', ',', 'Turkey'] --> none --> ['the', '507th', 'Air', 'Refueling', 'Wing']\n", + "['mid', 'January'] --> none --> ['the', 'EU']\n", + "['two', 'months'] --> none --> ['The', 'group']\n", + "['the', '2', 'leaders'] --> CoLocated --> ['The', 'Summit']\n", + "['the', 'Euphrates', 'River', 'Valley'] --> none --> ['al', 'Qaeda', 'in', 'Iraq']\n", + "['Iraqi', 'Army'] --> CoLocated --> ['the', 'Al', 'Qa', '’', 'im', 'region']\n", + "['Iraqi', 'Army'] --> none --> ['al', 'Qaeda', 'in', 'Iraq']\n", + "['al', 'Qaeda', 'in', 'Iraq'] --> none --> ['the', 'Al', 'Qa', '’', 'im', 'region']\n", + "['civil', 'society'] --> none --> ['the', 'moderate', 'opposition']\n", + "['civil', 'society'] --> none --> ['Syrian', 'children']\n", + "['the', 'moderate', 'opposition'] --> none --> ['independent', 'media']\n", + "['30', 'November'] --> none --> ['an', 'area', 'outside', 'Damascus']\n", + "['30', 'November'] --> none --> ['Israeli', 'jets']\n", + "['Israeli', 'jets'] --> CoLocated --> ['Lebanese', 'airspace']\n", + "['a', 'VBIED'] --> CoLocated --> ['an', 'ISIL', 'tactical', 'vehicle']\n", + "['an', 'ISIL', 'tactical', 'vehicle'] --> CoLocated --> ['a', 'VBIED']\n", + "['Tharthar', 'Barrage'] --> none --> ['a', 'senior', 'Iraqi', 'general']\n", + "['April', '2015'] --> none --> ['a', 'senior', 'Iraqi', 'general']\n", + "['The', 'Islamic', 'State'] --> FightingAgainst --> ['120', 'Iraqi', 'soldiers']\n", + "['The', 'Islamic', 'State'] --> FightingAgainst --> ['a', 'senior', 'Iraqi', 'general']\n", + "['The', 'Islamic', 'State'] --> InChargeOf --> ['the', 'Tharthar', 'Barrage']\n", + "['The', 'European', 'Union'] --> none --> ['Syria']\n", + "['Syria'] --> none --> ['September']\n", + "['a', 'group'] --> CoLocated --> ['the', 'very', 'site']\n", + "['24', 'hours'] --> none --> ['the', 'oil', 'infrastructure']\n", + "['tactical', 'units'] --> none --> ['the', 'oil', 'infrastructure']\n", + "['the', 'coalition'] --> none --> ['the', 'last', '24', 'hours']\n", + "['the', 'coalition'] --> FightingAgainst --> ['tactical', 'units']\n", + "['the', 'coalition'] --> FightingAgainst --> ['the', 'oil', 'infrastructure']\n", + "['the', 'international', 'community'] --> none --> ['The', 'United', 'Nations', 'Security', 'Council']\n", + "['U.S.', 'forces'] --> CommWith --> ['Syrian', 'Democratic', 'Forces']\n", + "['Syrian', 'Democratic', 'Forces'] --> CommWith --> ['U.S.', 'forces']\n", + "['Syrian', 'Arab', 'forces'] --> none --> ['U.S.', 'forces']\n", + "['Syrian', 'Arab', 'forces'] --> CoLocated --> ['the', 'northern', 'part', 'of', 'the', 'country']\n", + "['Jordan'] --> none --> ['U.S.', 'forces']\n", + "['Syrian', 'government', 'forces'] --> CoLocated --> ['some', 'outlying', 'residential', 'areas']\n", + "['Syrian', 'government', 'forces'] --> AlliesOf --> ['their', 'Hezbollah', 'allies']\n", + "['Syrian', 'government', 'forces'] --> CoLocated --> ['all', 'the', 'farmland', 'around', 'the', 'suburb']\n", + "['their', 'Hezbollah', 'allies'] --> CoLocated --> ['some', 'outlying', 'residential', 'areas']\n", + "['their', 'Hezbollah', 'allies'] --> AlliesOf --> ['Syrian', 'government', 'forces']\n", + "['their', 'Hezbollah', 'allies'] --> CoLocated --> ['all', 'the', 'farmland', 'around', 'the', 'suburb']\n", + "['Syrian', 'Arab', 'Coalition'] --> none --> ['Syrian', 'Arab', 'Coalition']\n", + "['Syrian', 'Arab', 'Coalition'] --> none --> ['more', 'than', '18,000', 'square', 'kilometers', 'of', 'territory']\n", + "['the', 'past', '20-plus', 'months'] --> none --> ['Syrian', 'Arab', 'Coalition']\n", + "['the', 'enemy'] --> none --> ['Counter-Daesh', 'Coalition']\n", + "['the', 'enemy'] --> FightingAgainst --> ['the', 'SDF']\n", + "['the', 'SDF'] --> InChargeOf --> ['more', 'than', '18,000', 'square', 'kilometers', 'of', 'territory']\n", + "['the', 'SDF'] --> FightingAgainst --> ['the', 'enemy']\n", + "['the', 'Syrian', 'Arab', 'Coalition'] --> BelongsTo --> ['the', 'SDF']\n", + "['Karim'] --> CommWith --> ['Sadiq']\n", + "['Sadiq'] --> CommWith --> ['Karim']\n", + "['The', 'Syrian', 'Computer', 'Society'] --> CoLocated --> ['Syria']\n", + "['the', 'international', 'coalition'] --> AlliesOf --> ['the', 'Iraqi', 'Security', 'Forces']\n", + "['The', 'Arab', 'element', 'of', 'the', 'SDF'] --> CoLocated --> ['the', 'area']\n", + "['the', 'Netherlands'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'Kingdom'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'States'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'coalition', 'troops'] --> CoLocated --> ['the', 'city', 'of', 'Ubaydi']\n", + "['four', 'more', 'days'] --> none --> ['the', 'city', 'of', 'Ubaydi']\n", + "['four', 'more', 'days'] --> none --> ['the', 'city', 'of', 'Ubaydi']\n", + "['an', 'ISIL', 'tactical', 'unit'] --> CoLocated --> ['Al', 'Huwayjah']\n", + "['an', 'ISIL-held', 'building'] --> CoLocated --> ['Al', 'Huwayjah']\n", + "['Coalition', 'military', 'forces'] --> FightingAgainst --> ['ISIL']\n", + "['Coalition', 'military', 'forces'] --> CoLocated --> ['Syria']\n", + "['Coalition', 'military', 'forces'] --> InChargeOf --> ['strikes']\n", + "['ISIL'] --> CoLocated --> ['Syria']\n", + "['ISIL'] --> CoLocated --> ['Iraq']\n", + "['Syria'] --> none --> ['23']\n", + "['Iraq'] --> none --> ['Coalition', 'military', 'forces']\n", + "['Iraq'] --> none --> ['ISIL']\n", + "['strikes'] --> none --> ['Jan.', '10']\n", + "['strikes'] --> CoLocated --> ['Syria']\n", + "['the', 'Wadi', 'Barada', 'area'] --> none --> ['militias', 'like', 'Hezbollah']\n", + "['the', 'Asad', 'regime'] --> none --> ['militias', 'like', 'Hezbollah']\n", + "['militias', 'like', 'Hezbollah'] --> none --> ['the', 'Wadi', 'Barada', 'area']\n", + "['government', 'officials'] --> CoLocated --> ['Iraq']\n", + "['Later', 'in', 'the', 'day'] --> none --> ['U.S.', 'service', 'members']\n", + "['U.S.', 'service', 'members'] --> CoLocated --> ['Qayyarah', 'Airfield', 'West']\n", + "['Qayyarah', 'Airfield', 'West'] --> CoLocated --> ['northern', 'Iraq']\n", + "['Iraqi', 'government', 'forces'] --> none --> ['a', 'tribal', 'leader']\n", + "['a', 'tribal', 'leader'] --> AlliesOf --> ['Iraqi', 'government', 'forces']\n", + "['six', 'men'] --> none --> ['Iraqi', 'government', 'forces']\n", + "['six', 'men'] --> CoLocated --> ['the', 'same', 'village']\n", + "['six', 'men'] --> CoLocated --> ['a', 'vehicle']\n", + "['A', 'non-governmental', 'security', 'analyst'] --> none --> ['IS', 'fighters']\n", + "['The', 'UK'] --> Likes --> ['its', 'allies']\n", + "['international', 'partners'] --> BelongsTo --> ['the', 'UN']\n", + "['Palmyra', ',', 'Syria'] --> none --> ['The', 'CJTF-OIR', 'commanding', 'general']\n", + "['Palmyra', ',', 'Syria'] --> none --> ['other', 'military', 'hardware']\n", + "['the', 'tanks'] --> none --> ['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend']\n", + "['the', 'tanks'] --> none --> ['the', 'CJTF-OIR', 'spokesperson']\n", + "['the', 'tanks'] --> CoLocated --> ['Tiyas', 'Military', 'Airfield']\n", + "['The', 'CJTF-OIR', 'commanding', 'general'] --> none --> ['the', 'tanks']\n", + "['The', 'CJTF-OIR', 'commanding', 'general'] --> IsSynonymOf --> ['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend']\n", + "['The', 'CJTF-OIR', 'commanding', 'general'] --> InChargeOf --> ['the', 'CJTF-OIR', 'spokesperson']\n", + "['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend'] --> IsSynonymOf --> ['The', 'CJTF-OIR', 'commanding', 'general']\n", + "['Lt.', 'Gen.', 'Stephen', 'J.', 'Townsend'] --> none --> ['Tiyas', 'Military', 'Airfield']\n", + "['other', 'military', 'hardware'] --> CoLocated --> ['Tiyas', 'Military', 'Airfield']\n", + "['the', 'CJTF-OIR', 'spokesperson'] --> none --> ['the', 'tanks']\n", + "['the', 'CJTF-OIR', 'spokesperson'] --> none --> ['The', 'CJTF-OIR', 'commanding', 'general']\n", + "['Tiyas', 'Military', 'Airfield'] --> none --> ['the', 'CJTF-OIR', 'spokesperson']\n", + "['ISIL', 'fighters'] --> CoLocated --> ['a', 'building', 'on', 'the', 'hospital', 'complex']\n", + "['ISIL', 'fighters'] --> FightingAgainst --> ['Iraqi', 'forces']\n", + "['ISIL', 'fighters'] --> none --> ['the', 'Coalition']\n", + "['ISIL', 'fighters'] --> none --> ['Dec.', '7th']\n", + "['ISIL', 'fighters'] --> InChargeOf --> ['machine', 'gun']\n", + "['ISIL', 'fighters'] --> InChargeOf --> ['rocket', 'propelled', 'grenade']\n", + "['the', 'hospital', 'complex'] --> none --> ['Iraqi', 'forces']\n", + "['the', 'hospital', 'complex'] --> none --> ['machine', 'gun']\n", + "['Iraqi', 'forces'] --> none --> ['machine', 'gun']\n", + "['the', 'Coalition'] --> none --> ['ISIL', 'fighters']\n", + "['the', 'Coalition'] --> none --> ['rocket', 'propelled', 'grenade']\n", + "['machine', 'gun'] --> CoLocated --> ['a', 'building', 'on', 'the', 'hospital', 'complex']\n", + "['machine', 'gun'] --> none --> ['rocket', 'propelled', 'grenade']\n", + "['rocket', 'propelled', 'grenade'] --> CoLocated --> ['a', 'building', 'on', 'the', 'hospital', 'complex']\n", + "['Representative', 'Federica', 'Mogherini', '’', 's'] --> none --> ['Syria']\n", + "['the', 'country'] --> none --> ['This', 'week']\n", + "['the', 'country'] --> none --> ['High']\n", + "['ISIL'] --> FightingAgainst --> ['members', 'of', 'the', 'Al', 'bu', 'Nimr', 'tribe']\n", + "['70'] --> none --> ['members', 'of', 'the', 'Al', 'bu', 'Nimr', 'tribe']\n", + "['members', 'of', 'the', 'Al', 'bu', 'Nimr', 'tribe'] --> none --> ['The', 'UN']\n", + "['members', 'of', 'the', 'Al', 'bu', 'Nimr', 'tribe'] --> HasAttrOf --> ['70']\n", + "['members', 'of', 'the', 'Al', 'bu', 'Nimr', 'tribe'] --> CoLocated --> ['Anbar', ',', 'Iraq']\n", + "['Anbar', ',', 'Iraq'] --> none --> ['The', 'UN']\n", + "['The', 'United', 'States'] --> none --> ['injured']\n", + "['on'] --> none --> ['The', 'United', 'States']\n", + "['The', 'jihadist', 'group'] --> InChargeOf --> ['military', 'barracks']\n", + "['Paris'] --> none --> ['30', 'Britons']\n", + "['the', 'Middle', 'East'] --> none --> ['Sousse']\n", + "['Sousse'] --> none --> ['30', 'Britons']\n", + "['30', 'Britons'] --> CoLocated --> ['Sousse']\n", + "['13', 'November'] --> none --> ['Resolution']\n", + "['Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant'] --> none --> ['our', 'allies']\n", + "['the', 'Yazidi', 'village', 'of', \"Ba'idn\"] --> none --> ['the']\n", + "['The', 'United', 'States'] --> Dislikes --> ['ISIL']\n", + "['The', 'United', 'States'] --> none --> ['market', 'place']\n", + "['The', 'United', 'States'] --> Likes --> ['the', 'Iraqi', 'people']\n", + "['Diyala', 'Province'] --> none --> ['The', 'United', 'States']\n", + "['Monday'] --> none --> ['ISIL']\n", + "['people'] --> HasAttrOf --> ['Iraqi', 'people']\n", + "['ISIL'] --> FightingAgainst --> ['Diyala', 'Province']\n", + "['ISIL'] --> FightingAgainst --> ['market', 'place']\n", + "['ISIL'] --> FightingAgainst --> ['the', 'Iraqi', 'people']\n", + "['market', 'place'] --> CoLocated --> ['Baghdad']\n", + "['market', 'place'] --> none --> ['ISIL']\n", + "['market', 'place'] --> none --> ['the', 'Iraqi', 'people']\n", + "['the', 'Iraqi', 'people'] --> FightingAgainst --> ['ISIL']\n", + "['Iraqi', 'Security', 'Forces'] --> FightingAgainst --> ['ISIL', 'fighters']\n", + "['the', 'Al', 'Salem', 'hospital', 'complex'] --> CoLocated --> ['the', 'heart', 'of', 'East', 'Mosul']\n", + "['ISIL', 'fighters'] --> InChargeOf --> ['the', 'Al', 'Salem', 'hospital', 'complex']\n", + "['CTF', '50'] --> none --> ['Royal', 'Navy', 'assets']\n", + "['CTF', '50'] --> none --> ['early', 'next', 'year']\n", + "['CTF', '50'] --> none --> ['NAVCENT', 'CTFs']\n", + "['SDF', 'forces'] --> CoLocated --> ['areas', 'north', 'and', 'west', 'of', 'Raqqah']\n", + "['areas', 'north', 'and', 'west', 'of', 'Raqqah'] --> none --> ['7.5', 'kilometers']\n", + "['Abboud', 'Qanbar'] --> IsSynonymOf --> ['Abu', 'Haidar']\n", + "['Arabic'] --> none --> ['Iraqi']\n", + "['Abu', 'Haidar'] --> IsSynonymOf --> ['Abboud', 'Qanbar']\n", + "['U.S.', 'Secretary', 'of', 'State', 'Kerry'] --> none --> ['ISIL', 'Coalition']\n", + "['U.S.', 'Secretary', 'of', 'State', 'Kerry'] --> none --> ['U.S.', 'Secretary', 'of', 'Defense', 'Ash', 'Carter']\n", + "['U.S.', 'Secretary', 'of', 'State', 'Kerry'] --> CommWith --> ['foreign', 'and', 'defense', 'ministers']\n", + "['the', 'Global', 'Coalition', 'to', 'Counter', 'ISIL'] --> none --> ['July', '21', ',', '2016']\n", + "['ISIL', 'Coalition'] --> none --> ['July', '21', ',', '2016']\n", + "['ISIL', 'Coalition'] --> none --> ['U.S.', 'Secretary', 'of', 'Defense', 'Ash', 'Carter']\n", + "['ISIL', 'Coalition'] --> none --> ['foreign', 'and', 'defense', 'ministers']\n", + "['Washington'] --> none --> ['U.S.', 'Secretary', 'of', 'State', 'Kerry']\n", + "['Counter', 'ISIL', 'Coalition'] --> none --> ['U.S.', 'Secretary', 'of', 'Defense', 'Ash', 'Carter']\n", + "['U.S.', 'Secretary', 'of', 'Defense', 'Ash', 'Carter'] --> none --> ['ISIL', 'Coalition']\n", + "['U.S.', 'Secretary', 'of', 'Defense', 'Ash', 'Carter'] --> CommWith --> ['foreign', 'and', 'defense', 'ministers']\n", + "['foreign', 'and', 'defense', 'ministers'] --> BelongsTo --> ['the', 'Global', 'Coalition', 'to', 'Counter', 'ISIL']\n", + "['the', 'Deputy', 'Secretary'] --> BelongsTo --> ['@', 'ABlinken']\n", + "['Minister', 'for', 'the', 'Middle', 'East'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['ISIL'] --> Dislikes --> ['Yazidi']\n", + "['one', 'year'] --> none --> ['ISIL']\n", + "['Yazidi'] --> none --> ['Tobias', 'Ellwood']\n", + "['The', 'United', 'States'] --> Likes --> ['the', 'Iraqi', 'people']\n", + "['the', 'Iraqi', 'people'] --> none --> ['The', 'United', 'States']\n", + "['Special', 'Presidential', 'Envoy', 'for', 'the', 'Global', 'Coalition', 'to', 'Counter', 'ISIL', 'Brett', 'McGurk'] --> CoLocated --> ['Berlin', ',', 'Germany']\n", + "['Special', 'Presidential', 'Envoy', 'for', 'the', 'Global', 'Coalition', 'to', 'Counter', 'ISIL', 'Brett', 'McGurk'] --> InChargeOf --> ['an', 'interagency', 'delegation']\n", + "['November', '17'] --> none --> ['an', 'interagency', 'delegation']\n", + "['key', 'members', 'of', 'the', 'Counter', 'ISIL', 'Coalition'] --> CoLocated --> ['Berlin', ',', 'Germany']\n", + "['key', 'members', 'of', 'the', 'Counter', 'ISIL', 'Coalition'] --> none --> ['November', '17']\n", + "['an', 'interagency', 'delegation'] --> none --> ['Special', 'Presidential', 'Envoy', 'for', 'the', 'Global', 'Coalition', 'to', 'Counter', 'ISIL', 'Brett', 'McGurk']\n", + "['an', 'interagency', 'delegation'] --> CoLocated --> ['Berlin', ',', 'Germany']\n", + "['the', 'Government', 'of', 'Iraq'] --> none --> ['all', 'international', 'forces']\n", + "['all', 'international', 'forces'] --> none --> ['the', 'Government', 'of', 'Iraq']\n", + "['Col.', 'Travis', 'Caughlin'] --> IsSynonymOf --> ['507th', 'Maintenance', 'Group', 'Commander']\n", + "['507th', 'Maintenance', 'Group', 'Commander'] --> IsSynonymOf --> ['Col.', 'Travis', 'Caughlin']\n", + "['former', 'Iraqi', 'police', 'officers'] --> CoLocated --> ['a', 'building']\n", + "['December', '25', ',', '2011'] --> none --> ['the', 'United', 'Nations']\n", + "['the', 'camp', '’', 's', 'residents'] --> none --> ['the', 'perpetrators']\n", + "['the', 'camp', '’', 's', 'residents'] --> none --> ['the', 'Government', 'of', 'Iraq']\n", + "['The', 'westernmost', 'ripple', 'of', 'the', 'greater', 'Zagros', 'mountains'] --> IsSynonymOf --> ['the', 'Hamrin', 'mountains']\n", + "['northern', 'Salah', 'ad', 'Din', 'Province'] --> none --> ['the', 'Tigris', 'river']\n", + "['northern', 'Salah', 'ad', 'Din', 'Province'] --> none --> ['the', 'Hamrin', 'mountains']\n", + "['northern', 'Salah', 'ad', 'Din', 'Province'] --> none --> ['southern', 'Kirkuk', 'Province']\n", + "['the', 'Hamrin', 'mountains'] --> IsSynonymOf --> ['The', 'westernmost', 'ripple', 'of', 'the', 'greater', 'Zagros', 'mountains']\n", + "['the', 'Hamrin', 'mountains'] --> none --> ['the', 'Tigris', 'river']\n", + "['the', 'Hamrin', 'mountains'] --> none --> ['northern', 'Salah', 'ad', 'Din', 'Province']\n", + "['a', 'kidney', 'transplant', 'unit'] --> none --> ['a', 'physiotherapy', 'unit']\n", + "['core'] --> IsSynonymOf --> ['the']\n", + "['84', 'MW'] --> none --> ['the', 'Tigris', 'River']\n", + "['hydro-electricity', 'station'] --> none --> ['84', 'MW']\n", + "['the', 'dam'] --> none --> ['hydro-electricity', 'station']\n", + "['the', 'Tharthar', 'depression'] --> none --> ['the', 'Tigris', 'River']\n", + "['the', 'Foreign', 'Secretary'] --> CoLocated --> ['the', 'Gulf']\n", + "['the', 'Foreign', 'Secretary'] --> CoLocated --> ['Saudi', 'Arabia']\n", + "['Saudi', 'Arabia'] --> none --> ['the', 'Gulf']\n", + "['the', 'Coalition', 'military'] --> none --> ['local', 'people']\n", + "['the', 'UK'] --> Likes --> ['the', 'Iraqi', 'Army']\n", + "['the', 'UK'] --> AlliesOf --> ['the', 'Coalition', 'military']\n", + "['suicide', 'bombers'] --> none --> ['I']\n", + "['Global', 'Coalition'] --> FightingAgainst --> ['Da', '’', 'esh']\n", + "['people', 'of', 'Iraq'] --> none --> ['the', 'government']\n", + "['President', 'Hollande'] --> CoLocated --> ['Prime', 'Minister']\n", + "['Prime', 'Minister'] --> CoLocated --> ['President', 'Hollande']\n", + "['foreign', 'terrorist', 'fighter', 'network'] --> CoLocated --> ['this', 'headquarters']\n", + "['ancient', 'city'] --> none --> ['the', 'entrance', 'to', 'the', 'ancient', 'city']\n", + "['The', 'Ishtar', 'Gate'] --> CoLocated --> ['the', 'entrance', 'to', 'the', 'ancient', 'city']\n", + "['the', 'entrance', 'to', 'the', 'ancient', 'city'] --> none --> ['the', 'ancient', 'city']\n", + "['the', 'entrance', 'to', 'the', 'ancient', 'city'] --> none --> ['575', 'B.C']\n", + "['the', 'river'] --> none --> ['western', 'Mosul']\n", + "['BAGHDAD'] --> none --> ['Peshmerga']\n", + "['Combined'] --> none --> ['Joint', 'Task', 'Force', '-', 'Operation', 'Inherent']\n", + "['Peshmerga'] --> none --> ['BAGHDAD']\n", + "['Mosul'] --> none --> ['BAGHDAD']\n", + "['24,000'] --> none --> ['Mosul']\n", + "['a', 'meeting', 'place'] --> none --> ['other', 'residents']\n", + "['Karim', 'Wasifi'] --> CoLocated --> ['the', 'United', 'States']\n", + "['the', 'United', 'States'] --> none --> ['Karim', 'Wasifi']\n", + "['Tobias', 'Ellwood'] --> IsSynonymOf --> ['The', 'Minister', 'for', 'Central', 'Asia']\n", + "['The', 'Minister', 'for', 'Central', 'Asia'] --> IsSynonymOf --> ['Tobias', 'Ellwood']\n", + "['Operation', 'Inherent', 'Resolve'] --> FightingAgainst --> ['the', 'ISIL', 'terrorist', 'group']\n", + "['airstrike'] --> FightingAgainst --> ['Abu', 'Jandal', 'al-Kuwaiti']\n", + "['Tabqa', 'Dam', ',', 'Syria'] --> none --> ['Raqqah']\n", + "['Abu', 'Jandal', 'al-Kuwaiti'] --> CoLocated --> ['Tabqa', 'Dam', ',', 'Syria']\n", + "['Abu', 'Jandal', 'al-Kuwaiti'] --> BelongsTo --> ['ISIL']\n", + "['Abu', 'Jandal', 'al-Kuwaiti'] --> CoLocated --> ['Raqqah']\n", + "['ISIL'] --> none --> ['Abu', 'Jandal', 'al-Kuwaiti']\n", + "['ISIL'] --> none --> ['Raqqah']\n", + "['Raqqah'] --> none --> ['Tabqa', 'Dam', ',', 'Syria']\n", + "['European', 'Union'] --> none --> ['other', 'international', 'players']\n", + "['other', 'international', 'players'] --> none --> ['European', 'Union']\n", + "['other', 'international', 'players'] --> none --> ['I']\n", + "['I'] --> CommWith --> ['Turkish', 'Foreign', 'Minister', 'Mevlüt', 'Çavuşoğlu']\n", + "['I'] --> CommWith --> ['the', 'UN', 'Special', 'Envoy', 'Staffan', 'de', 'Mistura']\n", + "['Turkish', 'Foreign', 'Minister', 'Mevlüt', 'Çavuşoğlu'] --> CommWith --> ['I']\n", + "['the', 'UN', 'Special', 'Envoy', 'Staffan', 'de', 'Mistura'] --> CommWith --> ['I']\n", + "['the', 'UN', 'Special', 'Envoy', 'Staffan', 'de', 'Mistura'] --> none --> ['Turkish', 'Foreign', 'Minister', 'Mevlüt', 'Çavuşoğlu']\n", + "['a', 'spokesman'] --> none --> ['the', 'Associated', 'Press']\n", + "['the', 'Royal', 'Navy'] --> InChargeOf --> ['the', 'task', 'force']\n", + "['FS', 'Forbin'] --> BelongsTo --> ['the', 'task', 'force']\n", + "['the', 'task', 'force'] --> none --> ['FS', 'Forbin']\n", + "['the', 'task', 'force'] --> none --> ['a', 'French', 'unit']\n", + "['Maj.', 'Gen.', 'Gary', 'J.', 'Volesky'] --> InChargeOf --> ['Combined', 'Joint', 'Forces', 'Land', 'Component', 'Command']\n", + "['Combined', 'Joint', 'Forces', 'Land', 'Component', 'Command'] --> none --> ['chemical', 'weapons']\n", + "['His', 'Chief', 'of', 'Staff'] --> IsSynonymOf --> ['Major', 'General', 'Hassan']\n", + "['His', 'Chief', 'of', 'Staff'] --> none --> ['his', 'Chief', 'of', 'Plans']\n", + "['His', 'Chief', 'of', 'Staff'] --> none --> ['Staff', 'Colonel', 'Abd', 'Alamir']\n", + "['his', 'Chief', 'of', 'Plans'] --> IsSynonymOf --> ['Staff', 'Colonel', 'Abd', 'Alamir']\n", + "['this', 'terrorist', 'group'] --> none --> ['Iraqi', 'security', 'forces']\n", + "['Iraqi', 'security', 'forces'] --> FightingAgainst --> ['this', 'terrorist', 'group']\n", + "['The', 'victims'] --> none --> ['one', 'child']\n", + "['one', 'child'] --> BelongsTo --> ['The', 'victims']\n", + "['I'] --> Likes --> ['Kurdish', 'and', 'Yezidi', 'fighters']\n", + "['the', 'end', 'of', 'last', 'year'] --> none --> ['EU', 'leaders']\n", + "['all', 'parties'] --> none --> ['other', 'terrorists']\n", + "['35°01′57″N', '43°38′47″E'] --> none --> ['35.0325', ';', '43.6463889']\n", + "['Abu', 'Ghadiya'] --> InChargeOf --> ['AQI', \"'s\", 'logistics']\n", + "['Abu', 'Ghadiya'] --> none --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['Abu', 'Ghadiya'] --> IsSynonymOf --> ['the', 'lead', 'Syrian', 'commander', 'for', 'AQI', \"'s\", 'logistics']\n", + "['the', 'United', 'States', 'Treasury', 'Department'] --> none --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['Abu', 'Musab', 'al-Zarqawi'] --> none --> ['the', 'United', 'States', 'Treasury', 'Department']\n", + "['Abu', 'Musab', 'al-Zarqawi'] --> InChargeOf --> ['Abu', 'Ghadiya']\n", + "['the', 'lead', 'Syrian', 'commander', 'for', 'AQI', \"'s\", 'logistics'] --> IsSynonymOf --> ['Abu', 'Ghadiya']\n", + "['the', 'lead', 'Syrian', 'commander', 'for', 'AQI', \"'s\", 'logistics'] --> none --> ['AQI', \"'s\", 'logistics']\n", + "['the', 'lead', 'Syrian', 'commander', 'for', 'AQI', \"'s\", 'logistics'] --> none --> ['Abu', 'Musab', 'al-Zarqawi']\n", + "['Iraqi'] --> none --> ['international', 'forces']\n", + "['Coalition', 'aircraft'] --> none --> ['Palmyra', ',', 'Syria']\n", + "['Thursday', ',', 'Dec.', '15'] --> none --> ['Palmyra', ',', 'Syria']\n", + "['Thursday', ',', 'Dec.', '15'] --> none --> ['ISIL', 'tanks']\n", + "['CJTF-OIR'] --> InChargeOf --> ['Coalition', 'aircraft']\n", + "['14'] --> none --> ['ISIL', 'tanks']\n", + "['ISIL', 'tanks'] --> none --> ['Palmyra', ',', 'Syria']\n", + "['ISIL', 'tanks'] --> none --> ['Palmyra', ',', 'Syria']\n", + "['ISIL', 'tanks'] --> none --> ['Coalition', 'aircraft']\n", + "['ISIL', 'tanks'] --> CoLocated --> ['Palmyra', ',', 'Syria']\n", + "['ISIL', 'tanks'] --> none --> ['CJTF-OIR']\n", + "['ISIL', 'tanks'] --> HasAttrOf --> ['14']\n", + "['a', 'team', 'of', 'Iraqi', 'preservationists'] --> none --> ['The', 'U.S.', 'Embassy']\n", + "['a', 'team', 'of', 'Iraqi', 'preservationists'] --> BelongsTo --> ['the', 'SBAH']\n", + "['I'] --> Likes --> ['the', 'Syrians']\n", + "['the', 'Netherlands'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'Netherlands'] --> none --> ['Saudi', 'Arabia']\n", + "['Saudi', 'Arabia'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['United', 'Arab', 'Emirates'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['United', 'Arab', 'Emirates'] --> none --> ['the', 'United', 'Kingdom']\n", + "['United', 'Arab', 'Emirates'] --> none --> ['the', 'United', 'States']\n", + "['the', 'United', 'Kingdom'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'States'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['The', 'Coalition'] --> Likes --> ['the', 'SDF']\n", + "['12.245', 'MHz'] --> none --> ['an', 'Arab', 'audience']\n", + "['an', 'Arab', 'audience'] --> none --> ['the', 'Americas']\n", + "['I'] --> CoLocated --> ['the', 'front', 'line']\n", + "['I'] --> CoLocated --> ['the', 'others']\n", + "['I'] --> CoLocated --> ['the', 'front', 'line']\n", + "['I'] --> CoLocated --> ['the', 'others']\n", + "['the', 'others'] --> CoLocated --> ['the', 'front', 'line']\n", + "['the', 'others'] --> CoLocated --> ['I']\n", + "['the', 'others'] --> CoLocated --> ['I']\n", + "['Ambassador', 'Robert', 'Ford'] --> CoLocated --> ['Washington']\n", + "['The', 'Yazidi', 'community'] --> CoLocated --> ['Mount', 'Sinjar']\n", + "['The', 'Yazidi', 'community'] --> Apart --> ['their', 'homes']\n", + "['his', 'friends'] --> CoLocated --> ['the', 'devastated', 'streets']\n", + "['allies', 'and', 'partners'] --> CommWith --> ['the', 'Coalition']\n", + "['the', 'Coalition'] --> none --> ['the', 'city']\n", + "['the', 'Coalition'] --> CommWith --> ['allies', 'and', 'partners']\n", + "['a', 'front-end', 'loader'] --> CoLocated --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['an', 'ISIL', 'tactical', 'unit'] --> none --> ['an', 'ISIL-held', 'building']\n", + "['an', 'ISIL-held', 'building'] --> CoLocated --> ['an', 'ISIL', 'tactical', 'unit']\n", + "['President', 'Bashar', 'al-Assad', \"'s\", 'forces'] --> none --> ['President', 'Bashar', 'al-Assad', \"'s\", 'forces']\n", + "['thousands', 'of', 'fighters'] --> none --> ['President', 'Bashar', 'al-Assad', \"'s\", 'forces']\n", + "['thousands', 'of', 'fighters'] --> AlliesOf --> ['President', 'Bashar', 'al-Assad', \"'s\", 'forces']\n", + "['one', 'hostage'] --> BelongsTo --> ['11', 'people']\n", + "['I'] --> CommWith --> ['Foreign', 'Minister', 'Al-Jubeir']\n", + "['Saudi', 'Arabia', '’', 's', 'leadership'] --> none --> ['Saudi', 'Arabia', '’', 's', 'leadership']\n", + "['State', 'television'] --> none --> ['the', 'army']\n", + "['Reuters', 'news', 'agency'] --> none --> ['the', 'army']\n", + "['the', 'army'] --> CommWith --> ['State', 'television']\n", + "['the', 'airport', 'compound'] --> none --> ['the', 'army']\n", + "['Joseph', 'Votel'] --> IsSynonymOf --> ['commander', 'of', 'U.S.', 'Central', 'Command']\n", + "['commander', 'of', 'U.S.', 'Central', 'Command'] --> IsSynonymOf --> ['Joseph', 'Votel']\n", + "['Air', 'Force', 'F-16', 'Fighting', 'Falcons'] --> FightingAgainst --> ['the', 'Islamic', 'State', 'of', 'Iraq', 'and', 'the', 'Levant']\n", + "['Air', 'Force', 'F-16', 'Fighting', 'Falcons'] --> BelongsTo --> ['the', 'Vermont', 'Air', 'National', 'Guard']\n", + "['Air', 'Force', 'F-16', 'Fighting', 'Falcons'] --> CoLocated --> ['here']\n", + "['the', 'Vermont', 'Air', 'National', 'Guard'] --> none --> ['Dec.', '10', ',', '2016']\n", + "['The', 'Members', 'of', 'the', 'Security', 'Council'] --> Likes --> ['Iraqi', 'President', 'Fuad', 'Masum']\n", + "['Coalition', 'military', 'forces'] --> InChargeOf --> ['strikes']\n", + "['Coalition', 'military', 'forces'] --> FightingAgainst --> ['ISIL', 'terrorists']\n", + "['Coalition', 'military', 'forces'] --> CoLocated --> ['Syria']\n", + "['Coalition', 'military', 'forces'] --> CoLocated --> ['Iraq']\n", + "['Jan.', '8'] --> none --> ['strikes']\n", + "['34'] --> none --> ['Coalition', 'military', 'forces']\n", + "['strikes'] --> HasAttrOf --> ['34']\n", + "['strikes'] --> CoLocated --> ['Syria']\n", + "['strikes'] --> CoLocated --> ['Iraq']\n", + "['ISIL', 'terrorists'] --> none --> ['strikes']\n", + "['ISIL', 'terrorists'] --> CoLocated --> ['Syria']\n", + "['ISIL', 'terrorists'] --> CoLocated --> ['Iraq']\n", + "['Syria'] --> none --> ['Jan.', '8']\n", + "['Iraq'] --> none --> ['ISIL', 'terrorists']\n", + "['Philip', 'Hammond'] --> none --> ['Syrian', 'regime', 'aircrew']\n", + "['Philip', 'Hammond'] --> none --> ['barrel', 'bombs']\n", + "['The', 'Foreign', 'Secretary', 'Philip', 'Hammond'] --> none --> ['Assad', '’', 's', 'regime']\n", + "['The', 'Foreign', 'Secretary', 'Philip', 'Hammond'] --> Dislikes --> ['Assad', '’', 's', 'regime']\n", + "['The', 'Foreign', 'Secretary', 'Philip', 'Hammond'] --> none --> ['Syria']\n", + "['The', 'Foreign', 'Secretary', 'Philip', 'Hammond'] --> none --> ['civilians']\n", + "['I'] --> none --> ['Assad', '’', 's', 'regime']\n", + "['I'] --> none --> ['helicopters']\n", + "['Assad', '’', 's', 'regime'] --> none --> ['barrel', 'bombs']\n", + "['Assad', '’', 's', 'regime'] --> none --> ['Philip', 'Hammond']\n", + "['Syrian', 'regime', 'aircrew'] --> none --> ['Syrian', 'regime', 'aircrew']\n", + "['Syria'] --> none --> ['Philip', 'Hammond']\n", + "['Syria'] --> none --> ['barrel', 'bombs']\n", + "['Syria'] --> none --> ['barrel', 'bomb']\n", + "['civilians'] --> none --> ['barrel', 'bombs']\n", + "['Abu', 'Ghadiya'] --> CoLocated --> ['the', 'Syrian', 'side', 'of', 'the', 'Iraqi', 'border']\n", + "['Local', 'Arab', 'fighters'] --> BelongsTo --> ['the', 'SDF']\n", + "['their', 'own', 'land'] --> none --> ['Local', 'Arab', 'fighters']\n", + "['United', 'States'] --> none --> ['the', 'United', 'States']\n", + "['an', 'ISIL', 'tactical', 'unit'] --> none --> ['a', 'fighting', 'position']\n", + "['a', 'fighting', 'position'] --> none --> ['a', 'fighting', 'position']\n", + "['a', 'fighting', 'position'] --> CoLocated --> ['Al', 'Shadaddi']\n", + "['Iraqi', 'forces'] --> InChargeOf --> ['Mosul', 'University']\n", + "['Iraqi', 'forces'] --> FightingAgainst --> ['so-called', 'Islamic', 'State']\n", + "['so-called', 'Islamic', 'State'] --> FightingAgainst --> ['Iraqi', 'forces']\n", + "['Secretary', 'Kerry'] --> CommWith --> ['Russian', 'Foreign', 'Minister', 'Lavrov']\n", + "['the', 'regime'] --> FightingAgainst --> ['opposition', 'forces']\n", + "['the', 'regime'] --> FightingAgainst --> ['innocent', 'civilians']\n", + "['the', 'regime'] --> none --> ['Russian', 'Foreign', 'Minister', 'Lavrov']\n", + "['innocent', 'civilians'] --> CoLocated --> ['Damascus', 'suburbs']\n", + "['Russian', 'Foreign', 'Minister', 'Lavrov'] --> CommWith --> ['Secretary', 'Kerry']\n", + "['Zahid', 'Mohammed', 'Rawi'] --> none --> ['A', 'local', 'physician']\n", + "['Zahid', 'Mohammed', 'Rawi'] --> none --> ['the', 'first', 'week']\n", + "['Lama', 'Fakih'] --> IsSynonymOf --> ['HRW', \"'s\", 'deputy', 'Middle', 'East', 'director']\n", + "['HRW', \"'s\", 'deputy', 'Middle', 'East', 'director'] --> IsSynonymOf --> ['Lama', 'Fakih']\n", + "['armed', 'forces'] --> none --> ['Lama', 'Fakih']\n", + "['Col.', 'David', 'C.', 'Lyons'] --> IsSynonymOf --> ['407th', 'AEG', 'commander']\n", + "['407th', 'AEG', 'commander'] --> IsSynonymOf --> ['Col.', 'David', 'C.', 'Lyons']\n", + "['Central', 'Asia'] --> none --> ['Minister', 'for', 'Central', 'Asia']\n", + "['Minister', 'for', 'Central', 'Asia'] --> IsSynonymOf --> ['Mr', 'Ellwood']\n", + "['Mr', 'Ellwood'] --> CommWith --> ['a', 'number', 'of', 'Ministers', 'and', 'senior', 'officials']\n", + "['a', 'number', 'of', 'Ministers', 'and', 'senior', 'officials'] --> none --> ['Minister', 'for', 'Central', 'Asia']\n", + "['a', 'number', 'of', 'Ministers', 'and', 'senior', 'officials'] --> CommWith --> ['Mr', 'Ellwood']\n", + "['people'] --> none --> ['the', 'rebel-held', 'Syrian', 'town', 'of', 'Azaz']\n", + "['people'] --> HasAttrOf --> ['43']\n", + "['car', 'bomb'] --> CoLocated --> ['the', 'rebel-held', 'Syrian', 'town', 'of', 'Azaz']\n", + "['the', 'rebel-held', 'Syrian', 'town', 'of', 'Azaz'] --> none --> ['43']\n", + "['43'] --> none --> ['the', 'Turkish', 'border']\n", + "['Foreign', 'Minister', 'Julie', 'Bishop'] --> CoLocated --> ['Canberra']\n", + "['Foreign', 'Minister', 'Julie', 'Bishop'] --> CommWith --> ['Minister', 'of', 'Foreign', 'Affairs', 'of', 'Iraq']\n", + "['Canberra'] --> none --> ['ISIL']\n", + "['12', 'February'] --> none --> ['Daesh']\n", + "['Daesh'] --> IsSynonymOf --> ['ISIL']\n", + "['ISIL'] --> IsSynonymOf --> ['Daesh']\n", + "['Minister', 'of', 'Foreign', 'Affairs', 'of', 'Iraq'] --> CommWith --> ['Foreign', 'Minister', 'Julie', 'Bishop']\n", + "['Minister', 'of', 'Foreign', 'Affairs', 'of', 'Iraq'] --> none --> ['Daesh']\n", + "['Minister', 'of', 'Foreign', 'Affairs', 'of', 'Iraq'] --> IsSynonymOf --> ['HE', 'Dr', 'Ibrahim', 'Al-Ja', '’', 'afari']\n", + "['HE', 'Dr', 'Ibrahim', 'Al-Ja', '’', 'afari'] --> CoLocated --> ['Canberra']\n", + "['HE', 'Dr', 'Ibrahim', 'Al-Ja', '’', 'afari'] --> IsSynonymOf --> ['Minister', 'of', 'Foreign', 'Affairs', 'of', 'Iraq']\n", + "['Sadiq', \"'s\", 'family', \"'s\", 'store'] --> InChargeOf --> ['Sadiq', \"'s\", 'family', \"'s\", 'store']\n", + "['The', 'UK'] --> BelongsTo --> ['the', 'Global', 'Coalition']\n", + "['The', 'UK'] --> Likes --> ['the', 'government', 'of', 'Iraq']\n", + "['the', 'Global', 'Coalition'] --> Likes --> ['the', 'government', 'of', 'Iraq']\n", + "['Foreign', 'Secretary', 'Boris', 'Johnson'] --> none --> ['eastern', 'Aleppo']\n", + "['the', 'UN'] --> Apart --> ['eastern', 'Aleppo']\n", + "['the', 'Foreign', 'Secretary', 'Boris', 'Johnson'] --> none --> ['eastern', 'Aleppo']\n", + "['Iraqi', 'security', 'forces'] --> none --> ['January', '13', ',', '2007']\n", + "['Iraqi', 'security', 'forces'] --> none --> ['the', 'Iraqi', 'commander']\n", + "['the', 'Iraqi', 'commander'] --> InChargeOf --> ['the', 'Baghdad', 'Operational', 'Command']\n", + "['the', 'Baghdad', 'Operational', 'Command'] --> InChargeOf --> ['Iraqi', 'security', 'forces']\n", + "['Gen.'] --> none --> ['Coe', '’', 's']\n", + "['Col.', 'John', 'J.', 'Thomas'] --> none --> ['Central', 'Command', 'spokesman']\n", + "['the', 'UN'] --> Apart --> ['eastern', 'Aleppo']\n", + "['a', 'courthouse'] --> CoLocated --> ['the', 'town']\n", + "['a', 'courthouse'] --> none --> ['the', 'Turkish', 'frontier']\n", + "['the', 'town'] --> none --> ['four', 'miles']\n", + "['Marines', 'from', '3rd', 'Battalion', ',', '6th', 'Marines', 'and', '2nd', 'Battalion', ',', '1st', 'Marines'] --> FightingAgainst --> ['insurgent-held', 'Karabilah']\n", + "['the', 'river', 'Tigris'] --> none --> ['key', 'bridges', 'crossing', 'the', 'river', 'Tigris']\n", + "['Foreign', 'Secretary', 'Boris', 'Johnson'] --> none --> ['19', 'September']\n", + "['Foreign', 'Secretary', 'Boris', 'Johnson'] --> Likes --> ['UN']\n", + "['an', 'aid', 'convoy'] --> CoLocated --> ['Urem', 'al-Kubra', ',', 'northern', 'Syria']\n", + "['Urem', 'al-Kubra', ',', 'northern', 'Syria'] --> none --> ['UN']\n", + "['19', 'September'] --> none --> ['UN']\n", + "['UN'] --> none --> ['Foreign', 'Secretary', 'Boris', 'Johnson']\n", + "['UN'] --> none --> ['Urem', 'al-Kubra', ',', 'northern', 'Syria']\n", + "['UN'] --> none --> ['UN']\n", + "['developers', ',', 'proliferators', 'and', 'users'] --> InChargeOf --> ['chemical', 'weapons']\n", + "['his', 'spiritual', 'adviser'] --> CoLocated --> ['the', 'safehouse']\n", + "['U.S.-led', 'coalition', 'aircraft'] --> FightingAgainst --> ['the', 'safehouse']\n", + "['U.S.-led', 'coalition', 'aircraft'] --> FightingAgainst --> ['his', 'spiritual', 'adviser']\n", + "['The', 'Minister'] --> CommWith --> ['British', 'businesses']\n", + "['The', 'Minister'] --> CommWith --> ['Uzbek', 'alumni', 'of', 'British', 'universities']\n", + "['British', 'businesses'] --> CommWith --> ['The', 'Minister']\n", + "['British', 'businesses'] --> none --> ['Westminster', 'International', 'University']\n", + "['British', 'businesses'] --> none --> ['Uzbek', 'alumni', 'of', 'British', 'universities']\n", + "['the', 'SRTF'] --> none --> ['our', 'international']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['Syria']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['ISIL']\n", + "['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood'] --> none --> ['Palmyra']\n", + "['ISIL'] --> none --> ['Minister', 'for', 'the', 'Middle', 'East', 'Tobias', 'Ellwood']\n", + "['ISIL'] --> CoLocated --> ['the', 'ancient', 'site', 'of', 'Palmyra']\n", + "['ISIL'] --> none --> ['all', 'Syrians']\n", + "['ISIL'] --> none --> ['Palmyra']\n", + "['Syria'] --> none --> ['ISIL', 'forces']\n", + "['Syria'] --> none --> ['the', 'UNESCO', 'World', 'Heritage', 'Site', 'at', 'Palmyra']\n", + "['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood'] --> Dislikes --> ['ISIL']\n", + "['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood'] --> none --> ['ISIL', 'forces']\n", + "['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood'] --> Dislikes --> ['ISIL']\n", + "['ISIL'] --> CoLocated --> ['the', 'ancient', 'site', 'of', 'Palmyra']\n", + "['all', 'Syrians'] --> none --> ['ISIL', 'forces']\n", + "['all', 'Syrians'] --> none --> ['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood']\n", + "['Palmyra'] --> none --> ['Foreign', 'Office', 'Minister', 'Tobias', 'Ellwood']\n", + "['2nd', 'Battalion'] --> BelongsTo --> ['5th', 'Special', 'Forces', 'Group']\n", + "['5th', 'Special', 'Forces', 'Group'] --> CoLocated --> ['Fort', 'Campbell', ',', 'Kentucky']\n", + "['Abu', 'Ghadiya'] --> BelongsTo --> ['AQI']\n", + "['1977-1979'] --> none --> ['Arabic']\n", + "['Arabic'] --> none --> ['Abu', 'Ghadiya']\n", + "['Mosul'] --> none --> ['Sunni']\n", + "['Mosul'] --> none --> ['al-Qaeda', 'in', 'Iraq']\n", + "['Mosul'] --> none --> ['al-Qaeda', 'in', 'Iraq']\n", + "['Mosul'] --> none --> ['Badran', 'Turki', 'Hishan', 'al-Mazidi']\n", + "['al-Qaeda', 'in', 'Iraq'] --> IsSynonymOf --> ['AQI']\n", + "['AQI'] --> none --> ['al-Qaeda', 'in', 'Iraq']\n", + "['Badran', 'Turki', 'Hishan', 'al-Mazidi'] --> none --> ['Mosul']\n", + "['Arabic'] --> none --> ['al-Qaeda', 'in', 'Iraq']\n", + "['HMS', 'Daring'] --> none --> ['DDG', '72']\n", + "['HMS', 'Daring'] --> BelongsTo --> ['U.K.', 'ships']\n", + "['HMS', 'Daring'] --> none --> ['HMS', 'Ocean']\n", + "['D', '32'] --> IsSynonymOf --> ['HMS', 'Daring']\n", + "['U.S.', 'ships'] --> none --> ['HMS', 'Daring']\n", + "['U.S.', 'ships'] --> none --> ['USS', 'Hopper']\n", + "['U.S.', 'ships'] --> CoLocated --> ['U.K.', 'ships']\n", + "['U.S.', 'ships'] --> none --> ['L', '12']\n", + "['USS', 'Hopper'] --> BelongsTo --> ['U.S.', 'ships']\n", + "['USS', 'Hopper'] --> none --> ['DDG', '72']\n", + "['DDG', '70'] --> IsSynonymOf --> ['USS', 'Hopper']\n", + "['DDG', '70'] --> none --> ['DDG', '72']\n", + "['USS', 'Mahan'] --> BelongsTo --> ['U.S.', 'ships']\n", + "['DDG', '72'] --> none --> ['DDG', '70']\n", + "['DDG', '72'] --> IsSynonymOf --> ['USS', 'Mahan']\n", + "['U.K.', 'ships'] --> CoLocated --> ['U.S.', 'ships']\n", + "['U.K.', 'ships'] --> none --> ['DDG', '70']\n", + "['HMS', 'Ocean'] --> BelongsTo --> ['U.K.', 'ships']\n", + "['L', '12'] --> none --> ['U.S.', 'ships']\n", + "['L', '12'] --> IsSynonymOf --> ['HMS', 'Ocean']\n", + "['a', 'large', 'contingent', 'of', 'the', 'wing'] --> none --> ['the', 'summer', 'of', '2017']\n", + "['Coalition', 'nations'] --> none --> ['the', 'United', 'States']\n", + "['the', 'Netherlands'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'Kingdom'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['the', 'United', 'States'] --> BelongsTo --> ['Coalition', 'nations']\n", + "['assembly', 'hall'] --> CoLocated --> ['The', 'cultural', 'center']\n", + "['UN'] --> none --> ['Gareth', 'Bayley']\n", + "['Gareth', 'Bayley'] --> IsSynonymOf --> ['the', 'UK', 'Special', 'Representative', 'for', 'Syria']\n", + "['the', 'UK', 'Special', 'Representative', 'for', 'Syria'] --> IsSynonymOf --> ['Gareth', 'Bayley']\n", + "['an', 'ISIL-held', 'building'] --> CoLocated --> ['Tal', 'Afar']\n", + "['UAV', 'construction', 'facilities'] --> CoLocated --> ['Tal', 'Afar']\n", + "['operations', 'group'] --> BelongsTo --> ['The', 'group', 'of', 'Reserve', 'Airmen']\n", + "['maintenance', 'group'] --> BelongsTo --> ['The', 'group', 'of', 'Reserve', 'Airmen']\n", + "['the', 'UN'] --> Apart --> ['eastern', 'Aleppo']\n", + "['eastern', 'Aleppo'] --> none --> ['The']\n", + "['pro-Assad', 'forces'] --> FightingAgainst --> ['the', 'front', 'line']\n", + "['Anas', 'Ahmad'] --> BelongsTo --> ['the', 'founders']\n", + "['a', 'former', 'civil', 'engineering', 'student'] --> none --> ['Anas', 'Ahmad']\n", + "['the', 'heroes'] --> InChargeOf --> ['This', 'great', 'scientific', 'and', 'cultural', 'building']\n", + "['religious', 'and', 'civic', 'leaders'] --> none --> ['the', 'Lebanese', 'religious', 'communities']\n", + "['Philip', 'Hammond'] --> none --> ['Syrian', 'government']\n", + "['Philip', 'Hammond'] --> none --> ['UN']\n", + "['Foreign', 'Secretary', 'Philip', 'Hammond'] --> none --> ['UN']\n", + "['UN'] --> none --> ['Philip', 'Hammond']\n", + "['UN'] --> none --> ['Syrian', 'government']\n", + "['the', 'victims'] --> none --> ['the', 'families', 'of', 'the', 'victims']\n", + "['the', 'families', 'of', 'the', 'victims'] --> none --> ['I']\n", + "['Camp', 'Hurriya'] --> none --> ['the', 'residents', 'of', 'Camp', 'Hurriya']\n", + "['the', 'residents', 'of', 'Camp', 'Hurriya'] --> CoLocated --> ['Camp', 'Hurriya']\n", + "['U.S.', 'service', 'members'] --> CoLocated --> ['Syria']\n", + "['graduate', 'doctors'] --> BelongsTo --> ['Syrian', 'universities']\n", + "['the', 'village', 'of', 'Safina'] --> none --> ['IS', 'fighters']\n", + "['the', 'village', 'of', 'Safina'] --> none --> ['15', 'civilians']\n", + "['Mr', 'Colville'] --> none --> ['28', 'miles']\n", + "['IS', 'fighters'] --> FightingAgainst --> ['15', 'civilians']\n", + "['the', 'international', 'community'] --> none --> ['The', 'United', 'States']\n", + "['tactical', 'units'] --> none --> ['ISIL-held', 'buildings']\n", + "['tactical', 'units'] --> none --> ['a', 'land', 'bridge']\n", + "['ISIL-held', 'buildings'] --> none --> ['ISIL', 'tactical', 'units']\n", + "['vehicle-borne', 'homemade', 'bombs'] --> none --> ['tactical', 'units']\n", + "['vehicle-borne', 'homemade', 'bombs'] --> none --> ['ISIL', 'vehicles']\n", + "['front-end', 'loaders'] --> none --> ['a', 'land', 'bridge']\n", + "['a', 'land', 'bridge'] --> none --> ['a', 'tunnel']\n", + "['a', 'supply', 'cache'] --> none --> ['tactical', 'units']\n", + "['a', 'supply', 'cache'] --> none --> ['rocket-propelled', 'grenades']\n", + "['supply', 'routes'] --> none --> ['ISIL', 'vehicles']\n", + "['ISIL', 'tactical', 'units'] --> none --> ['supply', 'routes']\n", + "['ISIL', 'vehicles'] --> none --> ['a', 'supply', 'cache']\n", + "['President', 'Obama'] --> none --> ['you']\n", + "['October', '29'] --> none --> ['93rd', 'anniversary']\n", + "['the', 'American', 'people'] --> none --> ['you']\n", + "['I'] --> none --> ['you']\n", + "['you'] --> none --> ['93rd', 'anniversary']\n", + "['you'] --> none --> ['President', 'Obama']\n", + "['May', '2006'] --> none --> ['October', '2005']\n", + "['October', '2005'] --> none --> ['the', 'tribunal']\n", + "['Scott', 'C.', 'Dayton'] --> none --> ['Targeted', 'Operations', 'Against', 'ISIL', 'Terrorists']\n", + "['Nov.', '24'] --> none --> ['northern', 'Syria']\n", + "['improvised', 'explosive', 'device'] --> none --> ['Operation', 'Inherent', 'Resolve']\n", + "['Targeted', 'Operations', 'Against', 'ISIL', 'Terrorists'] --> none --> ['Nov.', '24']\n", + "['Navy', 'Senior', 'Chief', 'Petty', 'Officer', 'Scott', 'C.', 'Dayton'] --> CoLocated --> ['Woodbridge', ',', 'Virginia']\n", + "['Navy', 'Senior', 'Chief', 'Petty', 'Officer', 'Scott', 'C.', 'Dayton'] --> CoLocated --> ['northern', 'Syria']\n", + "['The', 'Foreign', 'Secretary'] --> none --> ['Asad', 'regime']\n", + "['The', 'Foreign', 'Secretary'] --> Likes --> ['the']\n", + "['I'] --> none --> ['the']\n", + "['the'] --> none --> ['Asad', 'regime']\n", + "['the', 'aid', 'convoy'] --> none --> ['the']\n", + "['FS', 'Charles', 'de', 'Gaulle'] --> BelongsTo --> ['The', 'French']\n", + "['Samarra', 'Barrage'] --> none --> ['the', 'Tharthar', 'regulator']\n", + "['Samarra', 'Barrage'] --> none --> ['317,832', 'cu', 'ft/s']\n", + "['The', 'Samarra', 'Barrage'] --> none --> ['the', 'Tharthar', 'regulator']\n", + "['The', 'Samarra', 'Barrage'] --> CoLocated --> ['the', 'structure']\n", + "['the', 'Tigris'] --> none --> ['The', 'Samarra', 'Barrage']\n", + "['the', 'Tigris'] --> none --> ['7,000', 'm3/s']\n", + "['the', 'Tharthar', 'regulator'] --> none --> ['247,203', 'cu', 'ft/s']\n", + "['9,000', 'm3/s'] --> IsSynonymOf --> ['317,832', 'cu', 'ft/s']\n", + "['317,832', 'cu', 'ft/s'] --> none --> ['the', 'structure']\n", + "['36', 'gates'] --> CoLocated --> ['the', 'Tharthar', 'regulator']\n", + "['36', 'gates'] --> none --> ['317,832', 'cu', 'ft/s']\n", + "['36', 'gates'] --> none --> ['a', 'canal']\n", + "['17', 'gates'] --> CoLocated --> ['The', 'Samarra', 'Barrage']\n", + "['7,000', 'm3/s'] --> none --> ['Samarra', 'Barrage']\n", + "['7,000', 'm3/s'] --> IsSynonymOf --> ['247,203', 'cu', 'ft/s']\n", + "['the', 'Iraqi', 'Foreign', 'Minister'] --> none --> ['A', 'Foreign', 'Office', 'spokesman']\n", + "['the', 'Iraqi', 'Foreign', 'Minister'] --> IsSynonymOf --> ['Ibrahim', 'al-Jaafari']\n", + "['The', 'Foreign', 'Secretary'] --> CommWith --> ['Ibrahim', 'al-Jaafari']\n", + "['Ibrahim', 'al-Jaafari'] --> IsSynonymOf --> ['the', 'Iraqi', 'Foreign', 'Minister']\n", + "['Ibrahim', 'al-Jaafari'] --> CommWith --> ['The', 'Foreign', 'Secretary']\n", + "['The', 'Brandbergen', 'Mosque'] --> IsSynonymOf --> ['the', 'Islamic', 'Association', 'in', 'Brandbergen']\n", + "['The', 'Brandbergen', 'Mosque'] --> CoLocated --> ['Brandbergen', ',', 'Haninge', 'Municipality']\n", + "['the', 'Islamic', 'Association', 'in', 'Brandbergen'] --> IsSynonymOf --> ['The', 'Brandbergen', 'Mosque']\n", + "['the', 'Islamic', 'Association', 'in', 'Brandbergen'] --> none --> ['a', 'mosque']\n", + "['a', 'mosque'] --> none --> ['The', 'Brandbergen', 'Mosque']\n", + "['a', 'mosque'] --> none --> ['Brandbergen', ',', 'Haninge', 'Municipality']\n", + "['Brandbergen', ',', 'Haninge', 'Municipality'] --> none --> ['a', 'mosque']\n", + "['Brandbergen', ',', 'Haninge', 'Municipality'] --> none --> ['The', 'Brandbergen', 'Mosque']\n", + "['Brandbergen', ',', 'Haninge', 'Municipality'] --> CoLocated --> ['south', 'of', 'Stockholm', ',', 'Sweden']\n", + "['Brandbergen', ',', 'Haninge', 'Municipality'] --> none --> ['Brandbergen', ',', 'Haninge', 'Municipality']\n", + "['33.505716°N', '36.269635°E'] --> none --> ['33°30′21″N', '36°16′11″E']\n", + "['33.505716', ';', '36.269635'] --> none --> ['33°30′21″N', '36°16′11″E']\n", + "['the', 'ISF'] --> InChargeOf --> ['the', 'area']\n", + "['the', 'ISF'] --> none --> ['the', 'following', 'day']\n", + "['these', 'total', 'force', 'Airmen'] --> AlliesOf --> ['ground', 'forces']\n", + "['The', 'squadron'] --> CoLocated --> ['the', 'deployed', 'location']\n", + "['Coalition', 'partners'] --> CoLocated --> ['the', 'deployed', 'location']\n", + "['senior', 'officials', 'of', 'the', 'EU'] --> none --> ['United', 'Arab', 'Emirates']\n", + "['the', 'Department', 'of', 'the', 'Treasury'] --> none --> ['the', 'Syrian', 'government']\n", + "['both', 'countries'] --> none --> ['The', 'Foreign', 'Secretary']\n", + "['enemy', 'forces'] --> none --> ['two', 'years']\n", + "['Oct.', '17'] --> none --> ['the', 'city']\n", + "['two', 'years'] --> none --> ['the', 'city']\n", + "['the', 'Iraqis'] --> FightingAgainst --> ['enemy', 'forces']\n", + "['the', 'Government', 'of', 'Iraq'] --> none --> ['the', 'Iraqi', 'people']\n", + "['The', 'UK'] --> AlliesOf --> ['the', 'Government', 'of', 'Iraq']\n", + "['The', 'UK'] --> AlliesOf --> ['the', 'Iraqi', 'people']\n", + "Total number of pairs = 1223, positive examples = 546\n" + ] + } + ], + "source": [ + "from lab6_data.load_re3d import get_source_target_toks\n", + "\n", + "# load the relation data\n", + "# split into train and test -- same split as for NER\n", + "counter = 0\n", + "nonzero = 0\n", + "\n", + "tag_names_to_idx = {}\n", + "for idx, name in enumerate(tag_idx_to_names):\n", + " tag_names_to_idx[name] = idx\n", + "\n", + "sentences = [[tok for tok, lab in sent] for sent in train_set]\n", + "tags = [[tag_names_to_idx[lab] for tok, lab in sent] for sent in train_set]\n", + "\n", + "for s, sentence_rels in enumerate(train_rels):\n", + " for relation in sentence_rels: \n", + " source, target = get_source_target_toks(relation, sentences, tags, s)\n", + " \n", + " print(f'{source} --> {relation[2]} --> {target}')\n", + " \n", + " counter += 1\n", + " if relation[2] != 'none':\n", + " nonzero += 1\n", + " \n", + "print(f'Total number of pairs = {counter}, positive examples = {nonzero}')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To do relation extraction, we will train a classifier that takes pairs of entities as input and outputs a relation label. The pair of entities needs to be represented by a feature vector. The code below extracts a set of features, which will be treated like a bag of words. These include dependency parse and part of speech features.\n", + "\n", + "**TODO 2.1: Examine the code below and write down a list of features that the function extracts for each pair of entities.**" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "About 30,000 Iraqi security force personnel , Kurdish fighters , Sunni Arab tribesmen and Shia militiamen , assisted by US-led coalition air strikes , launched the long-awaited offensive to retake Mosul eight days ago .\n", + "HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /?properties=%7B%22outputFormat%22%3A+%22json%22%2C+%22annotators%22%3A+%22tokenize%2Cpos%2Clemma%2Cssplit%2Cdepparse%22%2C+%22ssplit.eolonly%22%3A+%22true%22%2C+%22tokenize.whitespace%22%3A+%22false%22%7D (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 61] Connection refused'))\n" + ] + }, + { + "ename": "CoreNLPServerError", + "evalue": "Could not connect to the server.", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mConnectionRefusedError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 169\u001b[0m conn = connection.create_connection(\n\u001b[0;32m--> 170\u001b[0;31m \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dns_host\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mport\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mextra_kw\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 171\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/util/connection.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[0;34m(address, timeout, source_address, socket_options)\u001b[0m\n\u001b[1;32m 95\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0merr\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 96\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 97\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/util/connection.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[0;34m(address, timeout, source_address, socket_options)\u001b[0m\n\u001b[1;32m 85\u001b[0m \u001b[0msock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msource_address\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 86\u001b[0;31m \u001b[0msock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msa\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 87\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0msock\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mConnectionRefusedError\u001b[0m: [Errno 61] Connection refused", + "\nDuring handling of the above exception, another exception occurred:\n", + "\u001b[0;31mNewConnectionError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m 705\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 706\u001b[0;31m \u001b[0mchunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchunked\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 707\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m 393\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 394\u001b[0;31m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mhttplib_request_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 395\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, body, headers)\u001b[0m\n\u001b[1;32m 233\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"User-Agent\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_default_user_agent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 234\u001b[0;31m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mHTTPConnection\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 235\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m 1251\u001b[0m \u001b[0;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1252\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1253\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36m_send_request\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m 1297\u001b[0m \u001b[0mbody\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'body'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1298\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1299\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36mendheaders\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m 1246\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1247\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1248\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36m_send_output\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m 1025\u001b[0m \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1026\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1027\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/http/client.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, data)\u001b[0m\n\u001b[1;32m 965\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 966\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 967\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36mconnect\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 199\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 200\u001b[0;31m \u001b[0mconn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_new_conn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 201\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_prepare_conn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 181\u001b[0m raise NewConnectionError(\n\u001b[0;32m--> 182\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Failed to establish a new connection: %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 183\u001b[0m )\n", + "\u001b[0;31mNewConnectionError\u001b[0m: : Failed to establish a new connection: [Errno 61] Connection refused", + "\nDuring handling of the above exception, another exception occurred:\n", + "\u001b[0;31mMaxRetryError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m 448\u001b[0m \u001b[0mretries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 449\u001b[0;31m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 450\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m 755\u001b[0m retries = retries.increment(\n\u001b[0;32m--> 756\u001b[0;31m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_pool\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_stacktrace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 757\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/urllib3/util/retry.py\u001b[0m in \u001b[0;36mincrement\u001b[0;34m(self, method, url, response, error, _pool, _stacktrace)\u001b[0m\n\u001b[1;32m 572\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mnew_retry\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_exhausted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 573\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mMaxRetryError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_pool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mResponseError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcause\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 574\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mMaxRetryError\u001b[0m: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /?properties=%7B%22outputFormat%22%3A+%22json%22%2C+%22annotators%22%3A+%22tokenize%2Cpos%2Clemma%2Cssplit%2Cdepparse%22%2C+%22ssplit.eolonly%22%3A+%22true%22%2C+%22tokenize.whitespace%22%3A+%22false%22%7D (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 61] Connection refused'))", + "\nDuring handling of the above exception, another exception occurred:\n", + "\u001b[0;31mConnectionError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36mextract_relation_BoW\u001b[0;34m(sent_index, rel_index, relations, sentences, tags)\u001b[0m\n\u001b[1;32m 21\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 22\u001b[0;31m \u001b[0mparse_tree\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtree\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtree\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdep_parser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_parse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 23\u001b[0m \u001b[0mparsed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mraw_parse\u001b[0;34m(self, sentence, properties, *args, **kwargs)\u001b[0m\n\u001b[1;32m 228\u001b[0m self.raw_parse_sents(\n\u001b[0;32m--> 229\u001b[0;31m \u001b[0;34m[\u001b[0m\u001b[0msentence\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproperties\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdefault_properties\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 230\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mraw_parse_sents\u001b[0;34m(self, sentences, verbose, properties, *args, **kwargs)\u001b[0m\n\u001b[1;32m 283\u001b[0m \"\"\"\n\u001b[0;32m--> 284\u001b[0;31m \u001b[0mparsed_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapi_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproperties\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdefault_properties\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 285\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mparsed_sent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mparsed_data\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"sentences\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mapi_call\u001b[0;34m(self, data, properties, timeout)\u001b[0m\n\u001b[1;32m 246\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 247\u001b[0;31m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 248\u001b[0m )\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mpost\u001b[0;34m(self, url, data, json, **kwargs)\u001b[0m\n\u001b[1;32m 589\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 590\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'POST'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 591\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m 541\u001b[0m \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 542\u001b[0;31m \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 543\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m 654\u001b[0m \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 655\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 656\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m 515\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 516\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 517\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mConnectionError\u001b[0m: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /?properties=%7B%22outputFormat%22%3A+%22json%22%2C+%22annotators%22%3A+%22tokenize%2Cpos%2Clemma%2Cssplit%2Cdepparse%22%2C+%22ssplit.eolonly%22%3A+%22true%22%2C+%22tokenize.whitespace%22%3A+%22false%22%7D (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 61] Connection refused'))", + "\nDuring handling of the above exception, another exception occurred:\n", + "\u001b[0;31mCoreNLPServerError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 76\u001b[0m \u001b[0mtrain_tags\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtag_names_to_idx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlab\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtok\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlab\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msent\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtrain_set\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 78\u001b[0;31m \u001b[0mfeatures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mextract_relation_BoW\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_rels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_sentences\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_tags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 79\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 80\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m\u001b[0m in \u001b[0;36mextract_relation_BoW\u001b[0;34m(sent_index, rel_index, relations, sentences, tags)\u001b[0m\n\u001b[1;32m 26\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 27\u001b[0m \u001b[0mserver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m \u001b[0mserver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/nltk/parse/corenlp.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(self, stdout, stderr)\u001b[0m\n\u001b[1;32m 151\u001b[0m \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 152\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 153\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mCoreNLPServerError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Could not connect to the server.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 154\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 155\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m60\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mCoreNLPServerError\u001b[0m: Could not connect to the server." + ] + } + ], + "source": [ + "# construct a feature vector for a pair of entities: word1, word2. \n", + "dep_parser = CoreNLPDependencyParser()\n", + "\n", + "def extract_relation_BoW(sent_index, rel_index, relations, sentences, tags):\n", + " features = {}\n", + " \n", + " relation = relations[sent_index][rel_index]\n", + "\n", + " source, target = get_source_target_toks(relation, sentences, tags, sent_index)\n", + " features['source_entity'] = \"_\".join(source)\n", + " features['target_entity'] = \"_\".join(target)\n", + " \n", + " pos_tagged_sent = nltk.pos_tag(sentences[sent_index])\n", + " features['source_pos'] = pos_tagged_sent[relation[0]][1]\n", + " features['target_pos'] = pos_tagged_sent[relation[1]][1]\n", + " \n", + " sent = \" \".join(sentences[sent_index])\n", + " sent = sent.replace('%', 'percent')\n", + " parsed = False\n", + " while not parsed:\n", + " try:\n", + " parse_tree = [tree for tree in dep_parser.raw_parse(sent)][0]\n", + " parsed = True\n", + " except Exception as e:\n", + " print(sent)\n", + " print(e)\n", + " server.stop()\n", + " server.start()\n", + " \n", + " \n", + " # traverse up the tree\n", + " nodes_on_path0 = []\n", + " current_node = parse_tree.nodes[relation[0] + 1]\n", + " while current_node['address'] != relation[1] + 1 and current_node['head'] is not None:\n", + " current_node = parse_tree.nodes[current_node['head']]\n", + " nodes_on_path0.append(current_node['address'])\n", + " \n", + " # and up from the other side \n", + " # traverse up the tree\n", + " nodes_on_path1 = []\n", + " current_node = parse_tree.nodes[relation[1] + 1]\n", + " while current_node['address'] != relation[0] + 1 and current_node['head'] is not None:\n", + " current_node = parse_tree.nodes[current_node['head']] \n", + " nodes_on_path1.append(current_node['address'])\n", + " \n", + " join_node_i = len(nodes_on_path0) - 1\n", + " join_node_j = len(nodes_on_path1) - 1\n", + " for i, node in enumerate(nodes_on_path0):\n", + " if node in nodes_on_path1:\n", + " join_node_i = i\n", + " join_node_j = np.argwhere(np.array(nodes_on_path1) == node).flatten()[0]\n", + " break\n", + " \n", + " dep_path_source = []\n", + " for node in nodes_on_path0[:join_node_i]:\n", + " dep_path_source.append(parse_tree.nodes[node]['rel'])\n", + " \n", + " features['dep_path_source'] = \"_\".join(dep_path_source)\n", + " \n", + " dep_path_target = []\n", + " for node in nodes_on_path1[:join_node_j]:\n", + " dep_path_target.append(parse_tree.nodes[node]['rel'])\n", + " \n", + " features['dep_path_target'] = \"_\".join(dep_path_target)\n", + " \n", + " features['dep_path_length'] = join_node_i + join_node_j # length of dependency path\n", + " \n", + " label = relation[2] # the class of the relation. 'none' is a negative example of unrelated entities.\n", + " return features, label\n", + " \n", + "tag_names_to_idx = {}\n", + "for idx, name in enumerate(tag_idx_to_names):\n", + " tag_names_to_idx[name] = idx\n", + " \n", + "train_sentences = [[tok for tok, lab in sent] for sent in train_set]\n", + "train_tags = [[tag_names_to_idx[lab] for tok, lab in sent] for sent in train_set]\n", + "\n", + "features = extract_relation_BoW(3, 1, train_rels, train_sentences, train_tags) \n", + "\n", + "print(features)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**TODO 2.2: Complete the function below to get a list of tuples for the training set, where each tuple is of the form (feature vector, label). The feature vector should be obtained by calling extract_relation_BoW() with the index of each sentence and relation in the training dataset.**" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0\n", + "100\n", + "200\n", + "300\n", + "400\n", + "500\n", + "600\n", + "700\n" + ] + } + ], + "source": [ + "def get_features_for_relations(rels, sentences, tags):\n", + " ### WRITE YOUR OWN CODE HERE\n", + " rels_with_feats = []\n", + " for s in range(len(rels)):\n", + " if s % 100 == 0:\n", + " print(s)\n", + " for r in range(len(rels[s])):\n", + " rel_feats, label = extract_relation_BoW(s, r, rels, sentences, tags)\n", + " rels_with_feats.append((rel_feats, label))\n", + " ###\n", + " return rels_with_feats\n", + " \n", + "train_set_rels = get_features_for_relations(train_rels, train_sentences, train_tags)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run the code below to train a naïve Bayes classifier using NLTK's library. The class is described here: https://www.nltk.org/book/ch06.html" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "classifier = nltk.NaiveBayesClassifier.train(train_set_rels)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The code below will generate the test set for relation extraction, then use the classifier to predict the relation types for each pair of entities in the test set:" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0\n", + "100\n" + ] + } + ], + "source": [ + "test_sentences = [[tok for tok, lab in sent] for sent in test_set]\n", + "test_tags = [[tag_names_to_idx[lab] for tok, lab in sent] for sent in test_set]\n", + "\n", + "test_set_rels = get_features_for_relations(test_rels, test_sentences, test_tags)\n", + "test_set_rels_no_labels = [rel for rel, lab in test_set_rels]\n", + "gold_labels = [lab for rel, lab in test_set_rels]\n", + "predicted_labels = classifier.classify_many(test_set_rels_no_labels)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**TODO 2.3: Use the [sklearn F1_score function](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) to compute the macro-average and per-class F1 scores for the relation extraction dataset.** \n", + "\n", + "The low performance on some classes may be due to the small dataset. " + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.22240187650614632" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# macro-average F1 score\n", + "### WRITE YOUR OWN CODE HERE\n", + "f1_score(gold_labels, predicted_labels, average='macro')" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([0.08333333, 0. , 0.22222222, 0.37288136, 0.06896552,\n", + " 0. , 0.22641509, 0.5 , 0.18181818, 0.38461538,\n", + " 0.2 , 0.42857143])" + ] + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# per-class F1 scores\n", + "### WRITE YOUR OWN CODE HERE\n", + "f1_score(gold_labels, predicted_labels, average=None)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**OPTIONAL TODO 2.4: Exclude or modify the feature vector and see if you can improve the performance of the RE classifier.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}