<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abbynewcomb on Medium]]></title>
        <description><![CDATA[Stories by Abbynewcomb on Medium]]></description>
        <link>https://medium.com/@abbynewcomb13?source=rss-321439f78670------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rXpJb4DFLqjSB5PI</url>
            <title>Stories by Abbynewcomb on Medium</title>
            <link>https://medium.com/@abbynewcomb13?source=rss-321439f78670------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 13:10:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abbynewcomb13/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fine-tuning a Question Answering model from BERT]]></title>
            <link>https://medium.com/@abbynewcomb13/fine-tuning-a-question-answering-model-from-bert-1c5efc046b4f?source=rss-321439f78670------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c5efc046b4f</guid>
            <category><![CDATA[question-answering]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Abbynewcomb]]></dc:creator>
            <pubDate>Tue, 03 Jan 2023 15:32:34 GMT</pubDate>
            <atom:updated>2023-01-03T15:32:34.673Z</atom:updated>
            <content:encoded><![CDATA[<p>To create a simple question answering interface, I used the Huggingface libraries and <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb#scrollTo=FSf3MSknTcrK">this tutorial</a> from Huggingface to download a model and run a few epochs of fine-tuning on the Squad dataset. I then wrote a small interface that will take as input some context and a question, and produce the answer using my fine-tuned model.</p><p>In this project, I used extractive question answering, in which the model returns a part of the context paragraph as the answer to the question. So the neural network actually outputs the start and end location of the answer in the context paragraph. These numbers are post-processed to find the model’s final answer, a string extracted from the context paragraph.</p><p>I’ll be using the Huggingface model “distilbert-base-uncased” as my base model, and using the accompanying tokenizers and such for training and testing. The original output layer of the model will be removed and replaced with a simple feed-forward layer as the question-answering head, which will output the location of the answer in the context paragraph.</p><p>First, I downloaded the Squad question answering dataset from Huggingface. This dataset is already divided into train and test, with a paragraph of context and an accompanying question in a separate field. The answer to the question is given for training as both regular text and the number of the character where the answer starts in the context paragraph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtsAD61AAQPjT6HaZQQwVw.png" /><figcaption>a single example of the squad train dataset, before preprocessing</figcaption></figure><p>In order to use the Squad dataset for training and testing, it must first be tokenized and the words must be mapped to their input ids, a unique number corresponding to each unique word or morpheme that is input to the model representing that word. I use the autotokenizer method from Huggingface to use the tokenizer corresponding to my model. In this step, the context paragraph may also be chunked into multiple inputs if it is too long to input into the model all at once. Finally, we preprocess the answer data so that the training dataset contains both the start and end position of the answer.</p><figure><img alt="a small section of the code for preprocessing the training data" src="https://cdn-images-1.medium.com/max/1024/1*4cm8J5cDE3_pV6alDJMkig.png" /><figcaption>a small section of the code for preprocessing the training data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cnnmi8tZhE3jUv3qXNNpxg.png" /><figcaption>the structure of the data, ready for training</figcaption></figure><p>After preprocessing, we basically just need to define the pretrained model we’re using and run the training using Huggingface’s codebase. My pretrained model trained for an additional 3 epochs, which took about 3 hours using the Google Colab GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*653wInkY1xdwYR-flKrb4Q.png" /><figcaption>a snippet of the code to instantiate and train the model</figcaption></figure><p>The model outputs two logit tensors which contain the unnormalized output of the network, or essentially the unnormalized probability that each character index in the input is the start and end of the answer. In order to choose our final answer, I perform some postprocessing of this output using functions from a Huggingface tutorial in order to choose the maximum probability section which is in the context part of the input and has the start token earlier than the end token. This is then converted from index positions to an actual string extracted from the context paragraph which hopefully answers the question correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7V3P5_8YPEK33CA2Fl_uw.png" /><figcaption>the process of running validation methods and the output in the last line</figcaption></figure><p>My results were as expected for this dataset and model since the tutorial I was following had a very similar result. The exact match number shows the percentage of 10,570 test examples for which my model produced the correct answer. My model thus has an accuracy of about 77%, which seems pretty good for only 3 hours of training.</p><p>I initially thought it was odd that the metrics provide an f1 score for this task, since the f1 is an average of precision and recall, which are evaluation methods for classification tasks. Since the extractive question answering task is not a classification task, this metric didn’t make sense to use. However, I found <a href="https://kierszbaumsamuel.medium.com/f1-score-in-nlp-span-based-qa-task-5b115a5e7d41">this medium article</a> which makes clear that in this context, true positives are the number of tokens which where correctly identified as part of the answer, false negatives are tokens which were incorrectly excluded from the answer, and false positives are tokens which were incorrectly included in the answer. Adjusting the meaning of these metrics from their traditional meaning in classification allows us to compute a meaningful f1 score of about 85% correctly identified words, which again is not bad after only 3 hours.</p><p>I did manage to upload my trained model to Huggingface so that I could redownload it using another script and use it without having to run 3 hours of training each time.</p><p>To further test my model, I wrote a small wrapping function so that I could write my own test examples for the model. The example below shows the output of these tests, which was very reasonable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGfwSs76IbWUsht-ypGNyw.png" /></figure><p>Overall, I learned a lot about working with the Huggingface library from this project. I had to do a lot of combing through Huggingface documentation in order to get everything to work, especially my ‘interact’ wrapper. I also learned about how complicated it is to work with pretrained models, and some of the requirements for both retraining and using a pretrained model. And I got to learn about how extractive question answering systems work, and made a model that works pretty well!</p><p>Here are the links to my notebooks:</p><ul><li><a href="https://colab.research.google.com/drive/1KwUDhjbHDb5E7yLNmQPbGajYuIqNtYhM?usp=sharing">training</a></li><li><a href="https://colab.research.google.com/drive/1pyKDWEIreTSpJ_SuHM5iaCwCs6diDYac?usp=sharing">testing</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c5efc046b4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embeddings to Evaluate Bias in News Sources]]></title>
            <link>https://medium.com/@abbynewcomb13/word-embeddings-to-evaluate-bias-in-news-sources-944c159c4362?source=rss-321439f78670------2</link>
            <guid isPermaLink="false">https://medium.com/p/944c159c4362</guid>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Abbynewcomb]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 22:57:27 GMT</pubDate>
            <atom:updated>2021-12-07T19:27:32.683Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://www.linkedin.com/in/abby-newcomb-a12b7819a/">Abby Newcomb</a>, <a href="https://www.linkedin.com/in/schatzki-mcclain/">Elise Schatzki-McClain</a>, and <a href="https://www.linkedin.com/in/sean-souksavath-64943a210/">Sean Souksavath</a></p><blockquote>Everyone is biased. It’s impossible to avoid. Journalists approach every story with unique life experiences that consciously and subconsciously influence the work they do, whether they identify as Left, Right, or anything in between. What’s more, the 24-hour news cycle operates on a business model that relies on inflaming the political divide, one that has already reached record levels of hyper-partisanship. — <a href="https://www.allsides.com/blog/does-unbiased-news-really-exist">Binion, Allsides.com</a></blockquote><p>A functioning democracy requires an informed voting population, which means reliable news is imperative for society. In today’s polarized climate, there is a wide variety of news sources with a wide variety of opinions, and which source you choose to read or listen to is often dictated by your political leaning. But we wanted to know whether the inherent bias of news sources could be seen by examining a word embedding trained on different news articles.</p><p>We were inspired by <a href="https://www.pnas.org/content/pnas/115/16/E3635.full.pdf">Garga, Schiebingerb, Jurafskyc, and Zoue’s work</a> on quantifying gender and ethnic stereotypes over the past 100 years using word embeddings. They used word embeddings to show how word associations with different ethnic groups and gender groups have changed over time. We followed their methodology to examine how words about gender and sexuality are associated in models trained on the articles of five different news sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*_KlVxkLU2uFclPI_" /><figcaption>2-dimensional representation of a word embedding model showing how similar words are related in the vector space</figcaption></figure><h3>Word Embeddings</h3><p>A word embedding is a language model in which each word has a location in a vector space. The distances between words have semantic meaning. Words that appear in similar contexts will appear closer together in the model.</p><p>There are many different ways to create these word embeddings using neural networks, but the method we used was skip-gram. In a skip-gram neural network, the input is a single word in a one-hot vector, and the output is the predicted two words before and two words after the input word in one-hot vectors. In between the input and the output, there is an embedding layer with weights and biases that the model will update as it trains in order to better predict the output. This weight matrix in the embedding layer of the model is the word embedding or vector space representation of the words. There is also a normal softmax layer after the embedding layer which serves to reduce the large output size from the embedding layer to a smaller output size.</p><p>As the model is trained, over time, it will place each word in the vector space closer to other words that appear in similar contexts, because this allows it to have a higher accuracy in predicting the next word. In this way, words with similar meanings are placed nearer to each other in the vector space. From this word embedding model, we can do analysis on the meaning of different words based on the distances between words in the vector space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*_ZsJl-2HzjWut75Y" /></figure><h3>Data</h3><p>We came across a <a href="https://components.one/datasets/all-the-news-2-news-articles-dataset/">Kaggle dataset</a> that currently has been expanded to contain 2.7 million news articles from 27 different American publications due to its popularity. This dataset contained articles written between the years of 2013 to 2020, with the number of articles per publication and per year varying widely. Since we really wanted to include Fox News in our dataset we decided to isolate our dataset to only articles from 2018 because it primarily had articles from the year 2018.</p><p>We chose 5 other news sources in total to investigate. Fox News is of interest because most Americans believe it is very right-leaning. Similarly, we chose CNN since it is quite left-leaning. We chose two news sources that are considered to be centrist publications, Reuters and the Hill. We also added the New York Times to our list of news sources as an additional data point for the left, but unfortunately couldn’t find another comparable dataset with enough articles on the right. However, we were happy with these news sources because we believe that we chose some at opposite ends of the political spectrum but also a couple of centrist publications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D6Zb5UU0OyvqNcvu" /></figure><h3>Model Architecture</h3><p>In the network itself, there are 2 main layers, with optional hidden layers in between the two. The first layer is the embedding layer, which is a layer that has a weight matrix V x D, where V is the size of the vocabulary or number of unique words in the training data, and D is the desired number of dimensions in the vector space for the word embeddings. Next are any additional hidden layers. The last layer in the network is a softmax layer that outputs the 4 other surrounding words in one hot vectors that are the same size as the input word one-hot vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*iB2BHfjR28vQ0xGW" /></figure><p>We used the Word2Vec model created by Mikolov, Chen, Corrado, and Dean and the gensim Python package to create our models. To create a word embedding model specific to a publication, we needed to preprocess the respective corpus of text for the model to train on. To do so, we lemmatized and removed stop words from the articles, which essentially just simplifies the text to meaningful and uniform words by removing things required by grammar but useless to our model, such as the<em> -ing</em> endings on verbs and the word <em>the</em>. We created a separate model for each of the five news sources and trained each model on its corresponding pre-processed corpus. Each of the models have the same architecture and parameters to keep the comparison between the news sources consistent. Each model has a word embedding layer with a 100 x C weight matrix, so the vector space for the word embedding is 100 dimensions. C is the number of unique words in the corpus for a given news source. The window is four, meaning that the model predicts the two words before and the two words after the given input word. In addition, the model does not include a word in the word embedding if it does not appear in the training data at least five times. This ensures that a word that is only used once or twice is not included in our model and thus not included in our analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*eE40WxkbojAKXFE8" /></figure><p>This type of word embedding when there is a single word as input and the model predicts the surrounding four words is called skip-gram. The opposite where the model predicts a single word based on an input of surrounding words is called Continuous Bag of Words (CBOW). Because we are not using the model to do analysis and instead we are using the trained model’s weights, it does not matter which direction in which we predict the words. CBOW generally trains faster and is very good and accurate for words that occur often in the dataset. We chose the skip-gram methodology however because the skip-gram is good at training on a small dataset, and it is specifically good for words that do not occur very often in the dataset. Some of the words surrounding the LGBT community may not appear very often in our data, thus we chose this model so that these words would hopefully be placed and organized well in the vector space. In addition, our datasets have varying sizes, but they are generally on the smaller size for creating a very high detail word embedding.</p><h3>Methods</h3><p>The first process we used took a <a href="https://www.pnas.org/content/pnas/suppl/2018/03/30/1720347115.DCSupplemental/pnas.1720347115.sapp.pdf">list of adjectives</a> (from the Garga, Schiebingerb, Jurafskyc, and Zoue’s work) and another list of related words. We calculated the average location of the related words and the distances between this average location and each of the adjectives in the list in the vector space. We then ranked the adjectives by distance and took the five closest adjectives to that average location in each model.</p><p>The other calculations use relative distances to compare word associations between two opposing words across news sources. The relative distance is how much closer a neutral word is to word1 than it is to word2, when word1 and word2 are opposite words. We obtain this distance as a percentage for each word by dividing its distance from the neutral word by the sum of the two distances. Relative distances allow us to easily compare word affiliations across news sources, because instead of giving a distance specific to a certain vector space or word embedding, the relative distance provides a percentage of how close words are relative to each other. These percentages can then be universally compared across multiple different embedding spaces created by different models, and in our case, different news sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6Bu2yu3ft4bf8dM6njQfQ.png" /></figure><p>In a perfect world with no bias, the neutral word should be placed equally close to both of these opposite locations, hence both words should have about 50% of the total distance to the neutral word. However, this is not the case all the time. In our analysis, we calculate the relative distances between the word and the two opposite locations and compare them across news sources for a list of related words surrounding a theme. In another test, we also use the previously mentioned adjectives as a list of neutral words to calculate relative distances, and we rank the five adjectives that are relatively closest to word1 and the five that are relatively closest to word2, and compare these adjectives across news sources.</p><h3>Results</h3><p>All of our analysis hinges upon the idea of distance between words in a word embedding. To reiterate, the word embedding model is a many-dimensional vector space, with words represented as points in the vector space. We can calculate the distance between words, and since the model places words that appear in similar contexts together, words that are semantically related will appear close together. For example, we would expect the words <em>smart</em> and <em>clever</em> to appear close to one another in the word embedding, since they are essentially synonyms. This could relate to bias because if the word <em>man</em> is placed closer to the word <em>clever</em> than the word <em>woman</em>, then in the data the model was trained on, the word <em>clever</em> has a looser association with the word <em>woman</em> and a stronger association with the word <em>man</em>. This in turn implies a bias in the text used to generate the embedding. The concept of words being ‘associated’ with one another by the word embedding is at the core of our analysis. The model merely tries to make its guesses about the next predicted word as accurate as possible, which generates the word embedding model.</p><p>When comparing the relative distances between <em>woman</em> and <em>man</em> for different news sources, there were some interesting findings. For words that had to do with intelligence, all news sources associated these words more with <em>man</em> than with <em>woman</em>. The most extreme association was from Reuters. Fox News had the least extreme association, and in some cases, such as with words <em>clever</em> and <em>intelligent</em>, Fox News associated these words slightly more with <em>woman</em> than with <em>man</em>. For words that are negative attributes about one’s character, i.e. <em>evil</em> or <em>dumb</em>, there was a higher association with <em>man</em> than with <em>woman</em> for all news sources except for Fox News. Fox News had an equal association for both <em>man</em> and <em>woman</em> for all of these negative attributes, unlike the other news sources. General trends were that Reuters consistently had the most extreme differences in association between <em>man</em> and <em>woman</em>, and that Fox News had the most equal association between <em>man</em> and <em>woman</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/0*52D9VZOKsNK6n9VB" /></figure><p>We looked at differences in word association with the words <em>homosexual</em> and <em>heterosexual</em> with words related to trust and familiarity, such as <em>friend</em>, <em>love</em>, <em>trustworthy</em>, and <em>babysitter</em>. CNN, The New York Times, and Reuters had a relatively equal association with these trust words and the words <em>homosexual</em> and <em>heterosexual</em>. However, this was not the case for The Hill and Fox News. The Hill had an extremely high association with <em>heterosexual</em> and these trust words, and Fox News had a less extreme, but still significant association between <em>heterosexual</em> and these trust words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*GIxZGXUAIYfdPu4Z" /></figure><p>We generated a list of the adjectives in our neutral word list that were most associated with the LGBT community. These are the adjectives nearest to the average location of the words <em>gay</em>, <em>lesbian</em>, <em>transgender</em>, <em>bisexual</em>, and <em>queer</em> in our dataset. The word <em>oppressed</em> occurs twice in the list, which makes sense because society tends toward heteronormativity and gender normativity. However, there are a few hard negative words that appear in CNN and Fox News’ lists such as <em>ungrateful</em>, <em>contemptible</em>, <em>disloyal</em>, and <em>callous</em>. Some of these words also may be associated with people who oppose the LGBT community, such as the word <em>intolerant</em>. The appearance of words such as <em>effeminate</em> and <em>masculine</em> could indicate a discourse about how LGBT people present themselves as perhaps more or less masculine or feminine than their cisgendered, heterosexual counterparts. These words also might come from a general discussion about stereotypes within different LGBT groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5ct0XiyhNrs9xZOZika_Q.png" /><figcaption>top 5 adjectives associated with members of the LGBT community as a whole by actual distance</figcaption></figure><p>Using the relative distance concept described under Methods, we generated a list of the adjectives with the highest degree of difference in distance from both words, meaning the adjectives which are most associated with one group over another. The words most associated with <em>homosexual</em> over <em>heterosexual</em> are generally a bit concerning because of the appearance of a lot of hard negative words. For instance, the words <em>hateful,</em> <em>barbaric,</em> and <em>cruel</em> all appear more than once. We believe that it is possible that these words are often mentioned in reference to bigots, since presumably the word <em>homosexual</em> is often mentioned in the news when there are substantial incidents of prejudice or hate. Generally, the list of words associated with <em>heterosexual</em> are not negative in character. They are either benign or positive such as <em>cheerful</em>, <em>tidy</em>, and <em>healthy</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zhy9DVj7iUG1x6WPUuZ5tA.png" /><figcaption>top 5 adjectives associated with homosexual and heterosexual by relative distance</figcaption></figure><h3>Limitations</h3><p>It must be noted that we had a different amount of data for each news source. Fox News had the least data with about 17 thousand articles the chosen year, 2018. It is possible that the word embedding created using this data may not be as good as the other models trained with more data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*fvIrCipfIMcWdExL" /><figcaption>the number of articles used to train a model for each news source</figcaption></figure><p>With more time, we would’ve liked to do more verification of our models before using them for this type of analysis. We do not know the groups or clusters of words that the model chose to group together. For example, it is possible that the word <em>homosexual</em> is used so infrequently in everyday English that the model placed it in an unusual or isolated location. Because embeddings are so complex and there is no perfect answer for where to place a word in the vector space, it is hard to know where to start in verifying our embedding.</p><h3>Conclusion</h3><p>Using word embeddings to quantify associations of words in a given text can be a powerful tool to find underlying biases and connections. This can be especially important when examining news articles from certain publications. With the help of Word2Vec’s skip-gram model architecture, we were able to generate several word embeddings for several popular American news publications such as CNN and Fox News using a corpus of articles from 2018. Using our word embedding models, we were able to find interesting associations between neutral words and words relating to gender and sexuality.</p><p>Our goal was to examine the narratives presented by the news to look for bias. While no news sources specifically stood out as being particularly biased towards a given group, our findings do show which narratives news sources often push about particular groups. Although the associations of the word embedding can be difficult to interpret, one can still use the findings from our research to make general inferences about the news’ treatment of gender and sexuality.</p><h3>Sources:</h3><p><a href="https://www.allsides.com/blog/does-unbiased-news-really-exist">https://www.allsides.com/blog/does-unbiased-news-really-exist</a></p><p><a href="https://www.pnas.org/content/pnas/115/16/E3635.full.pdf">https://www.pnas.org/content/pnas/115/16/E3635.full.pdf</a></p><p><a href="https://towardsdatascience.com/nlp-101-word2vec-skip-gram-and-cbow-93512ee24314">https://towardsdatascience.com/nlp-101-word2vec-skip-gram-and-cbow-93512ee24314</a></p><p><a href="https://towardsdatascience.com/how-to-train-the-word2vec-model-24704d842ec3">https://towardsdatascience.com/how-to-train-the-word2vec-model-24704d842ec3</a></p><p><em>The code used to generate these findings can be found on </em><a href="https://github.com/abbynewcomb/word_embeddings"><em>Github</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=944c159c4362" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>