<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Reinaldo Bergamaschi on Medium]]></title>
        <description><![CDATA[Stories by Reinaldo Bergamaschi on Medium]]></description>
        <link>https://medium.com/@rberga?source=rss-90c7ef27f79b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ke2MJN6TXPkRSTZJ08Ojng.png</url>
            <title>Stories by Reinaldo Bergamaschi on Medium</title>
            <link>https://medium.com/@rberga?source=rss-90c7ef27f79b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:31:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rberga/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Better Natural Language Understanding of Tabular Data through Prompt Engineering]]></title>
            <link>https://rberga.medium.com/better-natural-language-understanding-of-tabular-data-through-prompt-engineering-ea76d7d3dcbf?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea76d7d3dcbf</guid>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 02:01:18 GMT</pubDate>
            <atom:updated>2023-10-02T02:03:54.853Z</atom:updated>
            <content:encoded><![CDATA[<h4>We used Lanchain’s Pandas Dataframe Agent and Large Language Models (LLMs) to reason about tabular data. Questions that relied on reasoning did not get correct results until we applied prompt engineering to define the table semantics.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2nhuvHh8FDtvqGFEoyGCLQ.png" /><figcaption>This image was created with the assistance of DALL·E 2</figcaption></figure><p>In my last article (<a href="https://rberga.medium.com/searching-your-own-files-in-natural-language-a28557511214">Searching your own files in natural language</a>), I used text embeddings, vector databases and semantic search, together with Langchain chains and ChatOpenAI to store, search and reason about documents of different formats. In the case of tabular files (e.g., .csv) loaded with <a href="https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.csv_loader.CSVLoader.html">CSVLoader</a>, the reasoning was limited by the fact any non-obvious table semantics was not understood by the model. Moreover, rows were converted to text and any chain of thought that required looking at several values in columns would need to bring up too much data for it to be efficient (never mind the fact that the number of tokens would often exceed the maximum allowed by OpenAI).</p><p>Enter the concept of <em>Agents</em>. Suppose that instead of relying on the data currently present in an LLM (which could be out-of-date), you could search the internet for a particular topic, and then ask the LLM to summarize the search results, and answer any questions you may have about those results in a natural language form. Agents are a way of integrating LLMs with external tools, such as search, Python interpreters, Pandas functions, math libraries, or your own custom functions. Here is a clear <a href="https://www.youtube.com/watch?v=Xi9Ui-9qcPw">introductory video</a> about agents.</p><p>In this article I will explore the use of Langchain’s <a href="https://python.langchain.com/docs/integrations/toolkits/pandas">Pandas Dataframe Agent</a>. This Pandas agent uses the <a href="https://blog.research.google/2022/11/react-synergizing-reasoning-and-acting.html">ReAct framework</a>. The idea of the ReAct framework is to use the LLM to elaborate on what is needed to answer a question (the “thought” process), then decide if it has the necessary information or it needs to call some external function to get it (the “action” process). The LLM also elaborates how to call this external function, based on the functions “registered” with the agent. In this case, the external function is called and the output is fed back to the LLM as an “observation”, which is then submitted again to the “thought” process, and it iterates until the LLM considers that it “knows” the answer. The figure below illustrates this flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k2Y2LvvrwYmUw_HoeB9nDg.png" /></figure><p>The right side of this figure shows the (verbose) output of the <em>pandas_dataframe_agent</em> call to answer a question about attendance at FIFA Men’s Football World Cup games. As we will see later, I am using a <em>table of all men’s football </em>World Cup<em> games from 1930 to 2022</em>, scores, dates, location and attendance, which is read into a Pandas dataframe and used to initialize the agent. I will give you the code for it a little later, for now, let’s try to understand what the agent is doing.</p><p>Given a question shown in the figure, the LLM engages this <em>thought</em> process:</p><blockquote>Thought: I need to find the maximum attendance value in the dataframe</blockquote><p>The LLM+Agent <em>knows</em> it has access to data contained in a Pandas dataframe, and assumes that <em>attendance</em> is a column. Then, it kicks off an <em>action</em>, which invokes the Python shell and runs a Pandas command on the dataframe:</p><blockquote>Action: python_repl_ast<br>Action Input: df[‘attendance’].max()</blockquote><p>This action uses Pandas to find the maximum value in column ‘attendance’ in the input dataframe. The output of this command becomes an <em>observation</em> which is sent back to the LLM:</p><blockquote>Observation: 173850</blockquote><p>At this point the LLM has partial information, but it still needs to find out the date and the location, so it iterates, with new <em>thought</em>, <em>action </em>and <em>observation </em>steps, as shown below (dots added for easier visualization):</p><blockquote>Thought: I need to find the row with this maximum attendance<br>Action: python_repl_ast<br>Action Input: df[df[‘attendance’] == df[‘attendance’].max()]<br>Observation: <br>….. world_cup ……………… date……….. team 1……. team 2… score team 1 \<br>74: 1950 FIFA World Cup… 7/16/1950… Uruguay… Brazil….. 2 ……………</blockquote><blockquote>…. score team 2… penalties team 1… penalties team 2… Location…… phase \<br>74: 1………………. None……………. None…………………. Rio De Janeiro… Final</blockquote><blockquote>….. attendance… year… host <br>74: 173850..…… 1950… Brazil</blockquote><p>From this last <em>observation</em>, the LLM concludes that it knows the answer and returns to the user:</p><blockquote>Thought: I now know the final answer<br>Final Answer: The largest attendance was 173850 at the 1950 FIFA World Cup Final between Uruguay and Brazil in Rio De Janeiro on 7/16/1950.</blockquote><blockquote>&gt; Finished chain.</blockquote><blockquote><strong>‘The largest attendance was 173850 at the 1950 FIFA World Cup Final between Uruguay and Brazil in Rio De Janeiro on 7/16/1950.’</strong></blockquote><h3>It’s (almost) all in the prompt</h3><p>The Python code that invokes the Pandas agent is pretty straightforward. Here we are going to use OpenAI (i.e., you need a key) and a csv file containing the data for all FIFA Men’s World Cup games, which I collected from several places on the web and put together as one table. Data and code may be downloaded from <a href="https://github.com/rberga/SemanticSearch/tree/master/tabular_data_understanding/">my Github</a>. You may have to install some Python libraries — instructions are readily available on the web.</p><p>Let’s first load and look at the data.</p><pre>import pandas as pd<br><br>matches_df = pd.read_csv(&quot;./datasets/world_cup_matches_v1.csv&quot;)<br>matches_df = matches_df.replace(np.nan, None)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZlPXZzKK6sYk7Mp1gVMeQ.png" /></figure><pre># Import necessary packages<br>from langchain.llms import OpenAI<br>from langchain.agents import create_pandas_dataframe_agent<br>from langchain.agents.agent_types import AgentType<br><br># Create the Pandas Dataframe Agent<br>agent = create_pandas_dataframe_agent(<br>    OpenAI(temperature=0, model_name=&quot;text-davinci-003&quot;,openai_api_key=&lt;your_key&gt;),<br>    matches_df,<br>    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,<br>    verbose=True)</pre><p>The function <em>create_pandas_dataframe_agent</em> creates the agent, using OpenAI’s LLM ”<em>text-davinci-003</em>” (default), and <em>temperature = 0</em> (to get mostly reproducible results). The agent type for this LLM model is, by default, a <em>zero-shot</em>, which means the agent uses a <strong>prompt</strong> which does not rely on examples being given, and it relies on the <em>ReAct</em> framework to decide which tool to use, based solely on the tool’s <strong>description</strong>. To understand it better, let’s look at the default <strong>prompt</strong> associated with the <em>pandas_dataframe_agent</em>:</p><pre>agent.agent.llm_chain.prompt.template</pre><pre><strong>---- Default Prompt ----</strong><br>&quot;&quot;&quot;<br>You are working with a pandas dataframe in Python. The name of the dataframe is `df`.<br>You should use the tools below to answer the question posed of you:<br><br><strong>python_repl_ast</strong>: A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.<br><br><strong>Use the following format</strong>:<br><br><strong>Question</strong>: the input question you must answer<br><strong>Thought</strong>: you should always think about what to do<br><strong>Action</strong>: the action to take, should be one of [python_repl_ast]<br><strong>Action Input</strong>: the input to the action<br><strong>Observation</strong>: the result of the action<br>... (this Thought/Action/Action Input/Observation can repeat N times)<br><strong>Thought</strong>: I now know the final answer<br><strong>Final Answer</strong>: the final answer to the original input question<br><br><br>This is the result of `print(df.head())`:<br>{df_head}<br><br>Begin!<br><strong>Question</strong>: {<strong>input</strong>}<br>{agent_scratchpad}\<br>&quot;&quot;&quot;</pre><p>The question is, given a table with semantically-meaningful column names, can the agent and the model make sense of the data, starting from generically-worded questions? Let’s explore this now.</p><h3>Examples</h3><p>Let’s now try some questions with varying degrees of reasoning required. You may want to <a href="https://github.com/rberga/SemanticSearch/tree/master/tabular_data_understanding/datasets">download the table</a> of World Cup results to better understand the actions suggested by the Pandas agent.</p><p>To pass a question to the agent we use the <em>agent.run(“…”)</em> method.</p><blockquote><strong><em>Query 1<br>Simple selection and count</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJnR_8blWY57oSDMfkkucg.png" /></figure><p>Here we see the agent getting a sub-dataframe based on the value of column <em>‘year’= 1938</em>, and getting its number of rows, giving a correct answer.</p><p>Given that we are using OpenAI non-free models, it is useful to keep track of the number of tokens in each query, as this will determine the final cost. To get the number of tokens in a query we can use the function:</p><pre>from langchain.callbacks import get_openai_callback<br><br>def count_tokens(chain, query):<br>    with get_openai_callback() as cb:<br>        result = chain.run(query)<br>        print(f&#39;Spent a total of {cb.total_tokens} tokens&#39;)<br><br>    return result<br><br># Counting the tokens for a query<br>count_tokens(agent, &quot;how many games were there in the 1938 world cup?&quot;)<br><br>---- Output ----<br>Spent a total of 1252 tokens</pre><p>These <strong>1252 tokens</strong> were almost all due to the prompt, since there were not many characters exchanged as part of the <em>thought</em>, <em>action</em> and <em>observation</em> processes.</p><blockquote><strong><em>Query 2<br>Extracting semantics from the data</em></strong></blockquote><p>As humans (with a basic understanding of football), we can easily interpret our table of World Cup games. We know to look at scores to determine the winner and to look at the ‘phase’ column to determine if it is a final match, for example. But how much of these table-specific semantics are understood by our Pandas agent + LLM? All we have to do is ask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thdL6nOnInPljbw38qjOIw.png" /></figure><p>In this case, the model correctly figured out that comparing the values on ‘<em>score team 1</em>’ and ‘<em>score team </em>2’ columns was the way to determine if a team won, lost, or it was a tie. Note that it created a new internal column ‘result’ to store the outcome. The answer is not fully correct in case of a tie on a knockout match, in which case it would also need to look at the penalties columns.</p><p>Let’s see now its reasoning for figuring out the final match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zn952LiymtwD2gn6GYcqVQ.png" /></figure><p>The model correctly figured out how to interpret the table in order to find the ‘final’ match. Note that I deliberately did not use the word ‘final’ in the question.</p><p>In the case of knockout matches, if the result is a draw (possibly after extra-time), the winner is decided on penalty shoot-outs, and the table has two columns indicating the penalties scored by each team. Let’s see how the model interprets that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E4uKJ8pj5rEXICb8nG4G_A.png" /></figure><p>That is correct!</p><p>Let’s try a few more questions that rely on interpretation of the data.</p><blockquote><strong><em>Query 3<br>Interpreting the question</em></strong></blockquote><p>After trying out several questions, it was clear that the wording of the question mattered. How you ask has an impact on the actions requested by the agent. For an example, let’s ask about the 1994 World Cup final (hint: Italy and Brazil tied the game, and Brazil won in penalties).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*458Xu_Mw6p0h5SciElfX-Q.png" /></figure><p>This is not correct. It did find the right answer for the scores but it failed to consider the penalty scores in the case of a tie, and did not say who the winner was (Brazil won in penalties!), even though in the previous query it did say it should look at the penalties columns in case of a tie.</p><p>Moreover, it had to return all games in the 1994 World Cup in order to get the ‘Final’ game. This increased considerably the number of tokens in the query to <strong>3897</strong> tokens. It would have been better if the original action were the following: <em>df[df[‘year’] == 1994][df[‘phase’] == ‘Final’]</em>, which would have returned a single line as the <em>observation</em> and reduced the number of tokens.</p><p>If we ask the same question in a different way, we get the same wrong final answer but using a different thought process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JezmPjxyaL9POoWQcr_FQ.png" /></figure><p>Here again it found the answer based on scores alone, not penalties, which in this case was wrong. Moreover, it used the last row in the sub-dataframe (<em>df[df[‘year’] == 1994]</em>) as the final match, which by chance is correct, but there is no guarantee.</p><p>Here is another example of a wrong answer regarding the <em>1974</em> World Cup, using the same text as before (for the 1994 World Cup). In this case the action finds the right match, but it interprets the score wrongly (differently than its own reasoning, stated in the answer to <em>Query 2</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IoiOjSnhSZ0CRS8wsPqQhQ.png" /></figure><p>The <em>observation</em> brought the correct final match, but the last <em>thought</em> associated the scores with the wrong teams and declared the wrong winner (West Germany, as it was called then, won the match by the score of 2x1).</p><p>In fact, I repeated the same questions several times, and the answers would occasionally vary from correct to totally wrong, despite the <em>temperature</em> parameter being set to <em>0.0</em>.</p><h3>Prompt Engineering</h3><p>As we saw earlier, the default prompt instructs the model to use the dataframe and call the python interpreter with Pandas commands, if it would help coming up with an answer. But clearly the model does not have a good understanding of the semantics of the data (i.e., what each column means), being very erratic depending on how the question was worded. This is not what is being called a ‘hallucination’, as the model only uses the data provided in the dataframe. This is the case of the model not being able to understand the semantics of the table, given the generic default prompt that was given.</p><p>Now I am going to tweak the prompt with specific instructions as for what the columns mean and how they relate to each other. A new prompt can be set on the agent with the command below. Pay close attention to the instructions 1 through 9 passed in explicitly.</p><pre>agent.agent.llm_chain.prompt.template =\<br>&quot;&quot;&quot;<br>You are working with a pandas dataframe in Python. The name of the dataframe is `df`.<br>You should interpret the columns of the dataframe as follows:<br><br>1) Only answer questions related to the dataframe. For anything else, say you do not know.<br>2) Each row of the dataframe corresponds to a match or game.<br>3) The &#39;phase&#39; column states the stage of the match within each tournament. For example if asked for the final match in a given World Cup year, you can use the code &quot;df[df[&#39;year&#39;] == year][df[&#39;phase&#39;] == &#39;Final&#39;]&quot;.<br>4) Knockout matches are those that have any of the following values in the &#39;phase&#39; column: &#39;Final&#39;,&#39;Third-place&#39;, &#39;Semi-Finals&#39;, &#39;Quarter-finals&#39;, &#39;Round of 16&#39;.<br>5) Teams and scores are associated by the terms &#39;team 1&#39; or &#39;team 2&#39;. For example, the column &#39;team 1&#39; is always associated with the columns &#39;score team 1&#39; and &#39;penalties team 1&#39;, and similarly for &#39;team 2&#39;.<br>6) To determine the winner of a match you must compare the columns &#39;score team 1&#39; with &#39;score team 2&#39;, and for knockout matches, you may have to compare the columns &#39;penalties team 1&#39; and &#39;penalties team 2&#39;. If the value in &#39;score team 1&#39; is bigger than &#39;score team 2&#39; then the team in the &#39;team 1&#39; column wins the match. If the value in &#39;score team 1&#39; is smaller than &#39;score team 2&#39; then the team in the &#39;team 2&#39; column wins the match. If the value in column &#39;score team 1&#39; is equal to the value in &#39;score team 2&#39; then the game is a draw, unless it is a knockout match, in which case, the penalties columns need to be checked.<br>7) In a knockout match, if the columns &#39;score team 1&#39; with &#39;score team 2&#39; are equal, the winner is determined by the penalties scores, for which you must look at the &#39;penalties team 1&#39; and &#39;penalties team 2&#39; columns. If the value in &#39;penalties team 1&#39; is bigger than &#39;penalties team 2&#39; then the team in the &#39;team 1&#39; column wins the match. If the value in &#39;penalties team 1&#39; is smaller than &#39;penalties team 2&#39; then the team in the &#39;team 2&#39; column wins the match.<br>8) You only know about matches belonging to FIFA Men&#39;s Football World Cups. If asked about matches or games in any other tournament, competition or league, say you do not know.<br>9) If you do not know the answer, just say I don&#39; know.<br><br>You should use the tools below and the instructions above to answer the question posed of you:<br><br>python_repl_ast: A Python shell. Use this to execute python commands. Input should be a valid python command. When using this tool, sometimes output is abbreviated - make sure it does not look abbreviated before using it in your answer.<br><br>Use the following format:<br><br>Question: the input question you must answer<br>Thought: you should always think about what to do<br>Action: the action to take, should be one of [python_repl_ast]<br>Action Input: the input to the action<br>Observation: the result of the action<br>... (this Thought/Action/Action Input/Observation can repeat N times)<br>Thought: I now know the final answer<br>Final Answer: the final answer to the original input question<br><br><br>This is the result of `print(df.head())`:<br>{df_head}<br><br>Begin!<br>Question: {input}<br>{agent_scratchpad}\<br>&quot;&quot;&quot;</pre><p>Note that the new prompt does not include any actual data example, so this could still be considered ‘zero-shot prompt engineering’ as opposed to a ‘one-shot’ or ‘ ‘few-shot prompting’.</p><p>Now, let’s repeat the same questions as before and see if we get better results.</p><blockquote><strong><em>Query 1 (new prompt)<br>Simple selection and count</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZvXON8yKma-otYARVi0Bw.png" /></figure><p>Same correct result. The number of tokens in this query was <strong>2369 </strong>— considerably higher than the <strong>1252</strong> tokens used by the same query with the original prompt.</p><blockquote><strong><em>Query 2 (new prompt)</em><br><em>Extracting semantics from the data</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ETiwki8k656YwMU7XHFrwQ.png" /></figure><p>In this case, the reasoning was derived directly from the re-engineered prompt. It remains to be seen if it will actually and always use this reasoning when answering questions.</p><blockquote><strong><em>Query 3 (new prompt)<br>Interpreting the question</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L412arwenVGRCUVEVOPzuA.png" /></figure><p>In this case above, using the original prompt, the model was not able to give a complete correct answer, and it used a large number of tokens (<strong>3897</strong>). With the re-engineered prompt, the model gives a fully correct answer and it uses only <strong>2488 tokens</strong>, because, although the prompt is bigger, it finds the final match with one Pandas command that returns a single line back to the model as the <em>observation</em>.</p><p>The same question worded differently, also produced fully correct results, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUvNwWJwKLrw7rPK36JuJQ.png" /></figure><p>The final question about the 1974 World Cup final that had initially produced completely wrong results, now returns the correct result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IPambzDJIj-yX9T7Z3Ffw.png" /></figure><blockquote><strong><em>Query 4 (new prompt)<br>Complex queries</em></strong></blockquote><p>These queries require more complex language understanding with more complex Pandas actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GBAVq6T3P4Tjo-Gp3dubnA.png" /></figure><p>Italy did score 11 goals in the 1938 World Cup. Note that the <em>thought</em> process understood that it had to sum the goals scored by Italy as ‘team 1’ with the goals scored as ‘team 2’, and it called the correct Pandas functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ru8zQrM-3YU3UPq4wIhaSw.png" /></figure><p>Italy did suffer 5 goals in the 1938 World Cup. Here the <em>thought</em> process was to sum the goals scored by ‘team 2’ when Italy was ‘team 1’ with the goals scores by ‘team 1’ when Italy was ‘team 2’.</p><p>How about something requiring multiple sums, such as the total number of goals in a World Cup, like the question below. The model produced the correct result!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxuEDgivRt4kv9QvGxRlfQ.png" /></figure><p>Now a similar query with an additional complication — asking the model to include penalty goals. Again, the correct answer was produced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*taDwAlffkc1LVEvpIHZ8gQ.png" /></figure><p>The final query below requires looking at the whole table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsRbVruoelz5oayBec0flw.png" /></figure><p>Here the answer is not correct! The <em>thought</em> process interpreted the question correctly, but the Pandas functions applied in the <em>action</em> step were not correct. The <em>groupby</em> function needed to be applied to each subset of lines of a given year, and the results compared. For the record, the World Cup with most goals scored was the 2022 edition with 172 goals (not counting penalty shootouts).</p><h3>Summary and Conclusions</h3><p>This article explore the use of Langchain’s Pandas dataframe agent and language models for understanding of tabular data using natural language. I presented several queries and results using the default prompt (provided with the pandas_dataframe_agent) and contrasted them against using a re-engineered prompt.</p><p>The Pandas dataframe agent is a significant improvement step over the basic <a href="https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.csv_loader.CSVLoader.html">CSVLoader</a>, and the language model does a decent job at trying to understand the table and apply the correct Pandas functions to extract data from the table. However, it is not able to understand all semantics associated with the columns of the table and it produces the wrong answer to questions that need such understanding.</p><p>I used prompt engineering to <em>tell</em> the model the semantics of the columns and how they relate to each other. This improved significantly the correctness of the results, but it is still not perfect. This is a very active research area and models and agents still need to be improved in order to reach human-level understanding of tabular data.</p><p>As a final observation, the wording of the modified prompt also seem to matter considerably. I tried writing the same information about the meaning of the columns in several different ways, and the results were different. It seems that the more explicit an instruction is the better the understanding, and it also helps to repeat parts of an instruction in different contexts.</p><p>The csv file with the World Cup games and the Jupyter notebook used in this article can be found <a href="https://github.com/rberga/SemanticSearch/tree/master/tabular_data_understanding/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea76d7d3dcbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Searching your own files in natural language]]></title>
            <link>https://rberga.medium.com/searching-your-own-files-in-natural-language-a28557511214?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a28557511214</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[milvus]]></category>
            <category><![CDATA[semantic-search]]></category>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 15:11:29 GMT</pubDate>
            <atom:updated>2023-09-09T23:11:38.393Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to search and chat with your own files using LangChain, vector databases and LLMs. Useful, but with limitations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vK1710FNV4mq-j-FXgsUMA.png" /><figcaption>This image was created with the assistance of DALL·E 2</figcaption></figure><p>It is impossible today not to be overloaded with technical information. If you are like me, you probably receive dozens of links to interesting medium articles, journal and conference papers, technical reports, etc., many of which are worth reading. When you do not have time to read them immediately (who does?), you save them locally in your computer. Only to never find them again.</p><p>I have a constant problem of not finding “interesting” files that I saved. I usually remember what they were about, but the search on file content, available on Windows or Linux, always left me disappointed.</p><p>In the past, my company developed a full search system using Solr which could be deployed in a laptop and used to search on keywords on any type of file (pdf, csv, html, docx, xlsx, etc.). It would regularly scan all files in your computer, selected using a variety of filters, and index their contents as a search index. This worked very well, but it did not have the ability to pull together data from various sources and present a summarized version of the information available. For example, a typical search engine would return the lines and files containing the word “LLM”, but it would not understand a query like “Summarize the information about LLMs in my files and also return the sources”. Enter LangChain, vector databases and OpenAI/GPT.</p><p>In order to create such system we need to implement two flows: one responsible for collecting the information from our files and making them available in a searchable format, and the other responsible for understanding the user question and, using the content of our files, extract the relevant information and display it back to the user in a natural language format. These flows are illustrated below and explained in detail with corresponding Python implementations in the following sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YUHDa8lO22xIBxzUD-oeGQ.png" /><figcaption>Search System Indexing and User Flows</figcaption></figure><h4><strong>Frameworks and Tools</strong></h4><p>Before we go into details, let’s review the software environments that we will be using.</p><p><strong>OpenAI</strong><br>The LLMs (large-language models) that I will using are provided by OpenAI. These models are not free, and there are many free models from other providers that work just as well. But for our purposes the OpenAI models are relatively cheap and easy to use.</p><p>For the generation of the embeddings I will use model <a href="https://platform.openai.com/docs/guides/embeddings/what-are-embeddings"><em>text-embedding-ada-002</em></a><em> </em>which has a cost of <a href="https://openai.com/pricing#language-models">$0.0001 for 1K tokens</a>. For the user query LLM I will use <a href="https://platform.openai.com/docs/models/gpt-3-5">GPT3.5 Turbo</a> which has a cost between <a href="https://openai.com/pricing#language-models">$0.0015 and $0.002 for 1K tokens</a>. <a href="https://platform.openai.com/docs/introduction/tokens">According to OpenAI</a>, as a rough rule of thumb, 1 token is approximately 4 characters or 0.75 words for English text.</p><p><strong>Langchain</strong><br><a href="https://docs.langchain.com/docs/">LangChain</a> is a software framework with higher-level classes and methods for accessing and using language models and vector databases. It contains common classes that allow for more uniform interfaces to different language models, thus making it easier to develop new applications, not necessarily tied to one language model. Langchain has also higher-level methods to interact with different vector databases. In addition, LangChain has <em>chains,</em> which are combinations of methods that can be applied to specific use-cases, e.g., chatbots.</p><p><strong>Milvus (Vector DB)</strong><br><a href="https://milvus.io/">Milvus</a> is a vector database, where the data stored are primarily vector embeddings and data retrieval is done through vector similarity search algorithms, such as ANN (approximate nearest neighbors). Milvus is open-source software with an active and helpful community.</p><h3>Indexing Flow</h3><p>This flow is responsible for crawling a directory structure, collecting the files that we may be interested in searching, and storing the information. It can be run once a day, or more often depending on how fast the files change. The specific steps are shown in the figure above and described below.</p><h4>Step 1: List files</h4><p>This step, implemented by the <em>list_files</em> function below, accepts an initial directory and a list of file extensions and returns a list of all files under the initial directory and all its subdirectories, with extensions in the list.</p><pre>import os<br><br>def list_files(initdir: str, file_extensions: list):<br>    &#39;&#39;&#39;<br>    Returns a list of file under initdir and all its subdirectories<br>    that have file extension contained in file_extensions.<br>    &#39;&#39;&#39; <br>    file_list = []<br>    file_count = {key: 0 for key in file_extensions}  # for reporting only<br>    <br>    # Traverse through directories to find files with specified extensions<br>    for root, _, files in os.walk(initdir):<br>        for file in files:<br>            ext = file.split(&#39;.&#39;)[-1].lower()<br>            if ext in file_extensions:<br>                file_path = os.path.join(root, file)<br>                file_list.append(file_path)<br>                # increment type of file<br>                file_count[ext] += 1<br>    <br>    total = len(file_list)<br>    print(f&#39;There are {total} files under dir {initdir}.&#39;)<br>    for k, n in file_count.items():<br>        print(f&#39;   {n} : &quot;.{k}&quot; files&#39;)<br>    return file_list</pre><p>This function could be further enhanced by storing the name and timestamp or hash of each file and testing if the file has changed since the last visit, in which case it would need to be indexed again.</p><h4><strong>Step 2:</strong> <strong>Convert file to text + metadata and split it</strong></h4><p>In order to process non-plain-text files (e.g., docx, xlsx, pdf, etc.) we must first convert them to a textual representation. The text content may be broken up into chunks in a process called <em>splitting</em>. Splitting is needed because the embedding models are limited in the size of text (more specifically, tokens) they can process. Moreover, having a long text represented by several embedding vectors, instead of just one, allows a finer-grain representation of the full text, which can improve search results.</p><p>LangChain contains several classes for converting different types of files into textual representations. These classes, called <a href="https://python.langchain.com/docs/integrations/document_loaders/">document loaders</a>, use file converters to generate text, add associated metadata and create a list of objects of type <a href="https://api.python.langchain.com/en/latest/schema/langchain.schema.document.Document.html#langchain.schema.document.Document">Document</a>.</p><p>The following function takes a text splitter method and a file path, and depending on the type of file, it calls the appropriate text converter, followed by the text splitter, which creates a list of Documents. If there are no errors, this list of Documents is returned.</p><pre>def convert_and_split(text_splitter, file_path: str) -&gt; list:<br>    &#39;&#39;&#39;<br>    It converts the file to a textual representation, adds metadata, and<br>    calls the text_splitter to split long sequences.<br>    It returns a list of Documents or None is there is any error in the <br>    file conversion.<br>    &#39;&#39;&#39;<br><br>    texts = None<br>    try:<br>        if file_path.endswith(&#39;.pdf&#39;):<br>            loader = PyMuPDFLoader(file_path)<br>            pages = loader.load()<br>            texts = text_splitter.split_documents(pages)<br><br>        elif file_path.endswith(&#39;.docx&#39;) or file_path.endswith(&#39;.doc&#39;):<br>            loader = UnstructuredWordDocumentLoader(file_path)<br>            docs = loader.load()<br>            texts = text_splitter.split_documents(docs)<br>        <br>        elif file_path.endswith(&#39;.txt&#39;):<br>            loader = TextLoader(file_path)<br>            docs = loader.load()<br>            texts = text_splitter.split_documents(docs)<br><br>        elif file_path.endswith(&#39;.xlsx&#39;) or file_path.endswith(&#39;.xls&#39;):<br>            loader = UnstructuredExcelLoader(file_path)<br>            docs = loader.load()<br>            texts = text_splitter.split_documents(docs)<br><br>        elif file_path.endswith(&#39;.csv&#39;):<br>            loader = CSVLoader(file_path)<br>            docs = loader.load()<br>            texts = text_splitter.split_documents(docs)<br>        <br>        elif file_path.endswith(&#39;.pptx&#39;) or file_path.endswith(&#39;.ppt&#39;):<br>            loader = UnstructuredPowerPointLoader(file_path)<br>            docs = loader.load()<br>            texts = text_splitter.split_documents(docs)<br><br>        else:<br>            print(f&quot;Error: invalid file type: {file_path}&quot;)<br><br>    except Exception as e:<br>        print(f&quot;Error processing {file_path}: {e}&quot;)<br><br>    return texts</pre><p>There are several file converters, also called Document Loaders in LangChain. You can find descriptions and code examples of <a href="https://python.langchain.com/docs/integrations/document_loaders/">document loaders integrated with other systems</a> and <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/">document loaders for different file formats</a>.</p><p>There are several types of <a href="https://python.langchain.com/docs/modules/data_connection/document_transformers/">text splitters supported by LangChain</a>. A good discussion of different methods is presented <a href="https://towardsdatascience.com/how-to-chunk-text-data-a-comparative-analysis-3858c4a0997a">here</a>. After trying a few different splitters and checking the search quality, I chose to use the <a href="https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/split_by_token#nltk">NLTK-based splitter</a> with default parameters, but I recommend you experiment with your own data and see what works best. The definition of the text splitter I used is:</p><pre>from langchain.text_splitter import NLTKTextSplitter<br><br>text_splitter = NLTKTextSplitter(chunk_size=1000)</pre><h4>Steps 3+4: Generate embeddings and store data in Vector DB</h4><p>Embeddings are numerical representations of data, as dense vectors of real numbers. These vectors capture the relationships between different parts of entities within the data. Embeddings have been successfully used to represent images, videos, graphs, and text. For a good overview of embeddings, check out articles <a href="https://www.featureform.com/post/the-definitive-guide-to-embeddings">here</a> and <a href="https://encord.com/blog/embeddings-machine-learning/">here</a>.</p><p>Semantic search and many other natural language applications rely on embeddings to capture relationships between words and sentences. The actual embeddings are generated using neural network models. Here I will use <a href="https://platform.openai.com/docs/guides/embeddings/what-are-embeddings"><em>text-embedding-ada-002</em></a><em> provided by OpenAI</em>. LangChain has classes that <a href="https://python.langchain.com/docs/integrations/text_embedding/">integrate with several model providers</a>, both paid and open-source.</p><p>Our embedding function is defined as follows. You will need to create an account with OpenAI and get an API key. By default, the embedding vectors created using model <em>text-embedding-ada-002</em> have <strong>1536</strong> dimensions. For security reasons, it is better not to pass the API key directly in the code, use an environment variable.</p><pre>from langchain.embeddings.openai import OpenAIEmbeddings<br><br>embedding = OpenAIEmbeddings(model=&quot;text-embedding-ada-002&quot;, openai_api_key=&lt;insert-your-key-here&gt;)</pre><p>Once the embedding function is defined, we can take the list of Documents returned in Step 2 and pass it to the method that creates the embedding vectors for the Documents and stores them into the vector database.</p><p>Here I used the <a href="https://milvus.io/">Milvus Database</a>. I tried using <a href="https://www.trychroma.com/">Chroma DB</a> as well, and although it works, I found Milvus more intuitive, easier to install (using docker) and with a visual interface (Attu) that allowed me to inspect the stored entities and their vectors easily. Check <a href="https://github.com/binodsuman/nlp/blob/master/Installation_guide.txt">this link</a> for details on how to install Milvus and Attu using docker.</p><p>This function <em>add_docs_to_milvus</em> (below) calls method <a href="https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.milvus.Milvus.html#langchain.vectorstores.milvus.Milvus.from_documents"><em>Milvus.from_documents(…)</em></a> to actually insert the documents into the database. If a schema is not pre-defined, this method uses the metadata present in the first document to define the schema fields, and in addition, it adds to the schema a <em>primary key</em>, a <em>text</em> and a <em>vector</em> fields. This schema is defined as static.</p><p>This, however, causes an error if we try to insert a second document with fields that do not match the first one. This is exactly the case for when we convert files of different types. PDF files, when converted to LangChain Documents will have certain metadata fields which do not match with the fields of the converted MS Excel file.</p><p>In order to avoid this error, and make sure all types of files processed by the system end up stored with the same fields in Milvus (important for search), we need to define a schema with precisely the desired fields and make sure that all file types, when converted, contain these desired fields.</p><p>For the purposes of doing semantic search on documents, we need basically the embeddings vector, the actual text that creates the embeddings, and the full path of the file. All LangChain file loaders create Documents that have a primary key, an embedding vector, a text field, and a source file (containing the file path). Hence we need a schema that defines these four fields.</p><pre>def create_collection_schema(collection_name: str):<br>    &#39;&#39;&#39;<br>    Defines a schema for a collection, with 4 fields.<br>    &#39;&#39;&#39;<br>    # define the necessary fields and their types<br>    fields = [<br>        FieldSchema(name=&quot;pk&quot;, dtype=DataType.INT64, is_primary=True, auto_id=True),<br>        FieldSchema(name=&quot;source&quot;, dtype=DataType.VARCHAR, max_length=512),<br>        FieldSchema(name=&quot;text&quot;, dtype=DataType.VARCHAR, max_length=65535),<br>        FieldSchema(name=&quot;vector&quot;, dtype=DataType.FLOAT_VECTOR, dim=1536)<br>    ]<br><br>    # create a collection schema with fields<br>    schema = CollectionSchema(<br>        fields, <br>        description=&quot;Schema for different file types&quot;,<br>    )<br><br>    collection = Collection(collection_name, schema)<br>    return collection</pre><p>With the schema defined for a collection, we can add the documents to Milvus with the following function. Note that we only need to pass the collection name, being the same for which the schema was associated with.</p><pre>def add_docs_to_milvus(docs, embedding, collection_name, file_path):<br>    &#39;&#39;&#39;<br>    Store embedding vectors for docs int Milvus DB, under collection_name.<br>    Return vector_db if successful, otherwise return None<br>    &#39;&#39;&#39;<br>    n = 0<br>    vector_db = None<br><br>    # now generate embeddings and store texts and vectors into vector database<br>    try:<br>        vector_db = Milvus.from_documents(<br>            documents=docs,<br>            embedding=embedding,<br>            collection_name=collection_name,<br>            connection_args={&quot;host&quot;: &quot;localhost&quot;, &quot;port&quot;: &quot;19530&quot;},<br>        )<br>        <br>    except Exception as e:<br>        print(f&quot;Error storing {file_path} into Milvus: {e}&quot;)<br>        vector_db = None<br><br>    finally:<br>        # flush new entities added to collection. <br>        # Not really needed. Just a way to keep <br>        # track of collection size as files are processed.<br>        collection = Collection(collection_name)<br>        collection.flush()<br>        print(f&#39;Collection {collection_name} has {collection.num_entities} entities after processing file {file_path}&#39;)<br>    <br>    return vector_db</pre><h4>Putting it all together</h4><p>Using the functions define above, we can define a function that will visit all files under a certain directory and index them all into Milvus.</p><pre>def index_files_milvus(filelst: list, collection_name: str):<br>    &#39;&#39;&#39;<br>    Given a list of files and a collection name, visit all files, convert them<br>    to text and metadata, split them into chunks and store their embeddings<br>    and metadata into Milvus for future semantic search.<br>    &#39;&#39;&#39;<br>    # define the text_splitter and chunk size<br>    text_splitter = NLTKTextSplitter(chunk_size=1000)<br>    <br>    # define the language model to be used for generating the vector embeddings<br>    # For security reasons, it is better not to pass the API key directly in<br>    # the code, use an environment variable.<br>    embedding = OpenAIEmbeddings(<br>        model=&quot;text-embedding-ada-002&quot;,<br>        openai_api_key=&#39;insert-your-OpenAI-api-key-here&#39;<br>        )<br><br>    # create a schema and a collection using it<br>    create_collection_schema(collection_name)<br><br>    # bookeeping variables<br>    error_files = []    # list of files that failed<br>    cnt = 0             # number of files successfully stored<br><br>    # Process each file by calling the appropriate file loader, depending on the file type<br>    for file_path in filelst:<br>        cnt += 1<br>        print(f&#39;=&gt; processing file {cnt} / {len(filelst)}: {file_path}&#39;)<br><br>        # convert the file into a list of Documents to be stored<br>        texts = convert_and_split(text_splitter, file_path)<br><br>        if not texts:<br>            error_files.append(file_path)   # keep track of files that failed<br>            cnt -= 1                        # count only successful conversions<br>            continue<br><br>        # now generate embeddings and store texts and vectors into vector database<br>        vector_db = add_docs_to_milvus(texts, embedding, collection_name, file_path)<br><br>        if not vector_db:<br>            error_files.append(file_path)   # keep track of files that failed<br>            cnt -= 1                        # count only successful conversions<br>            continue<br>        pass   # end of for loop<br><br>    # print summary<br>    print(f&#39;[index_files] Generated embeddings for {cnt} / {len(filelst)} under dir {initdir}&#39;)<br>    if len(error_files) &gt; 0:<br>        print(f&#39;              Files with problems:&#39;)<br>        for f in error_files:<br>            print(f&#39;                 {f}&#39;)<br>    else:<br>        print(f&#39;              All files successfully processed&#39;)<br>    <br>    pass  # end of function</pre><p>The code below runs everything, given a list of file extensions and an initial directory.</p><pre>import os<br>import openai<br><br>from langchain.vectorstores import Milvus<br>from langchain.embeddings.openai import OpenAIEmbeddings<br>from langchain.text_splitter import NLTKTextSplitter<br>from langchain.document_loaders import TextLoader<br>from langchain.document_loaders import UnstructuredExcelLoader<br>from langchain.document_loaders import UnstructuredWordDocumentLoader<br>from langchain.document_loaders import UnstructuredPowerPointLoader<br>from langchain.document_loaders import CSVLoader<br>from langchain.document_loaders import PyMuPDFLoader<br><br>from pymilvus import Collection, FieldSchema, CollectionSchema, DataType, connections<br><br>if __name__ == &quot;__main__&quot;:<br><br>    file_extensions = [&#39;pdf&#39;, &#39;doc&#39;, &#39;docx&#39;, &#39;xlsx&#39;, &#39;xls&#39;, &#39;ppt&#39;, &#39;pptx&#39;, &#39;txt&#39;, &#39;csv&#39;]<br>    initdir = &#39;add_your_initial_directory&#39;<br>    <br>    # Connect to the milvus server<br>    conn = connections.connect(host=&quot;127.0.0.1&quot;, port=19530)<br><br>    # define a collection name (it can be any name)<br>    collection_name=&#39;testfiles_repo&#39;<br><br>    # visit directory structure and create list of files with given extensions<br>    filelst = list_files(initdir, file_extensions)<br>    <br>    # visit all files, convert them to text and index them into Milvus<br>    index_files_milvus(filelst, collection_name)<br>    pass</pre><h4>Test data and entities stored in Milvus</h4><p>I created a simple directory structure with subdirectories containing files of all allowed types. I chose files freely available on the web and you can download them from <a href="https://github.com/rberga/SemanticSearch">my github</a> if you want to duplicate exactly my environment. These files are relatively small and I encourage you to give them a quick look — it will help understand the queries I will make about them later on.</p><pre><strong>Description of files used for indexing and searching</strong><br><br><strong>CSV</strong><br>All Premier League football games in the 2022-2023 season.<br><em>./testfiles/csv/premier_league_all_matches_2022-2023-season.csv</em><br>Downloaded from <a href="https://www.kaggle.com/datasets/sinansaglam/premier-league-last-5-seasons-match-scores-and-xgs">https://www.kaggle.com/datasets/sinansaglam/premier-league-last-5-seasons-match-scores-and-xgs</a><br>and removed some columns to decrease the amount of data.<br><br><strong>EXCEL</strong><br>Results of the 2020 Presidential election by state and candidate<br><em>./testfiles/excel/2020_presidential_elections.xlsx</em><br>Downloaded from <a href="https://www.kaggle.com/datasets/tunguz/us-elections-dataset">https://www.kaggle.com/datasets/tunguz/us-elections-dataset</a> <br>as csv, removed data from other elections, removed non-relevant columns to<br>decrease the amount of data, and saved as .xlsx format.<br><br><strong>PDF</strong><br>Two articles about Generative Adversarial Networks, available on Arxiv<br><em>./testfiles/pdf/InsetGAN.pdf</em>  (<a href="https://arxiv.org/pdf/2203.07293.pdf">https://arxiv.org/pdf/2203.07293.pdf</a>)<br><em>./testfiles/pdf/garmentGAN.pdf</em> (<a href="https://arxiv.org/pdf/2003.01894.pdf">https://arxiv.org/pdf/2003.01894.pdf</a>)<br><br><strong>PPT</strong><br>Presentation by Washington State Department of Social and Health Services on Generations in the workplace.<br><em>./testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx</em><br>Downloaded from <a href="https://www.dshs.wa.gov/sites/default/files/DDA/dda/documents/training/Generations%20in%20the%20Workplace%20PPT%20%28Final%29.pptx">https://www.dshs.wa.gov/sites/default/files/DDA/dda/documents/training/Generations%20in%20the%20Workplace%20PPT%20%28Final%29.pptx</a><br><br><strong>TEXT</strong><br>Blog post about Characteristics of Generation X Professionals<br><em>./testfiles/text/genx_characteristics.txt</em><br>Downloaded from <a href="https://www.indeed.com/career-advice/career-development/generation-x-professional-characteristics">https://www.indeed.com/career-advice/career-development/generation-x-professional-characteristics</a><br>and saved as text file.<br><br><strong>WORD</strong><br>Made up reviews received from a hypothetical journal about a hypothetical <br>manuscript submitted to the journal.<br><em>./testfiles/word/anon_reviews.docx</em></pre><p>By running the <em>main</em> code above on this directory structure (<em>./testfiles</em>), we get the following log. The only reason for including this log is to draw your attention to the number of entities added to the vector database for each file.</p><pre>There are 7 files under dir ./testfiles.<br>   2 : &quot;.pdf&quot; files<br>   0 : &quot;.doc&quot; files<br>   1 : &quot;.docx&quot; files<br>   1 : &quot;.xlsx&quot; files<br>   0 : &quot;.xls&quot; files<br>   0 : &quot;.ppt&quot; files<br>   1 : &quot;.pptx&quot; files<br>   1 : &quot;.txt&quot; files<br>   1 : &quot;.csv&quot; files<br>=&gt; processing file 1 / 7: ./testfiles/csv/<strong>premier_league_all_matches_2022-2023-season.csv</strong><br>File ./testfiles/csv/premier_league_all_matches.csv <strong>added 380 entities<br></strong>to collection testfiles_repo, to a total of 380 entities.<br><br>=&gt; processing file 2 / 7: ./testfiles/excel/<strong>2020_presidential_elections.xlsx</strong><br>Created a chunk of size 6016, which is longer than the specified 1000<br>Created a chunk of size 21291, which is longer than the specified 1000<br>File ./testfiles/excel/2020_presidential_elections.xlsx <strong>added 3 entities<br></strong>to collection testfiles_repo, to a total of 383 entities.<br><br>=&gt; processing file 3 / 7: ./testfiles/pdf/<strong>garmentGAN.pdf</strong><br>File ./testfiles/pdf/garmentGAN.pdf <strong>added 46 entities</strong> <br>to collection testfiles_repo, to a total of 429 entities.<br><br>=&gt; processing file 4 / 7: ./testfiles/pdf/<strong>InsetGAN.pdf</strong><br>File ./testfiles/pdf/InsetGAN.pdf <strong>added 88 entities</strong><br>to collection testfiles_repo, to a total of 517 entities.<br><br>=&gt; processing file 5 / 7: .testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx<br>Created a chunk of size 1875, which is longer than the specified 1000<br>File ./testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx <strong>added 19 entities</strong><br>to collection testfiles_repo, to a total of 536 entities.<br><br>=&gt; processing file 6 / 7: ./testfiles/text/genx_characteristics.txt<br>File ./testfiles/text/genx_characteristics.txt <strong>added 7 entities</strong><br>to collection testfiles_repo, to a total of 543 entities.<br><br>=&gt; processing file 7 / 7: ./testfiles/word/anon_reviews.docx<br>File ./testfiles/word/anon_reviews.docx <strong>added 8 entities</strong><br>to collection testfiles_repo, to a total of 551 entities.<br><br>[index_files] Generated embeddings for 7 / 7 under dir ./testfiles<br>              <strong>All files successfully processed</strong></pre><p>CSV file <strong>premier_league_all_matches_2022–2023-season.csv</strong> has 380 rows (excluding headers) and got mapped by <a href="https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.csv_loader.CSVLoader.html"><strong>CSVloader</strong></a> into 380 entities, or one entity per csv row. For example, the first line in this csv file gets stored in Milvus as follows (visible in <a href="https://milvus.io/docs/v2.0.x/attu.md">Attu</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZzZBr9O9nlJ7lXDPMRJAg.png" /></figure><p>This line-by-line splitting and storing <em>is not usually the case</em> for other file types. PDF and text files, for example get split by chunks of tokens or sentences. Excel files, when using the <a href="https://api.python.langchain.com/en/latest/document_loaders/langchain.document_loaders.excel.UnstructuredExcelLoader.html"><strong>UnstructuredExcelLoader</strong></a>, get split in a way that several rows (or even half rows) are placed into one document, and consequently into one Milvus entity. For example, Excel file <strong>./testfiles/excel/2020_presidential_elections.xlsx</strong>, which has 547 rows (excluding headers), got split into 3 Documents, stored in Milvus as 3 entities. The first entity included 81 rows, the second entity had 312 rows and the third, 154 rows. Considering that all data in each entity gets mapped to a single vector, the information is stored with widely different granularities, depending on the file type, the file loader and the text splitter used. This has a direct impact in the quality of the search, as we will see in the next section.</p><h3>User Query Flow</h3><p>In order to make a user query on the context defined by the files that were indexed we need to follow the following steps.</p><h4>Step A: Load vector database</h4><p>In order to search, we first need to load the vector database that was indexed and stored by the <em>Indexing Flow</em>. This is done by creating a <a href="https://python.langchain.com/docs/integrations/vectorstores/milvus">Milvus</a> object, pointing to the previously created collection. The <em>embedding_function</em> parameter is the same as defined in the previous section.</p><pre>from langchain.vectorstores import Milvus<br><br>vector_db = Milvus(embedding_function=embeddings,<br>                   collection_name=&#39;testfiles_repo&#39;,<br>                   connection_args={&quot;host&quot;: &quot;localhost&quot;, &quot;port&quot;: &quot;19530&quot;})</pre><h4>Step B: Create a Retriever from the vector DB</h4><p>The <a href="https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.base.VectorStoreRetriever.html"><strong>retriever</strong></a> model encodes questions and contexts into the same vector space. It encodes questions to be compared to the context vectors in a <em>vector database</em> to <em>retrieve</em> the most relevant contexts. Using the vector_db we can define the retriever as:</p><pre>&#39;&#39;&#39;<br>This define a retriever from the Milvus database. Specifying that 5 documents<br>should be returned. Other parameters as default.<br>&#39;&#39;&#39;<br>retriever = vector_db.as_retriever(search_kwargs={&quot;k&quot;: 5})<br>print(retriever)</pre><pre><strong>---- Output ----</strong><br>   tags=[&#39;Milvus&#39;, &#39;OpenAIEmbeddings&#39;] <br>   metadata=None vectorstore=&lt;langchain.vectorstores.milvus.Milvus object at 0x7fcea5e24c10&gt;<br>   search_type=&#39;similarity&#39;<br>   search_kwargs={&#39;k&#39;: 5}</pre><h4>Step C: Configure a Retrieval Chain</h4><p><a href="https://python.langchain.com/docs/modules/chains/">LangChain defines <em>chain</em></a> as a way to <em>combine multiple components together to create a single, coherent application</em>.</p><p>The chain that we need in our application takes user input, formats it with a PromptTemplate, passes the prompt augmented with the context retrieved from a vector database to an LLM and returns the LLM’s response.</p><p><strong>Prompt Template</strong></p><p>The prompt combines instructions to the language model with a question (query). If we want the model to respond to queries using exclusively data from the context (i.e., the contents of the vector database), then we need to tell it. Failure to do so may result in the model returning data from its own training (not from the vector db) if the vector db search does not find anything relevant.</p><pre>from langchain.prompts import PromptTemplate<br><br>prompt_template = \<br>&quot;&quot;&quot;<br>Use the following pieces of context to answer the question at the end. <br>If you don&#39;t know the answer based on the context only, say you do not<br>know the answer.<br><br>{context}<br><br>Question: {question}<br>&quot;&quot;&quot;<br>PROMPT = PromptTemplate(<br>    template=prompt_template, input_variables=[&quot;context&quot;, &quot;question&quot;]<br>)<br><br>chain_type_kwargs = {&quot;prompt&quot;: PROMPT}</pre><p><strong>Retrieval Chain</strong></p><p>We used <a href="https://python.langchain.com/docs/use_cases/question_answering/how_to/vector_db_qa">LangChain’s RetrievalQA chain</a> which is designed for Q&amp;A type of applications. The parameters to this function allow us to specify the language model to be used (for submitting the query and generating answers), the chain type, the retriever function and the prompt to be used.</p><pre>from langchain.chains import RetrievalQA<br>from langchain.chat_models import ChatOpenAI<br><br>qa_chain = RetrievalQA.from_chain_type(<br>    llm=ChatOpenAI(temperature=0.2, model_name=&quot;gpt-3.5-turbo&quot;, openai_api_key=&lt;insert-your-key-here&gt;),<br>    chain_type=&quot;stuff&quot;, <br>    retriever=retriever, <br>    chain_type_kwargs=chain_type_kwargs,<br>    return_source_documents=True<br>)</pre><p>The <em>llm</em> parameter points to the API that interfaces with the language model. In this case we are using <a href="https://api.python.langchain.com/en/latest/chat_models/langchain.chat_models.openai.ChatOpenAI.html">ChatOpenAI</a> and the <a href="https://platform.openai.com/docs/models/gpt-3-5">gpt-3.5-turbo model</a> with a low temperature value in order to force the replies to be deterministic and closer to the content provided.</p><p>The <a href="https://python.langchain.com/docs/modules/chains/document/stuff"><em>chain_type=”stuff”</em></a><em> </em>parameter performs the basic function of taking a list of documents, inserting them all into the prompt and passing the prompt to the LLM. The documentation indicates that this chain type is better-suited for applications where documents are small and only a few are passed in for most calls.</p><p>We also need to pass the <em>prompt </em>through the <em>chain_type_kwargs</em> parameter. Finally, we also set <em>return_source_documents</em> to True to get a list of the source documents whose parts were used in the query.</p><h4>Step D: Submitting the User Query</h4><p>We can simply use the retrieval chain (<em>qa_chain</em>) defined above to query the model, as follows. For illustrative purposes, we used function <em>process_llm_response</em> to print the results and the source files used if any.</p><pre>def process_llm_response(llm_response):<br>    print(llm_response[&#39;result&#39;])<br>    print(f&#39;{len(llm_response[&quot;source_documents&quot;])} Source documents:&#39;)<br>    # print(llm_response[&#39;source_documents&#39;])<br>    srcset = set()<br>    for source in llm_response[&quot;source_documents&quot;]:<br>        srcset.add(source.metadata[&#39;source&#39;])<br>    if srcset:<br>        print(&#39;\n\nSources:&#39;)<br>        for s in srcset:<br>            print(s)</pre><blockquote><strong>Query 1</strong> <br>Content from specific lines of the .csv file.</blockquote><pre>query1 = &quot;What were the scores of the games between Arsenal and Aston Villa?&quot;<br>llm_response = qa_chain(query1)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>Based on the given context, the scores of the games between Arsenal and Aston Villa were:<br><br>- On 8/31/2022, Arsenal won with a score of 2 x 1.<br>- On 2/18/2023, Aston Villa lost with a score of 2 x 4.<br><br>Sources:<br>./testfiles/csv/premier_league_all_matches_2022-2023-season.csv</pre><blockquote><strong>Query 2<br></strong>Content from specific lines of the .csv file.</blockquote><pre>query2 = &quot;Which team has its home at &#39;The American Express Community Stadium&#39;?&quot;<br>llm_response = qa_chain(query2)<br>process_llm_response(llm_response)</pre><pre>---- Output ----<br>Brighton<br><br>Sources:<br>./testfiles/csv/premier_league_all_matches_2022-2023-season.csv</pre><blockquote><strong>Query 3<br></strong>Tricky question — to make sure only the indexed content is used. According to the prompt, it should say it does not know the answer.</blockquote><pre>query3 = &quot;When was the Penicillin discovered?&quot;<br>llm_response = qa_chain(query3)<br>process_llm_response(llm_response)<br><br>---- Output ----<br>Based on the given context, there is no information provided about the <br>discovery of Penicillin. Therefore, I do not know the answer to the question.</pre><blockquote><strong>Query 4<br></strong>Summarizing specific info about GANs from .pdf files.</blockquote><pre>query4 = &quot;What can you tell me about Generative adversarial networks?&quot;<br>llm_response = qa_chain(query4)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>Based on the given context, generative adversarial networks (GANs) are a <br>successful image generation paradigm. They consist of two neural networks, <br>a generator and a discriminator. The generator learns the distribution of <br>real images by generating images that are indistinguishable from real images, <br>while the discriminator learns to classify the images into real and fake. <br>GANs have impressive capabilities for synthesizing realistic content and are <br>commonly used in various synthesis tasks, such as text-to-image and <br>image-to-image tasks. GANs have also been used for person image generation, <br>which is a challenging task due to the high variability of human pose, shape, <br>and appearance.<br><br>Sources:<br>./testfiles/pdf/InsetGAN.pdf<br>./testfiles/pdf/garmentGAN.pdf</pre><blockquote><strong>Query 5<br></strong>Asking about specific data present in .pdf files — note that the output includes data from the paper references.</blockquote><pre>query5 = &quot;Can you give me several examples of AI systems using GANs?&quot;<br>llm_response = qa_chain(query5)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>Based on the given context, several examples of AI systems using GANs are:<br><br>1. StyleGAN: It is a method used for creating near photorealistic images for multiple classes such as human faces, cars, and landscapes.<br>2. BigGAN: This architecture is often used for class-conditional image generation on the ImageNet dataset.<br>3. TileGAN: It is a method used for the synthesis of large-scale non-homogeneous textures.<br>Please note that these examples are mentioned in the context, and there may be other AI systems using GANs that are not mentioned here.<br><br>Sources:<br>./testfiles/pdf/garmentGAN.pdf<br>./testfiles/pdf/InsetGAN.pdf</pre><blockquote><strong>Query 6<br></strong>Asking for a summary of one of the .pdf files.</blockquote><pre>query = &quot;Can you give me a brief summary of garmentGAN?&quot;<br>llm_response = qa_chain(query)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>GarmentGAN is a new algorithm that uses generative adversarial methods to <br>perform image-based garment transfer. It allows users to virtually try on <br>clothing items before purchase and can handle complex body poses, hand <br>gestures, and occlusions. The algorithm requires two input images: a picture <br>of the target fashion item and an image of the customer. The output is a <br>synthetic image where the customer is wearing the target apparel. GarmentGAN <br>improves on existing methods in terms of the realism of generated imagery and <br>solves problems related to self-occlusions. It incorporates additional <br>information during training, such as segmentation maps and body key-point <br>information, to synthesize photographs at high levels of realism. <br>The algorithm comprises two separate GANs: a shape transfer network and an <br>appearance transfer network. It also uses a geometric alignment module and a <br>method of masking semantic segmentation maps to handle complex body poses and <br>occlusions. GarmentGAN preserves the identity of the reference person and <br>clothing items that should remain unchanged during the try-on task. <br>The algorithm has been evaluated and shown to outperform other state-of-the-art<br>models in terms of realism and quality of generated images.<br><br>Sources:<br>./testfiles/pdf/garmentGAN.pdf</pre><blockquote><strong>Query 7<br></strong>Querying the data present in files .txt and .pptx.</blockquote><pre>query7 = &quot;How is a generation defined?&quot;<br>llm_response = qa_chain(query7)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>A generation is defined as a group of individuals born and living <br>contemporaneously who share common knowledge and experiences that affect <br>their thoughts, attitudes, values, beliefs, and behaviors.<br><br>Sources:<br>./testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx<br>./testfiles/text/genx_characteristics.txt</pre><blockquote><strong>Query 8<br></strong>Querying specific information from files .txt and .pptx. This info is present in 3 places with different values: pages 6 and 13 of .pptx file, and in .txt file. It chose to return the data from page 13 of .pptx file.</blockquote><pre>query8 = &quot;When were Baby Boomers born?&quot;<br>llm_response = qa_chain(query8)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>Baby Boomers were born between 1946 and 1968.<br><br>Sources:<br>./testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx<br>./testfiles/text/genx_characteristics.txt</pre><blockquote><strong>Query 9<br></strong>Querying information with semantic understanding, present in files .pptx and .txt.</blockquote><pre>query9 = &quot;who are the groups that preceeded and succeeded Generation X?&quot;<br>llm_response = qa_chain(query9)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>The groups that preceded Generation X are the baby boomers, born between 1943<br>and 1964. The group that succeeded Generation X is the millennials, born<br>between 1981 and 2000.<br><br>Sources:<br>./testfiles/powerpoint/Generations in the Workplace PPT (Final).pptx<br>./testfiles/text/genx_characteristics.txt</pre><blockquote><strong>Query 10<br></strong>Asking for a summary of separate pieces of information contained in .docx file.</blockquote><pre>query10 = &quot;Summarize the reviews of paper entitled &#39;The Best Search System&#39; and tell me if the paper was accepted or not.&quot;<br>llm_response = qa_chain(query10)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>The reviews of the paper entitled &#39;The Best Search System&#39; are mixed. <br>Review 4 gave a strong reject, Review 1 gave an accept, Review 2 gave a reject,<br>Review 3 gave a weak accept, and the metareview recommended rejection. <br>Therefore, the paper was not accepted.<br><br>Sources:<br>./testfiles/word/anon_reviews.docx</pre><p><strong>Queries that did not work well and why</strong></p><p>Queries that required aggregating data from a higher number of documents did not work. We defined our vector db retriever in <strong>Step B </strong>with <strong>k = 5</strong>, that is, the search will return up to <strong>5 documents</strong>, the most similar to the query, where a document is likely to be a fragment of an original file. Hence, depending on the 5 documents that are retrieved, the retrieval chain may not be able to compose a complete answer (if at all). Let us look at some examples.</p><blockquote><strong>Query 11<br></strong>This query requires all rows in the .csv file to be returned by the query (since each row gets mapped into a vector db entity), but since we are only returning 5 documents, the answer is wrong.</blockquote><pre>query11 = &quot;What match had the highest attendance among all Premier League matches?&quot;<br>llm_response = qa_chain(query11)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>The match with the highest attendance among all Premier League matches is the<br>one between Manchester City and Everton on 12/31/2022 at the Etihad Stadium,<br>with an attendance of 53444.<br><br>Sources:<br>./testfiles/csv/premier_league_all_matches_2022-2023-season.csv</pre><p>The answer above is clearly wrong. By looking at the .csv file, one can get that the highest attendance was 75546 people, at the Manchester United 2 x 1 Manchester City game at the Old Trafford on 01/14/2023. If you ask it directly about this game, it does return the correct attendance.</p><blockquote><strong>Query 12<br></strong>Asking the for specific information from .xlsx files. As we discussed in section <strong><em>Test data and entities stored in Milvus</em></strong>, the .xlsx file is split and stored as 3 entities only, with high numbers of tokens (much higher than the <em>chunk_size=1000</em> specified in the text splitter). As a result the context and query end up containing too many tokens for the OpenAi model to process (under my plan) and we get an error about exceeding the number of tokens.</blockquote><pre>query12 = &quot;How many votes did Biden get in Alabama in 2020?&quot;<br>llm_response = qa_chain(query12)<br>process_llm_response(llm_response)</pre><pre><strong>---- Output ----</strong><br>...<br>...<br>InvalidRequestError: This model&#39;s maximum context length is 4097 tokens. <br>However, your messages resulted in 16202 tokens. Please reduce the length of <br>the messages.</pre><p>This file would have to be split into smaller chunks for this query to work. However the function <em>UnstructuredExcelLoader</em> used with the <em>NLTKTextSplitter</em> is not able to do that, even if smaller <em>chunk_size</em> is used. If you need to search on a row information a better option would be to convert the .xlsx to .csv and use the <em>CSVLoader</em> function.</p><h4>Conclusion</h4><p>In this article I explained how to create a search engine for your own files using natural language. The availability of LangChain and large language models from different companies makes it easier for complex applications using be developed quickly. However, the crux is in the details, and as I tried to show, querying a language model, even using limited content, does not always return <em>satisfying</em> results, although it does offer significantly more powerful ways of searching than using keyword-based search.</p><p>Many of the methods that we used in this article accept different parameters which can directly affect the quality of the results. I did not go into details on how to tweak these parameters, but I included enough links to the documentation so the interested reader is able to do that. I used OpenAI models which are not free — however for all calls (several hundred) that I made to the models while developing this article, the cost did not reach $2 dollars.</p><p>All code and test files presented here may be downloaded from this <a href="https://github.com/rberga/SemanticSearch">github</a> repository.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a28557511214" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analysis: Comparing Results of AWS, IBM and Google]]></title>
            <link>https://rberga.medium.com/sentiment-analysis-comparing-results-of-aws-ibm-and-google-70175a3bfa25?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/70175a3bfa25</guid>
            <category><![CDATA[aws-comprehend]]></category>
            <category><![CDATA[google-ai-platform]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[ibm-watson-nlu]]></category>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 19:45:11 GMT</pubDate>
            <atom:updated>2023-04-10T19:45:11.536Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>We ran 6000 phrases through the basic sentiment analysis APIs of three of the main cloud systems. The results were not exactly consistent.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RmGTGArVOwtznuxpSpY5EA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@artbyhybrid?utm_source=unsplash&amp;utm_medium=referral">Madison Oren</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>My initial goal was to create a large dataset of social media comments properly categorized for sentiment, as positive, negative or neutral. This labeled dataset was to be used for fine-tuning our own sentiment analysis algorithms; it consisted of data collected by our social media monitoring tool, and it had to be in the order of 100K entries. Our first target language was Portuguese.</p><p>Instead of engaging in the expensive and time-consuming steps of manually classifying phrases for sentiment, I first decided to evaluate if I could use existing systems to classify phrases and form a dataset based on majority voting. If the accuracy of existing systems proved to be good enough (over 70%) then I could use existing systems to label our 100K dataset for fine-tuning our own algorithms.</p><p>To verify the accuracy of existing commercial systems we needed a manually classified dataset. For that, we manually labeled the sentiment of 6000 comments, collected using our <a href="https://www.odysci.com/">social media monitoring tool</a>. We found it important to use our own data instead of existing open-source datasets, because we had more control over the types of comments collected. For example, comments made about a product, or a store can be quite different than comments made about politicians, and we wanted to know that when analyzing the results.</p><p>We submitted this dataset of 6000 comments to three different commercial systems:</p><p>· <a href="https://aws.amazon.com/comprehend/">AWS Comprehend</a></p><p>· <a href="https://cloud.google.com/natural-language">Google Natural Language AI</a></p><p>· <a href="https://www.ibm.com/cloud/watson-natural-language-understanding">IBM Watson Natural Language Understanding</a></p><p>All three systems can perform several other analyses related to natural language understanding, but for our purposes we only used the APIs for sentiment analysis. They all use advanced machine learning techniques — although it is not clear exactly what algorithms are used, it is likely that several different techniques are used depending on the type of analysis to be performed.</p><h4><strong>Background</strong></h4><p>Sentiment analysis is a major component of any natural language processing (NLP) toolbox. Its evolution in the last 15 years went from dictionary/lexicon-based methods [1, 2], to traditional supervised machine learning methods (e.g., SVM, Naive Bayes) [3], to deep learning methods [4].</p><p><strong><em>Dictionary/Lexicon-based Sentiment Analysis</em></strong><em><br></em>This method uses dictionaries of words and expressions with an associated sentiment. For example, the word “healthy” would be positive, while “unhealthy” would be negative. Expressions are a little trickier, e.g., the word “high” could by itself be positive, as in “high quality”, but “high price” could be negative. The text to be categorized is broken down into words or expressions and their sentiments (from the dictionary) counted. The simplest way is to compute a sentiment score by subtracting the number of negative words from the positive ones — if the result is positive, the overall sentiment is considered positive, otherwise negative. There are many ways to improve on this basic method [1].</p><p><strong><em>Traditional ML-based Sentiment Analysis</em></strong><br>This refers to supervised learning families of algorithms using traditional algorithms (e.g., SVM, Random Forests, Naïve Bayes), not neural networks. These methods require labeled datasets and model fitting. The input text is first tokenized (in words or n-grams) and encoded in some numeric form, such as one-hot or tf-idf [5]. The encoded phrases are used for fitting the model according to the chosen algorithm. The fitted model can then be used for inferring the sentiment of new sentences. There are many variations of these algorithms [6].</p><p><strong><em>Deep-learning-based Sentiment Analysis</em></strong><br>These are also supervised learning algorithms that use deep neural networks for inferring the sentiment of text. These DL networks need to be trained or fine-tuned if using a pre-trained model (usually at a higher cost and time than ML algorithms). DL algorithms have reportedly achieved better accuracy than previous methods, especially with the advent of transformers and pre-trained language models [7].</p><h4><strong>Our Labeled Dataset</strong></h4><p>Out of the 6000 original samples, 5798 were successfully ran through the sentiment analysis systems, hence our analysis from now on will consider a dataset of 5798 phrases.</p><p>These graphs show the frequencies of comments based on their manually labeled sentiments and on their length in characters. Although the sentiments are not perfectly balanced, they are representative of the overall comments on social media, which tend to weigh more on the neutral and negative sides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8qeEo8wqZVsz4T9a36MLDg.png" /><figcaption>Manually-labeled dataset characteristics</figcaption></figure><p>The vast majority of the comments are short, i.e., less than 300 characters, with almost half (2573) having 50 characters or fewer. Again, this is representative of social media comments in general, which tend to be short.</p><h4><strong>Commercial Systems</strong></h4><p>There are many commercial NLP systems capable of doing sentiment analysis. We opted to use the offerings from Amazon AWS, Google and IBM, for their ease of set up and ability to use directly calling an API without having to fine -tune it first. Noted, these systems can be fine-tuned, and probably would have produced better results. But for our purposes of verifying their base-level accuracy and appropriateness of using them as majority voting, we decided to use them “out-of-the-box”, without any additional configuration.</p><p>In all three systems, some initial setup is required for obtaining access, such as creating an account with their cloud environments and obtaining the necessary access keys. These steps are well documented, and we will not repeat them here. In the code fragments below, we give some simple examples of how we used the APIs to get the sentiment of a given text.</p><h4><strong>Using AWS Comprehend for sentiment analysis</strong></h4><p>We used the <a href="https://docs.aws.amazon.com/comprehend/latest/dg/using-api-sync.html#get-started-api-sentiment">AWS Comprehend SDK for Python API</a>, which requires the Boto3 library, easily installed using pip:</p><pre>pip install boto3</pre><p>The code below contains the function we used for analyzing the sentiment of a single phrase using AWS Comprehend:</p><pre>import boto3<br><br>def aws_analyze_sentiment(text, lang):<br>    &quot;&quot;&quot;<br>    Returns the sentiment of text using AWS Comprehend API<br>    Args:<br>      text: The text content to analyze<br>      lang: the text language<br>    &quot;&quot;&quot;<br><br>    # First get the AWS Comprehend client<br>    client = boto3.client(&#39;comprehend&#39;)<br>    # call the detect_sentiment method<br>    response = client.detect_sentiment(Text=text, LanguageCode=lang)<br>    <br>    return response</pre><p>The response returned by <em>aws_analyze_sentiment</em> is a json object. For example, calling:</p><pre>response = aws_analyze_sentiment(text=&#39;This product is top notch.&#39;, <br>                                 lang=&#39;en&#39;)</pre><p>returns:</p><pre>{&#39;Sentiment&#39;: &#39;POSITIVE&#39;, <br> &#39;SentimentScore&#39;: {&#39;Positive&#39;: 0.9995675683021545,<br>                    &#39;Negative&#39;: 7.027297397144139e-05,<br>                    &#39;Neutral&#39;:  0.00033828799496404827,<br>                    &#39;Mixed&#39;:    2.3856197003624402e-05},<br> &#39;ResponseMetadata&#39;: {&#39;RequestId&#39;: &#39;...&#39;,<br>                      &#39;HTTPStatusCode&#39;: 200,<br>                      &#39;HTTPHeaders&#39;: {&#39;...&#39;},<br>                      &#39;RetryAttempts&#39;: 0}<br>}</pre><p>The <em>SentimentScore</em> part contains the magnitude of each sentiment, as determined by the API. The higher value determines the final sentiment — in this case <em>Positive . I</em>f the <em>Mixed</em> score is high, the text expresses both positive and negative sentiments without a dominant one, and for our purposes, it will be considered the same as <em>Neutral</em>.</p><p>For example, the call:</p><pre>response = aws_analyze_sentiment(text=&#39;This product is top notch. But delivery sucks.&#39;,<br>                                 lang=&#39;en&#39;)</pre><p>returns:</p><pre>{&#39;Sentiment&#39;: &#39;MIXED&#39;, <br> &#39;SentimentScore&#39;: {&#39;Positive&#39;: 0.0008746279636397958,<br>                    &#39;Negative&#39;: 0.004289490170776844,<br>                    &#39;Neutral&#39;:  6.699292862322181e-05,<br>                    &#39;Mixed&#39;:    0.9947688579559326},<br> &#39;ResponseMetadata&#39;: {&#39;RequestId&#39;: &#39;...&#39;,<br>                      &#39;HTTPStatusCode&#39;: 200,<br>                      &#39;HTTPHeaders&#39;: {&#39;...&#39;},<br>                      &#39;RetryAttempts&#39;: 0}<br>}</pre><p>We called the function <em>aws_analyze_sentiment</em> on the full manually-labeled dataset and computed the accuracy.</p><blockquote>The overall accuracy using the AWS Comprehend sentiment analysis was <strong>71.8%</strong>.</blockquote><h4><strong>Using Google Cloud NLP for sentiment analysis</strong></h4><p>We used the <a href="https://cloud.google.com/natural-language/docs/analyzing-sentiment#language-sentiment-string-python">Google Cloud Natural Language API for Sentiment Analysis</a>, which requires the google-cloud-language library, installed using:</p><pre>pip install --upgrade google-cloud-language</pre><p>The <a href="https://cloud.google.com/natural-language/docs/analyzing-sentiment#language-sentiment-string-python">documentation</a> provides the following code for analyzing the sentiment of a single phrase using Google Cloud NLP:</p><pre>def google_analyze_sentiment(text, lang):<br>    &quot;&quot;&quot;<br>    Returns the sentiment of text using Google Cloud API<br>    Args:<br>      text: The text content to analyze<br>      lang: the text language<br>    &quot;&quot;&quot;<br><br>    client = language_v1.LanguageServiceClient()<br><br>    # Available types: PLAIN_TEXT, HTML<br>    type = language_v1.Document.Type.PLAIN_TEXT<br><br>    # create document dictionary with the data to be analyzed<br>    document = {&quot;content&quot;: text, &quot;type_&quot;: type, &quot;language&quot;: lang}<br><br>    # Available values: NONE, UTF8, UTF16, UTF32<br>    encoding_type = language_v1.EncodingType.UTF8<br><br>    response = client.analyze_sentiment(request = {&#39;document&#39;: document, &#39;encoding_type&#39;: encoding_type})<br>    <br>    # return the response structure returned by google<br>    return response</pre><p>As an example, calling:</p><pre>response = google_analyze_sentiment(&quot;This product is top notch. But delivery sucks.&quot;, &quot;en&quot;)</pre><p>Returns the following structure:</p><pre>document_sentiment {<br>  magnitude: 1.8<br>  score: 0.0<br>}<br>language: &quot;en&quot;<br>sentences {<br>  text {<br>    content: &quot;This product is top notch.&quot;<br>  }<br>  sentiment {<br>    magnitude: 0.9<br>    score: 0.9<br>  }<br>}<br>sentences {<br>  text {<br>    content: &quot;But delivery sucks.&quot;<br>    begin_offset: 27<br>  }<br>  sentiment {<br>    magnitude: 0.9<br>    score: -0.9<br>  }<br>}</pre><p>Note that Google breaks down the sentence and analyzes both of them separately, as well as providing a joint score and magnitude. According to <a href="https://cloud.google.com/natural-language/docs/basics#interpreting_sentiment_analysis_values">Google’s documentation</a>:</p><blockquote>The score of a document’s sentiment indicates the overall emotion of a document. The magnitude of a document’s sentiment indicates how much emotional content is present within the document, and this value is often proportional to the length of the document.</blockquote><p>To be able to compute the accuracy of Google’s results against our manually labeled dataset, we need to attribute a sentiment class (Neutral, Positive, or Negative) to each text based on the response Score and Magnitude. Google’s documentation recommends that we define an evaluation function based on Score and Magnitude, that maximizes the accuracy over the full test dataset and then use this function on new datasets.</p><p>We actually ran a simple search optimization to find the best combination of score and magnitude. The code below contains the evaluation function and the non-parallel version of the optimization loop. The algorithm iterates over all combinations of discrete thresholds, ranging from -1 to +0.11 for negative thresholds, and from +1 to -0.11 for positive thresholds. For each combination, it computes the overall accuracy, and keeps the best.</p><pre>def eval_google_sentiment(score, magnitude, neg, pos):<br>    &#39;&#39;&#39;<br>    If magnitude is very small (&lt;= 0.101), return Neutral. Otherwise,<br>    check the score. If score &lt; neg, return Negative, If score &gt; pos,<br>    return Positive, otherwise, return Neutral. The small magnitude <br>    threshold (0.101) was determined empirically.<br>    &#39;&#39;&#39;<br>    if magnitude &gt; 0.101:<br>        if (score &lt; neg):<br>            return &#39;Negative&#39;<br>        elif (score &gt; pos):<br>            return &#39;Positive&#39;<br>        else:<br>            return &#39;Neutral&#39;<br>    else:<br>        return &#39;Neutral&#39;<br>    pass</pre><pre>&#39;&#39;&#39;<br>In the code below, resultsdata is a dataframe with all the responses from <br>calling google_analyze_sentiment on the manually-labeled dataset.<br>resultsdata[&#39;ManualSentiment&#39;] contains the manually-labeled value<br>resultsdata[&#39;Gscore&#39;] contains the score given by Google, which can vary<br>from -1.0 to +1.0.<br>resultsdata[&#39;Gmagnitude&#39;] contains the magnitude given by Google, which is<br>always &gt;= 0<br>&#39;&#39;&#39;<br>def compute_best_thresholds():<br>    best = 0<br>    bestneg = 0<br>    bestpos = 0<br>    for vneg in range(-100, 12, 1):<br>        neg = vneg/100.<br>        for vpos in range(-11, 101, 1):<br>            pos = vpos/100.<br>            <br>            resultsdata[&#39;Gsentiment&#39;] = resultsdata.apply (lambda row: gen_google_sentiment(row.Gscore, row.Gmagnitude, neg, pos), axis=1)<br>            nmatches = ( resultsdata[&#39;Gsentiment&#39;] == resultsdata[&#39;ManualSentiment&#39;] ).sum()<br>            if nmatches &gt; best:<br>                best = nmatches<br>                bestneg = neg<br>                bestpos = pos<br>    <br>    print(f&#39;Best values: matches = {best}, neg = {bestneg}, pos = {bestpos}&#39;)<br>    resultsdata[&#39;Gsentiment&#39;] = resultsdata.apply (lambda row: gen_google_sentiment(row.Gscore, row.Gmagnitude, bestneg, bestpos), axis=1)<br>    nmatches = ( resultsdata[&#39;Gsentiment&#39;] == resultsdata[&#39;ManualSentiment&#39;] ).sum()<br>    print(f&#39;Google matches = {nmatches} = {100*nmatches/len(resultsdata):3.1f}% of {len(resultsdata)} samples&#39;)<br><br>    # return a tuple with the best negative and positive thresholds<br>    return (bestneg, bestpos)</pre><p>The above loop tries 12544 combinations, and after all 12544 evaluations over the full dataset, it resulted in the best values being: <br><strong>negative threshold = 0.01 <br>positive threshold = 0.41</strong></p><blockquote>The overall best accuracy using the Google sentiment analysis and these thresholds was <strong>70.3%</strong>.</blockquote><h4>Using IBM Watson NLU for sentiment analysis</h4><p>We used <a href="https://cloud.ibm.com/apidocs/natural-language-understanding?code=python#sentiment">IBM Watson Natural Language Understanding Sentiment Analysis</a> API in Python. The required libraries may be installed with the command:</p><pre>pip install - upgrade &quot;ibm-watson&gt;=6.1.0&quot;</pre><p>The code below was used to call the API on each sample of our dataset:</p><pre>def ibm_analyze_sentiment(text, lang):<br>    &quot;&quot;&quot;<br>    Returns the sentiment of text using IBM Watson NLU API<br>    Args:<br>      text: The text content to analyze<br>      lang: the text language<br>    &quot;&quot;&quot;<br>    nlu = NaturalLanguageUnderstandingV1(version=&#39;2022-04-07&#39;,<br>                                         IAMAuthenticator(your-api-key))<br>    nlu.set_service_url(https://medium.com/your-service-url)<br><br>    response = nlu.analyze(language=lang,<br>                           text=text,<br>                           features=Features(sentiment=SentimentOptions())<br>                          ).get_result()<br>    <br>    return response</pre><p>As an example, calling:</p><pre>response = ibm_analyze_sentiment(text=&#39;This product is top notch.&#39;,<br>                                 lang=&#39;en&#39;)</pre><p>returns the following structure:</p><pre>{<br>  &quot;usage&quot;: {<br>    &quot;text_units&quot;: 1,<br>    &quot;text_characters&quot;: 26,<br>    &quot;features&quot;: 1<br>  },<br>  &quot;sentiment&quot;: {<br>    &quot;document&quot;: {<br>      &quot;score&quot;: 0.819643,<br>      &quot;label&quot;: &quot;positive&quot;<br>    }<br>  },<br>  &quot;language&quot;: &quot;en&quot;<br>}</pre><p>Now, notice how the result changes if we add a second negative phrase:</p><pre>response = ibm_analyze_sentiment(text=&#39;This product is top notch. But delivery sucks&#39;,<br>                                 lang=&#39;en&#39;)</pre><p>returns:</p><pre>{<br>  &quot;usage&quot;: {<br>    &quot;text_units&quot;: 1,<br>    &quot;text_characters&quot;: 46,<br>    &quot;features&quot;: 1<br>  },<br>  &quot;sentiment&quot;: {<br>    &quot;document&quot;: {<br>      &quot;score&quot;: -0.443754,<br>      &quot;mixed&quot;: &quot;1&quot;,<br>      &quot;label&quot;: &quot;negative&quot;<br>    }<br>  },<br>  &quot;language&quot;: &quot;en&quot;<br>}</pre><p>Now the whole sentence is classified as negative (slightly only), and it add a mixed flag with value ‘1’, indicating mixed emotions on the full sentence. In all cases, we use the “label” as the final sentiment.</p><p>We called the function <em>ibm_analyze_sentiment</em> on the full manually-labeled dataset and computed the accuracy.</p><blockquote>The overall accuracy using the IBM Watson NLU sentiment analysis was <strong>67.1%</strong>.</blockquote><p>The accuracies of all three systems are not very different (AWS=71.8%, Google=70.3%, IBM=67.1%), although AWS Comprehend has a slight edge. Now, let’s have a detailed look at their results compare and how they handled each type of sentiment class.</p><h3>Detailed Analysis</h3><p>As I mentioned in the beginning, the initial purpose of this experiment was to determine if the average result, based on majority voting, would be acceptable as a way to generate a much larger dataset automatically (i.e., without manual labeling). The base criteria was to have accuracy higher than 70% (which is <em>not much</em>, but a base value).</p><p>The majority voting result was given by the sentiment class matched by 2 or more systems, and if they did not agree at all, it was considered <strong><em>inconclusive</em></strong>. There were only 211 samples (3.6%) rendered <em>inconclusive</em>.</p><p>The following graphs summarize the accuracy results and the sentiment distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y4w8u-uExoxarf417zHH0g.png" /><figcaption>Accuracy and Sentiment Distribution by Classifier</figcaption></figure><p>The sentiment distributions are somewhat similar for AWS and Google, and very different from IBM. The Average Sentiment was considerably better and its sentiment distributions much closer to the labeled dataset.</p><p>Comparing the sentiments of all three systems together and pairwise, we get the following matches, considering the total of 5798 samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8eZ9-Tc5BYCyIHpeq4mY9A.png" /><figcaption>Matching predictions, pairwise and overall</figcaption></figure><p>From this we can infer:</p><ul><li>AWS and Google matched the most pairwise, followed by AWS and IBM, and last IBM and Google.</li><li>AWS, Google and IBM matched their results on 2804 samples (48.4%), and all three matched the manual dataset on 2513 samples (43.3%).</li></ul><p>The confusion matrices for each system, shown below, allow a detailed look at how each system handled each type of sentiment class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4GVlgPLcFmWLxYeFyDxw_w.png" /><figcaption>Confusion Matrices</figcaption></figure><p>Using these numbers we can compute the multi-class precision, recall and F1-score for each system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AI7oFSlF_gkHl9Yw1OeMWw.png" /><figcaption>Precision, Recall, F1-score</figcaption></figure><p>AWS and Google had the best Positive and Negative precision and good recall, while IBM had lower precision, but much higher recall on both sentiment classes. Conversely, on Neutral, IBM had high precision and low recall, while AWS and Google had lower precision and higher recall. Overall the F1-scores were not that different, except for IBM Neutral F1-Score which was much lower, due to its very low recall.</p><p>Comparing the majority-voting/average results with the manually-labeled results, we get the following. The average results got the best F1-score when compared to the three systems individually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WM0x-Q6WT0GpDDJVxShMxQ.png" /><figcaption>Confusion matrix, precision, recall and F1-score for Average results</figcaption></figure><h3>Summarizing</h3><p>We compared the performance of the sentiment analysis API of three of the main commercial systems available (AWS Comprehend, Google Cloud NLP and IBM Watson NLU), using a dataset consisting of 5798 sentences in <strong>Portuguese</strong>, collected from various social media accounts (Facebook, Twitter and YouTube, all public data). We used the standard configuration of these tools, without any training, except for selecting the language.</p><p>The accuracies of the three systems were not very different (from 67.1% for IBM to 71.8% for AWS). The average of the results (by majority voting) performed much better, scoring 73.8%.</p><p>The three systems behave somewhat differently depending on the prevailing sentiment. AWS and Google do well on positive and negative samples (good precision and recall), while IBM is more precise on neutral samples (although with smaller recall).</p><p>We plan to use this approach for automatically classifying (by majority voting) a much larger dataset, which in turn, we will use for fine-tuning our own sentiment analysis algorithms.</p><h4>References</h4><ol><li>T. Hardeniya and D. A. Borikar, <a href="https://ijaems.com/upload_images/issue_files/1462896418-13%20IJAEMS-APR-2016-89-Dictionary%20Based%20Approach%20to%20Sentiment%20Analysis%20-%20A%20Review.pdf">Dictionary Based Approach to SentimentAnalysis — A Review</a> (2016), International Journal of Advanced Engineering, Management and Science (IJAEMS), Vol-2, Issue-5, May, 2016.</li><li>M. Taboada, J. Brooke, M. Tofiloski, K. Voll and M. Stede, <a href="https://direct.mit.edu/coli/article/37/2/267/2105/Lexicon-Based-Methods-for-Sentiment-Analysis">Lexicon-Based Methods for Sentiment Analysis</a> (2011), Computational Linguistics (2011) 37 (2): 267–307.</li><li>S. Rana and A. Singh, <a href="https://ieeexplore.ieee.org/abstract/document/7877399">Comparative analysis of sentiment orientation using SVM and Naive Bayes techniques</a> (2016), 2nd International Conference on Next Generation Computing Technologies (NGCT 2016).</li><li>P. C. Shilpa, R. Shereen, S. Jacob and P. Vinod, <a href="https://ieeexplore.ieee.org/document/9388382">Sentiment Analysis Using Deep Learning</a> (2021), Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV).</li><li>Z. Keita, <a href="https://towardsdatascience.com/text-data-representation-with-one-hot-encoding-tf-idf-count-vectors-co-occurrence-vectors-and-f1bccbd98bef">Text data representation with one-hot encoding, Tf-Idf, Count Vectors, Co-occurrence Vectors and Word2Vec</a> (2021), Towards Data Science/Medium.</li><li>Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu and Lifang He, <a href="https://dl.acm.org/doi/abs/10.1145/3495162">A Survey on Text Classification: From Traditional to Deep Learning</a> (2022), ACM Transactions on Intelligent Systems and Technology, 13 (2).</li><li>L. Mathew and V. Bindu, <a href="https://link.springer.com/chapter/10.1007/978-981-16-2594-7_69">Efficient Classification Techniques in Sentiment Analysis Using Transformers</a> (2022), International Conference on Innovative Computing and Communications. Advances in Intelligent Systems and Computing, vol 1387. Springer, Singapore.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70175a3bfa25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting election results in Brazil based on social media data, some basic assumptions and a…]]></title>
            <link>https://rberga.medium.com/predicting-election-results-in-brazil-based-on-social-media-data-some-basic-assumptions-and-a-516235f6f477?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/516235f6f477</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[election-predictions]]></category>
            <category><![CDATA[opinion-polls]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Thu, 16 Feb 2023 18:42:26 GMT</pubDate>
            <atom:updated>2023-03-01T16:28:00.766Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Predicting election results in Brazil based on social media data, some basic assumptions and a little bit of AI</strong></h3><h4><em>Our analytical election prediction model, applied to the 2022 Brazilian Presidential runoff election, came within 0.5% of the actual results.</em></h4><p>We have been monitoring daily social media data (only public data) for about 5 years now, from all types of accounts, e.g., politicians, celebrities, influencers, products, companies, sports teams, colleges, you name it. Everything that happens in the <em>real world</em> gets commented, explored, faked, blown up, discredited, on social media. Still, one thing that has become very clear is that people (real, not bots) are honestly blunt when engaging or posting something on social media, especially if they feel strongly about some product, company, or politicians.</p><p>For a while now, we have been fascinated by the relationship between social media and elections. We have analyzed the social media contents of candidates in several elections in Brazil and the US and looked for correlations between social media indicators and the final election results. At the end of this article, you can find links to our analyses of past elections.</p><p>In this article we will detail our approach for predicting elections results, based solely on social media data, some basic assumptions about the size of the electorate and the candidates, and a little bit of AI (for data mining and sentiment analysis). In the last Brazilian presidential election in 2022, our prediction on the eve of the runoff election came within 0.5% difference from the final result, which was more accurate than all, but one opinion poll published on the day before. This article explains our methodology for coming up with the prediction.</p><p>In the last several years, with the increasing popularity and accessibility of machine learning technologies, many scholars have published methods to predict the outcome of elections based on data. Not data collected in an interview or phone call, but data <em>volunteered</em> by people, by means of <em>social media</em>.</p><p>Research [1] has shown that many people feel more at ease to express their personal views online than when talking to a pollster. For better or worse, interactions on social media tend to be more polarized and less susceptible to social norms that are usually present in the work or family environment. Thus, perhaps more representative of the person’s real intention at the ballot box.</p><p>The machine learning methods for predicting election results use a variety of social media data, most often, number of fans and followers, number of likes and shares, and number of comments mentioning a candidate positively or negatively. Sentiment analysis is widely used for assessing whether a comment is favorable to a given candidate or not.</p><p>Early studies were able to correlate the volume of social media posts associated with a given candidate or party, and the sentiment of the comments posted about the candidates with the election results [2, 3].</p><p>More recently, researchers have used more sophisticated models, considering various metrics such as number of posts, likes, shares, and comments (number and sentiment) and using machine learning algorithms to predict election results [4, 5].</p><p>Fake news and bots make the monitoring and understanding of real engagement and opinions more difficult. Moreover, the language used in social media tends to be very informal, sarcastic, ironic, full of slangs, lots of grammar and spelling mistakes, which make any type of analysis (e.g., detecting sentiment or opinion) much less precise.</p><p>Politicians, especially, the very polarizing ones, usually have huge numbers of followers and very high engagement on their social media posts. The majority of fans and followers are normally the <em>converted</em> ones, that is, the people who will vote for the candidate anyway, and do not represent the undecided electorate. In very competitive elections, the undecided voters (if they choose to vote) will <em>decide</em> for a candidate closer to the election and may swing the victory to a candidate, even if they have a lot fewer total fans and followers. Thus, considering total quantities in social media, as an indicator of electorate trends, can be very misleading.</p><blockquote>The <strong>incremental</strong> change in social media indicators, especially closer to the elections, are much better indicators of the trends among the undecided voters.</blockquote><p>This effect can be estimated quantitatively by looking at the variations of fans and followers along time, comparatively for all candidates in an election. When a person, who is not yet a fan or follower of any candidate, starts following a given candidate, especially close to the election, it likely means that something in that candidate’s recent message resonated with the person — enough for them to engage with the candidate on social media (and perhaps vote for).</p><p>This can be measured as follows: consider an election with 4 candidates, and on a given day, there are 100 new people that follows the 4 candidates (summing up the new followers). Let’s say that candidate A had 40 new followers, B had 25, C had 20 and D had15. From this we can estimate that, of all “undecided” people, on this day, 40% chose candidate A, 25% chose candidate B, 20% chose C and only 15% chose D. At this point, if we have an estimate on the number of undecided voters, we could estimate how their votes will be allocated to each candidate.</p><p>To give an idea of this daily effect, we monitored the total number of fans and followers, and their daily incremental change, for the top four presidential candidates in the 2022 Brazilian elections, for a period prior to the 1st round election, including a period when candidates were very active on TV and campaigning. The figures below show the details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5aZoBsHAlpWs3dbPVYcEQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6JDPxvgC1d0FEuq2F9rTYQ.png" /></figure><p>Both graphs above show the same information, the number of new daily fans/followers gained by each candidate, as a percentage of the total of new daily fans/followers.</p><p>The first graph is annotated with specific events that triggered distinct reactions. For example, on the day(s) after each candidate was interviewed by “Jornal Nacional” (JN), a popular TV news program, their number of new fans/followers went up. On Aug/22, the incumbent president and candidate Jair Bolsonaro was interviewed by JN, and immediately after (Aug/23) he gathered 75% of the new fans/followers that day. Similarly, Lula was interviewed by JN on Aug/25, and on Aug/26 he collected 71% of the new fans that day. On Aug/28 there was a Presidential Debate on TV, in which candidate Simone Tebet was considered to have won, and on the day after, Aug/29, Simone collected 30% of the new fans, more than any other candidate that day, and more than she collected on any other day. On Sep/7, president Bolsonaro participated in a large event celebrating Brazilian Independence Day, and on the following day he gathered 61% of the new fans.</p><p>These daily variations also show that the effect of an event is usually short lived, that is, the spotlight candidate gets proportionally more fans immediately after the event (and the other candidates proportionally fewer), but on later days their levels go back to normal. Nevertheless, we found that the average of these daily variations does correlate with the amounts of undecided voters that make a decision for a candidate. We will detail how to use it quantitatively later.</p><h4><strong>Analytical Model for Election Prediction</strong></h4><p>Our analytical model for election prediction consists of the following steps:</p><ol><li><strong>Initial Estimates</strong> of:<br>a. Total number of eligible voters (<em>eV</em>)<br>b. Number of actual voters (<em>aV</em>), i.e., people who actually cast a vote<br>c. Number of valid votes (<em>vV</em>)<br>d. Number of invalid votes (<em>iV</em>)</li></ol><p>For the purposes of estimating the election results, we will consider the number of valid votes only, in a fully proportional voting system. These numbers can be taken from the official electoral sites and estimated based on previous elections. They can clearly change for each election, but in Brazil, where voting is compulsory, the percentage of valid votes with respect to eligible voters has not changed drastically over the last 3 presidential elections.</p><p>2. <strong>Percentage of potential votes</strong> (<em>ppv</em>)<br>In the days leading to election day (this could be one day, one week, one month, depending on the granularity and reliability of the data available), each candidate is associated with a percentage of potential votes (ppv). For example, before a 1st round election (or single round), since we do not have any “real” numbers, we could use the average of the latest opinion polls for each candidate. If there is a 2nd round/runoff election, we should use the actual number of votes that each candidate got in the 1st round election.<br>Let <strong>ppvₖ</strong> denote this percentage of potential votes for candidate <strong>k</strong>, then</p><blockquote><strong>pVₖ = aV * ppvₖ</strong></blockquote><p>is the potential votes to be received by candidate k in the election (1st or single round) or actual votes received by candidate k in a 1st round election.</p><p>3. <strong>Definite and Undecided Votes</strong><br>Out of the potential votes for a given candidate <strong>k</strong> (item 2 above), we estimate the percentage that will definitely go to candidate <strong>k</strong> on election day (percentage of <strong>definite votes</strong> = <em>pdv</em>), and the percentage that are still undecided (percentage of <strong>undecided votes</strong> = <em>puv</em>). These percentages can be taken from opinion polls, which normally ask the voters how certain they are about their chosen candidate and whether they may change. Hence the definite votes (<strong>dVk</strong>) and undecided votes (<strong>uVk</strong>) associated with candidate <strong>k</strong> can be estimated as:</p><blockquote><strong>dVₖ = pVₖ * pdvₖ<br>uVₖ = pVₖ * puvₖ</strong></blockquote><p>The total number of undecided votes will be given by the sum for all N candidates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*pAqESkkJzD5Ly6gVXwN8hA.png" /></figure><p>4. <strong>Vote Transfer<br></strong>In a 2-stage election, only the top 2 candidates in the 1st round will compete in the runoff election (if none gets more than 50% of the valid votes). The voters who voted for the other candidates will them have to choose between the 2 remaining candidates (or abstain or invalidate the ballot). Some of these voters may stay undecided for longer (the <strong>uVₖ</strong> factor explained in item 3.), and the remaining ones will <strong>transfer</strong> their votes. In this step we estimate this transfer.</p><p>If candidates A and B are the top 2 in the 1st round, and candidate C is left behind, the percent transfer of C to A and C to B, denoted as pt_c<em>-&gt;A)</em>, or <br>pt_c-&gt;B, is estimated based on 2 factors:</p><ul><li>a) How aligned is the party of candidate C with the party of A and B? If C is totally aligned with A (e.g., two right-wing parties, or two left-wing parties), then pt_c-&gt;A = 100% or pt_c-&gt;B = 0%. So, some political context is needed here.</li><li>b) In the absence of a definite alignment (e.g., a center party and a left/right wing party), we will estimate the transfer percentage based on <em>data mining and sentiment analysis</em>. See details below.</li></ul><p>In the formulas below, we will call this percentage of votes transfer from candidate <strong>j</strong> to candidate <strong>k </strong>as as <strong>pt_j-&gt;k</strong>. Let’s assume that in a 1st round election with N candidates, each candidate <strong>i</strong> got <strong>Vᵢ</strong> votes and the number of undecided votes is <strong>uVᵢ</strong>. In the event of a runoff election with the first two candidates (1 and 2, to make the formulas cleaner), the total transferred votes from candidates 3 to N to candidates 1 and 2 will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*tXOsb37Qwy-qTzYotLpFKQ.png" /></figure><p>5. <strong>Assigning the Undecided Votes<br></strong>Assuming that these voters (still undecided in the days leading to the election) are going to the polls, they will have to choose between the remaining candidates. We will estimate the percentage of undecided votes that go to each candidate based on the <strong>average percentage of new daily fans/followers</strong> (<strong>pdF</strong>) gathered by each candidate in the time interval being considered (these daily values can be seen in the figures above).<br>The Undecided Votes that “decide” for Candidate <strong>i</strong> is denoted as <strong>udVᵢ:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*AXAs6PmjBSnwJcnCLI20GA.png" /></figure><p>6. <strong>Final Tally<br></strong>The final total estimated number of votes that a candidate <strong>i</strong> will receive in the election will be given by the sum of direct votes, transferred votes and part of the undecided votes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*iSS-2VLXS4SIGQa1ou6Tag.png" /></figure><p>The final percentage of valid votes estimated for candidate <strong>i</strong> will be given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*Pd6Xe9s7XsmamrhKUtZoMg.png" /></figure><p>where <strong>m</strong> is the final number of candidates (e.g., 2 in a runoff election)</p><p><strong>Putting it together</strong></p><p>The figure below illustrates how the final number of votes is composed, according to the 6 steps presented above, in a particular case of a runoff election, after a 1st round election with 4 candidates, assuming candidates A and B are the top 2. For example, the final votes predicted for candidate A (fVA) will be the sum of: (1) votes directly transferred from the 1st round election (dVA), (2) the portion of undecided votes that go to A based on the average percentage of new daily fans/followers gathered by A (udVA) and (3) the votes transferred from candidates C (tV_C-&gt;A) and D (tV_D-&gt;A).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q9BXFudh8fV7GE-gp1df0g.png" /></figure><h4><strong>A little bit of AI</strong></h4><p>As indicated in item 4b. above, we use data mining and sentiment analysis to estimate the percentage of votes transferred from any candidate who does not make it to a runoff election to another who does.</p><p>We have collected the posts and the comments made directly onto the candidates’ social media accounts (Facebook and Twitter) for several months before the elections. In particular, if we want to evaluate the transfer of votes, we need to look at the comments between the day after the 1st round election and the day before the runoff election posted on the social media accounts of the losing candidates. It is very common that a supporter of a losing candidate will post comments telling their candidate to support A or B in the runoff election; or the comment will directly criticize or support candidate A or B or their parties.</p><p>First, we created a long list of word expressions associated with the winning candidates and their parties. Then we apply data mining to further select only those comments that contain any expression similar to those in the list. Lastly, we run sentiment analysis on those selected comments. The proportion of positive comments associated with candidate A, plus negative comments associated with B, will represent the percentage of votes transferred to candidate A, and reversely for candidate B. The figure below illustrates these steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5OXaVTSg17-E5ewB5jLjw.png" /></figure><h4><strong>Our Predictions for the 2nd round Brazilian Presidential Election 2022</strong></h4><p>The 1st round of the elections was held on Sunday, October 2, 2022, with 11 candidates. The table below shows the actual results. We will use these numbers as a starting point for our 2nd round predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*9kzEGWbk4iuIe2Y2W8fMCQ.png" /></figure><p>The top 2 candidates, Lula and Bolsonaro would compete in the runoff election on October 30, 2022.</p><p>On October 29 we evaluated our prediction model to come up with the likely winner on the following day. Our assumptions were the following.</p><blockquote><strong>Definite and Undecided Votes</strong></blockquote><p>Since this was an extremely polarized election, we chose to use 100% as the percentage of definite votes for Lula and Bolsonaro. That is, we assumed that everyone that voted for Lula or Bolsonaro in the 1st round would vote in the same way in the 2nd round.</p><p>According to our modeling, the votes received by the other candidates in the 1st round would be either transferred to one of the top candidates, or go into a pool of undecided votes. The percentages used and our reasoning are given in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*s2taVhC2IwxdH3vYcTcrXQ.png" /></figure><p>The percentages of undecided votes (puv) in the table above are input parameters to our model and were defined based on our assessment of the political parties and alliances being formed in the days before the 2nd round election. They are certainly subjective.</p><blockquote><strong>Vote Transfer</strong></blockquote><p>As presented in step 4. above, the votes given to a non-top-2 candidate become either undecided or transferred to one or both of the top-2 candidates according to a percentage of vote transfer (pt_i-&gt;1, pt_i-&gt;2). This percent transfer is either an input parameter (green cells in the tables below) to the model (when the political alignment is clear), or estimated based on data mining and sentiment analysis of the comments posted on the social media accounts of the non-top-2 candidates (gray cells).</p><p>We performed data mining and sentiment analysis on all comments posted in the social media accounts (Facebook, Twitter and YouTube) of candidates <em>Simone Tebet</em> and <em>Ciro Gomes</em> by their fans/followers. The other candidates had considerably fewer comments and their party alignment was more well defined, thus we used estimated values. The tables below present the vote transfer percentages. The total votes to be transferred from each non-top-2 candidate is the total votes received in the 1st round minus the number of undecided votes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*L6FbCIjjWi7y12XpAcrULQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*uRdKKYLdEvWS_VkuX9LjrA.png" /></figure><blockquote><strong>Assigning the Undecided Votes</strong></blockquote><p>As explained in Step 5. above, the undecided votes are “assigned” to one of the top-2 candidates based on the average percentage of new daily fans/followers gathered by each candidate.</p><p>The figure below shows the daily incremental change in the total number of fans and followers in the social media accounts of Lula and Bolsonaro (considering the sum of Facebook, Instagram and Twitter) in the period between the 1st round and the 2nd round elections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hbw0Vk8w-a3gK4YaU0ruTw.png" /></figure><p>Using the average values from the figure above and the total number of undecided votes, we assign the undecided votes to each candidate as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*OmLJrYPC5S4GyiDmFIzMVQ.png" /></figure><p>Now we are able to estimate the final number of votes to be received by Lula and Bolsonaro in the runoff election, using the estimated values of Definite Votes, Transfer Votes and Undecided Votes and the formula on Step 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nCftNW-HAjdLGmRmVJtqbg.png" /></figure><p>The whole process can be visualized in the following diagram. According to our projections, when going from the 1st round to the runoff election, Lula would gain additional 3.5 million votes, and Bolsonaro would collect another 6.3 million. However, due to Lula’s advantage in the 1st round, Lula would still win the election at the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fLumQkzziF2UXptpRBBNYA.png" /></figure><blockquote><strong>Final Results and Opinion Polls</strong></blockquote><p>We produced this analysis on the eve of the runoff election (Oct/29): Lula would defeat Bolsonaro by a small margin, 51.41% against 48.59%. On the same day, there were several opinion polls released. The table below compares our estimates with the other opinion polls and the official election results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*EAlvN67qlFupMtFgTaNieA.png" /></figure><p>As the table shows, our predictions were better than all but one of the polls (see references [6,7,8] for the sources of these numbers).</p><h4><strong>Final Considerations</strong></h4><p>This article presented a methodology for estimating the results of elections based on 3 main components:</p><ol><li>Daily fans/followers collected from the social media accounts of the candidates.</li><li>Data mining and sentiment analysis of comments posted on those accounts.</li><li>Basic assumptions about the candidates’ initial positions (through polls or 1st round elections), and their political alignments.</li></ol><p>Using these 3 components, we estimated the number of decided votes, transfer of votes and votes from previously undecided people.</p><p>Several simplifying assumptions were taken, such as: (1) nationwide proportional voting (as it is the case in Brazil), (2) reasonable expectancy on the number of valid votes on election day (based on compulsory voting and historical trends), and (3) very polarized election where voters felt very strongly for one candidate or another. These assumptions were true in Brazil in 2022.</p><p>Our estimates were as good as the best 2 opinion polls published in Brazil on the eve of the runoff election, at a considerably lower cost. The price for conducting opinion polls in Brazil in 2022 varied from over five hundred thousand Brazilian Reais (approximately one hundred thousand USD) for the more prestigious ones (Datafolha, IPEC) with bigger sample population, to fifty thousand Reais (about 10K USD) for lesser known companies and smaller samples [9].</p><p>Our methodology is fully based on data collected from social media [13] and can be evaluated daily at no extra cost, after the initial setup. The main cost is associated with the social media data collection, which is significantly lower than the cheapest opinion poll, while giving consistently more accurate results.</p><p>Our simplifying assumptions may not be directly applicable to different election systems (such as the Electoral College process in the US), or to elections that are much less polarized. For US Presidential elections, we would need to collect social media data on a state by state basis, which is not always available (note that all data we collect from social media is public — there is no personal information used). As for elections that are significantly less polarized, we could adapt our input parameters to handle that, perhaps at some cost in accuracy (this is also a problem for opinion polls).</p><p>We have tested our methodology in the Brazilian elections for President 2018 [10], and 2022 (this article), and for Mayor of São Paulo in 2020 [11], as well as the US Georgia Senate elections in 2021 [12]. In all cases we obtained results comparable to the best opinion poll results.</p><p>Social media continues to draw a lot of criticism for its negative effects of overuse (especially in kids), potential for cyberbullying, dissemination of fake news, among many others. However, in the particular case of expressing opinions about candidates and elections, we have found social media to be fairly representative of the general population (subtracting the fake news and bots, of course) and a useful source of data in predicting results of elections. Our analytical methodology described here shows that is feasible.</p><h4>References</h4><ol><li>Andrea Ceron, Luigi Curini and Stefano Iacus, <a href="https://www.washingtonpost.com/news/monkey-cage/wp/2016/12/21/using-these-4-steps-pollsters-could-use-social-media-data-to-improve-election-forecasts/?noredirect=on">How pollsters could use social media data to improve election forecasts</a> (2016), <em>Washington Post</em>, Dec. 21, 2016.</li><li>A. Tumasjan, T. O. Sprenger, P. G. Sandner, and I. M. Welpe, <a href="https://ojs.aaai.org/index.php/ICWSM/article/view/14009">Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment</a> (2010), <em>Fourth International AAAI Conference on Weblogs and Social Media</em>.</li><li>B. O’Connor, R. Balasubramanyan, B. R. Routledge, and N. A. Smith, <a href="https://ojs.aaai.org/index.php/ICWSM/article/view/14031">From tweets to polls: Linking text sentiment to public opinion time series</a> (2010), Fourth<em> International AAAI Conference on Weblogs and Social Media</em>.</li><li>Kellyton dos Santos Brito and Paulo J. L. Adeodato, <a href="https://kellyton.com.br/assets/papers/PredictingBrazilianAndUSElections.pdf">Predicting Brazilian and U.S. Elections with Machine Learning and Social Media Data</a> (2020), <a href="https://ieeexplore.ieee.org/document/9207147"><em>2020 International Joint Conference on Neural Networks (IJCNN)</em></a>.</li><li>Z. Zhou, M. Serafino, L. Cohan, G. Caldarelli and H. Makse, <a href="https://journalofbigdata.springeropen.com/articles/10.1186/s40537-021-00525-8">Why polls fail to predict elections</a> (2021), <em>Journal of Big Data</em> <strong>8</strong>, 137 (2021).</li><li><a href="https://resultados.tse.jus.br/oficial/app/index.html#/eleicao/resultados">Official Election Results — Brasil</a> (in Portuguese)</li><li>Gabriel Sestrem, <a href="https://www.gazetadopovo.com.br/eleicoes/2022/pesquisa-eleitoral/pesquisas-eleitorais-acertam-mais-ganhadores-mas-erram-acima-das-margens-estimativas-numericas/">As pesquisas eleitorais acertaram ou erraram no 2.º turno?</a> (2022), <em>Jornal Gazeta do Povo</em>, Oct. 30, 2022 (in Portuguese).</li><li><a href="https://especiaisg1.globo/politica/eleicoes/2022/pesquisas-eleitorais/presidente/2-turno/">Pesquisas eleitorais</a> (2022), <a href="https://g1.globo.com/"><em>G1 Globo web site</em></a> (in Portuguese).</li><li><a href="https://www.em.com.br/app/noticia/politica/2022/10/29/interna_politica,1413748/lula-x-bolsonaro-veja-as-ultimas-pesquisas-que-serao-divulgadas-hoje-29.shtml">Lula x Bolsonaro: veja as últimas pesquisas que serão divulgadas hoje (29)</a> (2022), Jornal Estado de Minas, Oct. 29, 2022 (in Portuguese).</li><li><a href="https://www.linkedin.com/pulse/parte-9-podem-m%C3%ADdias-sociais-prever-o-resultado-das-para-bergamaschi/">Podem Mídias Sociais Prever o Resultado das Eleições para Presidente? (31/out/2018)</a>, <em>LinkedIn article (</em>in Portuguese).</li><li><a href="https://www.odysci.com/pt/blog/prevendo-o-resultado-das-eleicoes-em-sao-paulo/">Prevendo o Resultado das Eleições em São Paulo</a> (2020), <em>Odysci Blog</em> (in Portuguese).</li><li><a href="https://www.odysci.com/en/blog/the-polls-are-dead-long-live-social-media/">The Polls Are Dead — Long Live Social Media</a> (2021), Odysci Blog.</li><li>All social media data was collected using the <a href="https://www.odysci.com/">Odysci Media Analyzer</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=516235f6f477" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Polls Are Dead — Long Live Social Media]]></title>
            <link>https://rberga.medium.com/the-polls-are-dead-long-live-social-media-8411e1c12044?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8411e1c12044</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[election-polls]]></category>
            <category><![CDATA[opinion-polls]]></category>
            <category><![CDATA[georgia-elections]]></category>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 14:07:34 GMT</pubDate>
            <atom:updated>2021-01-11T18:28:23.193Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Polls Are Dead — Long Live Social Media</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4iN5APGSX15Ut3pkN-BVg.png" /></figure><p><strong>Georgia’s runoff elections are over </strong>with the Democrats winning both seats and taking control of the US Senate. These were extremely competitive races in which both incumbents lost. Jon Ossoff beat David Perdue by just over 50 thousand votes, and Rev. Raphael Warnock beat Kelly Loeffler by almost 90 thousand votes.</p><p>The traditional pollsters’ results were all over the place. <a href="https://www.odysci.com/en/blog/election-day-georgia-social-media-predictions/">We also made predictions</a> based on social media. Let’s see how they all compare.</p><p>The site <a href="https://projects.fivethirtyeight.com/georgia-senate-polls/">FiveThirtyEight</a> lists several poll results by different organizations and computes the average of those results (last updated on the morning of January 5). We followed the links and put the results in the table below, along with our results and the actual election results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*YbyyFEoB4E9QH0mj.png" /></figure><p>In the regular election (Ossoff, Perdue), the group <a href="https://www.axios.com/republican-poll-shows-statistical-tie-in-georgia-7c817124-5c12-40a5-a170-2db1bd105209.html">Preserve America</a>, came the closest to the actual result, with <a href="https://www.odysci.com/en/blog/election-day-georgia-social-media-predictions/">Odysci</a> in second.</p><p>In the special election (Warnock, Loeffler), <a href="https://www.pollsmartmr.com/latest-polls-1/poll-the-georgia-runoffs-are-a-dead-heat">Targoz Market Research</a> was the best, with Odysci second again. For the sake of ranking the polls, we are not considering FiveThirtyEight numbers here because they are weighted averages of several polls.</p><p>In both races, Odysci’s social-media-based predictions were consistently accurate, and ended up the best overall, when adding up the errors. Odysci used only social media data and the November 3 election results to predict the outcome, whereas all other polls presented here used the traditional method of asking people directly or over the phone.</p><p>Traditional polling methods have been suffering from inaccuracies for a while; certainly the 2016 presidential election was one of the most notable snafus. <a href="https://www.bostonglobe.com/2020/11/04/metro/among-losers-election-night-were-americas-pollsters-why-were-they-wrong-again/">Pollsters argue that is increasingly difficult to recruit and weigh likely voters</a>, especially in the age of cell phones, when respondents are less likely to answer calls from people they don’t know. In addition, <a href="https://www.bostonglobe.com/2020/11/13/opinion/problem-with-polling/">people may lie on purpose to pollsters</a> as a means of rebelling against the intrusion.</p><p>Perhaps it is time to rely on different methods to predict election results. Big data and social media are increasingly viable alternatives.</p><p><a href="https://twitter.com/virtualtout">Tom Miller</a> of Northwestern University <a href="https://news.northwestern.edu/stories/2021/01/new-prediction-survey-forecasts-georgia-runoff-elections/">used data analysis combined with polling to predict Georgia’s runoff election results</a> much better than traditional polls.</p><p><a href="https://www.odysci.com/en/blog/election-day-georgia-social-media-predictions/">Odysci used a simpler approach based on social media data</a>. The estimation starts with the number of votes received by each candidate in the November 3 election, considering all candidates that ran for the Georgia US Senate seats (regular and special elections). Then using these numbers of votes, and the percentage numbers of new fans and followers that each candidate gathered in the last six weeks prior to the election, we predicted the likely percentage of votes that each candidate would receive on election day, January 5. Odysci’s estimation does not use any type of polling. Instead, we rely on supporters expressing their preferences on social media.</p><p>Odysci’s models have been tested in Georgia’s runoff elections and in a couple of Brazilian elections before, all with results that were as accurate or better, than traditional polls. Data-driven election prediction is here to stay. The methods are still being developed and tested but the results so far indicate a more accurate future than pollsters can expect with their current methods.</p><blockquote><em>There was a time when polls had a lot of power and were able to sway undecided voters, mostly because people like to be on the winning side. Perhaps their power should be vanquished and anointed to more accurate and representative methods.<br></em><strong><em>The polls are dead, long live social media.</em></strong></blockquote><p>Links to Poll Data:<br><a href="https://projects.fivethirtyeight.com/georgia-senate-polls/">FiveThirtyEight Averages</a><br><a href="https://projects.fivethirtyeight.com/polls/20210104_GA_AtlasIntel.pdf">Atlas Intel</a><br><a href="https://www.axios.com/republican-poll-shows-statistical-tie-in-georgia-7c817124-5c12-40a5-a170-2db1bd105209.html">Preserve America</a><br><a href="https://busr.ag/georgia-senate-poll">BUSR/Univ. of Nevada Las Vegas Lee Business School</a><br><a href="https://www.pollsmartmr.com/latest-polls-1/poll-the-georgia-runoffs-are-a-dead-heat">Targoz Market Research</a><br><a href="https://www.odysci.com/en/blog/election-day-georgia-social-media-predictions/">Odysci — Social Media</a><br><a href="https://www.nytimes.com/interactive/2021/01/05/us/elections/results-georgia-runoffs.html">January 5 Election Results</a></p><p>Cover image <a href="https://iconscout.com/icons/dead-king">Dead king emoji icon</a> by <a href="https://iconscout.com/contributors/epangelald">Evan Gerard</a>.<br> Cover image background by <a href="https://www.freepik.com/photos/background/">Aopsan — Freepik</a>.</p><p><em>Originally published at </em><a href="https://www.odysci.com/en/blog/the-polls-are-dead-long-live-social-media/"><em>https://www.odysci.com</em></a><em> on January 11, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8411e1c12044" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Georgia’s Runoff Elections Using Social Media (Dec/21st)]]></title>
            <link>https://rberga.medium.com/predicting-georgias-runoff-elections-using-social-media-dec-21st-4d28857d8903?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d28857d8903</guid>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Wed, 23 Dec 2020 04:20:25 GMT</pubDate>
            <atom:updated>2020-12-23T14:37:33.512Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article I originally published on the </em><a href="https://www.odysci.com/en/blog/predicting-georgias-runoff-elections-using-social-media-dec21/"><em>Odysci Blog</em></a><em> on December 23, 2020.</em></p><p><strong>The last two, yet undecided, US Senate seats will be chosen by Georgians </strong>on January 5th. Given the accuracy of the polls (or lack thereof) on predicting the outcomes of the November elections, no one really knows <a href="https://www.nytimes.com/2020/12/17/us/politics/georgia-senate-polls.html">which way the electorate is tending to</a>. The lack of reliable polls is forcing the candidates to rely less on poll numbers, and concentrate on their own messages, fending off attacks, and turning out voters.</p><p>Continuing <a href="https://www.odysci.com/en/blog/has-social-media-got-georgia-on-its-mind/">our previous analysis of the candidates’ social media accounts</a>, we will present updates to their social media data as of December 21st, and use a simple metric based on the relative number of new fans per candidate to estimate the percentage of likely votes to each candidate. We used the <a href="https://www.odysci.com/">Odysci Media Analyzer</a> to monitor the social media pages on Facebook and Twitter of Jon Ossoff (D), David Perdue (R), Reverend Raphael Warnock (D) and Kelly Loeffler (R). We collected daily data for four weeks between November 24th and December 21st.</p><p><strong>Fans and Followers</strong></p><p>Let’s begin by revisiting the total number of fans and followers of each candidate on Facebook and Twitter, as of December 21st at 10am EST. The relative positions remained the same as in <a href="https://www.odysci.com/en/blog/has-social-media-got-georgia-on-its-mind/">our previous analysis on Dec/7</a>, although David Perdue and Kelly Loeffler had significant gains on Twitter (as a percentage of their Dec/7 values).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pH1Yi_iuaM-dxmOY.png" /></figure><p>The figure below shows the number of fans and followers that each candidate gained in each of the past four weeks, and the percentage increase over the base number at the beginning of each week. During the weeks of Dec/08-Dec/14 and Dec/15-Dec/21, all candidates gained fewer fans and followers when compared to the previous two weeks, both in absolute terms and percentually. Warnock was the leader in the last two weeks, followed by Ossoff, then Loeffler and lastly, Perdue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ibVkM6bc_dDYy_fd.png" /></figure><p>We also compared the percentage of new fans and followers going for each candidate. For example, for each 100 new people who started following any of the candidates, how many followed each candidate? The following figures look into these numbers pairwise for each senate race for the last four weeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*6ZGogGIZDpoVBekv.png" /></figure><p>The figure above breaks down the distribution of new fans and followers for both candidates, Jon Ossoff (D) and David Perdue (R), between November/24 and December/21, as well as the total in the period. For example, on Dec/15, for each 100 new fans and followers we had 80 following Jon Ossoff (D) and 20 following David Perdue (R). Although there are some daily fluctuations, Ossoff is clearly maintaining a steady lead on the percentage of new fans gained daily. Overall, for the last 4 weeks (up to Dec/21), for each new 100 fans and followers, 82 sided with Jon Ossoff (D) and 18 sided with David Perdue (R).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*lObtpF7Ctw_QIGXA.png" /></figure><p>The figure above shows the same metric for candidates Reverend Warnock (D) and Kelly Loeffler (R). For example, on Dec/17, for each new 100 fans and followers we had 57 following Reverend Warnock (D) and 43 following Kelly Loeffler (R). In this race, we see strong fluctuations from one day to the next, but it seems that after Dec/6, Reverend Warnock was able to keep the lead, although not always by high margins. Overall, for the last 4 weeks (up to Dec/21), for each new 100 fans and followers, 61 sided with Reverend Warnock (D) and 39 sided with Kelly Loeffler (R).</p><p><strong>Using Social Media to Forecast the Election</strong></p><p>We developed a methodology that starts with the number of votes received by each candidate in the Nov/3 election, considering all candidates that ran for the Georgia US Senate seats (regular and special elections). Then using these numbers of votes, and the percentage numbers of new fans and followers that candidate gathered in the last four weeks, we predict the likely percentage of votes that each candidate would receive if the election were held on Dec/21. Details of the methodology are as follows:</p><ol><li>Get the numbers of votes received by all candidates in the first election (Nov/3). We will assume that all voters that cast a valid vote on Nov/3 will do so again on Jan/5.</li><li>We assume that the people who voted, in the first election, for Jon Ossoff, David Perdue, Reverend Warnock and Kelly Loeffler, will do so for the same candidate again in the runoff election.</li><li>In the regular election, besides Ossoff and Perdue there was only one other candidate, Shane Hazel, a LIbertarian, who got about 115 thousand votes. For now, we will consider these 115 thousand voters as <strong>undecided</strong>, but people who will vote again on Jan/5.</li><li>In the special election, besides Warnock and Loeffler, there were 18 other candidates, from various parties. We will now “reassign” the votes for those 18 candidates to Warnock or Loeffler. We will assume that a high percentage (90%) of people who voted for a Democratic candidate will transfer their vote to Warnock, and the same percentage of people who voted for a Republican candidate will vote for Loeffler. The 10% remaining from each group of voters (pro Democratic or pro Republican) will be considered as <strong>undecided</strong>, or people who may switch camps. The people who voted for candidates in other parties (Libertarian, Green, Independent) will all be assumed to be <strong>undecided</strong>.</li><li>We assume that the number of <strong>undecided</strong> voters, in each election, will be the sum of the <strong>undecided</strong> voters as explained in steps <strong>3.</strong> and <strong>4.</strong> above, plus a number of new voters (who did not vote in the first round), which <a href="https://www.ajc.com/politics/75k-new-georgia-voters-registered-before-us-senate-runoffs/H3CXAFIKFVCKHJNW5MBFZKQDZU/">we will assume to be 75 thousand</a>. This estimate is based on the number of new voter registrations between Nov/3 and Dec/7 (registration deadline).</li><li>Based on the steps above, we estimate the number of <strong>undecided</strong> voters in the regular runoff election as 190 thousand, whereas for the special election we estimate the number of <strong>undecided</strong> voters as 376 thousand.</li><li>We will distribute these <strong>undecided</strong> voters per day in the 4 weeks considered in this analysis, as if they were making up their minds, a few each day; and we will “assign” them to each candidate (in each race, separately) based on the percentage of new fans and followers that they acquired each day, as shown in the graphs above entitled “<em>For each new 100 new fans and followers, how many go to each candidate per day?</em>”</li></ol><p>The figures below illustrate the percentage of likely votes per candidate, per day, and the final value for the regular and special elections if they were held on Dec/21. The inicial percentages on Nov/23 are based on the numbers of votes that each candidate received on Nov/3, plus voter migration as explained in step <strong>3.</strong> and <strong>4.</strong> above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/0*yGO2WE9wKMlPfR-c.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*Rm9tU3hbB7nc40EA.png" /></figure><p>For the Ossoff vs. Perdue race, we estimate that Ossoff leads Perdue by <strong>0.66</strong> percentage points on Dec/21. For the Warnock vs. Loeffler race, our estimate shows Warnock leading Loeffler by <strong>1.40</strong> percentage points on Dec/21.</p><p>This is a simplified analysis using the results of the first election (Nov/3), a few assumptions about voter migration and new undecided voters, and the relative gain of new fans and followers by each candidate on Facebook and Twitter in the last 4 weeks. In both races the edge of one candidate over another is within the margin of error of election polls (usually 2% to 4%), so it can certainly go either way. Although this analysis lacks statistical rigor, we believe it shows actual trends of the electorate, and could be used as validation or augmentation of other polling methods.</p><p>We plan to post one more update on the eve of the election on Jan/5.</p><p><em>Originally published at </em><a href="https://www.odysci.com/en/blog/predicting-georgias-runoff-elections-using-social-media-dec21/"><em>https://www.odysci.com</em></a><em> on December 23, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d28857d8903" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Has Social Media Got Georgia On Its Mind?]]></title>
            <link>https://rberga.medium.com/has-social-media-got-georgia-on-its-mind-bce1ecf0d749?source=rss-90c7ef27f79b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bce1ecf0d749</guid>
            <category><![CDATA[georgia-elections]]></category>
            <category><![CDATA[georgia]]></category>
            <category><![CDATA[georgia-senate-runoff]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[us-senate]]></category>
            <dc:creator><![CDATA[Reinaldo Bergamaschi]]></dc:creator>
            <pubDate>Thu, 10 Dec 2020 21:33:25 GMT</pubDate>
            <atom:updated>2020-12-14T16:10:27.246Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Article I originally published on the </em><a href="https://www.odysci.com/en/blog/has-social-media-got-georgia-on-its-mind/"><em>Odysci Blog</em></a><em> on December 10, 2020.</em></p><p>With the presidential election behind us (or most of us, at least) all eyes have turned to Georgia and the Senate run-off elections, as they will decide which party will eventually have control of the Senate — and, of course, which candidates will better defend Georgia’s interests in Washington. In this article we take a closer look at the social media pages of the candidates.</p><p>Since Obama’s reelection in 2013, researchers have used social media to try to understand and forecast election results. Obama’s campaign in 2012 was a milestone in how to use social media to mobilize its base, but at that time social media was not yet used extensively to compare candidates. Trump’s campaign in 2015 stepped up the use of social media to new levels. In 2015, we also saw the use of fake news as an effective way of spreading disinformation, and that applies in different degrees to sympathizers of both parties. Since then, candidates across the board have become more aware of the importance of social media, and started using them a lot more. Considering that a large percentage of the US population (79% in 2019 according to <a href="https://www.statista.com/statistics/273476/percentage-of-us-population-with-a-social-network-profile/">Statista</a>) is active on social media, what the fans and followers say on social media has become an important indicator of trends among the electorate.</p><p>Research has shown that many people feel more at ease expressing their personal views online than when talking to a pollster. That happens because the person may be embarrassed to openly support a more controversial candidate, or because of lack of trust in the institutions responsible for the polling ( <a href="https://www.washingtonpost.com/news/monkey-cage/wp/2016/12/21/using-these-4-steps-pollsters-could-use-social-media-data-to-improve-election-forecasts">How pollsters could use social media data to improve election forecasts</a>). The opinions expressed on social media tend to be more polarizing and less susceptible to social norms that may prevail in the family or work environment.</p><p>Social media played an even more significant role in this year’s elections. Due to the pandemic, big rallies were avoided by most candidates, who had to rely on social media to divulge their platforms and bring new sympathizers to their side.</p><p>This blog post analyzed the social media accounts of the four candidates to the Senate in Georgia’s runoff elections on January 5th, 2021. We used the Odysci Media Analyzer to monitor daily the social media pages on Facebook and Twitter of Jon Ossoff (D), David Perdue (R), Reverend Raphael Warnock (D) and Kelly Loeffler (R). We monitored Facebook and Twitter because these are the social networks most used by the candidates, although they all have Instagram and YouTube accounts as well. The data reported here were collected for two weeks between November 23rd and December 7th.</p><h4><strong>Fans and Followers</strong></h4><p>To start off, let’s have a look at the plain sizes of fans and followers of each candidate’s social media pages. The numbers were collected on December 7th at 8am.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*RtdTShuQQ0ZrOj2X.png" /></figure><p>The figure above shows Jon Ossoff leading in the overall number of Facebook fans and Twitter followers with just over 836 thousand, and Ossoff also leads separately on Facebook and Twitter. Reverend Warnock takes 2nd place in overall numbers, 2nd on Twitter and 3rd on Facebook. Kelly Loeffler is 3rd overall, 3rd on Twitter and 4th on Facebook. Finally, David Perdue is 4th in overall fans and followers, although he is 2nd on Facebook. Perdue is also the only candidate who has more Facebook fans than Twitter followers.</p><p>The overall number of fans and followers changes daily, but it does not say much about the rate in which new fans and followers are obtained, nor does it say anything about their engagement with the candidate. For that, we will now look at how many fans and followers each candidate gained in the last two weeks, as well as the amount of engagement that the candidates’ posts and tweets received.</p><p>The figure below shows the number of fans and followers that each candidate gained in each of the past two weeks, and the percentage increase represented over the base number starting each week. Here we can see that Warnock gained the most fans and followers in the week of Nov/24–10, with Ossoff in second; whereas Ossoff was the winner in the week of Dec/01–07 with Warnock in second. However, in terms of percentage increase we saw Kelly Loeffler leading with a 13.3% increase in the first week, and 13.1% increase in the second week.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*p4GzNt4fy--Dgc-n.png" /></figure><p>It is also important to compare the percentage of new fans and followers going for each candidate. For example, for each 100 new people who started following any of the candidates, how many followed each candidate? The following figures look into these numbers pairwise for each senate race.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*VhKnwS66euf2B833.png" /></figure><p>The figure above breaks down the distribution of new fans and followers for both candidates, Jon Ossoff (D) and David Perdue (R), between November/24 and December/7, as well as the total in the period. For example, on Nov/26, for each 100 new fans and followers we had 88 following Jon Ossoff (D) and 12 following David Perdue (R). Overall, for the 2 weeks, for each new 100 fans and followers, 81 sided with Jon Ossoff (D) and 19 sided with David Perdue (R).</p><p>Although there are a few daily variations, the overwhelming majority of new social media fans and followers are going for Jon Ossoff. We saw a small decrease in number on Dec/6, possibly explained by a boost to Perdue from Trump’s rally on Dec/5; only to see a bump in the numbers again on Dec/7, possibly explained by the debate in which Perdue was a no-show and Ossoff had all the coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*h-nbeivkCKRsi7RU.png" /></figure><p>The figure above shows the same metric for candidates Reverend Warnock (D) and Kelly Loeffler (R). For example, on Nov/28, for each new 100 fans and followers we had 78 following Reverend Warnock (D) and 22 following Kelly Loeffler (R), but on Dec/6 we had 35 for Reverend Warnock and 65 for Kelly Loeffler. In this race, we see strong fluctuations from one day to the next and it’s unclear why. Overall, for the 2 weeks, for each new 100 fans and followers, 57 sided with Reverend Warnock (D) and 43 sided with Kelly Loeffler (R).</p><p><strong>Engagement</strong></p><p>Engagement is a measure of how much the fans and followers interact with the social media pages of the candidates. This interaction may vary from simply looking at the candidates’ page, to liking a page, post or tweet, to writing a comment, to sharing the post or retweeting it. Here we will measure engagement based on the number of comments that fans and followers produce in association with a candidate’s social media page. Just to clarify the methodology: we count as a comment when someone posts something on the candidate’s Facebook page; when someone writes a tweet containing a mention <em>@candidate-page;</em> or when someone re-tweets something the candidate had tweeted.</p><p>The figures below show the total number of posts and tweets, as well as the total number of comments (as defined above) received by each candidate in the two weeks considered (Nov/24-Dec07). Kelly Loeffler posts the most, especially on Twitter. In second place comes Jon Ossoff, followed by Warnock and Perdue. On the receiving side, we see a different picture with Jon Ossoff way in front on Twitter having received more than 100 thousand mentions and retweets. Warnock comes second with over 60 thousand. Kelly Loeffler is third, followed by Perdue in fourth. On Facebook alone, Perdue has the lead in number of comments received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*49Koh063YfAjGjHg.png" /></figure><p>More important than the absolute number of comments received, is the ratio of comment-per-post, and comment-per-post-per-1000-fans. Usually, candidates that post more will receive more feedback, and the same is true for pages with more more fans and followers. Thus, to be able to compare we need to compute the ratios of comments per post/tweet, and comments per post/tweet per 1000 fans/followers, as shown below (note that by “comments” we mean comments, mentions, replies and retweets).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x6Nc-XNOCX_glDNA.png" /></figure><p>These figures show a few interesting aspects. On Facebook, Perdue has more engagement than the others. He posted less than the others but received a larger number of comments, making his Facebook metric Comments-per-Post the highest, followed by Loeffler. But once we divide by the number of fans, Loeffler’s engagement shoots up (to 10.7), mainly because Loeffler has the fewest number of Facebook fans. On Twitter, Ossoff and Warnock have the highest Comments-per-Post metrics, but that goes down once we divide by their larger number of followers.</p><p>Another interesting aspect of engagement, especially in politicians’ pages, is that the people who comment on the page or add a @mention to the candidate are not necessarily supporting the candidate. This means that having received a significant number of comments, or having high engagement does not necessarily represent support for whatever the candidates posted.</p><p><strong>What They Said and What They Heard</strong></p><p>The words used by the fans and followers in their comments on Facebook and mentions on Twitter are excellent indicators of the topics most relevant to the users tuned to the candidates’ social media pages. Candidates who are responsive to their constituencies should pay close attention to that. Moreover, it is also important to correlate what the candidates post with what their fans comment.</p><p>Using the <a href="https://www.odysci.com/">Odysci Media Analyzer</a> we mined the most frequent terms in the candidates’ posts and tweets, as well as in the comments, replies and retweets of their fans and followers. A word on methodology, for each post or tweet, we analyzed up to 500 comments, replies or retweets. Most posts and tweets have fewer than 500 comments, but when they exceeded this number, we collect 500 of them for this analysis. When collecting the frequent terms, we counted together terms with similar meaning, for example, “corrupt” and “corruption” were counted together. Finally, we also removed certain terms which tend to have very high frequencies, such as, the candidates’ names, ‘Georgia’, ‘Trump’, ‘Senate’, and other politicians’ names. This was done in order to let the actual words representing issues important to Georgians appear closer to the top.</p><p>To visualize these frequent terms, we created word clouds, where the inner cloud contains the words most frequent in the candidate’s posts and tweets, and the outer cloud contains the words most frequent in the comments, replies and retweets received by the candidate’s social media pages. The bigger the word in the cloud, the more frequent it appeared. The following figures show these <em>What They Said x What They Heard</em> word clouds.</p><p><strong>Jon Ossoff (D)</strong></p><p>His postings focused on attacking his opponent on alleged irregular stock sales and corruption. In addition, he focused on business, jobs, clean energy and health care. The comments Ossoff received focused mainly on corruption, Covid-19, health care, and stocks, followed by many other terms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/0*qwgFd3R_NtWDMf6r.png" /></figure><p><strong>David Perdue (R)</strong></p><p>Perdue’s posts emphasized business, jobs, taxes, supreme court, and attacked socialism and illegal immigration. The comments received talked about election fraud, the pandemic, covid-19, and attacked him about stock sales and corruption, among many other different topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*Jub4N_cQ77NMXvJa.png" /></figure><p><strong>Reverend Warnock (D)</strong></p><p>Raphael Warnock posted mainly about the pandemic, business, and health care. He received comments about Covid-19, corruption, health care, racism, stocks, fraud and many others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/0*CSq8EDkxbAYXSWTu.png" /></figure><p><strong>Kelly Loeffler (R)</strong></p><p>Her postings focused on business, taxes, and the economy. The comments received talked about election fraud, Covid-19, stocks, corruption, taxes, the pandemic, and several other issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*_LpLzpfIx8nGRyKu.png" /></figure><p><strong>Using Social Media to Forecast the Election</strong></p><p>Based on the polling results in the 2016 and 2020 elections, it has become clear that polls are no longer able to predict the actual outcome of the voting within a reasonable margin of error. In 2016, polls wrongly underestimated Donald Trump’s popularity in some states, which caused surprise with the subsequent outcome of the election (i.e., most polls had predicted Hillary Clinton would win). In 2020, many polls also underestimated Trump’s reach and predicted that Biden would take some of the battleground states with higher margins than he actually got. Polls did not get the final winner of the 2020 election wrong, but <a href="https://theconversation.com/an-embarrassing-failure-for-election-pollsters-149499">they got their numbers considerably off in some states</a>.</p><p>No convincing explanation has been put forward for why polls seem to be getting less accurate in the last 2 presidential elections. Some say it is the behavior of the Trump voter, but many are not convinced this is an issue with the Trump constituencies, and <a href="https://www.rollingstone.com/politics/politics-features/polling-failure-2020-election-1091176/">point to more structural problems in polling.</a></p><p>Social media has established itself as an additional component that can help predict the outcome of elections, although it is not clear yet which social metrics work better. Early research based on the <a href="https://www.tandfonline.com/doi/abs/10.1080/15377857.2014.959690">elections in New Zeland in 2011</a> and in <a href="https://www.aclweb.org/anthology/W11-3702.pdf">Ireland in 2011</a> indicated correlation between election results and the size of the social networks as well as the amount of activity on the social networks of the candidates. Although this was the case for those elections, it is no longer the case today, as candidates, winners and losers, have amassed huge number of fans and followers as well as extremely high engagement. So, number of fans and engagement are no longer a reliable metric.</p><p>According to <a href="https://www.socialmediatoday.com/news/can-social-media-data-predict-the-winner-of-the-2020-us-presidential-electi/587693/">SocialMediaToday</a>, looking at the 2020 presidential elections, comparing the absolute numbers of fans, engagement, likes and comments, between candidates is not an effective way of predicting the election outcome. It suggests that looking at recent growth of these metrics in isolation may better indicate momentum for one candidate or the other.</p><p>Our own studies, looking at the Brazilian presidential election of 2018, concluded that in polarized 2-sided elections, the daily incremental gain in number of fans and followers was a good metric that could be used to measure relative change in support of a candidate. ( <a href="https://www.linkedin.com/pulse/parte-9-podem-m%C3%ADdias-sociais-prever-o-resultado-das-para-bergamaschi/">The results of this study were published in Portuguese</a>).</p><p>We will used a similar method here to estimate the chances of each candidate in Georgia’s senate elections, and we will compare our numbers with the average poll numbers collected by the site <a href="https://projects.fivethirtyeight.com/georgia-senate-polls/">FiveThirtyEight</a> (updated on Dec/6).</p><p>This was our methodology:</p><ol><li>Get the numbers of votes received by each candidate in the first election, and compute the number of votes received by all other candidates</li><li>We assume that the people who voted, in the first election, for Jon Ossoff, David Perdue, Reverend Warnock and Kelly Loeffler, will do so for the same candidate again in the runoff election.</li><li>We assume that the number of undecided voters will be the sum of the votes for all other candidates, plus a number of new voters (who did not vote in the first round), something between 0 and 100 thousand. This may be a very rough estimate (or overestimate), but considering the stakes and the efforts by DEM and GOP to win the senate, we may see more people voting.</li><li>We will distribute these undecided voters per day in the 2 weeks considered in this analysis, as if they were making up their minds, a few each day; and we will “assign” them to each candidate (in each race, separately) based on the percentage of new fans and followers that they acquired each day, as shown in the graphs above entitled “<em>For each new 100 new fans and followers, how many go to each candidate per day?</em>”</li><li>To compare with the average poll numbers collected by the site <a href="https://projects.fivethirtyeight.com/georgia-senate-polls/">FiveThirtyEight</a>, we start off on the same day, November 23rd, with the same percentage points to each candidate. Then using the steps above we compute the gain/loss of each candidate daily. At the end of our analysis period, December 7th, we have the percentage points for each candidate, if the elections were held on that day.</li></ol><p>The figures below illustrate the values generated by our method, in comparison with the values collected by FiveThirtyEight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*yK_rZ_YcuYkzCo8-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*_kAGGOcQgkrv58tF.png" /></figure><p>For the Ossoff vs. Perdue race, the poll average computed by FiveThirtyEight, says that on Dec 6, the difference between the candidates was <strong>0.8%</strong>, whereas we came up with <strong>0.62%</strong>.</p><p>For the Warnock vs. Loeffler race, the poll average computed by FiveThirtyEight, says that on Dec 6, the difference between the candidates was <strong>2.2%</strong>, whereas we came up with <strong>1.28%</strong>. One thing to note is that FiveThirtyEight percentages do not add up to 100% and ours do.</p><p>In both races the edge of one candidate over another is within the margin of error of election polls (usually 2% to 4%), so it can certainly go either way. Although this is a simplified analysis, without statistical rigor, we believe it shows actual trends of the electorate and could be used as validation or augmentation of other polling methods.</p><p><strong>Wrapping Up</strong></p><p>We looked at the main social media metrics of all four candidates in Georgia’s runoff senate elections, as of Dec/7. Here are the main takeways:</p><ol><li>Ossoff has the most combined number of Twitter followers and Facebook fans, followed by Warnock, Loeffler and Perdue.</li><li>Between Nov/24 and Dec/7, Ossoff gained 66,824 new fans and followers, with Warnock in close second with 66,111. Loeffler is third with 50,766, and Perdue a distant fourth with only 16,039 new fans and followers.</li><li>Considering the number of new fans and followers going to candidates in each race daily (between Nov/24 and Dec/7), Ossoff has consistently gained more than Perdue, about 81 to 19 (for each 100 new fans/followers). These values for Warnock and Loeffler are very different, with huge daily oscillations, and overall, for the period, Warnock gained 57 to Loeffler’s 43 (for each 100 new fans/followers).</li><li>All four candidates published similar numbers of posts and tweets, with Loeffler being the most prolific. Ossoff and Warnock received the most mentions and replies on Twitter, with Perdue and Loeffler leading in comments received on Facebook. The engagement per 1000 fans/followers was the highest for Loeffler and Perdue on Facebook, in part because of their lower number of fans.</li><li>Analyzing the actual words most frequent in their posts and tweets, as well as in the comments and replies received, we see a familiar pattern: Covid-19, Corruption, Fraud, Business, Pandemic, and Health Care were among the most popular, although the top one for each candidate varied.</li><li>Finally, we used the incremental daily fans/followers counts to estimate the percentage gain or loss of voting intention for each candidate. Both races appear very tight on Dec/7 with Ossoff leading Perdue by 0.82%, and Warnock leading Loeffler by 1.3%.</li></ol><p>We plan to post updates to the data presented here as we get closer to the election.</p><p><em>Originally published at </em><a href="https://www.odysci.com/en/blog/has-social-media-got-georgia-on-its-mind/"><em>https://www.odysci.com</em></a><em> on December 10, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bce1ecf0d749" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>