<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Brockman on Medium]]></title>
        <description><![CDATA[Stories by Matt Brockman on Medium]]></description>
        <link>https://medium.com/@mattbrockman-78128?source=rss-a3d23264e777------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*i0Ha1spzJug4pHG2KHsZCA.jpeg</url>
            <title>Stories by Matt Brockman on Medium</title>
            <link>https://medium.com/@mattbrockman-78128?source=rss-a3d23264e777------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:59:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mattbrockman-78128/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deploying an Extension to the Chrome Store in 30 minutes]]></title>
            <link>https://mattbrockman-78128.medium.com/deploying-an-extension-to-the-chrome-store-in-30-minutes-55be733992e9?source=rss-a3d23264e777------2</link>
            <guid isPermaLink="false">https://medium.com/p/55be733992e9</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <category><![CDATA[twitter]]></category>
            <dc:creator><![CDATA[Matt Brockman]]></dc:creator>
            <pubDate>Sat, 26 Nov 2022 01:07:03 GMT</pubDate>
            <atom:updated>2022-11-26T01:07:03.166Z</atom:updated>
            <content:encoded><![CDATA[<p>Permissions and manifests and messaging, oh my! Deploying a Chrome Extension can seem like a daunting task but you can submit to the store in less than half an hour and have it live as soon as the next day!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3KN2mx3ZLc2k84P_Wzk5w.png" /><figcaption>I literally just deployed this at <a href="https://chrome.google.com/webstore/detail/fixedui/ccjnldpmocfclkchljacnjmbgdfcgbmi">https://chrome.google.com/webstore/detail/fixedui/ccjnldpmocfclkchljacnjmbgdfcgbmi</a></figcaption></figure><p>So why write about this? Earlier This week, a Twitter “Intern” (I think he technically is) asked if people could go and hack the Twitter search menu to make it less painful.</p><h3>George Hotz 🐀 on Twitter: &quot;Not sure I have authority for this, but if amazing React coder/designer wants internship to help me fix search...or someone working @Twitter alreadyI want to type &quot;from:&quot; in the Search Twitter box and have it tokenize and autocomplete.&quot;from:&quot; should not show me @TayFromCA pic.twitter.com/PHuhHg2hMT / Twitter&quot;</h3><p>Not sure I have authority for this, but if amazing React coder/designer wants internship to help me fix search...or someone working @Twitter alreadyI want to type &quot;from:&quot; in the Search Twitter box and have it tokenize and autocomplete.&quot;from:&quot; should not show me @TayFromCA pic.twitter.com/PHuhHg2hMT</p><p>Within a few days, a bunch of people across the internet went and created quick scripts to inject code into Twitter to make the experience better. I had a shot at it which you can see below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FerbwW5SOgHE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DerbwW5SOgHE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FerbwW5SOgHE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3d055e8362e2953c01d6e0d35c97fa6/href">https://medium.com/media/b3d055e8362e2953c01d6e0d35c97fa6/href</a></iframe><p>However, having people inject random scripts into their browser can be a bit dangerous, so taking this to the next level, it makes sense to be able to stick this in a Chrome Extension which is a bit more safer for users. I went and did it for my fun demo, and figured I’d write up a guide on how to do it for people that want to make an extension quickly themselves.</p><p>There’s three major components to submitting an extension to the chrome store: a Chrome Developer account, a privacy policy, and of course an extension (with the script you want to inject). Then take all of that and stick it in the Chrome store! So here’s how to go about getting these done.</p><h3>Create your Chrome Developer Account</h3><p>This is the easiest step. You just need to pay Google $5; it used to be a bit pricier but it looks like Google’s dropped the prices. Just go to <a href="https://chrome.google.com/webstore/devconsole/register">https://chrome.google.com/webstore/devconsole/register</a> and pay Google your $5 and it lasts forever. You’ll need to come back here later to actually upload the extension but we can worry about that later.</p><h3>Grab a domain and create a quick landing page</h3><h4>Step 1 (optional): Find the domain.</h4><p>There’s quite a few hosting providers out there. I just used Google for this and went to <a href="https://domains.google.com/registrar/">https://domains.google.com/registrar/</a> to find myself a domain. It costs $12 for a .com for a year for <a href="https://www.fixedui.com/">https://www.fixedui.com/</a>.</p><h4>Step 2: Create a quick landing page.</h4><p>For a fast landing page, <a href="https://github.com/ixartz/Next-JS-Landing-Page-Starter-Template">https://github.com/ixartz/Next-JS-Landing-Page-Starter-Template</a> is pretty good. You can just fork it and then download it and customize it however you want. The main page is in the filebase.tsx , and you can just swap in whatever info you want (or leave the defaults, Google doesn’t care). You can see the one I threw up at <a href="https://github.com/matthewlouisbrockman/FixedUI_Frontend">https://github.com/matthewlouisbrockman/FixedUI_Frontend</a>.</p><h4>Step 3: Host the landing page</h4><p>The above template has an easy to click option in the Readme.md to deploy directly to <a href="https://vercel.com/">https://vercel.com/</a>. This lets you instantly deploy your website as you make changes to it. If you completed Step 1, you can then set it up to deploy to your custom domain via the A-record.</p><h4><strong>Step 4: Grab a privacy policy</strong></h4><p>This part’s kind of long and I’m not sure the best way to do it so I just used <a href="https://termly.io/products/privacy-policy-generator/">https://termly.io/products/privacy-policy-generator/</a> to go and make one. I copied the code into a file in my repo called privacy.html which I then stuck into the public folder in my nextjs app. This means that you can go to wherever it’s hosted /privacy.html and it’ll be there automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*QBlzofXb0AuTIOFn_B3bOg.png" /><figcaption>privacy.html in my code, viewable at <a href="https://www.fixedui.com/privacy.html">https://www.fixedui.com/privacy.html</a></figcaption></figure><h3>Create Your Extension</h3><p>For the challenge, the goal was to create a script that would modify Twitter search to be less bad. I already had my code written, so I dropped it into another template, <a href="https://github.com/lxieyang/chrome-extension-boilerplate-react">https://github.com/lxieyang/chrome-extension-boilerplate-react</a>. That’s a great template for deplying an extension and it includes a bunch of files you might want to use later. However, for now all that matters is getting your script into there (you can see my modification at <a href="https://github.com/matthewlouisbrockman/FixedUI_Extension">https://github.com/matthewlouisbrockman/FixedUI_Extension</a>).</p><p>My script was called fixer.js , so I created a folder in the pages directory called Twitter and then dropped my fixer.js file in there. Then I just had to go to webpack.config.js and update the entry object to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*2mYCnao1uQYCmWqzHWlhyA.png" /><figcaption>fixer.js in the Twitter folder in pages</figcaption></figure><pre>entry:{<br>...,<br>twitter: path.join(_dirname,&#39;src&#39;, &#39;pages&#39;,&#39;Twitter&#39;, &#39;fixer.js&#39;)<br>}</pre><p>What this does is create a singled bundled javascript file called [name.bundle.js] at build time that has all the dependancies you’ll need.</p><p>Then you need to edit manifest.json , which is the core piece of a Chrome extension that tells chrome what files you’re loading where. I deleted all the content scripts and instead just injected mine to only fire on Twitter -</p><pre>  &quot;content_scripts&quot;: [<br>    {<br>      &quot;matches&quot;: [&quot;https://twitter.com/*&quot;],<br>      &quot;js&quot;: [&quot;twitter.bundle.js&quot;],<br>      &quot;run_at&quot;: &quot;document_end&quot;<br>    }<br>  ],</pre><p>As you can see, the matches parameter is what sites the script will inject on, js is the bundle created by the webpack, and run_at says to inject once the page loads.</p><p>To test it in real time, you just do npm start , and to create a production build folder, you run NODE_ENV=production npm run build . (This is all in the readme that comes with the starter code as well). We’re now ready to submit this to the chrome store! We simply zip the build folder made by running that last step and go back to the Chrome store (<a href="https://chrome.google.com/webstore/devconsole/register">https://chrome.google.com/webstore/devconsole</a>) and submit it.</p><p>For testing it, you just go to chrome://extensions , make sure Developer Modeis switched on and import the build folder using the load unpacked button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WFiZO1drD58YAlZXrEVOjg.png" /></figure><h3>Finalizing the Submission</h3><p>We’re now ready to submit to the chrome store! We drag the zip file into the Chrome store submission box to start. We need to fill in all the fields they ask about, include a quick video or description if we want, and link to the website and privacy policy we created above. And now we’re done!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uqy4iR5EjUCWrYPiNqWNIQ.png" /><figcaption>It took a day for Google to approve my extension</figcaption></figure><p>At this point, it can take a day or more for Google to check your code and see if there’s any security or privacy issues. You’ll need to do this each time you resubmit but hopefully the hard part of figuring out what you need to do to get started is done!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55be733992e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[1+1=3. Wait, no, 1+1=2. How to have GPT sanity check itself.]]></title>
            <link>https://medium.com/data-science/1-1-3-wait-no-1-1-2-how-to-have-gpt-sanity-check-itself-136e846987bf?source=rss-a3d23264e777------2</link>
            <guid isPermaLink="false">https://medium.com/p/136e846987bf</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[naturallanguagegeneration]]></category>
            <dc:creator><![CDATA[Matt Brockman]]></dc:creator>
            <pubDate>Wed, 30 Jun 2021 18:29:28 GMT</pubDate>
            <atom:updated>2021-06-30T22:46:25.963Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A Python walkthrough of using GPT to double-check its answer using GPT-J in Colab</em></h4><p>Large language models have a problem where they tend to just make stuff up. This can be because of the training data, the prompt, or even just ambiguity. This can be mitigated by engineering the prompt to have GPT sanity check its output, and it works with both GPT3 and GPT-J (the latter of which you can use for free).</p><p>What do I mean by a sanity check? Well, it turns out by setting up your prompts to double check the output using something like:</p><pre>Initial Answer: 6394 + 250 = 6643<br>Double Checking: 6643 looks wrong. 6394 + 250 = 6644</pre><pre>Initial Answer: {a} + {b} =</pre><p>you can improve performance compared to using a traditional prompt that looks more like:</p><pre>6394 + 250 = 6644<br>{a} + {b}=</pre><p>Also, for those like me who don’t read good, you can skip this post and just <a href="https://colab.research.google.com/github/matthewlouisbrockman/LearnGPT3/blob/main/GPT_J_6B_planning_sharable1.ipynb">run the code using GPT-J yourself using the linked colab</a>.</p><p>For background, there are a few other methods that people use to mitigate this sort of problem. The most popular is just manually curating for answers you like. That’s boring, and doesn’t really help if you’re trying to automate processes. There’s another method which involves having a model show its work through <a href="http://gptprompts.wikidot.com/linguistics:word-in-context">intermediary steps</a> which can <a href="https://blog.andrewcantino.com/blog/2021/05/28/how-to-dramatically-improve-the-reasoning-ability-of-GPT-3/">improve reasoning</a>, which is cool. However, I figure it’s worth taking a look at self correcting to see how well GPT can look at what it spat out and decide whether it needs improving because it doesn’t seem like people have looked into it much (although there were some cool tweets about anecdotally <a href="https://twitter.com/kleptid/status/1284098635689611264">guess-and-checking random algebraic solutions</a> last year).</p><p>For a toy example, I went with arithmetic. Why arithmetic? Well, first of all, we can measure if an answer’s correct which is often good to know. Second, while there’s an argument that the reason language models don’t do great at arithmetic is due to the way GPT’s tokens are encoded, I suspect that there’s a search issue with multi-digit arithmetic where you have to add from right to left to carry digits, and when you’re just predicting ahead it’s hard to guess where the digits are gonna get carried. GPT isn’t great at that type of search, and if search is contributing to this issue, then doing a double check should help it.</p><p>Anyway, in this post I’ll show you how to generate contexts with between 1–30 examples, run their completions using a standard few shot and an improved self-checking few shot, and then we’ll plot the errors (figure 1) and see that indeed, the self-checking prompts have a lower error and a higher exact match than the traditional few shot. Here’s what the graph’ll look like (Note for exact match we’re getting 16% top accuracy, whereas GPT3 large with commas you can get over 90% so there can be quite a difference).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*eMCQgARRgqeBOVuRm2QWtQ.png" /><figcaption>Figure 1: A self-correcting prompt has lower mean absolute percent error (left) and higher exact match (right) than regular few shots on 4 digit arithmetic using GPT-J.</figcaption></figure><h4>Setting up the query function</h4><p>This works on the GPT-J as well the GPT-3 models, you’ll just have to set up your query function differently, which is where we’ll start. When we query a completion for this sort of task, we want to use 0 temperature because we want to get the most likely tokens instead of randomly throwing in other numbers. Here’s what my query function in J looks like (borrowed from the infer function in the <a href="https://colab.research.google.com/github/kingoflolz/mesh-transformer-jax/blob/master/colab_demo.ipynb">GPT-J demo notebook</a> which is pretty cool):</p><pre>def query(context, top_p=0, temp=0, gen_len=50):<br>  tokens = tokenizer.encode(context)<br>  provided_ctx = len(tokens)<br>  pad_amount = seq - provided_ctx<br>  padded_tokens = np.pad(tokens, ((pad_amount, 0),)).astype(np.uint32)<br>  batched_tokens = np.array([padded_tokens] * total_batch)<br>  length = np.ones(total_batch, dtype=np.uint32) * len(tokens)<br>  output = network.generate(batched_tokens, length, gen_len, {&quot;top_p&quot;: np.ones(total_batch) * top_p, &quot;temp&quot;: np.ones(total_batch) * temp})<br>  samples = []<br>  decoded_tokens = output[1][0]<br>  for o in decoded_tokens[:, :, 0]:<br>    samples.append(tokenizer.decode(o))<br>  return samples[0]</pre><p><strong>Setting up the Prompt</strong></p><p>Now we’ll go ahead and set up our prompt. We’ll create 30 random examples to be our prompt and 100 examples as the test set.</p><pre>import pandas as pd, json, random</pre><pre>random.seed(42)</pre><pre>fourDigitDict = []<br>fourDigitTest = []</pre><pre>for i in range(30):<br>  a = int(random.random()*10**4)<br>  b = int(random.random()*10**4)<br>  fourDigitDict.append({&#39;first&#39;: a, &quot;second&quot;: b, &#39;sum&#39;: a+b})<br>fourDigitTrainDF = pd.DataFrame(fourDigitDict)</pre><pre>for i in range(100):<br>  a = int(random.random()*10**4)<br>  b = int(random.random()*10**4)<br>  fourDigitTest.append({&#39;first&#39;: a, &quot;second&quot;: b, &#39;sum&#39;: a+b})<br>fourDigitTestDF = pd.DataFrame(fourDigitTest)</pre><p>For the pure/traditional case, we’d just have the examples all lined up as the prompt, but instead what we’ll do here for the self-correcting is randomly set up some of these to be wrong. We’ll call the wrong answers (or right ones) <em>originalSum</em> so that we can tell the model to correct it to the right answer later.</p><pre>fourDigitTrainDF[&#39;randomlyWrong&#39;] = fourDigitTrainDF[&#39;sum&#39;].apply(lambda x: random.random() &lt; .5)</pre><pre>fourDigitTrainDF[&#39;offset&#39;] = fourDigitTrainDF[&#39;randomlyWrong&#39;].apply(lambda x: .5-random.random() if x==True else 0)</pre><pre>fourDigitTrainDF[&#39;offset&#39;] = fourDigitTrainDF[&#39;offset&#39;] * 2000</pre><pre>fourDigitTrainDF[&#39;originalSum&#39;] = fourDigitTrainDF[&#39;sum&#39;] + fourDigitTrainDF[&#39;offset&#39;]</pre><p>Next we just create our prompts. We’ll have one set of prompts for the pure few shots and one set of prompts for the corrected ones. And to see how many examples we need, we’ll just do a grid search and literally try running from 1–30 examples.</p><pre>correctionPrompts = {}<br>purePrompts = {}</pre><pre>for i in range(1,30):<br>  correctionPrompt = &quot;&quot;<br>  purePrompt = &quot;&quot;<br>  for row in fourDigitTrainDF[:i].iterrows():<br>    correctionPrompt += &#39;Initial Answer: {} + {} = {}&#39;.format(row[1][&#39;first&#39;], row[1][&#39;second&#39;], int(row[1][&#39;originalSum&#39;]))<br>    correctionPrompt += &#39;\n&#39;<br>    interjection = &#39;looks correct.&#39; if not row[1][&#39;randomlyWrong&#39;] else &#39;looks off by a bit.&#39;<br>    correctionPrompt += &#39;Double Checking: {} {} {} + {} = {}&#39;.format(int(row[1][&#39;originalSum&#39;]), interjection, row[1][&#39;first&#39;], row[1][&#39;second&#39;], row[1][&#39;sum&#39;])<br>    correctionPrompt += &#39;\n\n&#39;</pre><pre>    purePrompt += &#39;{} + {} = {}&#39;.format(row[1][&#39;first&#39;], row[1][&#39;second&#39;], row[1][&#39;sum&#39;]) + &#39;\n&#39;<br>  <br>  correctionPrompt += &#39;Initial Answer: &#39;<br>  correctionPrompts[i] = correctionPrompt<br>  purePrompts[i] = purePrompt</pre><p>Now we’ve got all of our prompts set up, time to try it against the test set!</p><p><strong>Running the Test</strong></p><p>To run the test, we’ll begin by going and setting up the actual query (a + b =) and then adding it to the end of each of our prompts (either with self correction or not). Then we’ll just run the whole thing and sit back as it does the 6,000 queries. We’ll also dump it to JSON each time we go through the test set in case things break so we can recover.</p><pre>import json<br>fourDigitTestDF[&#39;formatted&#39;] = fourDigitTestDF.apply(lambda x: &quot;{} + {} =&quot;.format(x[&#39;first&#39;], x[&#39;second&#39;]), axis=1)</pre><pre>correctionResults = {}<br>pureResults = {}</pre><pre>#for each size of example length in 1-30, run on the test set<br>for trainSize in range(1,30):<br>  if trainSize not in correctionResults:<br>  print(trainSize)<br>  correctionResults[trainSize] = []<br>  pureResults[trainSize] = []<br>  for example in fourDigitTestDF.formatted:<br>    correctionResults[trainSize].append( query(correctionPrompts[trainSize]+example, gen_len=50))<br>  pureResults[trainSize].append(<br>query(purePrompts[trainSize]+example, gen_len=50))<br>  with open(&#39;correctionResults.json&#39;, &#39;w&#39;) as fh:<br>    json.dump(correctionResults, fh)<br>  with open(&#39;pureResults.json&#39;, &#39;w&#39;) as fh:<br>    json.dump(pureResults, fh)</pre><p><strong>Evaluation</strong></p><p>K, so now that we got our 6,000 files, we’ll evaluate it. I’ll start by renaming the test set to <em>test</em> because… why not. Since I haven’t figured out how to do a stop sequence with GPT-J (My degree’s in journalism, math’s hard) we’ll have to parse out the answer we’re trying to get. For the traditional method, that’s just the first ‘word’ out the door. For the self-correcting method, it’s the last ‘word’ of the first set of outputs.</p><pre>def parsePureInt(x):<br>  base = x.split(&#39;\n&#39;)[0]<br>  try:<br>    return int(base)<br>  except:<br>    return 0</pre><pre>def parseCorrectedInt(x):<br>  base = x.split(&#39;\n\n&#39;)[0].split(&#39; &#39;)[-1]<br>  try:<br>    return int(base)<br>  except:<br>    return 0</pre><p>Now we’ll apply this to all of our results and calculate the errors.</p><pre>for key in pureResults.keys():<br>  test[key] = pureResults[key]<br>  test[key] = test[key].apply(lambda x: parsePureInt(x))</pre><pre>pureMape = pd.DataFrame()<br>  for col in test.columns[3:]:<br>  pureMape[col] = (abs(test[col] - test[&#39;sum&#39;]))/test[&#39;sum&#39;]</pre><pre>pureEM = pd.DataFrame()<br>for col in test.columns[3:]:<br>  pureEM[col] = test[col] == test[&#39;sum&#39;]</pre><pre>for key in correctionResults.keys():<br>  test[key] = correctionResults[key]<br>  test[key] = test[key].apply(lambda x: parseCorrectedInt(x))</pre><pre>correctedMape = pd.DataFrame()<br>for col in test.columns[3:]:<br>  correctedMape[col] = (abs(test[col] - test[&#39;sum&#39;]))/test[&#39;sum&#39;]</pre><pre>correctedeEM = pd.DataFrame()<br>  for col in test.columns[3:]:<br>  correctedeEM[col] = test[col] == test[&#39;sum&#39;]</pre><p>And now we just plot the results</p><pre>fig, axs = plt.subplots(ncols=2, nrows=1, figsize=(12,8), facecolor=&#39;white&#39;)<br>correctedMape.mean().plot(label=&#39;self correcting&#39;, ax=axs[0])<br>pureMape.mean().plot(ax=axs[0], label=&#39;pure few shot&#39;)<br>axs[0].legend()<br>axs[0].set_xlabel(&#39;# of examples&#39;)<br>axs[0].set_title(&#39;Mean Error&#39;)</pre><pre>correctedeEM.sum().plot(label=&#39;self correcting&#39;, ax=axs[1])pureEM.sum().plot(ax=axs[1], label=&#39;pure few shot&#39;)<br>axs[1].legend()<br>axs[1].set_xlabel(&#39;# of examples&#39;)<br>axs[1].set_title(&#39;Exact Match&#39;)<br>fig.suptitle(&#39;4 Digit Arithmetic on 100 Tests&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*eMCQgARRgqeBOVuRm2QWtQ.png" /><figcaption>Figure 2. Wait, no, figure 1 again.</figcaption></figure><p><strong>In Closing</strong></p><p>You can run the above code with GPT3 as well as J and you’ll get the same sort of improvement. I leave it as an exercise to the reader to do it with commas as well (you can just use f”{x:,}” to format things with commas). Pretty cool that it works though.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*cRveRK6FlMZIDESyWfBTgg.png" /><figcaption>Also if you do it in GPT3 with commas you get something more like this depending on the model size / fine tune. Not sure what happened there at 17 examples though.</figcaption></figure><p>I think what’s going on here is that since it can’t predict ahead sometimes it starts making up random digits until it gets to the end, and then that gives it the context on checking for what the first couple digits should be the next time through. I’m not entirely sure though. These models are weird.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=136e846987bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/1-1-3-wait-no-1-1-2-how-to-have-gpt-sanity-check-itself-136e846987bf">1+1=3. Wait, no, 1+1=2. How to have GPT sanity check itself.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling News Coverage with Python. Part 3: Newspaper Coverage and Google Search Trends]]></title>
            <link>https://medium.com/data-science/modeling-news-coverage-with-python-part-3-newspaper-coverage-and-google-search-trends-dca50f60a9c8?source=rss-a3d23264e777------2</link>
            <guid isPermaLink="false">https://medium.com/p/dca50f60a9c8</guid>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[news-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[gdelt]]></category>
            <dc:creator><![CDATA[Matt Brockman]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 04:36:02 GMT</pubDate>
            <atom:updated>2019-10-14T04:46:39.520Z</atom:updated>
            <content:encoded><![CDATA[<h3>Modeling News Coverage with Python. Part 3: News site interactions with Google Search trends</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYwyLjbLN3HfNxramyszZw.png" /><figcaption>Fitting models of Google Search to Search Trends and News Articles</figcaption></figure><p>This post integrates data from a limited sample of newspaper coverage with Google Search trends to model interactions between the two. In these examples, the preliminary analysis finds news coverage useful for forecasting search trends but small and mixed results in the other direction. These examples were selected to be done with limited resources and time in a blog post, but everything’s general purpose enough that you can swap in your own topics or data and it should be enough to get you started.</p><p>This is the third post in a series of blog posts demonstrating modeling news with Python. The <a href="https://towardsdatascience.com/making-sense-of-the-news-part-1-introduction-f171d1da65a8">first post</a> of this series looked at what time series were and interactions within a medium can be modeled. The <a href="https://towardsdatascience.com/modeling-news-coverage-with-python-part-2-starting-news-forecasting-with-limited-data-5c1092de3ea9">second post</a> looked at news coverage of a few publications and themes to forecast coverage with different methods.</p><h4>Loading Articles</h4><p>As before, all the necessary code for the models is provided in this post. It’s formatted to be copied and pasted into a Jupyter Notebook file. The code will acquire the article data, but you will want to follow the instructions and link to download the Google Search trend data or just download it yourself if you’re familiar with it.</p><p>We’ll start by going and pulling 60 days of articles from GDELT using a Python package (<a href="https://www.gdeltproject.org/data.html#rawdatafiles">https://www.gdeltproject.org/data.html#rawdatafiles</a>). These articles are labelled with topics, personalities, and locations. This checks if the files are already downloaded and if not, downloads them, and formats them into a pandas dataframe so that we can manipulate the series.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/381eb7eceeea8542a2fc4ce7c895d8d1/href">https://medium.com/media/381eb7eceeea8542a2fc4ce7c895d8d1/href</a></iframe><p>So now we’ll run it through a SARIMA model with parameters like we eyeballed last time. In the last post, we used the mean square error to evaluate accuracy but this time we’ll use the mean absolute percentage error (MAPE). This lets us normalize error across time series of different values. We’ll keep track of the MAPE values in the mape_df dataframe to be able to compare results.</p><p>The first model we test will use both coverage of other countries within the publication and coverage of that country in other publication to fit against as done previously to make sure everything works again.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6410cc19b957a0a6a04a75e9e07e356a/href">https://medium.com/media/6410cc19b957a0a6a04a75e9e07e356a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTLbqQYgYa5Z7hqKWBEnWg.png" /></figure><p>We can then change the exogenous series and only select articles from within the same publication to fit the model with instead, saving the results to the mape_df dataframe column “WithinNewspapersOnly_predicted”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2343abb0625296dad1db73fa48ab8f41/href">https://medium.com/media/2343abb0625296dad1db73fa48ab8f41/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MRyzrqg4DaSF_Q0oNVssjA.png" /></figure><p>We can compare the difference in the mape_df betweeen the two:</p><pre>mape_df[&quot;news_dif&quot;] = mape_df[&quot;WithinNewspapersOnly_predicted&quot;] - mape_df[&quot;NewspapersOnly_predicted&quot;]</pre><pre>for country in myCountries:<br>  print(&quot;{} mean hat change of {}&quot;.format(country, mape_df[mape_df.index.map(lambda x: x.find(country) &gt; -1)][&quot;news_dif&quot;].mean()))</pre><pre>&gt;&gt;&gt; dprk mean hat change of -10.75<br>&gt;&gt;&gt; ukraine mean hat change of -1.5<br>&gt;&gt;&gt; russia mean hat change of 6.5<br>&gt;&gt;&gt; iran mean hat change of -3.0<br>&gt;&gt;&gt; china mean hat change of -13.75</pre><p>It looks as though using only the same publication outperforms using coverage from other publications most of the time. That being said, that’s for this dataset and a small period of coverage. Alright, so what happens with Google Search trends? In the next step we fit a model to the search trends, and then after that fit the news model to the news data and the search trends.</p><h4>Google Search Trends</h4><p>We first need to get the Google search data. Google publishes their search trends at <a href="https://trends.google.com/">https://trends.google.com</a>. There’s lots of fun features there if you aren’t familiar with it. To speed things up, I went and searched for the trends on the relevant countries (North Korea, China, Russia, Iran, and Syria) from 8/8/2019 to 10/6/2019 at <a href="https://trends.google.com/trends/explore?date=2019-08-08%202019-10-06&amp;geo=US&amp;q=%2Fm%2F05b7q,%2Fm%2F06bnz,%2Fm%2F0d05w3,%2Fm%2F07t21,%2Fm%2F03shp">this link</a>.</p><p>The page should look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsXmpv1kH91GI18fYtyCpA.png" /><figcaption>Google Trend Search Results</figcaption></figure><p>Google gives search trends as percents relative to the highest searched term in the search queries as opposed to raw search numbers. Here, Ukraine received the highest amount of search result so all the other data points follow as a percent of Ukraine’s maximum search. We can download this in CSV format by clicking the down arrow to the right of the “Interest over time” label.</p><p>We can then put the CSV into our current working directory (or some other location if you want to change the code) and load it in with pandas. Note: The first row of Google’s CSV is some sort of metadata label, so we skiprows it to get it the columns formatted properly when loading.</p><pre>gdata = pd.read_csv(&quot;multiTimeline.csv&quot;, skiprows=2)<br>gdata.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*VhQBK4Vh4x6GLk__HZ9T4w.png" /></figure><p>Visual inspection shows it looks the same as the original data on the website, so it seems to have loaded properly. Now, we can format it to be the same as our news media data frame before and put them together, creating a copy of the original time_series df as original just in case we mess something up and need to go back.</p><pre>gdata.columns = [x.replace(&quot;: (United States)&quot;, &quot;&quot;).replace(&quot;North Korea&quot;, &quot;dprk&quot;).lower() for x in gdata.columns]</pre><pre>gdata.day = pd.to_datetime(gdata.day)</pre><pre>gdata.set_index(&quot;day&quot;, inplace=True, drop=True)</pre><pre>original = time_series.copy()</pre><pre>time_series = time_series.join(gdata)</pre><p>And so now we can just go ahead and plot both the newspaper coverage and the Google Search trends to visually check how much they line up.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7c41cfc774a5e479027f806c9080a81/href">https://medium.com/media/d7c41cfc774a5e479027f806c9080a81/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lp5i9-8lhdgp1ATJ_vVnow.png" /></figure><p>We can make a few observations to begin with. There’s a clear weekly seasonality with some of the search trends (e.g. China), but with the others (e.g. Ukraine, Iran), the seasonality isn’t as important as those big spikes that show up with increased news coverage. This variance is going to be a bit problematic because the search seems to ignore news coverage trends for DPRK, react to the second spike with Ukraine, and react to the third spike with Iran.</p><p>We can check quickly if there’s an effect from the news on search. To just establish baseline error rates, we can go ahead and run the same SARIMA models on the search results as the news sites, using only previous Google Search trends and news media coverage of the same topic for the models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/706a1fb2823b2df6e9641bdeea80d8d1/href">https://medium.com/media/706a1fb2823b2df6e9641bdeea80d8d1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYwyLjbLN3HfNxramyszZw.png" /></figure><p>North Korea and Ukraine have the most error at this point. As we’d seen earlier, the Google search had diverged from the news results. Sure enough, the model learned from the first Ukraine spike and predicted that the search trends would spike again when they didn’t. Also, the search trends kind of ignore the news content as well for North Korea. We don’t really have anything to predict that sort of behavior in this data, so not going to bother doing decomposition but longer periods of data might let us know how normal that is.</p><p>Since total article count per day will generally be around the same for each publication, covering one country might lead to a decrease in coverage of another country (except where both are covered in the same article), so we can see if just including everything improves the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35b0fd463ec4ab2c5ec72413b235091c/href">https://medium.com/media/35b0fd463ec4ab2c5ec72413b235091c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqgkKIeHMl-0ZWGMG6w3ig.png" /></figure><p>It looks like throwing in all the other time series improved the predictions on North Korea and Ukraine while having no effect on Russia and negative effects on Iran and China. A longer amount of time would definitely help massage out what kind of data we really want but I don’t want to get into building a database in these blog posts and it gets the point across enough.</p><p>It looks like news trends help us predict search trends. Does the reverse hold?</p><h4>Adding Google Search back into the News Model</h4><p>So far we’ve looked at how news predicts Google search trends. But what about the other way around?</p><p>Since we already added the Google Search trends into the time_series dataframe, we can just re-run the original code, modifying it slightly to only include newspapers and then the dataframe column save the MAPE values into .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4ebb9b11c02e56cd833fbd794f6dd47/href">https://medium.com/media/e4ebb9b11c02e56cd833fbd794f6dd47/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uVWhso4BJvXT7dc4RU2yYw.png" /></figure><p>We can check how the new model differs in accuracy from the original,</p><pre>mape_df[&quot;overall_change&quot;] =mape_df[&quot;WithSearch_predicted&quot;] - mape_df[&quot;NewspapersOnly_predicted&quot;]</pre><pre>for country in myCountries:<br>  print(&quot;{} mean hat change of {}&quot;.format(country, mape_df[mape_df.index.map(lambda x: x.find(country) &gt; -1)][&quot;overall_change&quot;].mean()))</pre><pre>&gt;&gt;&gt; dprk mean hat change of 0.25<br>&gt;&gt;&gt; ukraine mean hat change of -2.5<br>&gt;&gt;&gt; russia mean hat change of 7.0<br>&gt;&gt;&gt; iran mean hat change of 5.0<br>&gt;&gt;&gt; china mean hat change of -2.25</pre><p>With these parameters and this data, adding in the Google Search trends has mixed but small impacts on the model. That’s fine though, with 60 days of data we’re not expecting great results anyway.</p><h4>In Closing</h4><p>We saw that sometimes the news can help predict Google Search Trends but not always. SARIMA models did worse on the topics where search trends diverged from news coverage trends, which should be expected. Without previous examples for the Ukraine spike, we wouldn’t expect the model to really guess that the coverage would diverge that much.</p><p>We did not get great results from adding the search results back into the news models. That’s fine. The parameters on the models weren’t tuned to the specifics of any of the search terms and the period of time covered is pretty short. It would be interesting to see how both the news -&gt; search and search -&gt; news relationships hold up over longer periods of time. Furthermore, these are pretty broad subjects. In the next post, we’ll look at the trends in the sub-themes publications cover within each of these issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dca50f60a9c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-news-coverage-with-python-part-3-newspaper-coverage-and-google-search-trends-dca50f60a9c8">Modeling News Coverage with Python. Part 3: Newspaper Coverage and Google Search Trends</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling News Coverage with Python. Part 2: Starting News Forecasting with Limited Data]]></title>
            <link>https://medium.com/data-science/modeling-news-coverage-with-python-part-2-starting-news-forecasting-with-limited-data-5c1092de3ea9?source=rss-a3d23264e777------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c1092de3ea9</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[media-analysis]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Matt Brockman]]></dc:creator>
            <pubDate>Thu, 10 Oct 2019 14:04:22 GMT</pubDate>
            <atom:updated>2019-10-11T20:59:53.453Z</atom:updated>
            <content:encoded><![CDATA[<p>Earlier <a href="https://towardsdatascience.com/making-sense-of-the-news-part-1-introduction-f171d1da65a8">I wrote a bit</a> about getting news stories to start modeling and now I’ll play around a bit with how to forecast coverage and measure if a model’s any good.</p><p>I’ll start this post using an off the shelf neural network from fast.ai to generate some models of publisher daily production using just some articles from a set of publishers, evaluate the fit of the model, and then show why these models aren’t that great for the limitations of the dataset we’re dealing with (entirely arbitrary limitations to make it easy to run the code, messing with granularity and adding a longer period of articles will get you better results). Then this’ll play a bit with auto-regression and moving averages and see how that outperforms the neural network for the limited data that we’re utilizing on this, resulting in generating trends for coverage for each publication like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQvwv-wbuX-MlYnUK4DBtA.png" /><figcaption>SARIMA models for select publications for select coverage with error on test predictions</figcaption></figure><p>Let’s get started!</p><h4>Checking if we still have the data</h4><p>Last time we pulled 60 days of news coverage from GDELT (<a href="https://www.gdeltproject.org/">https://www.gdeltproject.org</a>). We’ll start making by sure we still have it, and if we don’t, we’ll download it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f5029de16c18841bc3dc7924b84307f/href">https://medium.com/media/7f5029de16c18841bc3dc7924b84307f/href</a></iframe><p>Now that we’ve verified we have all the files we need, we can go through them and pull the articles for the New York Times, Washington Post, Fox News, and CNN like before.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8945ba48d6afa9ca40516d6af5b06078/href">https://medium.com/media/8945ba48d6afa9ca40516d6af5b06078/href</a></iframe><p>After running the code, we have the news articles in our df. Next we can go ahead and break down the coverage of each of the target countries per day per publisher.</p><p>First we’ll make sure the date’s a good datetime and set our index to the date.</p><pre>df.DATE = df.DATE.apply(lambda x: str(x))<br>df.DATE = pd.to_datetime(df.DATE)<br>df.fillna(&quot;&quot;, inplace=True)<br>df.set_index(&quot;DATE&quot;, drop=True, inplace=True)</pre><p>Next we’ll go and one-hot encode the df for whether the article’s been labelled as relevant to one of our countries and group by the publisher and date.</p><pre>df[&quot;dprk&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;North Korea&quot;) &gt; -1)<br>df[&quot;ukraine&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;Ukraine&quot;) &gt; -1)<br>df[&quot;russia&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;Russia&quot;) &gt; -1)<br>df[&quot;iran&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;Iran&quot;) &gt; -1)<br>df[&quot;china&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;China&quot;) &gt; -1)</pre><pre>loc_df = df.groupby([&quot;SOURCES&quot;, &quot;DATE&quot;])[[&quot;dprk&quot;, &quot;ukraine&quot;, &quot;russia&quot;, &quot;iran&quot;, &quot;china&quot;]].sum()</pre><p>So now we can create a big df we’ll call <em>time_series</em> to hold the date values for each publisher and country with the columns labelled as such.</p><pre>time_series = pd.DataFrame()<br>for publisher in mySources:<br>  time_series = pd.concat([time_series, loc_df.ix[publisher].add_prefix(“{}_”.format(publisher))], axis=1)</pre><pre>time_series.plot(figsize=(20,10))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-iwog5UpB1s1eFXIpAcDg.png" /></figure><p>Cool, so you should have something that looks like that now, where we can see there’s a weekly seasonality to the data. Also, the Ukraine stuff takes off around the 23rd of September.</p><p>So let’s run these time series through a neural network and see what it comes up with!</p><h4>Using Tabular Model from Fast.ai</h4><p>So now we’re going to use a tabular model (<a href="https://docs.fast.ai/tabular.html">https://docs.fast.ai/tabular.html</a>) from fastai. I’m using the tabular model because it’s quick and easy even though the time series is way too short to expect a great outcome and I’m not going to fiddle with getting the results nice, but it gets the point across.</p><p>Noticing the seasonality, we’re going to want the model to be looking at the last couple days of data. Since it’s weekly seasonal, we can just give it the last 7 days and hope that helps it figure out that there’s a seasonal component to the datas. Also, we can give it the day of the week that the article’s produced on and maybe that’ll help it out. For example, for Fox News on North Korea on Day X, we’re going to want all publication data from Day X-1, X-2… and so forth. So we can create a function to do that for us.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76d29486f18c578ab674362d337c6ad7/href">https://medium.com/media/76d29486f18c578ab674362d337c6ad7/href</a></iframe><p>We can then create a method that lets us take one of the series from our <em>time_series</em> dataframe and train a neural network to try to model it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9da8ff3db2fe5fb15e1378d179328229/href">https://medium.com/media/9da8ff3db2fe5fb15e1378d179328229/href</a></iframe><p>So for demonstration purposes, we can pick one of our time series and see what the model it generates looks like. We’ll just run through once at a pretty large learning rate just to see what it looks like. If you’re not familiar with fastai, go take the course. Otherwise, you basically want to see how well you can play golf with the fit_one_cycle numbers, although this is just an arbitrary example.</p><pre>learn = genNN(time_series, “foxnews.com_russia”)<br>learn.fit_one_cycle(1, 3e-02)</pre><p>So to visualize the results, we’ll want to pull the predictions out of our learner and graph them against the real results. We can color the training, validation, and testing sets differently to get a sense of how different parts of the model perform.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17423714c2b5a6cc3e3d1d5c61d10877/href">https://medium.com/media/17423714c2b5a6cc3e3d1d5c61d10877/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*4SQtX1qSUJFoPooh9iJzJw.png" /></figure><p>So is our model good or bad? The October accuracy is likely a fluke. If you noticed, we also went and pulled the mean of the training set so we can compare if we’re even doing better than chance so far. We’ll use root mean square error as the evaluation of how well the model’s doing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85458eda1f1caa10abfbc81fd8fd579b/href">https://medium.com/media/85458eda1f1caa10abfbc81fd8fd579b/href</a></iframe><pre>&gt;&gt;&gt; RMSE training  using mean: 7.255011530500299<br>&gt;&gt;&gt; RMSE training using model: 7.2815201128101155<br>&gt;&gt;&gt; RMSE testing using mean: 8.930425521776664<br>&gt;&gt;&gt; RMSE testing using model: 6.544865479426577</pre><p>So this model actually doesn’t do too horribly; it does a really bad job of predicting what’ll happen with the training sets, but somehow it figures out the early October spike in reporting. (I don’t know why). But let’s re-run the model generation and see if it happens again. We should get different results because the model starts off with everything randomized and we’re just following the gradient.</p><pre>learn = genNN(time_series, “foxnews.com_russia”)<br>learn.fit_one_cycle(1, 3e-02)</pre><p>And re-run the code to display the graph results in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*GnHIWYZJ82ESj8bpg4RTtw.png" /></figure><p>Re-running the evaluation, I get that it’s getting out performed by just taking the mean on the training set and doing a bit worse than the previous one.</p><pre>&gt;&gt;&gt; RMSE training  using mean: 7.255011530500299<br>&gt;&gt;&gt; RMSE training using model: 7.314209362904273<br>&gt;&gt;&gt; RMSE testing using mean: 8.930425521776664<br>&gt;&gt;&gt; RMSE testing using model: 7.068090866517802</pre><p>We can mess with the learning rates to make it nicer,</p><pre>learn = genNN(time_series, &quot;foxnews.com_russia&quot;)<br>learn.fit_one_cycle(1, 2e-02)<br>learn.fit_one_cycle(5, max_lr =3e-04)<br>...<br>&gt;&gt;&gt; RMSE training  using mean: 7.255011530500299<br>&gt;&gt;&gt; RMSE training using model: 9.782066217485792<br>&gt;&gt;&gt; RMSE testing using mean: 8.930425521776664<br>&gt;&gt;&gt; RMSE testing using model: 5.876779847258329</pre><p>But generally speaking, it’s not really all that good of a model. But then again, it shouldn’t be because it’s only using a couple publishers and a few general countries on 60 days of data.</p><p>That was fun. We trained a neural network, but it wasn’t very smart; it didn’t seem to figure out the weekly seasonality even though we hand fed it the days. Let’s move on to what happens if we look at auto-regression with moving averages using more traditional models. (Which I guess we should have started with).</p><h4>Seasonality, Auto-Regression, and Moving Average</h4><p>It’s pretty easy with python to decompose a time series. Assuming you still have the <em>time_series</em> dataframe loaded, we can decompose each column into its component observed, trend, seasonal, and residual pieces using statsmodels (<a href="https://www.statsmodels.org/stable/index.html">https://www.statsmodels.org/stable/index.html</a>) and plots them.</p><pre>import statsmodels.api as sm<br>res = sm.tsa.seasonal_decompose(time_series[&quot;foxnews.com_russia&quot;])<br>res.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*bihKmOiEvupFcszLA15NVA.png" /><figcaption>Foxnews Coverage including Russia Decomposed</figcaption></figure><p>Horray it decomposed!</p><p>We can also look at the autocorrelation, or how much values relate to previous values.</p><pre>from statsmodels.graphics.tsaplots import plot_acf<br>plot_acf(time_series[&quot;foxnews.com_russia&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*kbPyyaODapMq3ysgkzFP_A.png" /></figure><p>We can see the seasonal component in there as well as the trend.</p><p>And we can look at the moving average. We’ll take a 2 day simple average of the previous two days.</p><pre>f_ru = time_series[[&quot;foxnews.com_russia&quot;]].copy()<br>f_ru.columns = [&quot;actual&quot;]<br>f_ru[&quot;sma&quot;] = 0</pre><pre>for i in range(0,len(f_ru)-2):<br>  f_ru.iloc[i+2,1] = np.round(((f_ru.iloc[i,0]+<br>  f_ru.iloc[i+1,0])/2),1)</pre><pre>f_ru.plot(title=&quot;Fox Russia prior 2 days SMA&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*CWVGmCuyD7AYPn-r0DTCHg.png" /></figure><p>And we can check the RSME</p><pre>f_ru = f_ru[2:]<br>print(&quot;RMSE testing using SMA2: {}&quot;.format(sqrt(mean_squared_error(f_ru.actual, f_ru.sma))))</pre><pre>&gt;&gt;&gt; RMSE testing using SMA2: 7.666822836990112</pre><p>So we have a RSME that’s slightly worse than what we got above but for way less effort (aside from me forgetting I had to use .copy() to make Pandas not complain). Not too shabby, but it’s actually worse than just taking the mean. Probably. I didn’t actually check for the whole duration before as opposed to this time. So let’s integrate the seasonality, auto-regression, and moving average using the Python Statstools SARIMA package.</p><p>So we’re going to eyeball the parameters on this one. Basically, we’re going to need the autoregressive moving average component and the time series component as the order and seasonal orders for the model. We’ll pretend it’s a weekly seasonality as well which is generally true.</p><p>So here goes. First we declare and fit a model.</p><pre>from statsmodels.tsa.statespace.sarimax import SARIMAX<br>issue = “foxnews.com_russia”<br>train_l = 55</pre><pre>s_model = SARIMAX(endog = time_series[[issue]][:train_l][1:], <br> exog = time_series[[x for x in time_series.columns if x != issue]][:train_l].shift().add_suffix(“_l1”)[1:], order=(2,0,1), seasonal_order=(1,0,1,7)).fit()</pre><p>Now we want to go and get our predictions. We’re OK using the real values for previous counts because we’d have that in the real world unless we’re predicting out really far (but we didn’t do that earlier).</p><pre>f_ru = time_series[[issue]].copy()[1:]<br>f_ru.columns = [&quot;actual&quot;]</pre><pre>f_ru[&quot;predicted&quot;] = s_model.predict(end=datetime.datetime(2019, 10, 6), endog = time_series[[issue]][-5:],exog = time_series[[x for x in time_series.columns if x != issue]].shift()[-5:], dynamic= False)</pre><pre>#plotting<br>ax = f_ru[&quot;actual&quot;].plot(title=&quot;Russia Mentions in Fox&quot;, figsize=(12,8))<br>f_ru[&quot;predicted&quot;][:-5].plot(color=&quot;orangered&quot;)<br>f_ru[&quot;predicted&quot;][-6:].plot(color=&quot;red&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*mJwu5w2uLl-cq0qFBn5DbQ.png" /></figure><p>So we have a SARIMA model that looks pretty good! And how are the RMSEs?</p><pre>testing = f_ru[:-5].copy()<br>print(&quot;RMSE training using model: {}&quot;.format(sqrt(mean_squared_error(testing.actual, testing.predicted))))</pre><pre>testing = f_ru[-5:].copy()<br>print(&quot;RMSE testing using model: {}&quot;.format(sqrt(mean_squared_error(testing.actual, testing.predicted))))</pre><pre>&gt;&gt;&gt; RMSE training using model: 3.098788909893913<br>&gt;&gt;&gt; RMSE testing using model: 6.281417067978587</pre><p>So the SARIMAX here does way better than the neural network above. We can then re-run the code in a loop for all of the combinations we’re looking at if we want to as such (and you can add any combination of publishers and topics you like). Just a heads up, the code will spit out a lot of warnings.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cab96f0cbe22b510c4ae1d6fa6476964/href">https://medium.com/media/cab96f0cbe22b510c4ae1d6fa6476964/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQvwv-wbuX-MlYnUK4DBtA.png" /><figcaption>It doesn’t do too well on the Ukraine forecasting but that should be expected because it didn’t fit the previous trends</figcaption></figure><p>Try playing with the parameters of the models and changing around the publications and coverage topics. Or better yet, go back and download a longer period of articles and see how more data improves training and testing.</p><h4>Moving On</h4><p>Between <a href="https://towardsdatascience.com/making-sense-of-the-news-part-1-introduction-f171d1da65a8">Part 1: Introduction</a> and this, you should have a pretty good idea of how much we can understand about the interactions of newspapers on one another. By modeling the news coverage over time, we can evaluate if a model incorporates factors influencing news coverage better than other models.</p><p>What does this matter? Well, when people run around saying that the news is doing this or that, if we don’t have models to evaluate their claims it’s really hard to tell who’s telling the truth or not. But in just (hopefully) a few minutes we were able to grab data, run computational models, and forecast future coverage based on media coverage alone. This all plays into the political science/communications/journalism studies on agenda setting (e.g. <a href="https://www.tandfonline.com/doi/full/10.1080/15205436.2014.964871">https://www.tandfonline.com/doi/full/10.1080/15205436.2014.964871</a>) but actually predicting news coverage isn’t as widespread as I’d thought and I’m not sure why that is.</p><p>Now, there are undoubtably a lot of issues with the math here and I leave that to the reader to figure out improving it on their own (or writing up their results :)) There are also issues with the data; we used 4 of the 100,000 news sources available to us, 5 locations from the very large domain of possible combinations of themes, locations, and people, and only 60 days of the years of coverage. We also eyeballed a LOT of the parameters here; ultimately you don’t want to be looking at just one topic but constantly monitoring them all anyway. However, hopefully this was straightforward enough that anyone can pick it up and mess with whatever they’re interested in.</p><p>I’m not entirely sure what I’m going to cover in Part 3. Possibly including measures of public interest in issues operationalizing Google Search or various Twitter datas might be fun, although the hard part for a blog post is making sure the data’s easily accessible. Might also do longer trends. I just started writing these to procrastinate on studying for midterms so haven’t thought this through all the way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c1092de3ea9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-news-coverage-with-python-part-2-starting-news-forecasting-with-limited-data-5c1092de3ea9">Modeling News Coverage with Python. Part 2: Starting News Forecasting with Limited Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling News Coverage with Python. Part 1: Introduction]]></title>
            <link>https://medium.com/data-science/making-sense-of-the-news-part-1-introduction-f171d1da65a8?source=rss-a3d23264e777------2</link>
            <guid isPermaLink="false">https://medium.com/p/f171d1da65a8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[media-analysis]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Matt Brockman]]></dc:creator>
            <pubDate>Wed, 09 Oct 2019 04:26:42 GMT</pubDate>
            <atom:updated>2019-10-16T14:54:26.016Z</atom:updated>
            <content:encoded><![CDATA[<p>There are a lot of news outlets out there with a lot of diverse views. That’s good in one sense, but in another sense it’s rather impossible for anyone to read everything every day much less anyone that gets busy for a few days. However, the news is just a bunch of events happening over time, and so that suggests there’s a way to model it so we can understand some of what’s going on.</p><p>Forecasting news trends can be thought of as similar to what people already try to do with the stock market, where we know there are a bunch of factors that influence stock price, and if you get the factors right you can forecast out what the future price of a stock will be based on what’s going on today. Likewise, if we truly understand what’s going on in the news and the factors that influence news production, then we should be able to use that understanding to forecast future news production. While not necessarily of economic benefit, understand how coverage of events change over time may help to inform our understanding of why people perceive events differently and how things have developed to where they are.</p><p>This is the first blog post in a series of posts exploring news coverage modeling. Each post contains the code to run the analysis independently from the others. It uses 60 days of data for the purposes of blogging (just enough for the SARIMA to hopefully fit later) and using longer periods of time will perform better.</p><p>Posts written so far:</p><ul><li>Part 1: Introduction (This) randomly generates time series data, moves to look at some real Twitter data, and then looks at a few time series of news coverage and runs them through vector auto-regression just to show there are patterns to find.</li><li><a href="https://towardsdatascience.com/modeling-news-coverage-with-python-part-2-starting-news-forecasting-with-limited-data-5c1092de3ea9">Part 2: Starting News Forecasting With Limited Data</a> provides the basics to take an off the shelf neural network and forecast coverage using a few time series of news coverage, then does the same thing with a SARIMA model.</li><li><a href="https://towardsdatascience.com/modeling-news-coverage-with-python-part-3-newspaper-coverage-and-google-search-trends-dca50f60a9c8">Part 3: News Site Interactions with Google Search Trends</a> continues using SARIMA models for the interactions between news site coverage and Google Search trends. It uses the news data with search data to both predict search rates and forecast news coverage.</li></ul><h3>Quick Refresher on Playing with Time Series (Random Data)</h3><p>I’m doing this in python and the code is all provided. You should be able to paste the code into a python enabled notebook (e.g. <a href="https://colab.research.google/">https://colab.research.google</a>) and run it and it should work. I’ll go back and fix things as I have time but figured I’d go and throw this blog post out there in case anyone’s looking for a place to start on this sort of analysis.</p><p>So let’s get started! Using index to stand in for time, we can create a random exogenous (inputs from outside model) time series in Python using numpy’s random function:</p><pre>import pandas as pd<br>import numpy as np<br>exogenous = np.random.uniform(size=100)<br>df = pd.DataFrame(exogenous)<br>df.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*0Tv56U_94M7YgMQWw1siqg.png" /></figure><p>We can then create an endogenous time series that’s say… 1.5x our exogenous series.</p><pre>endogenous = 1.5 * exogenous<br>df =  pd.DataFrame([endogenous, exogenous]).T<br>df.columns = [&quot;endogenous&quot;, &quot;exogenous&quot;]<br>df.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Eb2gSMuNo35vljbb2jRj5g.png" /></figure><p>We can now run an OLS and see that the endogenous is 1.5x the exogenous</p><pre>import statsmodels.api as sm</pre><pre>model = sm.OLS(df.endogenous,df.exogenous)<br>results = model.fit()<br>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9xfjGa-vqeVCrgeKJ6xLPw.png" /><figcaption>OLS Regression Summary on Endo/Exo Series</figcaption></figure><p>Hooray! We get our 1.5 coefficient on the endogenous leading to the exogenous. However, what we really want to do is be able to predict ahead of time what’s going to happen. Since the values are random, we won’t get any lags, but we can change that by going in and just pushing shifting all the endogenous values down a unit and removing the first time series..</p><pre>df.endogenous = df.endogenous.shift()<br>df = df[1:]<br>model = sm.OLS(df.endogenous,df.exogenous)<br>results = model.fit()<br>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*AmH1jwPXbpxPSeT7DBaNBg.png" /><figcaption>OLS on shifted series</figcaption></figure><p>Well, we’ve got a coefficient but the R-squared and other measures are bad. Which is good because it’s now a random value for each. However, we want to be looking at previous values so let’s try a vector autoregression model, throwing in a bunch of lag values for fun.</p><pre>from statsmodels.tsa.api import VAR</pre><pre>model = VAR(df)<br>results = model.fit(maxlags=3)<br>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9YSB5nDz1uQ8-X6WlgESjw.png" /><figcaption>VAR results</figcaption></figure><p>Hooray! The model estimates that endogenous is just 1.5 exogenous from 1 lag like it should and exogenous is .43 +- some noise, although those are statistically significant (It should be .5, it picked up some randomness from the lags as well).</p><p>Oh, just for fun, we can tell pretty easily that we can use exogenous to predict endogenous but not endogenous to predict future exogenous. To test how much this holds, we can use Granger Causality, which basically runs an OLS individually on each series and then sees if adding in the second time series helps us get better numbers. So running the test checks if the second time series tends to help predict the first. (i.e. h_0 is going to be that the second variable does not cause the first, so if we can disprove h_0 that means the second does not cause the first, although disproving h_0 just shows there’s a correlation not actual causation).</p><pre>from statsmodels.tsa.stattools import grangercausalitytests</pre><pre>gc = grangercausalitytests(df[[&quot;endogenous&quot;, &quot;exogenous&quot;]], 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3IqltB9glBKdV1Z3gQp9w.png" /><figcaption>Granger causality of exogenous on endogenous</figcaption></figure><p>Running the granger causality from exogenous to endogenous at lag 1 gives us a better test result than lags 1 and 2 (because the lag 2 is just noise). Now we can check the other direction.</p><pre>gc = grangercausalitytests(df[[“exogenous”, “endogenous”]], 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFfxrtQw6vfR08iWu_lwGQ.png" /><figcaption>Granger causality of endogenous on exogenous</figcaption></figure><p>We see from the results that there’s no statistically significant relationship between in the endogenous -&gt; exogenous direction so cannot reject h_0.</p><p>So cool, we remember what time series are and can mess around with them a bit. There’s other stuff like normalization trends and seasonality and messing with auto correlations that matter but we can worry about that later, now let’s go look at news. Ultimately, whatever predicts better outside the training data’s going to be what we care about and adding in different functions just improves the accuracy that we’re using to figure out if a model’s good.</p><h3>Newsy Time Series the the Wild (Twitter)</h3><p>Before getting into the news (which is a pain to get unless you happen to have an archive of news articles handy for which I’ll cover a workaround in a bit), let’s take a look at some random newsworthy tweets. Thankfully, there’s a good archive of newsworthy, Trump tweets available to look at.</p><p>So we can go to <a href="http://www.trumptwitterarchive.com/archive">http://www.trumptwitterarchive.com/archive</a> and go to “download” -&gt; “json” to grab an archive of all of Trump’s tweets. We’ll go ahead and copy/paste the results that pop up into an actual .json file in our working directory; I called it “trump_tweets.json”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vovF70UqJVcTWYZkdaLPGA.png" /><figcaption>Going here, downloading Trump’s tweets</figcaption></figure><p>So we can go ahead and load that into pandas.</p><pre>df = pd.read_json(“trump_tweets.json”)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_H8apGUjjac7a-KHEPEwXQ.png" /><figcaption>Head of the json file</figcaption></figure><p>So we see we’ve got a bunch of data for the tweets. We’ve got a bunch of timestamps for created_at, how many likes each tweet has, the id string (which links to the original tweet by appending to the end of <a href="https://twitter.com/realDonaldTrump/status">https://twitter.com/realDonaldTrump/status</a>/). And most important, we’ve got the text.</p><p>So let’s ignore everything except text and created_at. Now, there’s lots of ways to mess with the text, but I don’t know how to do natural language programming very well at this point in the blog post. So what’s something interesting to see what he tweeted about?</p><p>Well, something that took place over a long time was the whole thing with Russian collusion and the Mueller report. So lets see how often he tweeted about Russia and Collusion from the beginning of 2017 through the Barr memo in March 2019 just using some basic string matching. We’ll start by taking his text and making it lower case</p><pre>#convert created_at to datetime</pre><pre>import datetime<br>df.created_at = pd.to_datetime(df.created_at)</pre><pre>df = df[(df[&quot;created_at&quot;] &lt; datetime.datetime(2019,3,1)) &amp; (df[&quot;created_at&quot;] &gt; datetime.datetime(2017,1,1))]</pre><pre># create a column for lower case<br>df[“text_lower”] = df.text.apply(lambda x:x.lower())</pre><pre>len(df)<br>&gt;&gt;&gt; 41420</pre><p>Cool, so we’ve got 41,000 tweets. Now lets do some labeling</p><pre>df[“russia”] = df.text_lower.apply(lambda x: x.find(“russia”) &gt; -1)</pre><pre>df[“collusion”] = df.text_lower.apply(lambda x: x.find(“collu”) &gt; -1)</pre><p>And we can select only those tweets that mention either of those terms</p><pre>#create df where the previous rows had either russia or collusion label</pre><pre>rc_df = df[df[&quot;russia&quot;] | df[&quot;collusion&quot;]].set_index(&quot;created_at&quot;)</pre><p>We can resample this to get the count per day and then go ahead and plot that</p><pre>rc_df_d = rc_df[[&quot;russia&quot;, &quot;collusion&quot;]].resample(&quot;d&quot;).sum()</pre><pre>rc_df_d.plot(figsize=(12,8)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*-AtW_lwLzBuUR73qxvjfHw.png" /></figure><p>We can run our VAR models</p><p>model = VAR(rc_df_d)<br>results = model.fit(maxlags=5)<br>results.summary()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGJRi4LWWcmxn8TIiAxGlg.png" /></figure><p>So we see the big coefficients we’re calculating for our models so far are whether he’s tweeted about that subject before, at least at the daily level.</p><p>But we don’t need to deal with only the daily granularity, we can look at the weekly granularity too.</p><pre>rc_df_w = rc_df[[“russia”, “collusion”]].resample(“w”).sum()<br>rc_df_w.plot(figsize=(12,8))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*dkiXuYqEuaRrDCQSCrzvkw.png" /></figure><p>Running the VAR, we get almost no relation, except collusion tweets potentially are linked to Russia tweets 2 weeks prior.</p><pre>model = VAR(rc_df_w)</pre><pre>results = model.fit(maxlags=3)</pre><pre>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*ChZzftrdtyaxDRzMbfFzeQ.png" /></figure><p>So at the weekly level, we’re starting to see that there’s a slight coefficient (.18) from tweets on Russia leading to tweets on Collusion at a 2 week lag. So we can add in more variables as well. Let’s take something random like, oh, china and throw it in and re-run these maths.</p><pre>df[“china”] = df.text_lower.apply(lambda x: (x.find(“china”) &gt; -1) or (x.find(“chinese”) &gt; -1))</pre><pre>rcc_df = df[df[“russia”] | df[“collusion”] | df[“china”]].set_index(“created_at”)</pre><pre>rcc_df_d = rcc_df[[&quot;russia&quot;, &quot;collusion&quot;, &quot;china&quot;]].resample(&quot;d&quot;).sum()</pre><pre>rcc_df_w.plot(figsize=(12,8))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*2tb8vOQTopHYTAFH5-FqTA.png" /><figcaption>Oh, wait, I think there’s a trend there ish. and a seasonality.I should have covered stationarity and decomp.</figcaption></figure><p>So we can check our VAR</p><pre>model = VAR(rcc_df_d)</pre><pre>results = model.fit(maxlags=5)</pre><pre>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*HhBH5hthg06L_rat19RoTA.png" /></figure><p>So not seeing much at the daily level for the first two, but seeing that there’s a coefficient of .12 on a 4 day lag for collusion.</p><p>And we can run our weekly VAR there and see what comes out.</p><pre>model = VAR(rcc_df_w)</pre><pre>results = model.fit(maxlags=3)</pre><pre>results.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqssO0XniqvUceZMXWul3g.png" /></figure><p>So for this model so far there’s a 2 week lag on on the China coefficients for Russia tweets, a 2 week lag on Russia for collusion, and a 1 week lag on Collusion for China. So there’s some more stuff to dig into here (and this covered the whole time period) as far as checking when these relations hold and how often, but fo now we’ll move on to actually looking at the news instead. Because this post is about the news, not about tweets. But now you see how to take some text from tweets, convert it into time series, and run whatever model you want to run on it.</p><h3>So finally getting around to actual News</h3><h4>Pre-Labelled Sets: GDELT</h4><p>GDELT [<a href="https://www.gdeltproject.org/">https://www.gdeltproject.org</a>] is a pretty cool project that checks ~100,000 news sites each day, labels all the articles, and makes them available. There’s a version 1 which pulls daily and version 2 that pulls every 15 minute with some better labels. The labels include themes (which they made up), people, locations, and some other stuff I don’t remember off the top of my head.</p><p>We’re gonna jump into this by downloading the last 60 days of data. If you know what you’re doing, you can go ahead and pull whatever you want from their BigQuery db which goes back to like… 2014 (although earlier articles are bad).</p><p>So, this is going to be a bit RAM intensive (15gb or so) because I’m going to keep everything in memory without doing anything sneaky. So if you’re using Google Colab (<a href="https://colab.research.google.com/">https://colab.research.google.com</a>), you’ll want to switch to a GPU and run out your RAM real quick to use the free 25GB RAM upgrade. If not, uh, you can really label all of these as you pull them and actually just select relevant publishers too but this is just supposed to get you started where you have everything you want available.</p><p>So let’s get started.</p><p>First, lets make sure we got the python package for gdelt installed</p><pre>!pip install gdelt<br>import gdelt<br>gd = gdelt.gdelt(version=1)</pre><p>Now, we’ll go ahead and pull the last 60 days of the CSV and save them as python pkl because why not</p><pre>#re-importing stuff from earlier in case skipped<br>from statsmodels.tsa.api import VAR<br>import pandas as pd</pre><pre>import os</pre><pre>os.makedirs(“data”,exist_ok=True)</pre><pre>import datetime<br>cur_date = datetime.datetime(2019,10,7)-datetime.timedelta(days=60)<br>end_date = datetime.datetime(2019,10,7)<br>while cur_date &lt; end_date:<br>  print(cur_date)</pre><pre>  year = cur_date.year<br>  month = str(cur_date.month)<br>  day = str(cur_date.day)<br>  if cur_date.month &lt; 10:<br>    month = “0”+month<br>  if cur_date.day &lt; 10:<br>    day = “0”+day<br>  results = gd.Search([‘%s %s %s’%(year, month, day)],table=’gkg’,coverage=True, translation=False)</pre><pre>  results.to_pickle(“data/%s-%s-%s.pkl”%(cur_date.year, cur_date.month, cur_date.day))</pre><pre>  cur_date+=datetime.timedelta(days=1)</pre><p>So now the dir “data” should contain 60 pkl files, each of which have a day. We can go ahead and load them all by just creating a big DF from each</p><pre>df = pd.DataFrame()<br>k = os.listdir(“data”)<br>for i in k:<br>  print(i)<br>  if i.endswith(“.pkl”):<br>    df = pd.concat([df, pd.read_pickle(“data/”+i)])</pre><p>Now, in case we break things we can save this big df to disk. I think at this point it looks like I’m taking up 6.5gb or so of RAM, saving it to PKL will crash it. You can also save to CSV or whatever format you prefer to minimize the RAM it takes, but really this step is just to speed up resetting without having to load all the individual files again.</p><pre>df.to_pickle(“first_60_days.pkl”)</pre><p>K, so let’s look at what we’ve got.</p><pre>print(len(df))<br>&gt;&gt;&gt; 5250547</pre><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QlysD1zswlC6mRO4fWt_g.png" /></figure><p>We’ve got DATE which is the DATE of the article, NUMARTS if it pulled multiple locations for the same entity (don’t worry about that for now), COUNTS (not important for now),</p><ul><li>DATE: date</li><li>NUMARTS: &gt; 1 if multiple articles (don’t worry)</li><li>COUNTS: terms with numbers associated with them (e.g. <em>*50*</em> protesters)</li><li>THEMES: NLP labeling of the general theme from BIG_MED_SML</li><li>LOCATIONS: Places</li><li>PERSONS: people + peopleish things</li><li>ORGANIZATIONS: orgs</li><li>TONE: Sentiment</li><li>CAMEOEVENTIDS: consistant ID vals for tracking events over time</li><li>SOURCES: sites articles from</li><li>SOURCEURLS: original URLs of actual articles</li></ul><p>So let’s take a look at the SOURCEURLS real quick. This is actually semi-colon delineated, but in total it looks like we have 262,171 unique combinations of sources.</p><pre>df.SOURCES.nunique()<br>&gt;&gt;&gt;262171</pre><p>we can break this down a little better by combining it all and splitting it again into a dict</p><pre>df.fillna(“”, inplace=True)</pre><pre>from collections import Counter<br>def join_col(x):<br>  big_string = “;”.join(x).split(“;”)<br>  myCount = Counter(big_string)<br>  return dict(zip(myCount.keys(), myCount.values()))</pre><pre>x = join_col(df.SOURCES)<br>x=pd.DataFrame([x]).T.sort_values(by=0, ascending=False)<br>x</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*R6RFXNbuq4tB1LXGXQ06og.png" /></figure><p>So we can see kind of what publishers there are. You can export this to an excel sheet easy enough if you want to have a spreadsheet version to mark publishers on rather than using a static array like I’ll use in a minute like this: (and then you can import the excel sheet with markings)</p><pre>x.to_excel(“sources.xlsx”)</pre><p>Anyway, we can choose a few random sites and make sure they got scraped</p><pre>mySources = [“nytimes.com”, “washingtonpost.com”, “foxnews.com”, “cnn.com”]</pre><pre>for source in mySources:<br>  print(&quot;{} is {}&quot;.format(source, &quot;good&quot; if source in x.index else &quot;bad&quot;))</pre><p>So once we know our sites are good, lets get rid of the majority of this giant dataframe that’s taking up all our RAM and look at just those sources that we care about.</p><pre>df = df[df[“SOURCES”].apply(lambda x: x in mySources)]<br>len(df)<br>&gt;&gt;&gt; 44112</pre><p>A very manageable 44k articles!</p><p>So one of the problems with the GDELT data is sometimes the date gets interpreted as an int and sometimes as a string, so we’ll quick cast everything to a string and then convert it into datetimes. (It’s easy to miss that, I just spent an hour trying to figure out why my pivot tables were messed up and it was because of types).</p><pre>df.DATE = df.DATE.apply(lambda x: str(x))<br>df.DATE = pd.to_datetime(df.DATE)</pre><p>So now we can quick see how many articles we have per day for each publisher. We can do this by grouping by source and then forgetting why we used a pivot table.</p><pre>ax = df.groupby([“SOURCES”, “DATE”])[[“SOURCEURLS”]].count().unstack().T.droplevel(level=0).plot(figsize=(12,8))<br>ax.legend(loc=2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*BnX5Sxn_udU9NW_L3kIP7w.png" /><figcaption>Coverage over Time</figcaption></figure><p>We can immediately see that there’s some seasonality to the publication, where there’s a higher amount of articles on weekdays and a lower amount on weekends.</p><pre>df.groupby([“SOURCES”, “DATE”])[[“SOURCEURLS”]].count().unstack().T.droplevel(level=0).plot.box(figsize=(12,8))</pre><p>We can look at the boxplot to see how things get distributed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*TLSfCndzYPcMmZBY8AEzGQ.png" /><figcaption>Boxplots</figcaption></figure><p>So let’s look at what different publications write about! So we can go through and pull out individual locations if we want to remembering that they’re stored as “;” deliniated</p><p>We could look for a specific country in LOCATIONS by trying to find an individual country as such:</p><pre>df[df[“LOCATIONS”].apply(lambda x: x.find(“Syria”) &gt; -1)].LOCATIONS.to_list()[0]</pre><p>But there’s no need to actually know the country code (unlike THEMES or PERSONS which have some NLP normalization); we can just do a search.</p><p>So let’s see how often these orgs write about various countries. We can go and one hot encode them for if they contain places we can think of off the top of our head:</p><pre>df[“dprk”] = df[“LOCATIONS”].apply(lambda x: x.find(“North Korea”) &gt; -1)<br>df[“ukraine”] = df[“LOCATIONS”].apply(lambda x: x.find(“Ukraine”) &gt; -1)<br>df[“russia”] = df[“LOCATIONS”].apply(lambda x: x.find(“Russia”) &gt; -1)<br>df[“iran”] = df[“LOCATIONS”].apply(lambda x: x.find(“Iran”) &gt; -1)<br>df[&quot;china&quot;] = df[&quot;LOCATIONS&quot;].apply(lambda x: x.find(&quot;China&quot;) &gt; -1)</pre><pre>loc_df = df.groupby([“SOURCES”, “DATE”])[[“dprk”, “ukraine”, “russia”, “iran”]].sum()</pre><p>And then we can go ahead and plot them</p><pre>fig, ((ax1, ax2, ax5), (ax3, ax4, ax6)) = plt.subplots(nrows=2, ncols=2)<br>fig.set_size_inches(12,8)<br>x=0<br>subplots = [ax1, ax2, ax3, ax4, ax5]<br>for column in loc_df.columns:<br>  loc_df[[column]].unstack(level=0).plot(ax=subplots[x])<br>  x+=1<br>  plt.tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*JQhhcn63sFxo496EqcvKQA.png" /></figure><p>As we saw before, we can run a VAR on these to get a rudimentary look at the interactions between these time series. We’ll just look at days because if we look at weeks we only have like 8 data points which isn’t going to be that good.</p><p>So let’s look at how coverage of country moves from publisher to publisher! Does one publisher lead the others in covering certain countries?</p><pre>loc_df = loc_df.unstack(level=0)<br>countries = [&quot;dprk&quot;, &quot;ukraine&quot;, &quot;russia&quot;, &quot;iran&quot;, &quot;china&quot;]</pre><pre>for country in countries:<br>  model = VAR(loc_df[[x for x in loc_df.columns if x.startswith(country)]], freq=&quot;d&quot;)<br>  results = model.fit(maxlags=3)<br>  display(results.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7-mebUufaxYRmSOu27fTw.png" /><figcaption>Part of results</figcaption></figure><p>Cool, so we see that a lot of publishers tend to follow Washington Post on DPRK (North Korea) coverage in this model. Likewise, they follow New York Times on Ukraine. Of course, we haven’t gotten around to evaluating if a model’s good or not but for now we’re starting to see some patterns popping out of the data.</p><p>So now let’s go ahead and look how each publication covers different countries.</p><pre>for source in mySources:</pre><pre>  model = VAR(loc_df[[x for x in loc_df.columns if x.endswith(source)]], freq=”d”)<br>  results = model.fit(maxlags=3)<br>  display(results.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOycz3HVhMNFaSkk4QVGQg.png" /><figcaption>Part of results</figcaption></figure><p>So we can see that we’ve got some interactions going here as well.</p><p>This should be enough to get someone started on messing with the GDELT data set. You can play with the interactions of LOCATIONS, THEMES, and PERSONS to identify different sorts of articles and see what comes out of it over time. In the next post I’ll see if I can cover decomposition, a bit on model evaluation, and play with SARIMA and neural networks.</p><h4>Scraping Your Own Sets:</h4><p>So the GDELT data was nice, but it’s missing the actual text of the articles. It’s also missing features like author name and where on the website the article is located. For this sort of thing, you might want to be maintaining your own feed of articles to analyze yourself.</p><p>News articles can be a bit hard to get, but there’s varying difficulty in doing so. The Guardian is the. easiest, with an API that allows you to just request every single article. The New York Times also has an API, but it returns the URLS and some labels for all of its articles. However, you can then follow the URLs back to the original URL to scrape the text yourself. This is similar to using the GDELT SOURCEURLS field to go back and get the original text which you can do for other publications too. However, with the GDELT SOURCEURLS, publishers can change up the URL for older files so it’s not guaranteed you’ll get the article.</p><p>One way to grab articles is to scrape news site homepages every day (Which is doable). But if you want archives of past articles, you need to be able to figure out where they are. You can create a Google Custom Search Engine to search for articles from a publication by day, but that can take a while. Another way to do it is to use the internal search engine on a news site. The Washington Post and Fox News for instance both have internal search engines that you can search for articles published on specific dates to find the articles to scrape.</p><p>Some news sites just don’t allow to search by date, in which case you probably do want to go the Custom Search Engine route. These include Breitbart News, Huffington Post, or CNN.</p><p>If you think you might want a good archive of news coverage, it may be worth starting to scrape home page articles every day at least so that way you can get a good sense of what news sites are promoting. News coverage changes fast and it’s not always easy to go back and find out what was covered in the past. Hope this has enough to get you started!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f171d1da65a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/making-sense-of-the-news-part-1-introduction-f171d1da65a8">Modeling News Coverage with Python. Part 1: Introduction</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>