<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harry Stevens on Medium]]></title>
        <description><![CDATA[Stories by Harry Stevens on Medium]]></description>
        <link>https://medium.com/@harry_stevens?source=rss-59d6e5d41d1f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*s34EB0p-WOV4y-4i6gdRXg.png</url>
            <title>Stories by Harry Stevens on Medium</title>
            <link>https://medium.com/@harry_stevens?source=rss-59d6e5d41d1f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:20:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harry_stevens/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[About IndieData]]></title>
            <link>https://medium.com/indiedata/about-indiedata-e7e36f20e169?source=rss-59d6e5d41d1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7e36f20e169</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[india]]></category>
            <dc:creator><![CDATA[Harry Stevens]]></dc:creator>
            <pubDate>Sun, 23 Jul 2017 10:16:39 GMT</pubDate>
            <atom:updated>2017-07-23T10:16:39.882Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/indiedata">IndieData</a> is an independent blog about working with open data in India.</p><h4>Who is it for?</h4><p>Data journalists, researchers, and technologists working with open data and source code.</p><h4>What are its goals?</h4><ul><li>Provide a forum for open data practitioners in India to share and catalogue their discoveries in a structured and organized way.</li><li>Document the problems and big ideas to enhance the overall ecosystem of open data.</li><li>Make each other aware of solutions to common problems and useful sources of data and information.</li><li>Foster a community of people working with open data in India.</li></ul><h4>Why is it needed?</h4><p>Working with open data in India is hard; IndieData will make it easier by letting people share discoveries and ideas. Though open data practitioners in India already collaborate with each other, they do not do so in the manner envisioned by IndieData.</p><p><a href="http://datameet.org/"><strong>DataMeet</strong></a> organizes <a href="http://odc.datameet.org/">open data camps</a>, shares lots of <a href="https://github.com/datameet">open source code on GitHub</a>, <a href="http://datameet.org/2017/02/16/how-to-make-an-election-interactive/">posts tutorials</a> from time to time, and even <a href="http://datameet.org/jobs/">hosts a jobs board</a>. IndieData is more focused. It’s just a blog. It’s hosted on Medium because this is the ideal platform for encouraging anyone to contribute. If you have a good idea for a post, you can contribute to IndieData.</p><p><a href="https://hackshackers.com/"><strong>Hacks/Hackers</strong></a> is also a vibrant and useful community, but it emphasizes in-person meetings over thoughtful digital communication. It is also international in scope, whereas IndieData focuses on India. Finally, while IndieData will post content useful for journalists, it will also be useful to people working in other fields.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7e36f20e169" width="1" height="1" alt=""><hr><p><a href="https://medium.com/indiedata/about-indiedata-e7e36f20e169">About IndieData</a> was originally published in <a href="https://medium.com/indiedata">IndieData</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing “party-time”: An npm package for converting Indian political party names]]></title>
            <link>https://medium.com/indiedata/introducing-party-time-a-node-package-that-makes-converting-indian-political-party-names-b0e9ba28d5ca?source=rss-59d6e5d41d1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0e9ba28d5ca</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[featured]]></category>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[hindustan-times]]></category>
            <category><![CDATA[tools]]></category>
            <dc:creator><![CDATA[Harry Stevens]]></dc:creator>
            <pubDate>Wed, 12 Jul 2017 15:33:53 GMT</pubDate>
            <atom:updated>2017-11-01T16:46:52.490Z</atom:updated>
            <content:encoded><![CDATA[<p>One thing I’ve had to do more than once in my work at the Hindustan Times is convert party acronyms to full party names, or vice versa. Imagine you have a page showing the <a href="http://www.hindustantimes.com/interactives/goa-election-results-2017-live-data/">results of the Goa state assembly elections</a>. On mobile devices, you want to only show the party acronyms (in this case, “INC” and “BJP”), but on wider screens you want to show the full party names (“Indian National Congress” and “Bharatiya Janata Party”). To do this, you need a function that will output either the acronym or the full name, after detecting the screen size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r5lNleMRICpqMkoqe9ZYew.png" /><figcaption>Show the parties on a wide screen…</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*EyozTZKDbI8IrAZjwJxpnQ.png" /><figcaption>…or on a narrower mobile device.</figcaption></figure><p>This problem is easy to solve, but I’ve had to write the code to do it more than once, which is time consuming. I also keep creating different libraries of party acronyms and their associated full names, and I always end up looking for them over and over again.</p><p>To solve these problems, I created <a href="https://www.npmjs.com/package/party-time">an npm package called <strong>party-time</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*oi1pW0jG6O30uiQQSLzXLA.gif" /><figcaption>It’s party-time!</figcaption></figure><p>party-time has just three functions: getType(), getInfo(), and convert(). The convert() function uses the first two, but those can also be called by themselves.</p><p>Right now, the library of parties is almost certainly not comprehensive. But as I (and, perhaps, you) use the library, the gaps will be filled in. To give it a test, you can install it with npm:</p><pre>npm i party-time -S</pre><p>Or you can use it in the browser by including the CDN:</p><pre>&lt;script src=&quot;<a href="https://unpkg.com/party-time/lib/party-time.min.js">https://unpkg.com/party-time/build/party-time.min.js</a>&quot;&gt;&lt;/script&gt;</pre><p>It’s pretty simple to use, but if you want to take a look at the documentation, head over to <a href="https://github.com/HindustanTimesLabs/party-time">the GitHub repo</a>. If you find a party I’m missing, <a href="https://github.com/HindustanTimesLabs/party-time/issues">open up an issue</a>.</p><p><strong>Last note:</strong> I think npm is really great. It makes creating and sharing reusable JavaScript code easy. If you have some code you think you (or someone else) might want to reuse, and you want to turn it into a package, I recommend this <a href="https://medium.com/@jdaudier/how-to-create-and-publish-your-first-node-js-module-444e7585b738">useful tutorial</a> by Joanne Daudier. I’ve been on a <a href="https://www.npmjs.com/package/jeezy">package</a>-<a href="https://www.npmjs.com/package/fsz">creating</a> binge lately.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0e9ba28d5ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/indiedata/introducing-party-time-a-node-package-that-makes-converting-indian-political-party-names-b0e9ba28d5ca">Introducing “party-time”: An npm package for converting Indian political party names</a> was originally published in <a href="https://medium.com/indiedata">IndieData</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I built a Twitter bot that tweets 100-year-old New York Times articles]]></title>
            <link>https://medium.com/@harry_stevens/how-i-built-a-twitter-bot-that-tweets-100-year-old-new-york-times-articles-5253365f9966?source=rss-59d6e5d41d1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5253365f9966</guid>
            <category><![CDATA[1917crowd]]></category>
            <category><![CDATA[twitterbots]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[1917live]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Harry Stevens]]></dc:creator>
            <pubDate>Fri, 28 Apr 2017 15:53:52 GMT</pubDate>
            <atom:updated>2017-08-19T16:11:34.958Z</atom:updated>
            <content:encoded><![CDATA[<p>It was with great interest that I learned of <a href="https://1917.rt.com/#!/en">RT 1917</a>, a project that thrusts us into the Russian revolution as its protagonists lived it, one hundred years ago to the day. It’s the next best thing to an actual time machine: a bunch of Twitter accounts purporting to be the revolution’s major players, live tweeting their “experiences.”</p><p><a href="https://twitter.com/VLenin_1917">Lenin is there</a>, having just arrived in Petrograd from exile, busy releasing his April Theses to the eager proletariat <a href="https://twitter.com/VLenin_1917/status/854079748280123392">one tweet at a time</a>. So are the former czar and czarina, <a href="https://twitter.com/nicholasii_1917">Nicholas Romanov</a> and <a href="https://twitter.com/EmpressAlix1917">Alexandra Fedorovna</a>, trapped in Tsarskoe Selo and watching in horror as their usurpers giddily dismantle everything they’ve ever cared about. And <a href="https://twitter.com/JoeStalin_1917">Stalin</a>, still largely unknown to the world, is plotting somewhere in the background, or else endlessly <a href="https://roadupward.wordpress.com/2014/06/28/why-did-stalin-make-doodles-of-wolves/">doodling wolf heads in his notebooks</a>.</p><p>Several people, not content merely to watch the events play out, have created their own Twitter accounts as part of the <a href="https://twitter.com/hashtag/1917CROWD">#1917CROWD</a>. Among the growing cast of characters, which now numbers in the several dozens, are <a href="https://twitter.com/HudoznikA">a Russian journalist in Paris</a>, <a href="https://twitter.com/Paleologue_1917">the French Envoy to the Russian Empire</a>, <a href="https://twitter.com/GMyasnikov_1917">a Bolshevik metalworker</a>, and <a href="https://twitter.com/GeorgeV_1917">the King of England</a>.</p><p>I, too, wanted to play a part in the drama. But I have neither the knowledge of history nor the time to create a character of my own and respond to events as people tweet about them (I don’t even do this very well in the present). Instead, I thought I’d write some software to do it for me. What follows is an account of how I did it.</p><h3>Step 1: Get New York Times articles about Russia from 100 years ago today.</h3><p>The New York Times has an <a href="https://developer.nytimes.com/article_search_v2.json">Article Search API</a> that lets you access all its articles from September 18, 1851, until today. All you have to do is sign up for <a href="https://developer.nytimes.com/signup">an API Key</a> and then write some code to get the articles. For this sort of thing, I’m most comfortable with Node.js, so that’s what I’m going to use. To set up my Node project, I open my Terminal and type:</p><pre>mkdir nyt-1917<br>cd nyt-1917<br>npm init</pre><p>Then I press enter until it creates my npm project. Right off the bat, I also need to install the request module.</p><pre>npm install request --save</pre><p>I also want to make my Javascript file.</p><pre>touch index.js</pre><p>Now I’ll open up Sublime Text and write a little code to get some articles.</p><pre>var request = require(&quot;request&quot;);</pre><pre>request.get({<br> url: &quot;<a href="https://api.nytimes.com/svc/search/v2/articlesearch.json">https://api.nytimes.com/svc/search/v2/articlesearch.json</a>&quot;,<br> qs: {<br>  &quot;api-key&quot;: your_api_key<br> },<br>}, function(err, response, body) {<br> body = JSON.parse(body);<br> console.log(body);<br>});</pre><p>This will return the most recent articles. But we want articles from 100 years ago today, so we’ll get today’s date and subtract 100 years from it. Javascript has a <a href="https://www.w3schools.com/jsref/jsref_obj_date.asp">native Date Object</a>, but it’s difficult to use, so we’ll use <a href="https://momentjs.com/">Moment.js</a> to deal with dates.</p><pre>npm install moment --save</pre><p>And then use it to get the date, 100 years ago.</p><pre>var moment = require(&quot;moment&quot;);</pre><pre>var date = moment().subtract(100, &quot;years&quot;); // &quot;today&quot;, but actually 100 years ago</pre><p>Wasn’t that easy? Now we can add a few properties to our request to get the articles from 100 years ago today.</p><pre>request.get({<br> url: &quot;<a href="https://api.nytimes.com/svc/search/v2/articlesearch.json">https://api.nytimes.com/svc/search/v2/articlesearch.json</a>&quot;,<br> qs: {<br>  &quot;api-key&quot;: your_api_key,<br>  &quot;begin_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br>  &quot;end_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br> },<br>}, function(err, response, body) {<br> body = JSON.parse(body);<br> console.log(body);<br>});</pre><p>This will return all sort of articles, many of which have nothing to do with Russia or the revolution. So we’ll add a property to filter the results by topic.</p><pre>var query = &quot;russia OR lenin OR trotsky OR germany OR czar OR socialism&quot;;</pre><p>This will get us stories about Russia and Russians, and we’ll also include stories about Germany, which are likely to be of interest to the folks back home in Petrograd because the two countries are currently at war. Add the query to the qs object.</p><pre>request.get({<br>  url: &quot;<a href="https://api.nytimes.com/svc/search/v2/articlesearch.json">https://api.nytimes.com/svc/search/v2/articlesearch.json</a>&quot;,<br>  qs: {<br>    &quot;api-key&quot;: your_api_key,<br>    &quot;fq&quot;: query,<br>    &quot;begin_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br>    &quot;end_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br>  },<br>}, function(err, response, body) {<br>  <br>  body = JSON.parse(body);<br>  console.log(body);</pre><pre>});</pre><p>Now we encounter an interesting thing about the Article Search API: it only returns 10 hits at a time. We can loop through the results 10 at a time by adding another property to the qs object called page, but first we need to know how many pages we’re looping through, which means we need to know how many total articles match our query. Fortunately, the API returns that information.</p><pre>var hits = JSON.parse(body).response.meta.hits;<br>var pages = Math.ceil(hits / 10);<br>console.log(&quot;Found &quot; + hits + &quot; articles on &quot; + pages + &quot; pages of results.&quot;);</pre><p>Now we know how many pages there are. So, after our initial request, we can write a function that takes as its argument the page number and loops through the pages from 0 (the page index begins at 0, like an array) to the final page, making a request for the articles from each page as it goes.</p><pre>function makeRequest(page){<br>  request.get({<br>    url: &quot;<a href="https://api.nytimes.com/svc/search/v2/articlesearch.json">https://api.nytimes.com/svc/search/v2/articlesearch.json</a>&quot;,<br>    qs: {<br>      &quot;api-key&quot;: your_api_key,<br>      &quot;fq&quot;: query,<br>      &quot;begin_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br>      &quot;end_date&quot;: date.format(&quot;YYYYMMDD&quot;),<br>      &quot;page&quot;: page<br>    },<br>  }, function(err, response, body) {<br>    <br>    body = JSON.parse(body);<br>    console.log(body);</pre><pre>  });<br>  <br>}</pre><p>Now, we could simply loop through those pages with a for loop. But so as to avoid putting unnecessary pressure on the New York Times’ servers, we’ll “throttle” the requests, spacing them out every ten seconds. For this, I’ve used an extension to the <a href="http://underscorejs.org/">Underscore.js</a> library. So we install underscore.</p><pre>npm install underscore --save</pre><p>Then we’ll add our rate limiting function. I didn’t write this function; <a href="https://gist.github.com/mattheworiordan/1084831">this guy did</a>.</p><pre>var _ = require(&quot;underscore&quot;);</pre><pre>// underscore rateLimit function<br>_.rateLimit = function(func, rate, async) {<br>  var queue = [];<br>  var timeOutRef = false;<br>  var currentlyEmptyingQueue = false;<br>  <br>  var emptyQueue = function() {<br>    if (queue.length) {<br>      currentlyEmptyingQueue = true;<br>      _.delay(function() {<br>        if (async) {<br>          _.defer(function() { queue.shift().call(); });<br>        } else {<br>          queue.shift().call();<br>        }<br>        emptyQueue();<br>      }, rate);<br>    } else {<br>      currentlyEmptyingQueue = false;<br>    }<br>  };<br>  <br>  return function() {<br>    var args = _.map(arguments, function(e) { return e; }); // get arguments into an array<br>    queue.push( _.bind.apply(this, [func, this].concat(args)) ); // call apply so that we can pass in arguments as parameters as opposed to an array<br>    if (!currentlyEmptyingQueue) { emptyQueue(); }<br>  };<br>};</pre><p>And now we’ll loop through the pages, making a request every ten seconds.</p><pre>// a rate-limited version of the request, where it runs<br>// every 10 seconds<br>var makeRequest_limited = _.rateLimit(makeRequest, 10000);</pre><pre>// loop through the pages<br>for (var i = 0; i &lt; pages; i++){<br>  makeRequest_limited(i);<br>}</pre><p>So we have our articles. Next, let’s turn them into tweets.</p><h3>Step 2: Turn the articles into tweets</h3><p>Within the request, we’ve parsed the body that the API returns by writing the line body = JSON.parse(body). But to get the actual articles, we’ll have to go a little deeper.</p><pre>var docs = body.response.docs; // the actual articles</pre><p>We can loop through these and create tweets out of them. Here’s an example that includes a hashtag at the end.</p><pre>//write filtered tweets to json<br>docs.forEach((d, i) =&gt; {<br>  <br>  // an empty object to store data<br>  var obj = {};  </pre><pre>  // create the tweet of the headline and a hashtag<br>  obj.headline = d.headline.main<br>  var tweet_start = obj.headline;</pre><pre>  var tweet_end = &quot;#1917LIVE&quot;;</pre><pre>  // the 1 is for the space between the tweet and the end<br>  var end_len = tweet_end.length + 1; </pre><pre>  // the tweet content is dependent upon the length of the headline<br>  // if the headline is too long, we&#39;ll cut it off, if necessary<br>  // adding 3 dots<br>  if (tweet_start.length &gt; (140 - end_len - 3)) {</pre><pre>     tweet_start = tweet_start.substr(0, (140 - end_len - 3));</pre><pre>     // put the elipses after the space<br>     var li = tweet_start.lastIndexOf(&quot; &quot;);<br>     tweet_start = tweet_start.substr(0, li) + &quot;...&quot;;</pre><pre>  }</pre><pre>  obj.tweet = tweet_start.toTitleCase() + &quot; &quot; + tweet_end;</pre><pre>});</pre><p>I’ve added a .toTitleCase() function to the tweet_start string so it’s not completely capitalized. This is my function (<a href="https://twitter.com/manas_sh">Manas Sharma</a> helped me with the regular expressions).</p><pre>String.prototype.toTitleCase = function() {<br> <br>    var x = this;</pre><pre>    var smalls = [];<br>    var articles = [&quot;A&quot;, &quot;An&quot;, &quot;The&quot;].forEach(function(d){ smalls.push(d); });<br>    var conjunctions = [&quot;And&quot;, &quot;But&quot;, &quot;Or&quot;, &quot;Nor&quot;, &quot;So&quot;, &quot;Yet&quot;].forEach(function(d){ smalls.push(d); })<br>    var prepositions = [&quot;As&quot;, &quot;At&quot;, &quot;Atop&quot;, &quot;By&quot;, &quot;Into&quot;, &quot;It&quot;, &quot;In&quot;, &quot;For&quot;, &quot;From&quot;, &quot;Of&quot;, &quot;Onto&quot;, &quot;On&quot;, &quot;Out&quot;, &quot;Over&quot;, &quot;Per&quot;, &quot;To&quot;, &quot;Unto&quot;, &quot;Up&quot;, &quot;Upon&quot;, &quot;With&quot;].forEach(function(d){ smalls.push(d); });<br>    <br>    x = x.split(&quot;&quot;).reverse().join(&quot;&quot;) + &quot; &quot;;</pre><pre>    x = x.replace(/[&#39;&quot;]?[a-z][&#39;&quot;]?(?= )/g, function(match){ return match.toUpperCase(); });</pre><pre>    x = x.split(&quot;&quot;).slice(0, -1).reverse().join(&quot;&quot;)</pre><pre>    x = x.replace(/ .*?(?= )/g, function(match){<br>      <br>      if (smalls.indexOf(match.substr(1)) !== -1) {<br>        return match.toLowerCase();<br>      }<br>      return match;<br>    });</pre><pre>    //smalls at the start of sentences shouldbe capitals. Also includes when the sentence ends with an abbreviation.<br>    x = x.replace(/(([^\.]\w\. )|(\.[\w]*?\.\. )).*?(?=[ \.])/g, function(match) {<br>      var word = match.split(&quot; &quot;)[1];<br>      var letters = word.split(&quot;&quot;);</pre><pre>letters[0] = letters[0].toUpperCase();<br>      word = letters.join(&quot;&quot;);</pre><pre>      if(smalls.indexOf(word) !== -1) {<br>        return match.split(&quot; &quot;)[0] + &quot; &quot; + word;<br>      }</pre><pre>      return match;<br>    });</pre><pre>    x = x.replace(/: .*?(?= )/g, function(match){ <br>      var first_letter = match.match(/\b[a-z]/);<br>      return match.replace(first_letter[0], first_letter[0].toUpperCase()); <br>    });<br>    <br>    return x;<br>}</pre><p>The New York Times Article Search API also returns keywords for each article. The keywords often include famous people, so we can check to see if any of the #1917CROWD are subjects of the article. If they are, we’ll mention them at the end of the tweet.</p><pre>var people = [<br>  {<br>    name: &quot;WILSON, WOODROW&quot;,<br>    handle: &quot;POTUS28_1917&quot;<br>  },<br>  {<br>    name: &quot;NICHOLAS II., CZAR OF RUSSIA&quot;,<br>    handle: &quot;NicholasII_1917&quot;<br>  },<br>  {<br>    name: &quot;NICHOLAS II.,&quot;,<br>    handle: &quot;NicholasII_1917&quot;<br>  },<br>  {<br>    name: &quot;ALEXANDRA FEODOROVNA, CZARINA OF RUSSIA&quot;,<br>    handle: &quot;EmpressAlix1917&quot;<br>  },<br>  {<br>    name: &quot;KERENSKY, ALEXANDER F.&quot;,<br>    handle: &quot;Kerensky_1917&quot;<br>  },<br>  {<br>    name: &quot;WILLIAM II., EMPEROR OF GERMANY&quot;,<br>    handle: &quot;Kaiser_1917&quot;<br>  },<br>  {<br>    name: &quot;TROTZKY, LEON&quot;,<br>    handle: &quot;LeoTrotsky_1917&quot;<br>  },<br>  {<br>    name: &quot;ALFONSO XIII., KING OF SPAIN&quot;,<br>    handle: &quot;AlfonsoXIII1917&quot;<br>  },<br>  {<br>    name: &quot;LENIN, NIKOLAI&quot;,<br>    handle: &quot;VLenin_1917&quot;<br>  },<br>  {<br>    name: &quot;GEORGE V., KING OF ENGLAND&quot;,<br>    handle: &quot;GeorgeV_1917&quot;<br>  },<br>  {<br>    name: &quot;MCADOO, WILLIAM GIBBS&quot;,<br>    handle: &quot;WillMcAdoo_1917&quot;<br>  },<br>  {<br>    name: &quot;MILUKOFF, PAUL N.&quot;,<br>    handle: &quot;Milyukov_1917&quot;<br>  },<br>  {<br>    name: &quot;LUXEMBURG, ROSA&quot;,<br>    handle: &quot;luxemburgquotes&quot;<br>  },<br>  {<br>    name: &quot;KORNILOFF&quot;,<br>    handle: &quot;GenKornilov1917&quot;<br>  },<br>  {<br>    name: &quot;GUCHKOFF, ALEXANDER J.&quot;,<br>    handle: &quot;Guchkov_1917&quot;<br>  },<br>  {<br>    name: &quot;BRUSILOFF, ALEXIS&quot;,<br>    handle: &quot;GenBrusilov1917&quot;<br>  },<br>  {<br>    name: &quot;BRUSILOFF, ALEXEI A.&quot;,<br>    handle: &quot;GenBrusilov1917&quot;<br>  },<br>  {<br>    name: &quot;LVOFF, GEORGE E.&quot;,<br>    handle: &quot;PrinceLvov_1917&quot;<br>  },<br>  {<br>    name: &quot;MOLOTOFF, VIACHESLAV MICHAELOVICH&quot;,<br>    handle: &quot;Molotov_1917&quot;<br>  },<br>  {<br>    name: &quot;RODZIANKO, MICHAEL&quot;,<br>    handle: &quot;MRodzianko_1917&quot;<br>  },<br>  {<br>    name: &quot;ALEXEIEFF, MICHAEL V.&quot;,<br>    handle: &quot;GenAlexeev_1917 &quot;<br>  },<br>  {<br>    name: &quot;FREUD, SIGMUND&quot;,<br>    handle: &quot;SigmundFreud_BP&quot;<br>  },<br>  {<br>    name: &quot;CLEMENCEAU, GEORGES&quot;,<br>    handle: &quot;Clemenceau_BP&quot;<br>  },<br>  {<br>    name: &quot;BERTIE, FRANCIS LEVESON&quot;,<br>    handle: &quot;WW1Bertie&quot;<br>  },<br>  {<br>    name: &quot;KROPOTKIN , PETER&quot;,<br>    handle: &quot;PKropotkin_1917‏&quot;<br>  },<br>  {<br>    name: &quot;KROPOTKIN, PETER&quot;,<br>    handle: &quot;PKropotkin_1917‏&quot;<br>  },<br>  {<br>    name: &quot;KROPOTKIN, PETER ALEXEIEVITCH&quot;,<br>    handle: &quot;PKropotkin_1917‏&quot;<br>  },<br>  {<br>    name: &quot;KROPOTKIN , PETER ALEXEIVICH&quot;,<br>    handle: &quot;PKropotkin_1917‏&quot;<br>  },<br>  {<br>    name: &quot;POINCARE, RAYMOND&quot;,<br>    handle: &quot;RPoincare_1917&quot;<br>  },<br>  {<br>    name: &quot;BUBLIKOFF, ALEXANDER ALEXANDROVITCH&quot;,<br>    handle: &quot;Bublikov_1917&quot;<br>  },<br>  {<br>    name: &quot;TSERETELLI&quot;,<br>    handle: &quot;Mensheviks_1917&quot;<br>  }<br>];</pre><p>These are the people I’ve found so far, but there may be more. Let’s write some code that goes in the docs loop to check if any of these are present in the keywords. If they are, we’ll add them to the end of the tweet.</p><pre>// figure out if any of the 1917crowd are mentioned<br>var persons = d.keywords.filter(function(key){<br>  return key.name == &quot;persons&quot;<br>}).map(function(person){<br>  return person.value<br>});</pre><pre>var lookup = people.map(function(p){<br>  return p.name;<br>})<br>var mentions = _.intersection(persons, lookup).map(function(p){<br>  return &quot;@&quot; + _.where(people, {name: p})[0].handle;<br>}); <br>// if they are add them to the end of the tweet<br>if (mentions.length &gt; 0){<br>  tweet_end = tweet_end + &quot; &quot; + mentions.join(&quot; &quot;);<br>}</pre><p>We also want to attach an image of the article itself to the tweet. For this we’ll use a module called <a href="https://www.npmjs.com/package/pdf-image">pdf-image</a> and a module called <a href="https://www.npmjs.com/package/cheerio">cheerio</a> that basically lets us use jQuery in a Node.js environment.</p><pre>npm install pdf-image --save<br>npm install cheerio --save</pre><p>Add them to the project, preferably somewhere way up at the top. You’ll also need to require http, which doesn’t need to be installed because it comes with Node.js.</p><pre>var cheerio = require(&quot;cheerio&quot;),<br>  http = require(&quot;http&quot;),<br>  PDFImage = require(&quot;pdf-image&quot;).PDFImage;</pre><p>And implement this by requesting the URL that hosts the PDF of the article, downloading that PDF to a temp directory, and converting that PDF file into an image. Again, we do this within the docs loop.</p><pre>obj.url = &quot;<a href="http://query.nytimes.com/mem/archive-free/pdf?res=">http://query.nytimes.com/mem/archive-free/pdf?res=</a>&quot; + d.web_url.split(&quot;res=&quot;)[1];</pre><pre>// some variables for creating a unique pdf file name<br>obj.date = d.pub_date.split(&quot;T&quot;)[0];<br>obj.page = page + 1;<br>obj.page_index = i + 1;</pre><pre>// this is the pdf file name<br>obj.pdf_file_name = &quot;temp/&quot; + obj.date + &quot;_&quot; + obj.page + &quot;_&quot; + obj.page_index + &quot;.pdf&quot;;</pre><pre>// a stream to write the pdf file for downloading<br>var file = fs.createWriteStream(output);</pre><pre>// get the html of the nyt page<br>request(obj.url, function(error, response, body){</pre><pre>  if (!error &amp;&amp; response.statusCode == 200){<br>    // load cheerio<br>    var $ = cheerio.load(body);</pre><pre>    // find the pdf url in the response<br>    var pdf = $(&quot;iframe&quot;).attr(&quot;src&quot;);</pre><pre>    // download the pdf<br>    var request = http.get(pdf, function(response) {<br>      <br>      // pipe the response to the file<br>      var stream = response.pipe(file);<br>      <br>      // when it&#39;s done, we&#39;ll convert to an image<br>      stream.on(&quot;finish&quot;, function(){</pre><pre>        // convert to image, with white background<br>        var pdfImage = new PDFImage(output, {<br>            convertOptions: {<br>                &#39;-background&#39;: &#39;white&#39;,<br>                &#39;-flatten&#39;: &#39;&#39;<br>            }<br>        });</pre><pre>        pdfImage.convertPage(0).then(function (imagePath) {</pre><pre>          console.log(imagePath);</pre><pre>        });</pre><pre>      });</pre><pre>    });<br>    <br>  } else {<br>    console.log(&quot;Error getting PDF&quot;);<br>    console.log(error);<br>  }</pre><pre>});</pre><p>Now that we’ve written some nice tweets and created images to go with them, it’s time to send them out into the world.</p><h3>Step 3: Send the tweet</h3><p>To gain access to Twitter’s API, you’ll need to <a href="https://apps.twitter.com/">create a Twitter app</a> and follow the steps until you have a consumer key, consumer secret, access token, and access token secret.</p><p>The next part’s fairly simple, thanks to the excellent <a href="https://www.npmjs.com/package/twit">Twit</a> module. Go ahead and install that first.</p><pre>npm install twit --save</pre><p>Now require the module and create a Twit instance using your Twitter app credentials.</p><pre>var Twit = require(&quot;twit&quot;);</pre><pre>// a new Twit instance<br>var T = new Twit({<br>  consumer_key: your_consumer_key,<br>  consumer_secret: your_consumer_secret,<br>  access_token: your_access_token,<br>  access_token_secret: your_access_token_secret<br>});</pre><p>And then, at the end of the docs loop, add some code.</p><pre>pdfImage.convertPage(0).then(function (imagePath) {                      <br><br>  // post a tweet with media<br>  var b64content = fs.readFileSync(imagePath, { encoding: &quot;base64&quot; });</pre><pre>  // first we must post the media to Twitter<br>  T.post(&quot;media/upload&quot;, { media_data: b64content }, function (err, data, response) {<br>    <br>    // now we can assign alt text to the media<br>    // for use by screen readers and<br>    // other text-based presentations and interpreters<br>    var mediaIdStr = data.media_id_string;<br>    var altText = obj.headline;<br>    var meta_params = { media_id: mediaIdStr, alt_text: { text: altText } };</pre><pre>   // create the tweet&#39;s metadata, which contains the image<br>   T.post(&quot;media/metadata/create&quot;, meta_params, function (err, data, response) {<br>      if (!err) {<br>        // now we can reference the media and post a tweet<br>        // (media will attach to the tweet)<br>        var params = { status: obj.tweet, media_ids: [mediaIdStr] }</pre><pre>        // post the tweet<br>        T.post(&quot;statuses/update&quot;, params, function (err, data, response) {<br>          if (!err){<br>            // we&#39;ll console log each tweet so we know it was sent       </pre><pre>            console.log(data.text);<br>            console.log(&quot; &quot;);</pre><pre>          } else {<br>            console.log(err.message);<br>          }<br>        });</pre><pre>      }<br>      <br>    });</pre><pre>  });<br>  <br>});</pre><h3>Step 4: Automatically run the code ever hour</h3><p>Now, if we run node index.js, we’ll tweet out everything at once. But that’s not how real newspapers use Twitter. Rather, they tweet their stories out over the course of the day. And that’s what we want our bot to do.</p><p>For this task, we’ll need to put our code on some server and run it every hour, only sending out 1/24th of the tweets each time. To do this, we’ll be using <a href="http://heroku.com/">Heroku</a>. If you’re new to Heroku, you should create an account and go through the <a href="https://devcenter.heroku.com/articles/getting-started-with-nodejs#introduction">tutorial on how to deploy a Node.js app</a>, as that will give you a basic understanding of what comes next.</p><p>We are going to be using an addon called Scheduler. If you’d like, you can read <a href="http://www.spacjer.com/blog/2014/02/10/defining-node-dot-js-task-for-heroku-scheduler/">a simple tutorial of how to use Scheduler here</a>. Now, in your terminal, you must login to Heroku.</p><pre>heroku login</pre><p>Create your app.</p><pre>heroku create nyt-1917</pre><p>Deploy your app.</p><pre>git push heroku master</pre><p>Alright, let’s now add the Scheduler addon.</p><pre>heroku addons:add scheduler</pre><p>To use Scheduler, you will need to provide a valid credit card. Don’t worry, you won’t be charged unless you go over a certain usage limit. I’ve been running my bot for a while, and my estimated monthly cost, which Heroku provides, is $0.00. In other words, free. Follow the link in your terminal to add your credit card details.</p><p>Once you’ve provided your credit card details, you’ll need to create a directory called bin and put a file in it called tweet (or any other name you’d like to use to describe the process you want to schedule). Note that there is no file extension.</p><pre>mkdir bin<br>touch bin/tweet</pre><p>Open up that tweet file and add a line at the top called a shebang.</p><pre>#!/usr/bin/env node</pre><p>And copy everything from index.js into the tweet file. Now, there’s just one more thing we must add to the code before we start up the Scheduler, because we need to decide which tweets to send out at any given hour of the day. To do that, we’ll first get the current hour. Obviously, this will change depending on when the app runs. Somewhere at the top of your code, make a variable for the current hour.</p><pre>var current_hour = moment().format(&quot;H&quot;);</pre><p>This will return the current hour in 24-hour time. We just need to assign an hour to every tweet, and when we run the code, we’ll only send out the tweets whose hours match the current_hour. Within the docs loop, add some code.</p><pre>obj.page = page + 1;<br>obj.page_index = i + 1;<br>obj.tweet_number = (page * 10) + obj.page_index;<br>        <br>// calculate the hour of the day that the tweet<br>// should go out, based on the total number of tweets<br>obj.hour_of_tweet = Math.round((obj.tweet_number * 24) / hits);</pre><p>Before you actually post the tweet, you can write a conditional statement to see if the obj.hour_of_tweet matches the current_hour .</p><pre>if (obj.hour_of_tweet == current_hour){<br>  // here is where you put the code that<br>  // 1. downloads the pdf<br>  // 2. converts it to an image<br>  // 3. posts the tweet<br>  // (see above for instructions on how to do this)<br>}</pre><p>Log in to Heroku, select your app, and, in the top left, under “Installed add-ons”, select Heroku Scheduler. Click “Add new job”, where it says rake do_something type tweet, and set the “FREQUENCY” to “Hourly”.</p><p>That’s it. Here’s <a href="https://twitter.com/NYT_1917">the bot’s Twitter page</a>, so you can see it in action, mixing it up with all those famous Russians from 100 years ago. And here’s <a href="https://github.com/HarryStevens/nyt-1917">the GitHub repo</a> with all the code.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5253365f9966" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>