<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Robert Coleman on Medium]]></title>
        <description><![CDATA[Stories by Robert Coleman on Medium]]></description>
        <link>https://medium.com/@rdcolema7?source=rss-f96f279bd2e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*az8qSpzR2gQ4kYzYzjrh5A.png</url>
            <title>Stories by Robert Coleman on Medium</title>
            <link>https://medium.com/@rdcolema7?source=rss-f96f279bd2e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:20:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rdcolema7/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mapping Historical Events on Demand with Claude]]></title>
            <link>https://medium.com/@rdcolema7/mapping-historical-events-on-demand-with-claude-a1550ce8da70?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1550ce8da70</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[history]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Wed, 15 Apr 2026 01:28:42 GMT</pubDate>
            <atom:updated>2026-04-15T01:28:42.571Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2020 I wrote a <a href="https://medium.com/swlh/creating-interactive-historical-maps-with-react-mapbox-and-wikipedia-26be1726a3a7">post</a> about a small React + Mapbox app I built for plotting historical events geographically and chronologically. It was essentially a simple date slider, with a map populated by events occurring around the selected year. The data was sourced from DBpedia, a structured knowledge base built on top of Wikipedia. It was somewhat fun to play with at the time, but the data was sparse and heavily skewed toward European military history and sports (go figure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilDIgRk5FEx9fcvuGKZWEg.png" /><figcaption>The new UX</figcaption></figure><p>Six years later the original looks pretty disappointing. The DBpedia SPARQL endpoint still responds, but the React version is three major versions behind and the Mapbox API has drifted significantly. The linear 0-2020 year slider in the old version of the UX was also more or less unusable for ancient history, among various other issues.</p><p>Recently, I “rehabilitated” another old side project of mine (a <a href="https://medium.com/@rdcolema7/resurrecting-a-dead-django-app-in-the-llm-era-30258362a648">Django portfolio tracker</a>) with Claude, and the process was smooth enough that I figured this app was worth another go too. This post walks through what changed in the timeline app, why I like the new architecture is better, and the tradeoffs of using LLMs for on-demand synthetic data.</p><h3>Why a historical maps app</h3><p>The original motivation hasn’t changed. History is usually taught sorted by topic, which leaves us without a good way to ask “what was happening around the world at this moment” for any given year. The American Civil War and the Taiping Rebellion, the two deadliest wars of the 19th century, happened in overlapping decades on opposite sides of the planet, but the two chapters about them in the average high school textbook are fifty pages apart and don’t mention each other (if they’re taught at all, that is). The original version of this app was my first attempt at looking at history a different way.</p><h3>The new architecture</h3><p>Instead of pre-populating a database from a fixed upstream source, the new version generates events on demand with Claude. Now you can switch the map into “generate” mode, click a region, and the backend sends a prompt to the Anthropic API with that grid cell’s coordinates, the current year range from the slider, and a list of nearby reference cities to help anchor the geography. Claude returns about 15-25 historically accurate events for the region and era, which get validated, inserted into SQLite, and rendered as map markers. Subsequent clicks on the same cell and time range should hit the cache and return instantly, in theory at least.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WvVfNylHTGsIoJGWs4pFYQ.png" /><figcaption>Event generation</figcaption></figure><p>This avoids the original data sparsity problem entirely. There’s no upstream data source with a built-in regional or categorical bias. If you want events for 12th-century Ethiopia, then they’re available, as long as the model can come up with some. Bronze Age Mesopotamia, same thing. Coverage is in principle as broad as Claude’s training data, which in practice is <em>much</em> broader than DBpedia’s event tables.</p><p>The rest of the stack is actually smaller than v1:</p><ul><li><strong>Frontend</strong>: Vite + React 19 + TypeScript + Tailwind + Zustand. No more create-react-app, no Bootstrap, no jQuery.</li><li><strong>Maps</strong>: MapLibre GL JS instead of Mapbox GL JS. Same API shape, MIT licensed, no API key needed. Tiles come from <a href="https://openfreemap.org/">OpenFreeMap</a>, which hosts the OpenStreetMap vector tiles for free with no account required.</li><li><strong>Backend</strong>: Express + better-sqlite3 + the Anthropic SDK. No MySQL, no Docker orchestration, no separate DB container. The database is a single file on disk.</li></ul><p>The timeline slider is the other big improvement. The original ran linearly from year 0 to year 2020, which meant ancient history was compressed into a tiny bucket at the left edge. The new one uses a piecewise non-linear scale with more pixel budget allocated to recent history (where there are more events per unit time) and less to deep antiquity. BCE dates are also fully supported now, so you should be able to drag to 480 BCE and see Thermopylae, which wasn’t possible before.</p><h3>The core loop</h3><p>When you set the slider to, say, 1015 CE, switch to generate mode, and click somewhere in central Iran, the server figures out which 10-degree grid cell that falls into, looks up any existing events in that cell for the time range, and sends Claude a prompt with a hint not to regenerate existing events. Claude returns a JSON array of events, each with a name, a 1-3 sentence factual description, a year, actual coordinates for the specific site (not a country centroid), a category, and a significance rating from 1 to 10. The server validates them (a bounding-box check catches coordinate hallucinations, the field types get checked, and invalid values get logged) and inserts the survivors with a unique constraint on (name, year) to prevent duplicates across overlapping cells.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4unPBUb7EHWHAEMa2D1ZnQ.png" /><figcaption>A topical example</figcaption></figure><p>The markers on the map are rendered through a single GeoJSON source with a MapLibre circle layer, not DOM markers. This matters because dense years can produce hundreds of points in a region and DOM markers choke above a few hundred. The circles are colored by category and sized by significance, with one extra dimension: each event has a location_precision field (exact, city, or region) that controls how the marker is drawn. Exact-precision events get a small sharp dot. City-level events get a slightly larger opaque circle. Region-level events (things like &quot;Buyid dynasty conflicts in central Persia&quot; that didn&#39;t happen at any one place ) get rendered as a larger, semi-transparent, blurred blob. The motivation here is that a lot of historical &quot;events&quot; are really processes spanning a region, so forcing them into a single coordinate felt off.</p><p>For the event detail view, I also added a “Generate deeper analysis” button that kicks off a second, separate Claude call to produce a 2-3 paragraph narrative summary of the event covering what happened, why it mattered, who was involved, how it connects to the broader world at the time, etc. That summary is also cached in SQLite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWzwlGQMG7eLyoHYoUJNwg.png" /><figcaption>Event detail view</figcaption></figure><h3>What’s next</h3><p>The coordinate accuracy from the LLM is decent but not perfect. The validation step rejects events that fall outside the bounding box of the grid cell they were requested for, which catches the gross hallucinations, but something like a geocoding pass that cross-references Claude’s coordinates against GeoNames would catch the subtler ones too.</p><p>Marker clustering at low zoom levels would also help in dense regions. In other words, if you generate a lot of events across western Europe in the same century, the circles pile up on top of each other. MapLibre supports cluster layers out of the box, so that should be doable.</p><p>Finally, the border overlay is a bit frustrating. I’d love a historical border layer that changes with the timeline (i.e. Roman Empire boundaries at 117 CE, Mongol Empire at 1279, Ottoman at 1683, etc). There’s some free data for the modern era (<a href="https://icr.ethz.ch/data/cshapes/">CShapes</a> goes back to 1886) but nothing great for earlier periods. <em>Maybe</em> this could be generated on demand too via Claude by asking for a GeoJSON polygon of a named empire at a given year? But that’s probably a rabbit hole, so will revisit that later.</p><p>The full source for v2 is on <a href="https://github.com/rdcolema/timeline">GitHub</a>. It runs locally with npm install &amp;&amp; npm run init-db &amp;&amp; npm run dev and an optional ANTHROPIC_API_KEY in .env for the generation features. If you don&#39;t set the key the map and UI still work, you just can&#39;t generate new events.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1550ce8da70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resurrecting a Dead Django App in the LLM Era]]></title>
            <link>https://medium.com/@rdcolema7/resurrecting-a-dead-django-app-in-the-llm-era-30258362a648?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/30258362a648</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Tue, 14 Apr 2026 17:05:13 GMT</pubDate>
            <atom:updated>2026-04-14T17:05:13.758Z</atom:updated>
            <content:encoded><![CDATA[<p>Early in 2020 I wrote a <a href="https://medium.com/@rdcolema7/make-your-own-investing-web-portal-with-python-and-django-89feb5ce0ddd">post</a> about a simple Django app I’d built to track my investments across accounts. The idea was pretty simple: I wanted to see all my positions in one place, with real quotes, without handing credentials to a third-party aggregator. It worked well enough for my needs and I used it occasionally, but then I mostly forgot about it.</p><p>A lot has changed since then. The IEX Cloud API the original version used to fetch quote data was shut down in 2024. The version of Bokeh I used for the analysis charts stopped shipping the bokeh.charts module back in 2017. Pandas dropped df.append(). The Django template syntax I was using has been deprecated twice over. If you cloned that repo today and tried to run it, nothing would work. The data layer is dead, the charting is dead, and half the view code raises on import.</p><p>But the bigger change is that we’ve officially entered the LLM era, and a bunch of things I would have wanted to build into that original app are now borderline trivial to add.</p><p>This post is the story of that refactor. I want to go through why I find a project like this useful in the first place, walk through the new architecture and what it actually does now, and wrap up with some thoughts on using coding agents to rehab old projects.</p><h3>Why bother with a personal portfolio tracker</h3><p>My portfolio is spread across a handful of brokerage accounts, and I’m (for better or worse) a stock picker, with somewhere north of 50 positions at any given time, most of them individual names rather than funds. I care about the breakdown of my holdings along several dimensions that no single broker UI gives me all at once: sector, theme, account, cost basis, day’s change, thesis, etc.</p><p>The commercial options do some of this, but in a very limited way. None of them let me tag a position with “AI infrastructure buildout” and “defense modernization” at the same time and then see my portfolio concentration along those axes.</p><p>The point of rolling my own is that I can make it exactly what I want. If I decide tomorrow that I want a view of my portfolio grouped by “positions I wrote a thesis for in the last six months,” I can add it in an afternoon. And now that I can plug Claude into it for thesis review and portfolio-level analysis, I can make it a kind of research assistant too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwi1I9gThviQKCQqK3kDuQ.png" /><figcaption>Dashboard overview</figcaption></figure><h3>The new architecture</h3><p>The bones of the app are still Django, but almost nothing else survived. I rewrote the data model, the data layer, the charting, the styling, the entire front-end, and the deployment story. Here’s what it looks like now:</p><p>For the data layer I swapped IEX Cloud out for <a href="https://github.com/ranaroussi/yfinance">yfinance</a>, which is free, unofficial, and stable enough for personal use. It gives me live quotes via fast_info (price, previous close, 52-week range) and fundamentals via the slower .info endpoint (forward P/E, trailing P/E, P/S, EV/EBITDA, market cap, sector, dividend yield). The quote layer fetches concurrently via a ThreadPoolExecutor and caches results in Django&#39;s local memory cache, with 15 minutes for quotes and 24 hours for fundamentals. A portfolio of 50–100 positions refreshes in about two seconds.</p><p>The data model is more expressive than the original. The v1 version had Account and Position, where a position was a single row with shares and cost_basis fields. That works for cumulative holdings but doesn&#39;t handle multiple buys at different prices, which is how I actually accumulate positions. The new version has Account, Position, and Lot. A position is a grouping, and the lots underneath it store individual buys with their own price and date. Cost basis and average cost are computed properties that aggregate over the lots.</p><p>I also added a Sector and Theme model. Sectors are taxonomic and each position has one; themes are cross-cutting tags and each position can belong to several, which is important since a typical investor’s mental model of &quot;AI infrastructure&quot; isn&#39;t a clean sector, but more like a pattern spanning Information Technology, Industrials, Utilities, Communication Services, etc.</p><p>For the front end I ripped out Bootstrap 3 and jQuery and replaced them with Tailwind CSS and HTMX, both loaded from a CDN with no build step. Tabulator.js handles the positions table, which is the workhorse view and needs to support per-column sorting, filtering, frozen columns, and a totals row. Plotly.js handles the charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjG-LlAKNP3gs1i7MFtO6A.png" /><figcaption>Positions table</figcaption></figure><p>Like the prior iteration, the positions page is still the most interesting view, but I’ve cleaned it up substantially and added a few columns and better aggregations. Clicking a symbol now also takes you to the detail page for that position.</p><p>The new analysis view has four tabs, each of which is a separate view of the same data. Concentration is the most visual, with four charts showing how the portfolio breaks down by position, sector, theme, and account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ezz8SD2YiYUukWCVjqPycQ.png" /><figcaption>Analysis page</figcaption></figure><p>Valuation is the one I’ve wanted for a long time. It pulls live multiples for every position and flags each one as Cheap, Fair, or Stretched based on per-sector thresholds (these are pretty flimsy, but wanted to start somewhere…a value investor&#39;s &quot;stretched&quot; is a growth investor&#39;s &quot;fair,&quot; so there&#39;s no universal answer of course). ETFs are excluded from the verdict since fund-level multiples don&#39;t mean the same thing as single-stock multiples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XlEQKvAFQ9T399UMiHa-8w.png" /><figcaption>Valuations tab</figcaption></figure><p>Movers and Alerts round out the analysis app. Movers shows today’s biggest day-changers alongside all-time winners and losers, sorted by percent. Alerts surfaces positions that exceed target weights, positions I haven’t written a thesis for, and any sector that’s taking up more than 25% of the portfolio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Im8t0vXkov11sNnvnokIbw.png" /><figcaption>Movers tab</figcaption></figure><p>The position detail page is where LLMs unlock a lot of new possibilities. Each position has a thesis and a bear case, which can be manually added or AI-generated. The thesis drives one of the alerts: if I haven’t touched it in 180 days, it starts showing up as stale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LcOY7UmmArhj2den1oJhSA.png" /><figcaption>Example JPM position detail view</figcaption></figure><h3>LLM integration</h3><p>The new app has an ai module that wraps the Anthropic API and adds four features: thesis check, thesis generation, bear case generation, and a weekly portfolio snapshot.</p><p>Thesis check is the most interesting one. The app bundles up the position’s symbol, sector, themes, current price, fundamentals, cost basis, current return, thesis, and sends all of that to Claude with a system prompt that essentially says: review this thesis for an experienced self-directed investor, push back if it’s weak or outdated, stress-test the reasoning on its own terms rather than imposing a different investing philosophy, keep it under 300 words (and, of course, no sycophancy…a dangerous LLM failure mode for investing-related inputs).</p><p>In the screenshot above, you can see Claude’s review of an example JPM thesis. It identified the parts of the argument that held up (the ROTC performance, the diversification, the valuation relative to peers), pushed back on the parts that were potentially glossed over (Dimon succession risk, NII compression under rate cuts, Basel III endgame capital requirements), and called out that the framing was “slightly complacent for a position this large.” That last bit is close to what I actually wanted from this feature: not a second opinion on whether to hold, but a specific pointer at the places in my reasoning where I might have been lazy or ignorant.</p><p>Finally, the weekly snapshot is the portfolio-level view. This sends Claude a compact JSON digest of every position (symbol, sector, market value, day change percent, total return percent, portfolio weight) along with a sector rollup. Claude writes back a short review covering today’s movers, all-time winners that look stretched, stale theses, and concentration concerns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E80EXwNC_D56se2jaXCt9g.png" /><figcaption>AI Insights page</figcaption></figure><h3>What’s next</h3><p>There are a few things I left out for now. I don’t have historical portfolio value tracking yet. Every view is a point-in-time snapshot, so if I want to see how my portfolio performed over the last year I still have to look at my broker’s statements. I could potentially add this as a nightly cron that writes a row to a PortfolioSnapshot table, something like that. Once that’s in, I could also add a longitudinal performance chart, and probably start computing things like dividend income over time, etc.</p><p>Bulk CSV import from broker statements would make the initial setup a lot less tedious too. Currently you have to add positions through the Django admin one at a time or via a script, which is fine for ongoing use but annoying when you’re first setting things up. A management command that reads a broker’s CSV export and creates the corresponding accounts, positions, and lots would eliminate the worst part of the setup experience.</p><p>The one thing I don’t plan to add is broker integrations. The original app’s pitch was that it didn’t require you to hand over your brokerage credentials, and I still think that’s the right call, especially with largely vibe-coded apps.</p><h3>A note on building things this way</h3><p>Full disclosure, I did most of this refactor via writing a detailed spec and handing it to Claude, then reviewing the result in several passes and pushing back where the implementation didn’t match my intent. The initial spec took a couple of hours to get right. I had to think through what I actually wanted the new data model to look like, which features were in scope, which belonged in a hypothetical v3, and how to describe the architecture precisely enough that a coding agent wouldn’t wander off into scope creep. The implementation was relatively fast after that, and mostly consisted of iterative testing and turn taking with the local agent.</p><p>I don’t think I could have built v2 this fast without an LLM. I also don’t think I would have built v2 <em>correctly</em> if I’d just accepted the first output. This workflow still feels a bit dirty to me to be honest, but it lets me actually finish some of the personal projects I previously abandoned in unfinished states. I plan on doing the same for other past projects, like my <a href="https://medium.com/swlh/creating-interactive-historical-maps-with-react-mapbox-and-wikipedia-26be1726a3a7">timeline</a> app.</p><p>The full source for v2 is on <a href="https://github.com/rdcolema/snapshot-finance">GitHub</a>. If you want to run it yourself, cp .env.example .env, fill in a username and password, optionally add an ANTHROPIC_API_KEY, and docker compose up. Add your accounts and positions through the Django admin at /admin/ and the rest of the app lights up.</p><p>And if you’ve got an old project sitting on your GitHub that rotted away like this one, it’s worth asking whether it’s time for a second pass.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30258362a648" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Interactive Historical Maps With React, Mapbox, and Wikipedia]]></title>
            <link>https://medium.com/swlh/creating-interactive-historical-maps-with-react-mapbox-and-wikipedia-26be1726a3a7?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/26be1726a3a7</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[dbpedia]]></category>
            <category><![CDATA[wikipedia]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[mapbox]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 14:53:51 GMT</pubDate>
            <atom:updated>2020-09-30T22:46:41.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZNJlWZi4QQ9EJmcs" /></figure><p>One of the biggest flaws with how schools teach history is that they’re rarely able to tie things together into a cohesive global picture. Students learn about individual events and figures in great detail, but few are able to simultaneously describe the state of the world across geographies in a given era.</p><p>I’m a visual learner, so I’ve been thinking about the sorts of web apps that could help engage with history in a more interactive and scalable way. At a high level, I wanted to make something that enables a user to:</p><ol><li>Control the geographical search space</li><li>Control the returned information chronologically</li><li>Easily navigate to finer-grain detail if desired</li></ol><p>Below I’ll describe the app I came up with — it’s very much a work in progress, but hopefully you’ll find the code, the APIs used, or any of the design decisions made so far useful. The full code is available in this <a href="https://github.com/rdcolema/timeline">github repo</a>.</p><h3>The Back End</h3><p>For an application like this to be successful, the first requirement is data — and a lot of it. While exploring any given time period, a user needs the app to return data about that period, with high-level information, links, and other attributes immediately available. The easiest way to structure this requirement is to create a model around historical events, which fit nicely into a database model and are easy to enumerate programmatically .</p><p>So how does one acquire a large amount of highly structured data about historical events and populate a database with those entities? Fortunately, a project called <a href="https://wiki.dbpedia.org/about">DBpedia</a> provides a public database built on top of Wikipedia entries that can be queried with a SQL-like language called SPARQL. This means that not only can the data be scraped once to build out an initial database, but our app’s back end can also refresh the database as new events are added to the public records.</p><p>The code for this step, implemented in Python and loaded into a MySQL database, can be found <a href="https://github.com/rdcolema/timeline/blob/master/server/refresh_events.py">here</a>. The key features in the SPARQL query itself are that we’re able to query event names, dates, location data, and links to additional resources; the results of this query can then easily be inserted into a database table with the same fields:</p><pre>PREFIX dbo: &lt;http://dbpedia.org/ontology/&gt; <br>PREFIX n1: &lt;http://www.w3.org/2003/01/geo/wgs84_pos#&gt; <br>SELECT <br>    (?Event_69 AS ?event) <br>    (?date_89 AS ?date) <br>    (AVG(?place_lat_158) AS ?latitude) <br>    (AVG(?place_long_159) AS ?longitude) <br>WHERE { <br>    ?Event_69 a dbo:Event . <br>    ?Event_69 dbo:date ?date_89 . <br>    ?Event_69 dbo:place ?place_92 . <br>    ?place_92 n1:lat ?place_lat_158 . <br>    ?place_92 n1:long ?place_long_159 . } <br>GROUP BY <br>    ?Event_69 <br>    ?date_89 <br>ORDER BY <br>    ?date_89</pre><p>Now that we have a database full of clean historical event data, we need a way to expose that data to the front end of the app. A natural solution for this is an API, which we’ll build out with <a href="https://expressjs.com/">Express</a>, a web framework for Node.js.</p><p>The most important piece of the API code is a simple handler for a GET request to the events endpoint:</p><pre>// List events<br>app.get(&#39;/events/:year&#39;, (req, res) =&gt; {<br><br>  const year = req.params.year;<br><br>  pool.query(`<br>    SELECT * FROM timeline_event WHERE date LIKE ?`,  [year + &quot;%&quot;], <br>    (err, results) =&gt; {<br>      if (err) {<br>        console.log(err);<br>        return res.send(err);<br>      } else {<br>        return res.send(results);<br>      }<br>    });<br>});</pre><p>The endpoint accepts a single parameter, year, which is used as a filter while querying the timeline_event table. An example request and response might look like:</p><pre>// <a href="http://127.0.0.1:8003/events/1950">http://&lt;base_url&gt;/events/1950</a></pre><pre>[<br> {<br> “id”: 24248,<br> “name”: “Battle of Bamianshan”,<br> “url”: “<a href="http://dbpedia.org/resource/Battle_of_Bamianshan">http://dbpedia.org/resource/Battle_of_Bamianshan</a>&quot;,<br> “date”: “1950–01–19”,<br> “latitude”: 37.4583,<br> “longitude”: 109.692<br> },<br>...<br>]</pre><p>With this API available, we can make queries like the above from our front-end code and use these results to populate a dynamic map.</p><h3>The Front End</h3><p>The primary feature of the front-end code is a WorldMap React component that serves as a wrapper for the open-source ReactMapGL component. The latter integrates with Mapbox’s public API, so it requires setting up an account with them and generating an <a href="https://docs.mapbox.com/help/how-mapbox-works/access-tokens/">access token</a>, but once that’s done it makes rendering an interactive map as easy as inserting a snippet like this into our React app:</p><pre>&lt;ReactMapGL<br>    {...viewport}<br>    width=&quot;90w&quot;<br>    height=&quot;80vh&quot;<br>    mapStyle=&quot;mapbox://styles/mapbox/outdoors-v11&quot;<br>    mapboxApiAccessToken={process.env.REACT_APP_MAPBOX_TOKEN}<br>&gt;&lt;/ReactMapGL&gt;</pre><p>A map alone isn’t of much use without a way to render our events and navigate through the historical timeline. For this, we’ll use a “range-slider” input that lets users drag a button in the web UI that updates the current year displayed on page.</p><p>The input year can then be used to make a GET request to the back end API we set up above, which will return the events for that year in our database. Each event has a latitude and longitude associated with it, making it possible to render the event as a Marker component on our map.</p><figure><img alt="Sample navigation within the Timeline app" src="https://cdn-images-1.medium.com/max/546/1*0wLQzcgu8mTaNrK13hHkog.gif" /><figcaption>Sample navigation within the Timeline app</figcaption></figure><p>The full code for the WorldMap component can be found <a href="https://github.com/rdcolema/timeline/blob/master/client/src/worldMap.js">here</a>; I’ve added some extra functionality such as popups with links to more info for each event, click event handling, and component state management.</p><h3>Orchestration and Deployment</h3><p>So how can we actually run any of this code and make the services we’ve created accessible to each other? I’ve been a fan of <a href="https://docs.docker.com/compose/">Docker Compose</a> lately for managing apps with multiple services, and it’s a pretty good fit for this case as well given that there’s a database, an API back end, and a front-end client involved.</p><p>Docker Compose lets you organize each service in a container and then manage dependencies, networking, environment variables, entrypoints, and more from a single YAML configuration file. For this example, there are three services, the database, the front-end client, and the back end API, named “mysql”, “client”, and “server” respectively:</p><pre>version: &quot;3.4&quot;<br><br>x-common-variables: &amp;common-variables<br>  MYSQL_USER: ${MYSQL_USER}<br>  MYSQL_PASSWORD: ${MYSQL_PASSWORD}<br>  MYSQL_DATABASE: ${MYSQL_DATABASE}<br>  REACT_APP_MAPBOX_TOKEN: ${REACT_APP_MAPBOX_TOKEN}<br>  REACT_APP_SERVER_PORT: ${REACT_APP_SERVER_PORT}<br><br>services:<br>  mysql:<br>    image: mysql:5.7<br>    environment:<br>      &lt;&lt;: *common-variables<br>      MYSQL_HOST: ${MYSQL_HOST}<br>      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}<br>    ports:<br>      - ${MYSQL_PORT}:3306<br>    restart: unless-stopped<br>    volumes:<br>      - ./db/ddl.sql:/docker-entrypoint-initdb.d/ddl.sql<br>      - mysql-data-volume:/var/lib/mysql<br>  client:<br>    build: ./client<br>    environment:<br>      &lt;&lt;: *common-variables<br>      NODE_PATH: src<br>    expose:<br>      - 3000<br>    ports:<br>      - ${CLIENT_PORT}:3000<br>    volumes:<br>      - ./client/src:/app/src<br>    stdin_open: true<br>    command: npm start<br>  server:<br>    build: ./server<br>    depends_on:<br>      - mysql<br>    expose:<br>      - ${REACT_APP_SERVER_PORT}<br>    environment:<br>      &lt;&lt;: *common-variables<br>      MYSQL_HOST_IP: mysql<br>    ports:<br>      - ${REACT_APP_SERVER_PORT}:${REACT_APP_SERVER_PORT}<br>    volumes:<br>      - ./server:/app<br>    links:<br>      - mysql<br>    command: [&quot;./run_server.sh&quot;]<br>volumes:<br>  mysql-data-volume:</pre><p>More details on the variables used here can be found in the <a href="https://github.com/rdcolema/timeline">github repo</a> for the app.</p><p>To build the containers run $ docker-compose build from the same directory as “docker-compose.yml”, and to start the services run $ docker-compose up -d.</p><p>This will build the docker images and run the containers locally, so navigating to “localhost:&lt;CLIENT_PORT&gt;” in your browser after executing these commands should display the running app.</p><h3>Final Notes</h3><p>There are a number of missing pieces and unsolved problems to the approach taken above. The most obvious is how sparse and unequally distributed the DBpedia data is; there are likely more comprehensive data sources out there, but I wasn’t able to find any readily available public alternatives with a more satisfactory size and level of detail. (If you’re aware of any please do reach out and let me know.)</p><p>Another weakness is that I haven’t yet accounted for B.C.E. history, or the fact that as history stretches further back in time, our record of events is spread thinner. A log scale of some sort for the input dates would likely help with this.</p><p>With all that said, putting this app together was a great intro to Mapbox and the public Wikipedia databases, as well as a fun exercise in React, and I hope that others will find these resources as fun to build with as I did.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26be1726a3a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/creating-interactive-historical-maps-with-react-mapbox-and-wikipedia-26be1726a3a7">Creating Interactive Historical Maps With React, Mapbox, and Wikipedia</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make Your Own Investing Web Portal With Python and Django]]></title>
            <link>https://medium.com/@rdcolema7/make-your-own-investing-web-portal-with-python-and-django-89feb5ce0ddd?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/89feb5ce0ddd</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Wed, 01 Jan 2020 21:51:03 GMT</pubDate>
            <atom:updated>2020-05-03T22:46:59.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSizhMtC3xBXeeLGIZgYuA.png" /></figure><p>Sometimes the hardest part of personal finance is knowing what you have. There’s a reason that apps like Mint and have become so popular — nobody wants to log in to a dozen different two-factor-authenticated accounts every time they want an update on their financial situation.</p><p>But consolidation sometimes comes at the expense of detail, particularly when it comes to monitoring and analyzing your stock portfolio across accounts. There are countless portfolio management tools on the market, but in this post I’ll demonstrate how you can take matters into your own hands and create your own web app for tracking your investments.</p><p>The advantages to this approach are three-fold:</p><ol><li>You can incorporate positions from any account, and you don’t need to worry about facilitating integrations between accounts (or the associated security risks there)</li><li>You can add whatever features, analyses, or visualizations you want to your dashboards</li><li>You can choose your data clients, so you can refresh data in real time if desired</li></ol><p>Plus, it’s free. So how can you get started?</p><h4>Creating a Django Project</h4><p>I’ve always leaned towards the Django web framework for prototyping websites; it’s clean, it’s easy to integrate with databases, it makes front-end templating easy, and it’s built with python, making it approachable for developers of all levels.</p><p>A typical project structure should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*DjFejnJZvl8-MQJ0VBT4-Q.jpeg" /></figure><p>There are two applications for this example portfolio tracker: “positions” and “analysis.” The former handles views and logic for the base metrics (quotes, cost basis, etc) and the latter handles more in-depth analysis and data visualization. There’s also a project directory to manage settings, and a home directory to store common front-end assets.</p><p>There are scores of Django tutorials available online if you want to start from scratch or organize your applications differently, and I’ve also made my code for this demo available <a href="https://github.com/rdcolema/snapshot-finance">here</a> for a quick start. For the purposes of this post, I’ll mostly be discussing the “positions” app, but the code for the “analysis” app is also available via the github repo in the link above.</p><h4>Configuring the Project</h4><p>Before testing the app in your local browser there are a few configurations that need to be added in your project’s “settings.py” file. Any applications need to be registered in the INSTALLED APPS lists (i.e. “positions” or “analysis”), and any databases used will need to be configured here. For a SQLite database stored in the project’s root directory, it might look like this:</p><pre>DATABASES = {<br>    &#39;default&#39;: {<br>        &#39;ENGINE&#39;: &#39;django.db.backends.sqlite3&#39;,<br>        &#39;NAME&#39;: os.path.join(BASE_DIR, &#39;db.sqlite3&#39;),<br>    }<br>}</pre><p>There are many Django database options available, but I’m using SQLite for this demo because it’s lightweight and easy to get up and running quickly for a proof of concept.</p><h4>Data Modeling</h4><p>Django makes working with databases easy through its object-relational mapping (ORM) layer. This means that for each application in your project, you’ll want to create a series of models to represent your data. Django manages these models through a “models.py” file where models are defined as classes that inherit from Django’s base Models class. Let’s look at an example for the “positions” app:</p><pre>from django.db import models<br><br><br>class Account(models.Model):<br>    &quot;&quot;&quot;A brokerage account&quot;&quot;&quot;<br><br>    ACCOUNT_TYPES = (<br>        (&#39;TRADITIONAL&#39;, &#39;Traditional IRA or 401(k)&#39;),<br>        (&#39;ROTH&#39;, &#39;Roth IRA or 401(k)&#39;),<br>        (&#39;STANDARD&#39;, &#39;Standard Brokerage&#39;),<br>    )<br><br>    name = models.CharField(max_length=100)<br>    account_type = models.CharField(max_length=100, choices=ACCOUNT_TYPES)<br>    cash_balance = models.FloatField(default=0.0)<br><br>    def __str__(self):<br>        return self.name<br><br><br>class Position(models.Model):<br>    &quot;&quot;&quot;A position, such as an equity holding&quot;&quot;&quot;<br><br>    name = models.CharField(max_length=100)<br>    symbol = models.CharField(max_length=10)<br>    shares = models.FloatField()<br>    cost_basis = models.FloatField(default=0.0)<br>    account = models.ForeignKey(&quot;Account&quot;, on_delete=models.CASCADE)<br><br>    def __str__(self):<br>        return self.symbol</pre><p>Here there are two main entities, an Account and a Position tied to an account. The choices for attributes and relations here are flexible, but think about how you’d want to query this data later (i.e. “give me all positions in my 401(k) where I’ve invested more than $5k”). For example, you could restructure the Position model to account for purchase blocks rather than cumulative holdings, or modify it to work for fixed income and commodity assets.</p><p>Once you’re satisfied with the data model for your project, <a href="https://docs.djangoproject.com/en/3.0/topics/migrations/">migrate</a> your changes and verify that the modifications are reflected in your local database.</p><h4>URL Routing</h4><p>Any website you create needs to know what assets to associate with which URL path on your site. Fortunately, defining this is in Django is a simple process, and can be done via a “urls.py” file — typically these are set at both the project level (in <em>&lt;project_root&gt;/&lt;project_name&gt;/urls.py</em>) and for each application in the project (in <em>&lt;project_root&gt;/&lt;app_name&gt;/urls.py</em>). The former defines resolution for the root URL, the admin URL, and for each app’s base URL:</p><pre>from django.conf.urls import include, url<br>from django.contrib import admin<br>from home import views<br></pre><pre>urlpatterns = [<br>    url(https://medium.com/r&#39;^r&#39;^$&#39;, views.index, name=&#39;index&#39;#39;,%20views.index,%20name=&#39;index&#39;),<br>    url(https://medium.com/r&#39;^analysis/&#39;,%20include(&#39;analysis.urls&#39;)),<br>    url(https://medium.com/r&#39;^positions/&#39;,%20include(&#39;positions.urls&#39;)),<br>    url(https://medium.com/r&#39;^admin/&#39;,%20admin.site.urls),<br>]</pre><p>The latter sets routes within a specific application, such as this simple configuration for the “Positions” app, which only has one view:</p><pre>from django.conf.urls import url<br>from . import views</pre><pre>urlpatterns = [<br>    url(https://medium.com/r&#39;^r&#39;^$&#39;, views.index, name=&#39;index&#39;#39;,%20views.index,%20name=&#39;index&#39;),<br>]</pre><h4>Adding Accounts and Positions Through Django Admin</h4><p>In order to customize which positions will be displayed through your app, you’ll need to populate the database your app connects to. An easy way to do this is via Django admin.</p><p>To access the admin section of your site locally, you’ll need to navigate to <a href="http://127.0.0.1:8000/positions/">localhost:8000/admin</a> and log in as a superuser. There you’ll find an intuitive UI for modifying your project’s back-end data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*TbByCCLHa6nQw57EPtFimQ.jpeg" /></figure><p>Before any models will show up in Django admin however, you need to explicitly register them for each application that contains its own models. This is done in an “admin.py” file in the application directory:</p><pre>from django.contrib import admin<br>from .models import Account, Position</pre><pre>admin.site.register(Account)<br>admin.site.register(Position)</pre><p>After that code is added, refreshing the admin page should display the registered models and allow you to modify their instances.</p><p>For this example, you’ll first need to add one or more accounts to hold new positions; then, when populating your positions, you can simply link them to an account via the form for adding a new position.</p><h4>Designing the Back End</h4><p>So now that you have a framework for organizing your investments, how can you link them to external data? For most use cases, you’ll need to:</p><ol><li>Request data from an API, or scrape it from the web</li><li>Parse and transform that data to make it usable for your front-end views.</li></ol><p>I found the <a href="https://iexcloud.io/docs/api/">IEX API</a> to be a practical (and free) way to request real-time stock data at low-volume, but the options for financial data via the web are deep. Scraping this data from sites like <a href="https://finance.yahoo.com/">Yahoo! Finance</a> is relatively straightforward, and API providers like <a href="https://www.quandl.com/">Quandl</a> provide a wealth of financial and economic metrics.</p><p>It’s usually a good idea to separate the main functional code blocks into separate modules rather than trying to cram them into Django’s views themselves, so in my example app I created a “portfolio_utils.py” file to hold all the scripts for requesting and parsing external stock data from the web. The full utils file can be found <a href="https://github.com/rdcolema/snapshot-finance/blob/master/positions/portfolio_utils.py">here</a> for examples on how to build dataframes of real-time stock data with the IEX API — later in this post, I’ll discuss how you can pass this data to the front-end templates on your site.</p><h4>Designing the Front End</h4><p>Django manages its front-end code via a templates directory for each installed app. In each of these directories, you’ll add the HTML files to be rendered by the views within that app. For the “positions” app, this is essentially just a space to display the tabular portfolio data that’s been pulled from the client and handled by your back-end code. There are a few extra pieces you might want to add, however, both for style and for functionality.</p><p>For instance, in the demo <a href="https://github.com/rdcolema/snapshot-finance/blob/master/positions/templates/positions/index.html">code</a> for this post, I added table styling, javascript for account filtering, position counts, cash totals, and more. I also organized a lot of reused HTML and javascript code into a <a href="https://github.com/rdcolema/snapshot-finance/blob/master/home/templates/home/base.html">base template</a> that can be reused across the site by adding a Django template command at the beginning of each template that will extend it: {% extends &quot;home/base.html&quot; %}. You can read more about Django’s template language <a href="https://docs.djangoproject.com/en/3.0/ref/templates/language/">here</a>.</p><h4>Writing Views</h4><p>Regardless of how much you customize these templates, the data they have access to is dictated by the code you provide in your Django views. These are typically classes or functions in a “views.py” file for each application in your project that take in a request, and return a response along with some context associated with that response. For the “Positions” app, the view should return the data needed to populate a table with all holdings across accounts, along with any additional context desired, such as cash balances.</p><pre>from django.shortcuts import render<br>from .models import Account, Position<br>from . import portfolio_utils as pu<br><br><br>def index(request):<br>    &quot;&quot;&quot;Primary view of positions held&quot;&quot;&quot;<br><br>    positions_summary = pu.get_position_summary(Position.objects.all())<br>    accounts = Account.objects.all()<br>    total_cash = sum((acct.cash_balance for acct in accounts))<br><br>    context = {<br>        &quot;positions&quot;: positions_summary.to_html(index=False, float_format=lambda x: &#39;%.2f&#39; % x),<br>        &quot;accounts&quot;: [acct.name for acct in accounts],<br>        &quot;cash_balances&quot;: {acct: acct.cash_balance for acct in accounts},<br>        &quot;total_cash&quot;: &quot;{:,.2f}&quot;.format(total_cash),<br>        &quot;total_value&quot;: &quot;{:,.2f}&quot;.format(total_cash + positions_summary[&quot;Market Value ($)&quot;].iloc[0]),<br>        &quot;num_positions&quot;: positions_summary.shape[0] - 1,<br>    }<br><br>    return render(request, &quot;positions/index.html&quot;, context)</pre><p>Recall that while routing URLs for the project earlier in this post, the root URL for the positions app was set to resolve to views.index — now that this view actually exists, you can navigate back to <a href="http://127.0.0.1:8000/positions/">localhost:8000/positions</a> in your local browser and the page should render successfully. Any data passed to this page via the keys and values in the context dictionary of your view can now be referenced on the template using Django’s template language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_NX861pvH-IY3NHm8TRmw.jpeg" /></figure><h4>What Next?</h4><p>Having your financial data organized in a platform that you own and control provides a lot of opportunities for future features. Anything from watchlists to price alerts to retirement projections becomes achievable once you have a way to aggregate and visualize your holdings alongside real-time market data, so the benefits to tackling a project like this one are manifold, even if you start with very basic functions.</p><p>And if you put enough work in, you’ll start spending less time with paper statements and password resets, and more time kicking yourself for not buying AMZN in 2001.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89feb5ce0ddd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying GPT-2 Models in Custom Applications]]></title>
            <link>https://medium.com/@rdcolema7/deploying-gpt-2-models-in-custom-applications-f8117e482837?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8117e482837</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Sun, 10 Nov 2019 04:00:03 GMT</pubDate>
            <atom:updated>2019-11-10T04:00:03.933Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever since OpenAI showed off their <a href="https://openai.com/blog/better-language-models/">GPT-2 language model</a> earlier this year, the machine learning community has been abuzz with demonstrations of the model’s potential. Where other language models in the past have struggled with issues of coherence and long-term dependencies over passages of text, GPT-2 is capable of generating convincing output throughout multiple paragraphs.</p><p>Until <a href="https://openai.com/blog/gpt-2-6-month-follow-up/">recently</a>, OpenAI’s fully trained model was kept under wraps, but lighter-weight versions trained on more limited datasets have been available for some time, making it easy to generate fun examples in auto-completion and Q&amp;A contexts.</p><h4>The Issue of Integration</h4><p>GPT-2 training and sampling tutorials are a dime a dozen, but less has been written about how you might actually incorporate a trained model into your own applications or websites. In this post I’ll walk through a sample integration with a Slackbot app (see my <a href="https://medium.com/@rdcolema7/building-a-python-slackbot-that-actually-works-2930663e20f3">previous post</a> for details on how to create your own) which will provide an easy interface for interacting with the model. I’ll be sampling a few different datasets as demos, including OkCupid dating profiles, rap lyrics, Trump speeches, and Ruby source code, but you can supply any training data you want for your own application.</p><h4>Creating a Training Environment</h4><p>First off, there are a handful of prerequisites and caveats for any deep-learning project involving sufficiently large data. To keep training times practical, you’ll likely want to make use of GPU acceleration — if you don’t have experience getting set up with CUDA for an ML framework like PyTorch or TensorFlow there are many tutorials available online, but that won’t be the focus of this post.</p><p>I will, however, specify the package versions I tested with, along with my CUDA and cuDNN versions, since getting the right combinations working there can be a bit tricky. These are listed in the “requirements.txt” file of the <a href="https://github.com/rdcolema/gpt2-slackbot">github repo</a> accompanying this writeup. I use Anaconda to manage my environments, but any Python virtual env tool should work.</p><p>In terms of project structure, I’m keeping the source and util files for the GPT-2 model in their own module, and I’m using separate directories for models, checkpoints, and samples, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*MBHc3jkbYn00SEPpHxLmbw.jpeg" /></figure><h4>The Training Strategy</h4><p>In order to get the best results on your custom dataset, which is most likely much, much smaller than the data OpenAI trained their model on, you can implement a shortcut known as fine-tuning. Fine-tuning is a technique that takes a pretrained model and updates the weights of that model via iterative training on a new dataset, rather than initializing the weights for the new model from scratch; this usually enables gains in both required training time and performance on new data. In other words, fine-tuning can produce high-quality results from much less input data, allowing you to eke out realistic generative models from only a few MB of training data.</p><p>A good starting point for the pretrained model in this case would be the 124 million parameter model open-sourced by OpenAI in February 2019 (Note: due to an early naming error this model is sometimes referred to as the 117M parameter model). The model can be downloaded <a href="https://github.com/openai/gpt-2/blob/master/download_model.py">here</a> or via the “download_model.py” script in the accompanying <a href="https://github.com/rdcolema/gpt2-slackbot">github repo</a>.</p><p>Fortunately, once you’ve downloaded the pretrained model, the heavy-lifting is done; all you need to supply is your own raw text to use for fine-tuning. Depending on your application, this could be as simple as a single file containing unstructured text data, or it could be a more specialized dataset containing question/answer pairs, stanzas of poetry, code blocks, ascii art, etc. The input format shouldn’t substantially change the training methodology however, since GPT-2 is capable of learning the long-term dependencies required for structured text representations.</p><p>When you’re ready to start training, provide a path to your data and a name for your custom model and kick off the fine-tuning process:</p><pre><strong>$ python train.py --dataset /path/to/dataset.txt --run_name &lt;your_custom_dataset_name&gt;</strong></pre><p>While the model is training, it will intermittently generate samples and save checkpoints so that you can monitor progress and resume training if interrupted. These outputs will show up in the “samples/” and “checkpoints/” directories respectively.</p><h4>Testing Your Model</h4><p>Once you have some checkpoints populated in your new model directory, you can generate samples from that model using the “generate_samples.py” script:</p><pre><strong>$ python generate_sample.py --dataset /path/to/dataset.txt --run_name &lt;your_custom_model_name&gt;</strong></pre><p>This will print the sample output to the console, but the script should be easy to modify to write to disk if preferred. The script also accepts an optional “sample_length” parameter if you’d like to produce shorter or longer samples.</p><h4>Adding Pipelines to Your Application</h4><p>Now that you have your custom model working, you’ll need to make a few different pieces available to the application you’d like to integrate it with:</p><ol><li>The checkpoint directory for your custom model</li><li>The model hyperparameters, which are included as a JSON file with the downloaded pretrained model)</li><li>Prime text, for the model to use as a starting point when predicting the sequence that follows it</li><li>An encoder, to convert the raw primetext string into the byte-pair-encoded format required by the model, and then to subsequently decode the BPE output back to a human-readable string</li></ol><p>Putting it altogether, a fully-connected sampling function might look something like the following code block (omitting imports, helper functions, etc):</p><pre>def generate_samples(run_name, sample_len=250, prime_text=None):<em><br>    </em>enc = get_encoder(&#39;117M&#39;)<br>    hparams = default_hparams()<br>    with open(os.path.join(MODEL_DIR, &#39;117M&#39;, &#39;hparams.json&#39;)) as f:<br>        hparams.override_from_dict(json.load(f))<br><br>    config = tf.ConfigProto()<br>    config.gpu_options.allow_growth = True<br><br>    with tf.Session(config=config) as sess:<br>        context = tf.placeholder(tf.int32, [1, None])<br><br>        tf_sample = sample_sequence(<br>            hparams=hparams,<br>            length=sample_len,<br>            context=context,<br>            batch_size=1,<br>            temperature=1.0,<br>            top_k=40<br>        )<br><br>        train_vars = [v for v in tf.trainable_variables() if &#39;model&#39; in v.name]<br><br>        saver = tf.train.Saver(<br>            var_list=train_vars,<br>            max_to_keep=5,<br>            keep_checkpoint_every_n_hours=2)<br>        sess.run(tf.global_variables_initializer())<br><br>        ckpt = tf.train.latest_checkpoint(os.path.join(CHECKPOINT_DIR, run_name))<br><br>        # restore saved weights<br>        saver.restore(sess, ckpt)<br><br>        # generate sample from model<br>        chunks = load_dataset(enc, DATASET_DIR.format(run_name, run_name), 50000)<br>        data_sampler = Sampler(chunks)<br><br>        if prime_text:<br>            context_tokens = enc.encode(prime_text)<br>        else:<br>            context_tokens = data_sampler.sample(1)<br><br>        out = sess.run(tf_sample, feed_dict={context: [context_tokens]})<br>        text = enc.decode(out[0])<br>        sess.close()<br><br>        # strip prime text from response (if desired)<br>        if prime_text and text.startswith(prime_text):<br>            text = text.split(prime_text)[1]<br><br>        return text</pre><p>This function can then be referenced elsewhere in the application with the optional length and primetext arguments supplied…</p><pre>generate_samples(“&lt;custom_model_name&gt;”, sample_len=100, prime_text=command)</pre><p>…which will return the decoded text output.</p><h4>Talking to Your Models</h4><p>Finally, the fun part — chatting with the models. Here are a few random samples using the datasets I chose for this post.</p><p>First, a cringe-worthy dating profile learned from the OkCupid profiles dataset (27.7 MB, 3k training steps):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*iTDviuyxw5dCNGp2_tIG5w.jpeg" /></figure><p>Next, some freestyling based on the rap lyrics dataset (80.9 MB, 5k training steps):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*jBpKsbtL73OvgkQ_lU2xPg.jpeg" /></figure><p>This particular model has a bit of a fresh mouth, so I had to do some censoring.</p><p>Then an obvious choice, a model trained on Trump’s public speeches (13.7 MB, 6k training steps):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ixPbMPLd8vQCbKHgGB0Rcw.jpeg" /></figure><p>And then last but not least, a more unique demonstration of GPT-2’s ability to learn structure in data — a model trained on the Ruby language’s source code downloaded from github (33.4 MB, 2k training steps):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*CErpPbIJLb4YH6hKgSg4zg.jpeg" /></figure><h4>Improvements, Scaling, Optimization</h4><p>This post described a very quick and dirty way to get a model up and running in a simple app, but there’s tons of low-hanging fruit to optimize how models are served up in your own projects.</p><p>First and foremost, speed — in this demo there’s a lot of repetition of object-loading (encoders, parameters, etc) that could likely be cached so that reusing a model doesn’t start the process from scratch.</p><p>Then, there are many opportunities to add custom logic and heuristics for controlling the prime text that’s sent to a model, and how its response incorporates that prime text. This could make the interactions themselves much smoother.</p><p>It also might make sense to create more distinct boundaries between your app and the service that handles the model’s input and output; I’ve kept the scripts packaged together for simplicity, but in a more complex project that might lead to difficulties later on.</p><p>Finally, in a frequently-used model, you might want to turn new inputs from users into an online learning process, where the model learns continuously from new data. This would require some more substantial modifications, but could provide worthwhile gains for cases where additional training data is valuable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8117e482837" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building A Python Slackbot That Actually Works]]></title>
            <link>https://medium.com/@rdcolema7/building-a-python-slackbot-that-actually-works-2930663e20f3?source=rss-f96f279bd2e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2930663e20f3</guid>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Robert Coleman]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 12:12:20 GMT</pubDate>
            <atom:updated>2019-11-09T20:10:16.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*yeIauTTyld2jt34iKuYr9g.png" /></figure><p>If you’re a Slack user, you’re probably no stranger to the various integrations floating around your workspace — webhooks, reminders, slash commands, apps, alerts, you name it. But it’s less common to find well-functioning bots that do much beyond delivering weather reports or canned responses to narrow inputs.</p><p>In this post I will share an approach to designing a more all-purpose python Slack integration, with use cases that hopefully surpass sheer novelty and extend into the realm of everyday utility. My goal is less to provide a finished product (though the code for this project is available <a href="https://github.com/rdcolema/slackbot">here</a>) and more to provide a framework for designing flexible bot applications with modular, customizable functionality.</p><h4>Creating a New App</h4><p>Before you start coding, you’ll need to create an app for your workspace, which you can do via this <a href="https://api.slack.com/start/overview">page</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*GMmth-6SZqbWToUenHeWzQ.jpeg" /></figure><p>I won’t spend too much time going step-by-step through the various configurations that can be set here, but be sure that you:</p><ol><li>Add a bot user for the app</li><li>Install the app</li><li>Note the bot user OAuth access token (found in the “OAuth &amp; Permissions” tab), since you’ll need this later</li></ol><p>After adding the integration, you should see a new entry in the “Apps” section of your Slack workspace with the display name you chose for your bot.</p><h4>Installing Dependencies</h4><p>The python packages used will vary based on your bot’s features. I’ve included a <em>requirements.txt</em> file in the github repo for this demo (installed in a python 3.7 virtual environment), but with the exception of <a href="https://github.com/slackapi/python-slackclient">slackclient</a> most of the names there are not strictly needed for a functioning bot. The slackclient library can be installed via pip or <a href="https://anaconda.org/conda-forge/slackclient">conda-forge</a>.</p><h4>Project Structure</h4><p>Now that your app and the dependencies are installed, you’ll need a place to hold the code that will control it. For this generic template, I put my project in a new directory called “slackbot” with a simple one-dimensional layout:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*CQFT-BbZDXonI3By17AvGw.png" /></figure><p>The bot’s main processes will be run out of a single python file, which I’ve called <em>slackbot.py</em> in my example, but you could also name this file something like <em>main.py</em> or <em>&lt;the_name_of_your_bot.py&gt;</em>. This is where you’ll use the slackclient python package installed earlier to connect to Slack’s API and monitor the data flow in your workspace in real time.</p><p>In the same directory, add three other files:</p><ol><li><em>triggers.py</em> for parsing and prioritizing commands</li><li><em>response_map.py</em> for mapping commands to responses</li><li><em>responses.py</em> for holding the bot’s actual response functions</li></ol><p>More on each of these files later.</p><h4>Running Your Bot</h4><p>At the top of <em>slackbot.py</em>, we’ll add the following imports…</p><pre>import os <br>import time </pre><pre>from slackclient import SlackClient</pre><p>…followed by a few global variables and a line instantiating the SlackClient class used to interact with Slack’s API:</p><pre># Bot settings &amp; connection handlers<br>BOT_NAME = &quot;&lt;your_bot_name&gt;&quot;<br>slack_client = SlackClient(os.environ.get(&quot;SLACKBOT_TOKEN&quot;))</pre><p>I’ve set my personal SLACKBOT_TOKEN as an environment variable to avoid posting it here, but this value should correspond to the bot user OAuth access token you noted in the “OAuth &amp; Permissions” tab earlier.</p><p>Next you need to define the main loop that will continuously poll your workspace for activity:</p><pre>if __name__ == &quot;__main__&quot;:<br>    if slack_client.rtm_connect():<br>        print(f&quot;{BOT_NAME} now online.&quot;)<br><br>        while True:<br>            text_input, channel = <strong>parse_slack_output</strong>(slack_client.rtm_read())<br>            if text_input and channel:<br>                <strong>handle_command</strong>(text_input, channel)<br>            time.sleep(1)  # websocket read delay<br><br>    else:<br>        print(&quot;Connection failed.&quot;)</pre><p>The above code runs an infinite loop that queries Slack’s real time messaging API for output, parses commands from the output, and then sends the commands off to be handled. The parse_slack_output and handle_command functions highlighted above still need to be defined and implemented, however.</p><h4>Parsing Output From the Real Time Messaging API</h4><p>Data from the real time messaging API is returned as a python dictionary object, which makes searching for metadata about the incoming messages pretty straightforward. Let’s look for objects with a ‘text’ key. These can be checked for commands directed at our bot.</p><pre>def parse_slack_output(slack_rtm_output):<br>    <em>&quot;&quot;&quot;Parses output data from Slack message stream&quot;&quot;&quot;<br><br>    </em># read data from slack channels<br>    output_list = slack_rtm_output<br><br>    if output_list and len(output_list) &gt; 0:<br>        for output in output_list:<br><br>            if output and &#39;text&#39; in output:<br>                text = output[&#39;text&#39;]<br><br>                # if bot name is mentioned, take text to the right of the mention as the command<br>                if BOT_NAME in text.lower():<br>                    return text.lower().split(BOT_NAME)[1].strip().lower(), output[&#39;channel&#39;]<br><br>    return None, None</pre><p>How you want to establish the syntax for interacting with your bot is up to you — here we’re just taking any text that comes after the bot’s name and considering that a command to the bot. It’s also helpful to track the channel that the message came from (using the ‘channel’ key in the output dict), which is why this function returns a tuple instead of a single object.</p><h4>Handling Commands</h4><p>Now let’s look at one way we can take the raw text passed to the bot and determine which responses the bot should return. In order to help with this we’ll introduce the three other modules mentioned earlier, <em>triggers.py</em>,<em> response_map.py</em>, and<em> responses.py</em>.</p><p><strong>triggers.py:</strong></p><pre>import re<br><br><br>match_triggers = (<br>    (re.compile(&quot;(hey|hi|hello|howdy|greetings)&quot;), &quot;hello&quot;),<br>    (re.compile(&quot;define&quot;), &quot;define&quot;),<br>    (re.compile(&quot;quote&quot;), &quot;quote&quot;)<br>)<br><br>search_triggers = (<br>    (re.compile(&quot;weather&quot;), &quot;weather&quot;),<br>    (re.compile(&quot;comic&quot;), &quot;comic&quot;),<br>    (re.compile(&quot;joke&quot;), &quot;joke&quot;),<br>    (re.compile(&quot;pics&quot;), &quot;pics&quot;),<br>    (re.compile(&quot;(search)|(look up)|(google)&quot;), &quot;search&quot;),<br>    (re.compile(&quot;synonyms&quot;), &quot;synonyms&quot;),<br>    (re.compile(&quot;(wordcount)|(word count)&quot;), &quot;wordcount&quot;),<br>    (re.compile(&quot;(play|video|youtube)&quot;), &quot;youtube&quot;),<br>)<br><br><br>def get_response_key(command, regex_type=&#39;match&#39;):<br>    regex = re.match if regex_type == &#39;match&#39; else re.search<br>    lookup = match_triggers if regex_type == &#39;match&#39; else search_triggers<br>    for key, value in lookup:<br>        if regex(key, command):<br>            return value<br>    return None</pre><p>The above implementation allows some commands to be require exact matches (i.e. “slackbot define &lt;word&gt;”) and others simply to require keywords (i.e. “slackbot tell me a good joke”). Introducing regex means it should be possible to define even complex pattern matching for your commands, and you can also order the patterns to determine the priority of potential matches.</p><p><strong>response_map.py:</strong></p><pre>import responses as rsp<br><br><br>response_dict = {<br>    &#39;comic&#39;: [<br>        rsp.comic<br>    ],<br>    &#39;define&#39;: [<br>        rsp.define<br>    ],<br>    &#39;hello&#39;: [<br>        &quot;Sup&quot;,<br>        &quot;hey guys&quot;,<br>        &quot;yo&quot;,<br>        &quot;what&#39;s the word&quot;,<br>        &quot;hi&quot;<br>    ],<br>    &#39;joke&#39;: [<br>        rsp.joke<br>    ],<br>    &#39;no_command&#39;: [<br>        &quot;I don&#39;t know what you want from me&quot;,<br>    ],<br>    &#39;pics&#39;: [<br>        rsp.get_pic<br>    ],<br>    &#39;quote&#39;: [<br>        rsp.get_quote<br>    ],<br>    &#39;search&#39;: [<br>        rsp.search_words<br>    ],<br>    &#39;synonyms&#39;: [<br>        rsp.synonyms<br>    ],<br>    &#39;weather&#39;: [<br>        rsp.check_weather<br>    ],<br>    &#39;wordcount&#39;: [<br>        rsp.word_count<br>    ],<br>    &#39;youtube&#39;: [<br>        rsp.get_video<br>    ],<br>}</pre><p>For details on how the example functions provided in <em>responses.py</em> are implemented you can reference the <a href="https://github.com/rdcolema/slackbot/blob/master/responses.py">github repo</a> for this sample bot, but the main idea is that you can provide a flexible lookup mechanism mapping commands to defined responses. An example response for, say, the ‘joke’ command above might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*LvrV9AeY8MhG0Vx1OMYDlA.png" /></figure><h4>Putting It All Together</h4><p>Finally, back in <em>slackbot.py</em> you can import the new modules and use them in a handle_command function to look up the proper response mapping, fetch the response itself, and then post that response to the target channel via the “chat.postMessage” endpoint. In this example, we check for exact matches first before searching keywords. I also set some default behaviors and define distinct logic between simple string responses and callable responses (namely the functions in <em>responses.py</em>).</p><pre># add these imports to the top of slackbot.py<br>from response_map import response_dict<br>from triggers import get_response_key<br></pre><pre>def handle_command(command, channel):<br>    <em>&quot;&quot;&quot;Take cleaned command, look up response, and post to channel&quot;&quot;&quot;<br><br>    </em># Check for exact matches first then check for contained keywords<br>    response_key = get_response_key(command, regex_type=&#39;match&#39;) or get_response_key(command, regex_type=&#39;search&#39;)<br><br>    # Default behavior if no response keys match the command<br>    if not (response_key or command):<br>        # If no command is given<br>        response_key = &#39;no_command&#39;<br>    elif not response_key:<br>        response_key = &#39;search&#39;<br>        command = &#39;search &#39; + command<br><br>    response = random.choice(response_dict[response_key])<br><br>    # If response is a function, call it with command as argument<br>    if callable(response):<br>        response = response(command)<br><br>    slack_client.api_call(&quot;chat.postMessage&quot;, channel=channel, text=response, as_user=True)</pre><h4>Running Your Bot</h4><p>Once all your command triggers and responses are defined above the main control loop in <em>slackbot.py</em>, you should be able to start the bot straight from the command line with:</p><pre><strong>$ python slackbot.py</strong></pre><p>If all goes well, a message will be logged to the console saying the bot is online, and the app in your Slack workspace should display a green icon next to its name indicating it’s online. From there, you’ll just need to invite the bot to any channel where you’d like for your app to listen for commands.</p><h4>What Next?</h4><p>Of course, for the app to persist past your shell session, you’ll need to host it somewhere more permanent, but for development purposes it’s useful to test a variety of interactions with the bot in a local environment.</p><p>For a more robust and secure app, you’ll also want to implement much more thorough logging and exception handling, and think more carefully about how your app processes raw text — from both a security standpoint and a functionality standpoint, it’s important to consider all the various ways that raw input strings can be abused. This is particularly true if your Slack workspace is open to the public.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2930663e20f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>