<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tristan Crockett on Medium]]></title>
        <description><![CDATA[Stories by Tristan Crockett on Medium]]></description>
        <link>https://medium.com/@tristancrockett?source=rss-459afaeb38d0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UEvouZ5rt3NI4vol.</url>
            <title>Stories by Tristan Crockett on Medium</title>
            <link>https://medium.com/@tristancrockett?source=rss-459afaeb38d0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 10:36:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tristancrockett/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimizing pandas.read_sql for Postgres]]></title>
            <link>https://medium.com/data-science/optimizing-pandas-read-sql-for-postgres-f31cd7f707ab?source=rss-459afaeb38d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f31cd7f707ab</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tristan Crockett]]></dc:creator>
            <pubDate>Sun, 17 Mar 2019 17:28:23 GMT</pubDate>
            <atom:updated>2019-03-17T23:30:45.605Z</atom:updated>
            <content:encoded><![CDATA[<p>Reading SQL queries into Pandas dataframes is a common task, and one that can be very slow. Depending on the database being used, this may be hard to get around, but for those of us using Postgres we can speed this up considerably using the COPY command. However, there are several ways of using the COPY command to get data from SQL into pandas, with different memory/speed tradeoffs. In this article, we’ll test several different methods against each other.</p><p>The test dataset is simply the first five million rows of a sample <a href="http://github.com/dssg/triage">Triage</a> predictions table, which is just one I had handy. I tried to use all thirteen million rows I had in my local Postgres database, but pandas.read_sql crashed so I decided to bring down the dataset to something it could handle as a benchmark.</p><p>Included which each method are three statistics:</p><p><strong>Peak memory</strong> — the highest amount of memory used during the sql read code. This is the important one to see if your program will crash!</p><p><strong>Increment memory</strong> — the amount of memory that is still used at the end of the sql read code. In theory, this would be the same for all of the methods, but memory leaks can make different methods retain more memory.</p><p><strong>Elapsed time</strong> — the clock time used by the program.</p><p>The pandas version used here is 0.24.1.</p><p>First, a quick rundown of the different methods being tested:</p><ul><li>pandas.read_sql — the baseline</li><li>tempfile — Using the tempfile module to make a temporary file on disk for the COPY results to reside in before the dataframe reads them in</li><li>StringIO — Using a StringIO instead of disk; more memory used, but less disk I/O</li><li>Compressed BytesIO, pandas decompress — Using a BytesIO instead of a StringIO, and compressing the data; should use less memory, but take longer</li><li>Compressed BytesIO, gzip decompress — Same as the other compressed bytesio, but using GzipFile to decompress rather than pandas</li><li>Compressed tempfile — Applying the compression idea to the diskfile; should reduce the disk I/O needed</li><li>Compressed BytesIO, low compression level — A lower compression level to try and split the difference between uncompressed and compressed methods</li></ul><h4>pandas.read_sql</h4><p>This is the baseline. Nothing fancy here.</p><p><strong>Peak memory: 3832.7 MiB / Increment memory: 3744.9 MiB / Elapsed time: 35.91s</strong></p><h4>Using a temporary file</h4><p>Here is our first attempt of using the COPY command. The data from the COPY command has to go do a filehandle: what simpler way to do this than using a temporary file?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29a63094a5860440b68ba188a4c0dea1/href">https://medium.com/media/29a63094a5860440b68ba188a4c0dea1/href</a></iframe><p><strong>Peak memory: 434.3 MB / Increment memory: 346.6 MB / Elapsed time: 8.93s</strong></p><p>That’s…much better. I’m not surprised that the elapsed time is far quicker than read_sql, but I’m a bit surprised that the memory usage is so much different. Anyway, let’s keep going</p><h4>Using a StringIO</h4><p>Disk I/O can be expensive, especially depending on what type disk is available. Can we speed it up by using a StringIO for the filehandle? This would take more memory, of course, but maybe that’s a tradeoff we can make.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfcde60b3a7510322a3a115a4c3f4f2f/href">https://medium.com/media/bfcde60b3a7510322a3a115a4c3f4f2f/href</a></iframe><p><strong>Peak memory: 434.2 MB / Increment memory: 346.6 MB / Elapsed time: 9.82s</strong></p><p>This is a surprising result. I would have expected this to use more memory and be faster, but it’s neither. My hypothesis would be that the peak memory used by the StringIO ends up being surpassed by a spike during the dataframe creation process.</p><p>Also of note: the increment memory was the same as the temporary file version, which probably tells us that the 346.6 MB is a good reference for what the baseline for that memory should be without any memory leaks.</p><h4>Using a Compressed BytesIO, pandas decompression.</h4><p>Could we bring down on the memory required for the in-memory option? Given the prior results this may seem like a fool’s errand, but I already wrote the code so I’m not going to cut off the test early! Python’s GzipFile interface wraps a filehandle (in this case, a BytesIO) and handles compression. We let pandas handle the decompression by passing `compression=’gzip’` to read_csv</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc599866321061a1c2aaae4388a0b1c7/href">https://medium.com/media/bc599866321061a1c2aaae4388a0b1c7/href</a></iframe><p><strong>Peak memory: 613.6 MB Increment memory: 525.8 MB, Elapsed time: 1:30m</strong></p><p>Not good! It actually used more memory (and leaked some) compared to the uncompressed versions.</p><h4>Using a Compressed BytesIO, Gzip decompression</h4><p>Same as the last one, except we bypass pandas’ decompression routines in case they introduced a problem. GzipFile can handle the decompression for us, too!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20a2c5c45f226f5e74213078ab579c41/href">https://medium.com/media/20a2c5c45f226f5e74213078ab579c41/href</a></iframe><p><strong>Peak memory: 504.6 MB Increment memory: 416.8 MB, Elapsed time: 1:42m</strong></p><p>Well, this is better RAM-wise than the pandas decompression, for sure, but this is still worse than the uncompressed versions.</p><h4>Using a Compressed tempfile</h4><p>The compression idea can also apply to the tempfile method from before. In this case, compression should help us cut down on disk I/O.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acbd4c67a5326f9aec94dd0d7a31b012/href">https://medium.com/media/acbd4c67a5326f9aec94dd0d7a31b012/href</a></iframe><p><strong>Peak memory: 517.2 MB Increment memory: 429.5 MB, Elapsed time: 1:35m</strong></p><p>Similar to the other gzip examples. Not a good option.</p><h4>Using a compressed BytesIO, low compression level</h4><p>Since we’re trying things out, we have one more avenue to explore: the gzip compression level. The default for all of the prior examples is 9, the highest compression possible. It’s possible that in doing this, it takes extra memory to do the compression in addition to extra time. What if we flip one of them to the lowest compression level (1)?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b14d5cd1555f855190f6045ce22eb4e/href">https://medium.com/media/1b14d5cd1555f855190f6045ce22eb4e/href</a></iframe><p><strong>Peak memory: 761.5 MB Increment memory: 673.8 MB, Elapsed time: 1:13m</strong></p><p>Slightly better on time, but worse on RAM: it appears that the gzipping process is using a bunch of memory no matter what and doesn’t stream well.</p><h3>Conclusion</h3><p>What did we learn here?</p><ol><li>pandas.read_sql is very bad, for both time and memory.</li><li>Using a StringIO or tempfile with COPY can perform similarly. It’s tempting to call tempfile the winner here, but I’ll stress that this is totally based on what is cheap for you. This test ran on my laptop, using local disk. Disk I/O might be more expensive to use depending on the setting! But both examples are very fast in general and should be your first choice!</li><li>The compression ideas seemed to be a red herring, at least as implemented with gzip.GzipFile, the internals of which I’m not totally familiar with. There may be other, more sophisticated ways that might work (e.g. zlib.compressobj, or another compression type entirely), but nothing here did.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f31cd7f707ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/optimizing-pandas-read-sql-for-postgres-f31cd7f707ab">Optimizing pandas.read_sql for Postgres</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unit Testing Python data visualizations]]></title>
            <link>https://medium.com/data-science/unit-testing-python-data-visualizations-18e0250430?source=rss-459afaeb38d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/18e0250430</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Tristan Crockett]]></dc:creator>
            <pubDate>Sat, 09 Mar 2019 18:30:48 GMT</pubDate>
            <atom:updated>2019-03-12T13:37:15.705Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8kPTC-P1_7xhxFW9F-1Wg.png" /></figure><p>Unit Testing Python code that interfaces with matplotlib can be challenging. How much is helpful to test, and what’s the best way of doing so?</p><p>This comes up a lot while doing development for <a href="https://dssg.github.io/triage">Triage,</a> a data science toolkit that, among other things, includes a decent amount of code that draws plots using matplotlib or other related Python libraries. These plots are often used in Jupyter notebooks, which have their own facilities to show the plots.</p><p>It doesn’t make a ton of sense to test visualization code in too much detail. Expectations about what a plot should look like are hard to express in a way that is easy to unit test. It usually makes sense to separate as much of the plot data generation from the plotting as possible, and test the plot data generation as you would any other generated data. But plot data is not a plot. We want to make sure that the plotting code works. How do we define ‘works’? One useful definition is if a test can answer the question:</p><blockquote>Did we plot something?</blockquote><p>This is a great start. Any code that is supposed to plot something should probably plot something, and hopefully it should be easy enough to figure this out automatically.</p><h4>Approach one: patch</h4><p>The approach I’ve used a lot in the past to do this is just to mock ‘plt.show’ and assert that it got called. An example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a6491045b7adeb01d38e275b8c99a99/href">https://medium.com/media/6a6491045b7adeb01d38e275b8c99a99/href</a></iframe><p>This works, generally. Because ‘plt.show’ is generally called at the end of plotting code, if something breaks leading up to plt.show (often what you’re trying to test), this mock won’t get called and the test will fail.</p><p>There are some shortcomings, though:</p><ul><li><em>What if plt.show isn’t called?</em> For code that’s often called from a Jupyter notebook (or any other display engine that may be handling the showing on its own layer). You can try and make internal guidelines to enforce that all plotting code calls plt.show, but at that point you’re just testing for your internal guidelines, not that the code works.</li><li><em>What if plt.show is called, but not in the module you’re importing?</em> Take seaborn plots, for example. They use matplotlib but wrap it, so you’re less likely to see plt.show in your code. What if somebody refactors a clunky matplotlib figure into a nice seaborn one? I don’t want my test to break in this case.</li><li><em>What if plt.show was called but without any content?</em> This is not super likely, but still possible.</li></ul><h4>Approach two: plt.gcf</h4><p>Luckily, matplotlib has another interface we can use to get around these shortcomings of patch: plt.gcf() (get current figure).</p><p>I’m not sure if this is how Jupyter’s inline plotting magic works, but it’s mighty useful. gcf returns <em>a reference to the current (last plotted) figure. </em>Better yet, it includes a number (plt.gcf().number) which lets you keep track of how many were plotted. We can inspect the current figure after our plotting code is run to make whatever level of assertions we want on it. So it doesn’t care if plt.show was called, and works as long as somewhere the figures got created, even some higher-level library that internally uses matplotlib.</p><p>When I first found gcf, it was tempting to “reset” the plot number before calling the code to try and assert what the number of the current figure should be exactly. I tried to “clear” the plot before doing so, with plt.close():</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45e430dde41c98c533686442cfcfd7bd/href">https://medium.com/media/45e430dde41c98c533686442cfcfd7bd/href</a></iframe><p>This worked….locally. However, when I pushed to Travis the ‘close’ didn’t seem to do anything and the value of gcf().number was affected by other test cases. I assume this has to do with my laptop having an actual display/window and Travis being headless. Maybe there’s a better way to do the ‘clearing’, but I wasn’t about to get into the weeds with displays, so instead I came up with an alternate way to make it deterministic:</p><blockquote>assert that the number of figures is higher than when you started the test</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c39740eaedbf61325c18d6a4bcd808c/href">https://medium.com/media/4c39740eaedbf61325c18d6a4bcd808c/href</a></iframe><p>This works the same on Travis as on my laptop. Another difference is that I’m not asserting a specific number of figures plotted. Whether or not you want to do this in your code (ultimately I think it may get too deeply into how matplotlib is handling things internally), you could using this method. But this by itself is a handy plot assertion tool. I even ended up making it into a context manager-aware method for easy reuse:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8cbd93c8ab9bc12dd8ef64dbca1229b2/href">https://medium.com/media/8cbd93c8ab9bc12dd8ef64dbca1229b2/href</a></iframe><p>What are the possible shortcomings of this method?</p><p>We’re asserting that whatever code we are calling to plot creates matplotlib figures. Is this brittle? Theoretically. It definitely sounds like it violates the “don’t test the implementation” guideline. But in all reality, I think that</p><blockquote>matplotlib is a common backend that many nice frontends that i may switch between will use</blockquote><p>is a safer assumption than</p><blockquote>seaborn is the best frontend and i will always use it and maybe it will switch backends away from matplotlib</blockquote><p>My team redoing a chart into a new, non-matplotlib-backed plotting library is certainly possible, but not very likely. So I’m fairly comfortable with this as a reasonably-generic Python data visualization assertion. What do you think? Are there other shortcomings I’m missing, or better ways to test Python plotting?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18e0250430" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/unit-testing-python-data-visualizations-18e0250430">Unit Testing Python data visualizations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Poster out of Bus Route Maps, pt 1]]></title>
            <link>https://medium.com/@tristancrockett/building-a-poster-out-of-bus-route-maps-pt-1-f40135d80492?source=rss-459afaeb38d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f40135d80492</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[poster-design]]></category>
            <category><![CDATA[bus]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[maps]]></category>
            <dc:creator><![CDATA[Tristan Crockett]]></dc:creator>
            <pubDate>Wed, 03 Jan 2018 23:25:07 GMT</pubDate>
            <atom:updated>2018-01-04T15:12:40.495Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I use Python image libraries and open data to pay tribute to the humble city bus.</em></p><p>I love mass transit. I really do. My family took a trip to London when I was eight, and I obsessed over figuring out how to navigate us on the Tube. Nowadays, sometimes this love takes the form of taking buses and trains pretty much everywhere. Sometimes it takes the form of reading about transportation and housing policy as it relates to ensuring mass transit is a viable option for as many people as possible. Sometimes it takes the form of building a poster for a wall using bus maps, which is what I’m writing about today.</p><p>The Chicago Transit Authority releases bus brochures on their website. They feature a mini map of the entire route, with not a ton of detail but mostly just the route represented as a solid line, with major intersections, transfers, and points of interest marked along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSzCM1OAknkIKyF7bVYJpQ.png" /><figcaption>A sample bus brochure. There is a great taqueria right by Halsted off of this route, but sadly restaurants are not considered ‘points of interest’ by the CTA.</figcaption></figure><p>The goal of the project is simple: Make a big poster of all of the little maps. Individually, they are all compact and similar in dimension; as you can see from the example above, they are oriented in different ways to make the map fit into the tall shape required by the brochure. This works out because it means the maps should line up reasonably well if I stick them in a grid.</p><p>The trickiest task is to extract the actual map from each PDF. I don’t even want the black bars, just the route #/name and the map itself. This is easy enough to do manually in an image program, but the CTA has well north of 100 bus routes, so I need something more automatic. Sure, maybe I could email the CTA and see if they have the originals and would send them to me. But I’m not going to do that. I’m going to write a script.</p><h4>Downloading all the images</h4><p>There has to be a list of all these brochures, right? Of course there is. It’s here: <a href="http://www.transitchicago.com/travel_information/bus_schedules.aspx">http://www.transitchicago.com/travel_information/bus_schedules.aspx</a></p><p>This page actually looks pretty easy to scrape. All the links we want are in neat lists, with very similar link texts. Here’s an example: <a href="http://www.transitchicago.com/asset.aspx?AssetId=856">http://www.transitchicago.com/asset.aspx?AssetId=856</a></p><p>All of the PDF links have this format (differing only by AssetId). Furthermore, all of the ones that link to schedules have text that starts with ‘Schedule’.</p><p>To do this, we’re going to use Beautiful Soup, which is a handy Python scraping library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c86cd5a5432d93837ae806cbcdd12932/href">https://medium.com/media/c86cd5a5432d93837ae806cbcdd12932/href</a></iframe><p>As mentioned above, the regular structure of the links on this page help us out; we can filter the links on the page to the ones we want to download by specifying prefixes for both the link target and link text. Once all the schedule links have been collected, they’re downloaded into memory with a standard call to ‘urlretrieve’. Of course, this code snippet by itself doesn’t do anything too useful yet.</p><h4>Cropping the route maps out of each PDF</h4><p>This next step is a bit tougher. There are a bunch of other things in the brochure, and we want to extract just the maps. Thankfully, there is some structure to it, but they aren’t totally uniform. The first page of the Pershing bus brochure above is divided into three equal width sections with the map on the rightmost section. Extracting from this layout shouldn’t be too bad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8pUxtPUg5A3DbThbmOOixA.png" /><figcaption>Another bus brochure layout</figcaption></figure><p>How about other maps? Another layout, like the 28 above, is four equal-width sections, with the map in the second section. With a little luck, there is some simple pattern to tease out here, maybe based on the image width.</p><p>To work with these images in code, I’ve decided to use <a href="http://docs.wand-py.org/en/0.4.4/">Wand</a>, a Python library based on ImageMagick that can work with PDF files. I haven’t used it before, but it seems to offer the necessary features and has good documentation. To start, I had a hunch that the page width might be important for distinguishing between different layouts, and decided to print them out.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66ee6481b8fa581c962af9bd48f0b2c8/href">https://medium.com/media/66ee6481b8fa581c962af9bd48f0b2c8/href</a></iframe><p>The output from this looked something like</p><pre>&lt;a href=&quot;/asset.aspx?AssetId=885&quot;&gt;Schedule - Route 165 - West 65th&lt;/a&gt;<br>(792, 612)<br>&lt;a href=&quot;/asset.aspx?AssetId=828&quot;&gt;Schedule - Route 169 - 69th/UPS Express&lt;\a&gt;<br>(504, 612)</pre><p>Every printout had a height of 612, and the widths all looked like 792, 504, and 1008. Upon further inspection I got a couple more widths. This seems to imply 2, 3, and 4 section layouts. The cropping mode in ImageMagick works by specifying ‘top’ and ‘left’ values to denote the top left corner of the image, and ‘width’ and ‘height’ to denote how far to the right and down the image extends. So I put together some cropping rules based on the image width.</p><p>Combining the cropping snippet with the earlier retrieval code, and some boilerplate code to save the results, I came up with this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d5142d7efbb80745492536296d0411f/href">https://medium.com/media/4d5142d7efbb80745492536296d0411f/href</a></iframe><p>Within these cropping rules, the ‘top’ and ‘height’ definitions are all extracting the full vertical span, but there is some variation in the ‘left’ and ‘width’. Each variation is divided into a certain number of equal-width sections, and some of them start in different sections. For example, for a width of 792, the left starts at `(page_width / 3) * 2` to target the rightmost section of the page.</p><p>The results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*ToM3HzEdvsvgSvQIjulNFQ.png" /></figure><p>One maybe-problem, as can be seen by the marble background I artificially added: some of the brochure maps were against a white background and some were against a transparent background! This may not be a problem when overlaid over white, but I’d still like to give them a consistent background, which can be done with one line in Wand:</p><pre>page.alpha_channel = False</pre><p>There’s still one other thing I’d like to do: Remove those black bars. I didn’t do anything fancy to achieve this, just played around with ‘top’ and ‘height’ values in the cropping rules until the results looked good. Here’s the full script at this point.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/544ed28d7ae2bc554f8e31303eadfe16/href">https://medium.com/media/544ed28d7ae2bc554f8e31303eadfe16/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*YfsModwn0xRRXOzld37hkA.png" /><figcaption>The 15 bus map, after removing the top/bottom black bars.</figcaption></figure><p>There are still occasionally pieces of text I would rather exclude like in the 15 bus above, but I can live with those. I consider the bus maps properly extracted.</p><h4>Assembling the maps into one big image</h4><p>So, how do I want to sort these? By route number? By route name? By northernmost point? By ridership? There are lots of ways to order the routes in the grid, but I’m going to keep it simple and order the routes how the CTA presents them, numerically sorted.</p><p>I still have to figure out a width and height for the grid. Ideally it would be a full grid, but the script produced 127 images, which isn’t conducive to evenly dividing. Close is still good, and I’d rather have a few missing spaces at the end of a line rather than a few extra on a new line. The code below creates an image by taking in the number of columns in the grid and filling in images from left to right, top to bottom.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fad3d9161ad7e444c09ddaa5b1d0b05/href">https://medium.com/media/2fad3d9161ad7e444c09ddaa5b1d0b05/href</a></iframe><p>I tried out a few different values for the number of columns, mostly in the 20s to approximate a golden rectangle shape. I ended up liking 22, because of the shape and minimal remainder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yD4fvhD5V2qC1y6W18DTzQ.png" /><figcaption>Version 1 of the bus map grid</figcaption></figure><p>It works! That’s as far as this post will go. There are two remaining problems that I can see, that I’ll want to fix before sending this off to get printed:</p><ol><li>Not quite high-res enough. The width of each of these is 252 pixels, not quite an inch under 300dpi. I would like to resample this at a higher resolution, but want to play around with different filters to keep sharpness as high as possible.</li><li>The route names don’t quite align perfectly. You can even see it at thumbnail size: check out 56 and 57, for example. This might be kind of complicated to take care of.</li></ol><p>I will cover both of these in a followup post.</p><p>The code I used to generate the poster is also at the repository below (with some minor reorganization). I’ll add any updates I make there, whether or not they are covered by a blog post.</p><p><a href="https://github.com/thcrock/bus-brochure-maps">thcrock/bus-brochure-maps</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f40135d80492" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Research Hub Updates: Skill Extraction]]></title>
            <link>https://medium.com/data-at-work/research-hub-updates-skill-extraction-9d7b39486683?source=rss-459afaeb38d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d7b39486683</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[labor-market]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[dataset]]></category>
            <dc:creator><![CDATA[Tristan Crockett]]></dc:creator>
            <pubDate>Tue, 21 Nov 2017 16:07:36 GMT</pubDate>
            <atom:updated>2017-11-21T16:07:36.485Z</atom:updated>
            <content:encoded><![CDATA[<p>Is Facebook a job skill? It depends.</p><p><em>This post regards a recent update to the Data@Work Research Hub, a collection of public datasets about the US labor market. It is produced by the Open Skills Project, an initiative based at the Center for Data Science and Public Policy at the University of Chicago. If you’re unfamiliar, check out our </em><a href="https://medium.com/data-at-work/introducing-the-data-work-research-hub-and-funding-opportunity-99ad9cb1dcfa"><em>introductory post here</em></a><em>.</em></p><p>When we launched the initial version of the <a href="http://dataatwork.org/data/research">Data@Work Research Hub</a>, we had a problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*Z_pVHX3H9GP5v3E6_AJGNA.png" /><figcaption>A snippet of job titles and their associated skills in the Research Hub</figcaption></figure><p>There are a few highly visible problems here. Why are Facebook, Twitter, and Instagram so popular? What is Forth? To understand why these show up so often, it helps to explain how skill extraction works in the first version of the Research Hub.</p><h4>Naive Skill Extraction</h4><p>The initial version of the skill extraction for the Research Hub is rudimentary. We are building off of the commonly-used ONET jobs and skills taxonomy, which is updated gradually based on survey data over time. For skills, ONET catalogs a mixture of hard skills, soft skills, and tools under a variety of categories they call <strong>Knowledge</strong>, <strong>Skills</strong>, <strong>Abilities</strong>, and <strong>Technology Skills. </strong>For<strong> </strong>the purposes of extraction, we pool these together into a broader list that we refer to as <strong>skills</strong>.</p><p>We take this combined list (about 20k in total, most of them technology skills) and look for members from the list in each job posting text. If one shows up, we tag it. This works in many cases, but sometimes the text shows up in job postings for other reasons, such as a link to more information about the company (facebook) or as a common word (forth).</p><h4>How do we fix this?</h4><p>Figuring out whether or not a matched word or phrase from the ONET taxonomy is a real skill in this context is not a simple problem. We are working on more robust solutions to this problem, which I’ll mention later in the post. To produce a much smaller list whose precision we can be more confident in, we can as a short-term fix start filtering by the <strong>job posting’s SOC (occupation) code.</strong></p><p>These job postings also have a code for the occupation. Our data partners classify the SOC code of the posting and send it to us. ONET publishes their list of skills per SOC code. By using this information, we can filter down the list of skills we found to what ONET considers skills relevant to the occupation. We will no longer indicate that (all jokes about work procrastination aside) cost accountants need to be well-versed in Facebook, but marketing managers still do. We also will not tell you that cashiers, RNs, and English professors need to know a <a href="https://en.wikipedia.org/wiki/Forth_(programming_language)">programming language invented in 1970</a>.</p><h4>Is this a good solution?</h4><p>By using this approach, the list of skills we get is quite small. So small, in fact we rarely extract any skills from the job postings. This solution pins us to what ONET currently knows about, when in reality we want to work towards extending ONET’s taxonomy by using insights gleaned from job postings.</p><p>We are improving precision at the expense of recall, and this highlights something that is important to understand: <strong>the language used in ONET rarely matches the language used in job postings</strong>. The skills are in there, and we need another way to extract them, but as a stopgap let’s be more picky.</p><p>If you really want the old version, <a href="https://open-skills-datasets.s3-us-west-2.amazonaws.com/v1.zip">it’s archived on the Research Hub</a>.</p><h4>What’s next?</h4><p>We’re working on some far better solutions for generating phrases that we think may be skills, untethered from the current ONET taxonomy: rule-based methods, word embedding, and conditional random fields. We’re also working on a web labeling solution to more effectively gauge precision and recall of various methods.</p><p>You’ll hear much more about these new approaches in later blog posts. Until then, don’t use version 1 of the Research Hub as an excuse to spend hours a day at work on Facebook: <a href="https://www.bustle.com/articles/143221-wasting-time-on-facebook-costs-an-absurd-amount-of-money-in-lost-productivity">the US economy will thank you for it.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d7b39486683" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-at-work/research-hub-updates-skill-extraction-9d7b39486683">Research Hub Updates: Skill Extraction</a> was originally published in <a href="https://medium.com/data-at-work">Data At Work</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>