<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Connor Waldoch on Medium]]></title>
        <description><![CDATA[Stories by Connor Waldoch on Medium]]></description>
        <link>https://medium.com/@waldoch?source=rss-3f5a9a48ccba------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ci8RBPuIVSMgweYG3ZdacA.jpeg</url>
            <title>Stories by Connor Waldoch on Medium</title>
            <link>https://medium.com/@waldoch?source=rss-3f5a9a48ccba------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:14:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@waldoch/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reading the Natural Gas Pipeline Tea Leaves]]></title>
            <link>https://medium.com/@waldoch/reading-the-natural-gas-pipeline-tea-leaves-38e09a871d63?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/38e09a871d63</guid>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[texas]]></category>
            <category><![CDATA[power]]></category>
            <category><![CDATA[natural-gas]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Thu, 03 Feb 2022 23:47:10 GMT</pubDate>
            <atom:updated>2022-02-03T23:47:10.998Z</atom:updated>
            <content:encoded><![CDATA[<p>A quick and dirty look at pipeline operations using only public sources</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCIwB3BNz9CXhU1ryFUEiA.png" /><figcaption>Hey, it’s the ERCOT logo, you can probably guess the regional focus of this post!</figcaption></figure><p>The place I currently work, <a href="https://leap.energy/">Leap</a>, is in the demand response (DR) and distributed energy resource (DER) space and in particular deploys lots of resources when a grid is under stress to help it cope with tight conditions. With another Texas winter storm on the horizon some of the more market oriented folks were writing up threads on different facets of the system, which is itself an amalgamation of a whole bunch of other systems. Since some of my colleagues were already getting into load forecasts, wind conditions, and potential divergences from day-ahead schedules across the board, I decided to go a step back from electricity and write up a little bit on natural gas. Specifically, how to use public sources of information to identify some of the current, and day-ahead operating patterns on pipelines and potentially at individual power plants. I’ve never been a gas trader or anything, and am by no means an expert, but understanding pipelines at least a bit was pretty important working on wholesale markets in NY and New England. Below is a lightly edited consolidation of my slack thread on how to use public information to identify conditions on natural gas pipelines.</p><h3>Talkin’ bout Maps</h3><p>To start, I’ll say that Texas is absolutely awash in fossil infrastructure, and here’s a map to prove it. It’s gas pipelines, except, this is only a fraction of the story because it is just the pipelines that have public reporting requirements due to crossing states, or that the EIA otherwise had access to for some reason. Like the grid, once you go interstate, standards rise. But those interstate pipelines are also the big ones, and some of the most critical. You might see these referred to as trunklines as well. Every color here is a different company’s pipeline (and there is probably some color duplication as well because the palette in the GIS software I’m using is limited). Also, this infrastructure map data is from the <a href="https://hifld-geoplatform.opendata.arcgis.com/">Homeland Infrastructure Foundation-Level Data portal</a>, which is a shockingly excellent source that updates far, far more frequently than the EIA’s shapefile repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVqvQiAyoHHFhYnbFBhssw.png" /><figcaption>Natural Gas Pipelines (the bigger ones, mostly)</figcaption></figure><p>The next map (below) image is those pipelines with the addition of fossil plants in Texas. Gas plants are those red stars, coal plants are the black circles, and the white circles are oil (although this is not considering plant capacity or availability in any way). In some states (like New York), lots of gas plants are behind the systems of Local Distribution Companies (LDCs) and do not have access to these trunkline systems. In most places, power plants are sited such that they can get direct access to gas at the lowest rates on these bulk transfer pipelines. Also, unlike the north east, Texas is not really supposed to face gas constraints the way NY/NE can in the winter because there is just dramatically more pipeline capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozmw43NelospJjrqKzbDaA.png" /><figcaption>Gas pipelines and fossil plants</figcaption></figure><p>This third map is power plants and transmission lines as an additional reference for “where are things”. Red lines are 345 kV lines, black lines are the 100kV classes and below, blue is lines in the 200–300 kV range, and the rare purple on there at the far eastern edge is 500kV. The higher kV, the more capacity a line has, i.e., can move more power, higher limits on what power plants can put on them before they melt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3t6vmHC2Op4fPLM6N_LSA.png" /><figcaption>Power plants and transmission lines</figcaption></figure><p>Folks in our slack were talking about specific infrastructure regions of potential interest, and the first area mentioned was Laredo to Beaumont, which sort of follows an arc a bit north of the gulf from the south western border of Texas to a bit east of Houston, which is represented on the maps above by that particularly thick bundle of natural gas pipelines in southern Texas. To take a closer look I’m going to zoom in on that section and layer in a bit more natural gas infrastructure.</p><p>There are two potentially critical pieces here, storage and compressor stations. Storage in case production falls off for some reason and compressor stations because they’re needed to keep up pressure on the system. This is a pretty critical distinction when it comes to gas vs liquid pipelines, the need for regular compressor stations to keep everything running. One compressor station is not the end of the world (although if it’s at the end of a pipeline that power plant or residential system might only get a trickle), but lose more than one and the system becomes dramatically less useful very quickly. And that’s just on the pressure side, natural gas is still very flammable and any part of the system being compromised can be super dangerous for continued operations.</p><p>The other (yes, there’s more) thing here is that Texas is producing a huge amount of the gas it’s consuming, so we need to think about where wells are located too, since those can also freeze and fail (and did!).</p><p>All of that aside, let’s layer in storage, compressor stations, and gas wells on top of that bundle of pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEfDtjanIp_wEqEb_hSAtA.png" /></figure><p>The small yellow dots are the wells (filtered to “gas” and “gas or oil” wells) and you can see how concentrated they are at the start of the arc mentioned above, around Laredo. This is why that area is important. You also notice an increase in the red triangles (compressor stations) as we leave the well-heavy area because demand is increasing and there are less well heads feeding onto the system and keeping pressure up that way. Finally, we have more storage (green diamonds) moving away from production as well because it’s more valuable to have the gas closer to demand, and the storage tends to be exhausted wells, which, you guessed it, were exhausted first because they’re closer to where the people are (and they’re not frac-ed wells, but we won’t get into that). One other thing I added to the above is pipeline names, which are just randomly labeled by some unknowable don’t overlap algorithm in QGIS, so not entirely useful, but this does begin to get at the business information case for all of this preamble.</p><p>So this data, these pieces of information are not necessarily simple to link together already, but with a little domain knowledge you can get going. This was all step one of my personal “how to know things about a pipeline” process. Now that we’ve gotten the GIS data, we can end the prelude and start getting into the actually business critical information (yes, this was all prelude).</p><h3>The Process</h3><p>1. Acquire natural gas pipeline shapefiles from either the EIA or homeland security’s GIS repositories.</p><p>2. Use those to align visually (or in code, if you want to go all Neo on it geopandas is a nice tool) with the area of interest.</p><p>3. Once you’ve identified the area you want and the overlapping pipelines get the operator names.</p><p>4. Start googling.</p><p>As I mentioned at the beginning, these interstate pipelines have associated public informational postings, and when you don’t know the ultimate parent company of a pipeline or their website before hand, that’s exactly what you search for “pipeline name informational postings”.</p><p>Returning to the map, let’s find a pipeline. First I’ll remove the wells since they’re kind of a hassle, then thicken up the pipelines a little and make the names bigger. When I click on one it turns pink-ish, so you can really make out how extensive an individual network is against the blue of everything else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObXUZeK8jE7Ki_Iuy5sI7g.png" /></figure><p>This particular network I selected is Kinder Morgan Texas Pipeline Co. <a href="https://www.google.com/search?q=kinder+morgan+texas+pipeline+co+informational+postings&amp;oq=kinder+morgan+texas+pipeline+co+informational+postings&amp;aqs=chrome..69i57.16j0j4&amp;sourceid=chrome&amp;ie=UTF-8">So we search that</a>. Which gets us <a href="https://pipeline2.kindermorgan.com/Default.aspx/?code=KMTP">here</a>, where we can select the one we want under Midstream at the top to get <a href="https://pipeportal.kindermorgan.com/PortalUI/DefaultKMIntra.aspx?TSP=KMTP">here</a> (these companies do truly the bare minimum to remain in compliance with reporting standards). Finally you end up at a super plain looking page with a map that has no legend and little idea where to go next. Well, there are two places: the notices, and Location Information. both on the left sidebar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEjbflHKWz8kswYkA5Sjww.png" /></figure><p>You might also notice my pipeline map doesn’t match their pipeline map and that’s because I dissolved the pipeline segments by owner-operator, and they might define the specific bits a bit differently in their internal pipeline classifications (also I dissolved on operator, not pipeline name).</p><h3>Interpreting Notices</h3><p>So now we get into really trying to track down specific things about these pipelines. First, let’s go with the notices, and the critical ones in particular. Critical is what we care about here because they’re basically notices of “hey, important thing is happening on this system and it’s going to affect gas deliveries” major outages, force majeure events, and similar will show up here. Unfortunately I can’t show any references from last year as this pipeline only shows notices within 90 days of the current date (although you might be able to coax the site to take a different date by intercepting some information it’s passing to the table retrieval but that’s a different topic). We can see that there is one notice related to the current cold weather situation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GKcPKqZR48OlBiRnHpk1mA.png" /><figcaption>Critical Notices for the Kinder Morgan Texas Pipeline</figcaption></figure><p>Clicking on it returns the following current pipeline rules which are in place until further notice:</p><ul><li>transportation agreements must stay balanced. Receipts must match deliveries.</li><li>Storage customers will be limited to their firm contractual rights.</li><li>Points with balancing agreements may be scheduled to match the actual volume flowing at the location.</li><li>Sales customers will be limited to their firm contractual limits.</li></ul><p>And what you have there is a whole lot of jargon. To interpret this you need to know what firm contracts on this pipeline typically are (and in fact what the heck is a firm contract? are there non-firm contracts?), what’s a balancing agreement? What’s a transportation agreement? Does all this balancing talk mean pipeline deliveries aren’t normally enforced strictly on a day-to-day basis? (yup). But most of this is out of the scope of my small primer here, suffice it to say, these notices indicate a heightened need to make sure the system balanced in real-time (or close to it, daily), and that everyone strictly adheres to their stated contractual values. While the gas pipeline system has a higher tolerance for mismatching receipt/delivery than the electric system’s generation/load, it still does need to be balanced, and the storage on the pipeline itself is not necessarily enough alone during tougher stretches of time with high demand and possible supply disruptions.</p><p>If I select a totally different pipeline form their Interstate list (El Paso), and go to critical notices we can find one that is a bit more typical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5bbFtA1L6nIz2yDGEgh7qg.png" /><figcaption>Critical Notices for El Paso Pipeline</figcaption></figure><p>Clicking on the first one of these we return:</p><blockquote>EPNG OPEN FOR CONFIRMATIONS — INTRADAY 3 CYCLE</blockquote><blockquote>EPNG is open for Gas Day February 2, 2022 Intraday 3 cycle confirmations.</blockquote><blockquote>Capacity reductions were performed at the following segment(s):</blockquote><blockquote>Segment 920 — PERMPLNS</blockquote><blockquote>Segment 940 — L2000</blockquote><blockquote>Segment 1600 — CORNL2K</blockquote><blockquote>Please call the EPNG scheduling Hot Line 800–238–3764 Option 1 with any questions.</blockquote><p>There are a couple of useful pieces of information here:</p><ul><li>There are multiple scheduling cycles for gas, and this notice is about Intraday 3 (which implies the existence of other intraday cycles and perhaps non-intraday schedules as well)</li><li>Segments have names and codes</li><li>Capacity can be reduced on individual segments.</li></ul><p>But where are those constraints? Well, El Paso has a map on their homepage with segments and a red highlighting for active constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*pb6AOtAabkPKeNyE9uMujg.png" /><figcaption>Map of the El Paso Pipeline</figcaption></figure><p>Unsurprisingly, some of those segments are in Texas, although not all, and not all parts of their Texas network are currently under constraints at all. How can we know the impact of those constraints and how the relate to business as usual? Well, it’s time to turn to locations and capacity. By navigating to the Capacity options on the left sidebar (sticking with El Paso), you can select “operationally available” and then point or segment. let’s stick with segment since we saw some in the above critical notice. This is where you begin to get hard data on pipeline operations as the design, operating, scheduled, and available capacity on the pipeline are displayed for each point or segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUuWO7rKdiqhHQUOJFgxvA.png" /><figcaption>Pipeline Capacity per Segment</figcaption></figure><p>These different capacities have a whole breakdown unto themselves:</p><ul><li>Design Capacity generally reflects the physical capacity of a segment as determined by a capacity model utilizing current contractual receipt/delivery quantities and a reasonable set of engineering assumptions and parameters.</li><li>Design Capacity may be updated periodically to reflect changes in contractual receipt/delivery patterns, paths that do not utilize the entire segment and/or capacity modeling parameters.</li><li>Operationally Available Capacity may be updated throughout the day to account for changes including, but not limited to: netting of bidirectional nominations, partial segment path nominations, facility outages, and imbalance management.</li><li>Total Scheduled Quantity is the resulting scheduled volume after netting scheduled volumes in both flow directions. This quantity for each segment is displayed only in the resulting net physical flow direction of the segment. Opposite of the physical flow direction will display a quantity of zero although there could be volumes scheduled. For further explanation, contact your Scheduling Representative.</li><li>The net Total Scheduled Quantity may result in a quantity different than Operating Capacity due to the confirmation process after scheduling.</li><li>Operating Capacity may be different than Design Capacity based on the periodic, non-permanent anticipated or observed operational pipeline conditions including, but not limited to: higher average thermal content of the gas, higher pressure from interconnecting pipelines, on-going maintenance, etc.</li><li>Operating capacity may be greater than the Design Capacity of the segment because operating conditions are more favorable than established design conditions, for instance, average thermal content of the gas is higher than assumed in design.</li></ul><p>If you actually read those you might have puzzled over bits about “opposite of physical flows”, well, you can “ship” gas up and downstream on the pipes contractually, so the concept of “flow” gets pretty weird really fast. They do have an overall direction, but contractually the gas can sometimes go in a kind of loop within the same pipeline depending on the location of receipt and delivery points for a specific shipping contract.</p><p>There is also a “Cycle Selection” box, best available would the most forward in time scheduling cycle for the date we select. Intraday 3 is the final cycle of the day and the “true-up” cycle, so that’s the one I would prefer to use for any day that’s reached the end of its scheduling passes. Downloading the Excel for for the day and filtering on one of the segments (L2000) from the critical notice we can see that it has a design capacity of 597k Million BTUs but is scheduled at 0 and has 0 capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3xXzzcVVJvT8DNJi9iAFA.png" /></figure><p>Now we finally begin to return to the world of electricity.</p><h3>Back to Power</h3><p>That L2000 segment is out in western Texas (and unfortunately I guess not a place we’re super concerned about for the purposes of this storm but it’s way too late for me to go back and cherry-pick something else), but there are still some gas plants out there (snippet of map attached, El Paso pipeline highlighted in pink).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYDTQZslaTOLY7QgEaSPwQ.png" /></figure><p>This time we’re going to download capacity by point and filter it down to the segment, L2000 (ID 940). And also compare it to the customer list as well as the map of power plants to try and cross reference some information. On the map there is a plant called “Permian Basin” right off of an El Paso pipeline. The plant is owned by Luminant, and if we search Luminant in the list of customers it returns a point on the L2000 segment, in the Texas-Permian location. This point has a capacity of a few hundred thousand MMBTUs is delivering 124k to the point today, and has an operating capacity of nearly double its design at the moment. This indicates that lots and lots of gas is being made available on the system right now, possibly to prepare for the storm. You can also back calculate the delivery amount into a rough calculation of “hey, is this a power plant or not?”</p><p>The plant data embedded in the GIS file lists this plant as having 366MW winter capacity, although the total capacity is 865MW on Luminant’s website, and is comprised of 5 CTs and two retired STs. The CTs are a bit over 30 years old at the youngest, so let’s give them a heat rate of 10. This means they would require 10 MMBtu’s of gas for every MWh of production. 124,000 MMBtu / (10*366) = 33 hours of max production, so the capacity is probably higher than that 366 value. 124,000 MMBtu / (10*865) = 14 hours of max production. So it could have ran at portion of the day flat out, or at some capacity between the 366 and 865 values all day. Either way, these back-of-the-envelope calculations do make a case for this point being one delivering to a power plant, and we got pretty far into the weeds of estimating their today fuel delivery all from public sources.</p><p>Hopefully this brief primer is helpful for someone else looking to interpret the natural gas system. As you can see there are multiple important points to be concerned about if the weather takes a turn, from wells, to compressors, to the power plants themselves. So far this storm doesn’t look remotely as severe as Uri was last year and that’s great.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38e09a871d63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Here and Back Again: A Public Blockchain’s Meager Journey]]></title>
            <link>https://medium.com/@waldoch/here-and-back-again-a-public-blockchains-meager-journey-df021f029d12?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/df021f029d12</guid>
            <category><![CDATA[utilities]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[nevada]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Fri, 28 Jan 2022 05:30:38 GMT</pubDate>
            <atom:updated>2022-01-28T14:58:12.693Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Nevada PUC’s Journey into the Cryptoverse</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uW8kiHm_uDQdet6peq88aA.jpeg" /></figure><p>More than three years ago the Public Utilities Commission of Nevada (NV PUC) began a use case investigation into blockchains. The goal was to potentially use a blockchain-based solution to track Portfolio Energy Credits (PECs). PECs are generated as a part of Nevada’s Renewable Portfolio Standard (RPS), and represent the “renewableness” of electricity generated from renewable energy producers. I say “renewableness” because the electricity itself is bought, sold, and accounted for on its own, it’s the renewable characteristic of the generating technology that creates the PEC and its value, the proof that it was generated from an eligible renewable resource.</p><p>Generally, renewable energy certificate (REC) programs are instantiated due to a need arising from state mandates that XX% of electricity is produced by renewable sources by time N. In order to comply, electric utilities must hold RECs equivalent to the mandated amount of electricity at the end of each compliance period (often a year). RECs can be generated by sources owned by the utility, or generated by another company (perhaps an independent power producer) and purchased from them. A quick example:</p><p>Electric Utility A serves customers 100,000 Megawatt-hours (MWh) in electricity sales a year. Utility A is in State Z, which has a 20% renewable energy mandate, so they must hold (0.2*100,000) = 20,000 MWh worth of “proof” (RECs)that they (or someone in the state) generated that amount via renewable resources. If the Utility is vertically integrated (i.e., they own power plants and sell electricity to homes and businesses), it’s possible that they generate all or most of these credits themselves. However, not all electric utilities own power generation, and some that do still mostly own older fossil units, which wouldn’t generate credits. So where do they get their RECs? Generally, states with these programs create a verification system and trading platform (sometimes all in one) where entities can buy and sell these credits, the renewable attributes of their generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*mj_Y543_szPBr-9azFk_TQ.png" /></figure><p>Back in Nevada (I swear we’re getting to the crypto part of this), a single PEC generally maps to one kilowatt-hour (kWh) of electricity generated, although certain system configurations and generation timings can range from 0.7 PECs/kWh up to 2.45 PECs/kWh. Now, a lot of what I’ve talked about above has probably perked up the ears of the crypto-friendly crowd: the need for transparency and verification of the attributes of an individual unit of electricity, then tracking that set of attributes along with millions more in real time while maintaining the associated information in an immutable context, those both sound crypto-y! Now to a timeline:</p><h4><a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/32973.pdf">September 2018</a></h4><p>The previous system for tracking PECs is getting a bit long in the tooth and is perceived as suboptimal, so a Commissioner proposes exploring a blockchain-based solution to deal with this aging system. The PUC of Nevada then opens a new docket, 18–09008 to explore alternative solutions, “including, but not limited to, blockchain-based solutions”.</p><h4><a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/37073.pdf">March 2019</a></h4><p>The NV PUC issues a puts out a Procedural Order and simultaneously releases a Notice of Investigation and Rulemaking, Notice of Request for Comments, and “Notice of Workshop. This Order lays out some of the questions the NV PUC is seeking to address, such as: How useful would this be to residential or commercial customers? Would the Alternative System have better functionality and be able to quantify its improvements in “efficiency, cost, tracking, and security . . .compared to the current systems?”, If the computer hardware requirements were high how would smaller players participate? Does a blockchain-based system require cryptocurrency or can PECs be monetized in USD? What do the costs look like? And so on.</p><h4>April 2019</h4><p><a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/37755.pdf">Blockchains files comment</a>s in support of the Order and seeks to answer some of the above questions. They propose potentially using an existing blockchain such as Ethereum to not only collect data and generate certificates but match sellers and buyers. They also state that Exelon, Duke, and PG&amp;E are already using this blockchain framework to “build their energy credit registries.” Ultimately, their proposal is for a “ground-up, purpose-built public blockchain solution.” Their residential-solar proposal involves using the computing power of smart meters to automatically generate, track, and sell PECs on the blockchain. This system would be automatically installed and enrolled in upon solar panel installation and “streamlines the prosumer-generator user experience, reduces administration costs, and is much more efficient than the current system.” This actually seems possible, because, as they note, the system at the time required residential customers to email amounts to PUC staff for verification on a monthly basis.</p><p>Of course, don’t forget this is still a blockchain, so cryptocurrency isn’t far behind, the comments also state: “To ensure only well-vetted tokens were used as payment methods within the Alternative System an [API] integrating from a reputable exchange could be used.”, and “A blockchain-based Alternative System would use executable distributed code (or so-called “smart”) contract, which would require the use of tokens to pay for the transaction costs. Tokens could then be exchanged for cryptocurrency, converted into fiat through exchange platforms, or the contract could initiate a payment process through a traditional accounting system, resulting in an ACH bank transfer or a physical check.”</p><p>Finally, the comments recommend a pilot program with a small group of consumers, point the Commission to the Energy Web Foundation and PJM Environmental Information Services pilot program state that it would take 5–6 developers ~1 year to implement an initial program.</p><h4>Let’s see what Google says</h4><p>Alright, we have some stuff to unpack here. First, a few google searches:</p><p>Exelon Utilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*8P9ObxxzQ85gwAvqS13s_A.png" /><figcaption>Searching “blockchain” returns 0 results.</figcaption></figure><p>Duke Energy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*XaNVLe7N_SArh8R1xLPwFw.png" /><figcaption>“Blockchain” returns a single result, which I can’t actually find on the page itself.</figcaption></figure><p>PG&amp;E:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*_ttdixm6xQk8yvpXTBSoiw.png" /><figcaption>“Blockchain” returns 24 results, 4 of which are from the last year</figcaption></figure><p>PG&amp;E’s website at least mentions blockchain somewhere, and it’s even within the past year. Although only only one mention represented a potentially real future project: Project #3.04 in the Electric Program Investment Charge 2020 Annual Report, which is to build a blockchain solution for “tracking the Company’s use of steel reels”, and “for the trading of incremental EV credits in the residential market for LCFS”. This project had no associated deliverables, metrics, or partners, an 18 month schedule, and $52 in spending to date. Its status is listed as “currently on hold”.</p><p>Suffice it to say, less than 3 years later none of the utilities cited in Blockchains’ filing seem to be pursuing anything like the extent of the projects they described, nor can <strong>anything</strong> be found on their respective websites matching the described projects.</p><h4>But what about the rest?</h4><p>With respect to the actual proposal, as someone who works with meter data and struggles with data access on a regular basis I would be deeply envious of the level of consumer control over their own meters Blockchains proposes here. Utilities frequently make it difficult for customers simply to acquire their own data, let alone provide it to third parties. Running ad hoc code on their smart meters is even more out there to most utilities, as it remains a struggle to get them to enable even the basic features and make them available to customers.</p><p>On the side of their proposed tracking and financial mechanisms, you may wonder why it’s more efficient to conduct transactions on a blockchain (with their typically high costs per transaction, slow time per transaction, and volatile token pricing) to buy and sell PECs, and then layer on an additional integration with a cryptocurrency exchange as well as traditional banking infrastructure just to get a check in the mail. You would be right to question this (and you can really just skip to the end if you want to know the outcome). In fact, let’s run through some scenarios using real numbers.</p><p>In 2020, Nevada’s renewable goal was 20% of electricity sales. Using EIA form 860 we can check the disposition of MWh for the state’s utilities. Subtracting total energy losses, we get a value of ~37.6 million MWh. 20% of this is 7.52 million MWh, which turns into a requirement for 7.52 billion PECs (as they each represent 1 kWh, of which there are 1000 in 1 MWh). Let’s take a spin through some PEC generation scenarios assuming Nevada‘s utilities are fully served in-state (i.e., not buying any PEC equivalents on an approved regional marketplace).</p><h4>Example Time!</h4><p>Crypto websites are full of whatever the reverse of FUD is (unicorns and roses?) but even they admit a typical Bitcoin transaction takes 10–20 minutes. Of course, transactions should be able to be processed simultaneously to some extent, and over the last 24 hours the hourly average was 10,649. Ethereum’s single transaction time is a bit easier to find (perhaps because it’s substantially quicker than Bitcoin, although still poor by any other metric) and ranges from seconds to minutes, but is then complicated by the need to pay “gas” to have transactions actually take place. In general terms, the more you are willing to pay in gas the quicker your transaction will be. What’s not too bad to find is the average transactions per hour over the previous 24 hours, which at the time of writing was 47,278. Bonus, EW Chain (the proposed tech) is based on the Ethereum stack, so we can probably get a useful comparison out of that knowledge. Time for a fantasy scenario.</p><ul><li>Let’s fulfill PEC needs with residential solar. 5kw systems generating 40 kWh/day. 40 x 365 = 14,600 PECs a year per home. They needed 7,520,000,000 to comply, so 7,520,000,000/14,600 = 515,068 homes generating this ideal amount. On any give day you would have to mint a lot of PECs, 515,068 x (40kWh / 5kW) = 64,384/hour, about 1,073/minute. Obviously, neither Bitcoin nor Ethereum could keep up with the pace in this residential setting. Bitcoin would fail to process even a 6th of the PECs, and while Ethereum does get substantially closer, it still doesn’t make it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3tNx8yN9Wx5TPWIB.jpg" /><figcaption>Solar! Maybe in Nevada! Perhaps generating PECs!</figcaption></figure><h4>Wait, but how do these programs actually work?</h4><p>The trick with the above is that any way you slice it it would be quite difficult for these blockchains (the largest, most popular, and most lauded) to keep up with PECs in any generation scenario. You still need to mint 7.52 billion per year, transact them at least once (to retire them), but often twice (to transfer them to a compliance entity before they retire them). And actually, it gets worse because compliance takes place at one moment in time and blockchains are notoriously bad when it comes to bulk processing a large number of transactions. I worked with a carbon compliance market for nearly five years and observed certain trends that would be tough for current blockchains to deal with. Since the compliance period is known in advance, trading is also likely to cluster near compliance deadlines, creating an elevated period of action likely to cripple most extant blockchains. I can hear the crypto enthusiast argument:</p><blockquote>“But what if you totally remove Bitcoin or Ethereum from the picture?! If you started something else it wouldn’t be this slow or require a pay-to-play structure. Surely a bespoke new blockchain would solve this!”</blockquote><p>Well, it would have to largely eschew the technology of those preceding blockchains (spoiler, the proposal does not) and make some radical decisions about its own functionality, and if it fails to sufficiently distance itself from the “best-in-class” blockchains (at least based on market cap), it would quickly run into the same technological limitations, since, as we detailed above the transaction count and timing are particularly difficult for this type of technology and the basic needs of a PEC-oriented blockchain would exceed the capacity of either Bitcoin or Ethereum. Blockchains themselves mentioned both Ethereum and existing exchanges in their original filing (although in future filings they will drop all reference to external crypto and eventually even explicitly make clear that this is not Bitcoin).</p><p>Before I continue I do want to say I’m not denigrating the NV PUC here. They were presented with real issues in the extant tracking system by NV Energy and were responsive to new technologies, engaging with them in good faith, which is cool. Unfortunately, particularly given how it ultimately ended, it’s difficult to believe the blockchain peddlers were operating in similarly good faith. Let’s dip back into the timeline of filings.</p><h4>September 2019</h4><p><a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/41618.pdf">NV Energy files a letter</a> on behalf of itself, the Energy Web Foundation, and Blockchains LLC with a list of follow up items ostensibly derived from issues discussed during a workshop earlier in the year. NV Energy also discloses that they had entered into a Memorandum of Understanding (MOU) with Blockchains LLC to “collaborate and incubate public blockchain technology” in October of 2018, which was after the NV PUC commissioner first filed in support of the concept but before the official opening of the docket the following spring. There are three proposed followups:</p><ol><li>Commission a summary report on how RECs are being tracked elsewhere and identify tracking methods that could be studied “in addition to the public blockchain, if any”</li><li>Schedule workshops to education stakeholders on a public blockchain solution</li><li>Charter a public blockchain based on PEC tracking proof of concept pilot project.</li></ol><h4>June 2020</h4><p>The <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45147.pdf">Nevada PUC asks</a> interested parties to file comments on potential next steps for the docket.</p><h4>July 2020</h4><p>NV Energy submits a filing providing an update on the proof of concept. The proposed pilot will use an Energy Web Chain, “EW Chain”, which will utilize smart contracts and integrated software to prove origin, amount generated, and issue digital PECs. This high level plan is supplemented by next steps which include: the need for technical engineering requirements for software development and integration, development of user interfaces, identification of volunteer households, <strong>complete development of the integration of a blockchain enabled smart meter</strong>, integration of a blockchain client with existing utility processes. I bolded the blockchain enabled smart meter because that’s a huge lift that seems a bit out of place with some of the other items. Like, perhaps a proof of concept outside of a smart meter first? The NV Energy + Blockchain group proposes a detailed pilot will be available by the end of summer 2020.</p><p>On the same day the NV PUC <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45505.pdf">staff file</a> and propose four items:</p><ol><li>An educational workshop (first suggested in the September, 2019 NV Entergy filing) on blockchain technology;</li><li>Workshops on non-blockchain technologies;</li><li>Asking for a detailed written status update on the NV Energy+Blockchains proof of concept pilot project including goals, timelines, metrics, and costs;</li><li>A public workshop on the NV Energy+Blockchains pilot project.</li></ol><p>The staff filing concludes with additional specific questions on benefits and costs of a blockchain-based solution vs other alternative solutions as well as the status quo, and makes clear that any new solution must substantially exceed the current process.</p><h4>October 2020</h4><p>NV PUC files <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45757.pdf">Procedural Order №3</a> which, in part requests information on the description and status of the blockchain pilot, a summary of the potential solutions, and updates on the cost and timeline.</p><h4>November 2020</h4><p>NV Energy and Blockchains file a <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45790.pdf">status update</a> indicating progress. They claim that Blockchains is working closely with Sensus, which makes smart meters, on integrating their software into a meter. There is also mention of a preliminary financial model, but inputs have not been populated yet, so cost estimates cannot be provided. Integration with the Sensus meter is expected by February of 2021, followed by validation in March. NV PUC staff <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45789.pdf">also filed</a> on this date, mostly indicating that they had yet to receive any information from NV Energy and Blockchains, but did expect to shortly.</p><h4>Now back to our friend, Google</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*3wzBiYzK2ehYw6_4XyOMdQ.png" /><figcaption>If you search for just blockchain (without the s) you do get 5 results, none of which are relevant or co-temporal with the November filings.</figcaption></figure><p>Hmm, so there’s nothing on Sensus’ site about integrating Blockchains’ technology into their meters. I also searched for “incubed”, which Blockchains stated was their software, but it also returned 0 results. You’re free to scrounge google results without the site filter, but:</p><ol><li>I didn’t see anything useful there myself, and;</li><li>If this was as big a deal as presented or something considered valuable or in use anywhere, it would probably be somewhere on Sensus’ website.</li></ol><h4>January 2021</h4><p>In January of 2021 <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/45983.pdf">a workshop is held</a> on the blockchain, it’s uses and a tease of what the final app might look like. What I find most interesting in this presentation is trying to think through the use cases presented and determine why a blockchain presents any incremental value over a centralized database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*pUV6WpAQvVDJ7Y0XckUVWA.png" /><figcaption>A decentralized identifier is just a UUID I guess?</figcaption></figure><p>Ultimately, everything described in the presentation has to refer back to some other “off-chain” repository of information for what energy assets can participate in in terms of market products or what compensation they may receive. Additionally, some market signal would have to arrive onto the chain and then slowly work its way to the correct devices. The entire way these immutable, public ledgers prevent man-in-the-middle attacks is by making them prohibitively expensive. This has the knock on effect of making any on-chain transaction quite slow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*FB3ummY3tFYkng5Qm5FEPQ.png" /><figcaption>All of this can be stored in a normal database for much lower costs and with much higher performance. Additionally, how do they envision confirming compliance with those local rules and regulations? It inherently requires interfacing with an off-chain centralized authority.</figcaption></figure><p>There was also quite a bit that was not in scope of the proof of concept at this point, which seems to have been downgraded from pilot during one of the interim periods between filings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*62nImmYh8GLR5k77LLXwyg.png" /></figure><p>The presentation ultimately claimed that the proof of concept would be implemented between January and summer of 2021.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*azghqmTmrJ7t1ZzBdYgc6Q.png" /></figure><h4>December 2021, The Beginning of the End</h4><p>There was no pilot or proof of concept before summer 2021, and in fact there never will be. In December, NV Energy made its <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/46415.pdf">final filing on behalf of the blockchain-pilot stakeholders</a>. This filing admits that very few of the previously proposed goals were met. It states that:</p><blockquote>“Blockchains designed and built a prototype application comprised of a cellphone application and web portal which would allow the owners of small renewable generation systems to register their PECs, and the Commission to manage these PECs.”</blockquote><p>Achievements included: user onboarding, certification of issuance of PECs, tracking balance of PECs, web portal development, test net validation of meter data, and “enable users with interface to track, claim, trade, and retire PECs on the blockchain.” Until the final item, which has questionable value to this program as a whole, most of these tasks were either already available in some form or within the capabilities of any half competent webdev shop to design and implement in a quarter or less.</p><p>What was not accomplished included: performing an exchange (i.e., any sale, transfer or retirement of PECs), enabling any small-scale device (presumably this means non-dedicated computer hardware) to connect to the blockchain and utilize a decentralized application, no Incubed integrated device was demonstrated and it is engineers from Sensus and Blockchains ceased work on the smart meter project due to “software incompatibility” and the high level of effort required to make it work, demonstration of advantages of a blockchain process compared to the status quo.</p><p>That last one is pretty damning, and I think it’s worth quoting some of the bullet:</p><blockquote>“Although the [proof of concept] Collaborators initially provided documentation of potential ways to achieve a streamlined solution incorporating these features, implementation of the [proof of concept] itself could not accomplish this objective. Any such digital transformation initiative is unlikely to prove more cost-efficient than a centralized solution.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*hjT4xoWzPKtpQsIAJKWkeg.gif" /></figure><p>Essentially their accomplishments were to get some users and PECs to register on the web and also generate the unique IDs that identified their PECs as the sole inhabitants of a blockchain wasteland. The data they used was post-processed results that NV Energy provided, and yet they still claim within the outcomes that the proof of concept demonstrated use of the blockchain to measure, manage and track POCs. The one positive outcome seems to be the automatic issuance of PECs to participants, but, again, the data was collected and pre-processed by NV Energy before any interaction with their test chain.</p><p>Finally, the outcomes return to cryptocurrency issues I raised many paragraphs ago: price volatility and gas fees (transaction costs). The gas fee in EW Chain’s native token is so volatile that they did not even attempt to determine potential costs of this project were it to move forward. Which makes sense, because, remember, EW Chain is derived from the Ethereum stack and retains all of its issues. Why would anyone pursue a technology that would be less efficient at compliance, leading to things like a <a href="https://coinmarketcap.com/alexandria/article/3-minute-tips-what-are-gas-wars">gas war</a>?. Either their confidence bands were so broad as to be useless, or their actual confidence in the project so low they determined it wasn’t even worth presenting the PUC with a number. Ultimately NV Energy states that:</p><blockquote>“based on the experience developing the [proof of concept], NV Energy does not view the [proof of concept] as a viable option for registering, tracking and retiring PECs, and the [proof of concept] Collaborators do not intend to pursue any further analysis beyond the conclusion of the [proof of concept]. NV Energy does not recommend that additional resources or time be dedicated to the POC.”</blockquote><h4>January 2022</h4><p>The Nevada PUC <a href="https://pucweb1.state.nv.us/PDF/AxImages/DOCKETS_2015_THRU_PRESENT/2018-9/46518.pdf">closes the docket</a> on 1/25/2022. This prompts almost no press (infinitesimally less than any of the breathless coverage of the latest meme coin’s price perturbations), and some social media ripples in interested communities, mostly of the “Huh, I wonder what happened there?” variety. I was one of those folks, except I like reading dockets and allowed that vague curiosity to get the better of me. Now that I’ve spent the entirety of my Thursday evening (and a tiny bit of Friday morning) satisfying that curiosity I think it’s time to go to bed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df021f029d12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring Neighborhood Diversity with Python]]></title>
            <link>https://medium.com/@waldoch/measuring-neighborhood-diversity-with-python-f32574f79ff0?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/f32574f79ff0</guid>
            <category><![CDATA[census]]></category>
            <category><![CDATA[arlington-county]]></category>
            <category><![CDATA[demographics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Mon, 28 Sep 2020 01:59:21 GMT</pubDate>
            <atom:updated>2020-09-28T01:59:21.082Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9hPDAVuMMPvp3NaV2GUig.png" /><figcaption>22204!</figcaption></figure><p>Whether I’m out for a run, chatting with neighbors, or <a href="https://parks.arlingtonva.us/2018/08/tyrol-hills-park-features-new-futsal-court-picnic-shelter-and-restroom/">playing soccer in the local park</a> I’m struck by the diversity of my neighborhood. Wedged at the far southwestern corner of Arlington county VA, along Columbia Pike, zip code 22204 is home to some of the busiest bus routes in the country and all kinds of people. Local restaurants are particularly varied, from longstanding Indian, Thai, and Chinese establishments to wholly Ethiopian strip malls, and plenty of pupuserias. Questions like “How diverse is my neighborhood?” tend to accumulate in the slow rock tumbler of my mind, not necessarily gaining much polish, by grinding against my curiosity until I give in and spend a Saturday evening corralling some data. So, how diverse is my neighborhood? But first, where do I start?</p><p>I know you aren’t supposed to use zip codes. I know they’re not real geography, that they’re far more arbitrary than a census tract, and that they have a whole host of other issues when working with them in a cartographic context, but people know zip codes. I wouldn’t be surprised if fewer than 1 in 25,000 adults knows their census tract, but most of them probably know their zip code. So, let’s just get the demographics of every zip code in America, but where do we get that? <a href="https://www.census.gov/">The Census Bureau</a>.</p><p>The census website is one of those that feels like it has undergone multiple revisions in just the past few years. The venerable Fact Finder was even <a href="https://www.census.gov/newsroom/press-releases/2020/american-factfinder-retiring.html">retired earlier this year</a>. Now, everything is available on <a href="https://www.census.gov/data.html">their data.gov platform</a>, but after some poking, prodding, and querying I didn’t come up with the “simple” flat file I was looking for, so I redirected my energy towards the API. Of course, the API was probably where I should have started. As with most federal data repositories with structured access some kind soul had already developed a Python library. I quickly found, installed, and started using <a href="https://pypi.org/project/CensusData/">CensusData</a>, a dead simple and effective way to access the Census Bureau’s API. One nice thing about this API is that unlike, say, <a href="https://www.eia.gov/opendata/">the EIA</a>, you don’t need a key or developer credentials to carry out any basic querying.</p><p>Instead of relying on querying the documentation I needed from the API, I downloaded the <a href="https://www.census.gov/programs-surveys/acs/technical-documentation/table-shells.html">Table Shells flat file</a> and used the table IDs there to build my request. After a little testing I came up with the query I wanted:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*h4PVBtB-sV0OLqgJM8tNYg.png" /></figure><p>I knew I was going to use the <a href="https://www.census.gov/programs-surveys/acs">American Community Survey (ACS)</a>, and from poking around on their data.gov site it seemed the most recent complete results on the tables I wanted were from 2018. Those table IDs are topline results by race, not delving into very specific countries or regions of origin, they are: Total, White, Black or African American, American Indian, Asian, Pacific Islander, Other Alone, Two or More, Hispanic or Latino. It’s important to understand that these labels and categories have changed over time, you can’t necessarily compare the values from 2018 to the ones form 1990 that used entirely different terms and grouping. Additionally, these values are based on self-reported data, and are therefore susceptible to whatever caveats you want to apply based on that. Regardless, it’s the best option we have, so we go with it.</p><p>Once I had the data I mapped in different column names and put the index in as a column, I also created a new “White” column by subtracting the Hispanic or Latino total, since at the topline that group is aggregated into “White”*:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*HifE6CgD-wMho3_8lJeDdA.png" /></figure><p>Then, I got the list of columns with the population values I wanted to analyze, and looped over them, appending new columns with their percent share of the total population:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*ihDB_l5swaYN9wQV5OmkMA.png" /></figure><p>Then I take the max value for each zipcode in terms of pct. This is the largest racial group in the zip code, and I reasoned that the smaller this value, the more diverse the zip code. For a first pass, this was pretty effective in defining the most diverse zip codes in America. I added a rank column to the data, filtered it to the top 100, added a column specifying what the max value race was, and appended zipcodes as an integer column before exporting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*v4y2upri3D6xCPc966OT4Q.png" /></figure><p>At this point I had a neat csv of zip codes and demographic data and was able to confirm that mine had made it onto the list at #70.Out of 33,120 zip codes, I live in the 70th most diverse (by my first metric). This is the top 0.2% of zip codes, but the top 0.6% in terms of population since the total population of these top 70 zip codes represents ~2.5x the population of the average zip code. One thing I’m not doing is removing tiny population zip codes. There are a handful of zipcodes in this top 100 with less than 1000 people, but I decided to go with completeness over an arbitrary filter that seemed reasonable to me.</p><p>Next, I cast about for a more nuanced metric. I don’t have a great sense of what statistics are used in demographic modeling, but I was able to find a <a href="http://www.csun.edu/~hcmth031/MDITUS.pdf">useful short paper</a> from 2002 in Math Horizons, a journal that explores math concepts for undergrads, which is a very reasonable level for me, someone who was in college a while ago and who very much did not major in math. This short overview article described two metrics I ended up adapting, Simpson’s diversity index:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*azvGkV_fj1vhX4hy-pMFdw.png" /><figcaption>Simpson’s diversity index</figcaption></figure><p>and Shannon’s diversity index:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*ewiVBAte9g4QZpPAJKRCpQ.png" /><figcaption>Shannon’s diversity index</figcaption></figure><p>Each of these uses a different representation of the rareness of a species (or, in my case, a demographic descriptor), and comes from the field of ecology. The rarer a species, the larger the R value. The one pitfall here is correcting for 0 values in the data where there is no presence of a specific race as this will cause an error in Shannon’s index. In the code this is simple, you just don’t include those values in the process which calculated. If you already exported everything to Excel and are tinkering manually, you can put in a tiny value for the 0s, such as 0.00000000000001. Larger numbers will dramatically affect the result, but a very small value returns nearly nothing and therefore has essentially no effect in this ad hoc implementation.</p><p>After applying these new indices to my curated list of 100 zip codes, my neighborhood ranked 51st using Simpson’s, and 56th with Shannon’s. If we average the three rankings, it ends up at #59. Anywhere in the top 100 I think easily qualifies for one of the most diverse zip codes in America, but I was satisfied to see that my intuition was correct. Of course, you don’t have to go very far to find the opposite. Northern Arlington is very expensive, and, relatedly, quite white:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HOaeNDr3mqSalskP2LHsTg.png" /></figure><p>If we look at zip code 22207, it ranks 10,916th on the original metric, falling into the vast middle swathe of American zip codes with a 75%+ White majority. My hometown is in 60134 (a far Chicago suburb), and with a 91% White majority comes in at 18,671st.</p><p>Returning to the top 100 most diverse zipcodes there are a few interesting tidbits to close out:</p><ul><li>California has the most zip codes in the top 100, with 29, but New York is third with 10. Hawaii is second, at 14, which makes sense if you think about it, but might not have been readily apparent. Hawaii also has 4 of the 5 zip codes where the dominant race is “Two or More”. The fifth of these is a tiny 63 person zip code in California, so perhaps Hawaii is the only state with any zip codes that have that majority outside the bounds of the error bars;</li><li>Only 22 states (and DC) are represented at all;</li><li>The West Coast has 50 of these zip codes, the Mid-Atlantic/East Coast 25, the South 13, and the Midwest and Interior West have 6 each;</li><li>Hispanic or Latino are most commonly the largest group, with 34, then White at 33, Asian at 17, Black or African American at 10, Two or More at 5, and Other Alone with 1.</li></ul><p>That’s it! I satisfactorily answered a question I’ve had for a while, and hopefully this little writeup can encourage other people to explore some of these data resources.</p><p>*This is not precisely correct, but is close enough for the purposes of my exploration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f32574f79ff0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fuel Through Time, EIA API+Geopandas]]></title>
            <link>https://medium.com/swlh/fuel-through-time-eia-api-geopandas-3cfaf9f2866a?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cfaf9f2866a</guid>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[virginia]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[geopandas]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 20:16:01 GMT</pubDate>
            <atom:updated>2020-07-18T02:07:18.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wradWN3Z5l7r_DdWhSXhEQ.png" /><figcaption>Static image for the header, gif below.</figcaption></figure><p>I’ve been doing some python mapping recently, and am now getting around to instantiating the GitHub repos for these projects. I try to write my README files a bit narratively, enough so that I think they’re useful content for Medium as well. Here’s the first of those, which examines fuel use and generation at Virginia power plants from 2001–2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DTCqiXY0vmW6rnmbgzbQrg.gif" /></figure><h3>Intro</h3><p><a href="https://github.com/cwaldoch/fuel_through_time">This repo contains the code and data for making a map (in gif form) of generation by Virginia power plants 2001–2019</a>. The plotted circles are colored by fuel type and sized by the amount of generation. While the data in this repo is only for Virginia, the script I used to go through the EIA’s API and collect the data is also included, as well as a reference file from EIA form-860 and a national version of the Virginia data. Let’s run everything down.</p><h3>Data Collection</h3><p>I’ve already done some manual processing with the full EIA-860 form and placed that output in this repository. That form has data on individual generators, including their location. It also has proposed and canceled or retired plants. I collated those tables into one file and also added a “color” column for a different project, which I use here as well. These colors are assigned based on the “Reported Fuel Type Code” which you can find in the File Layout tab of EIA forms that contain fuel codes. However, Form-860 only has capacity, and the annual 923 forms are too tedious to grab individually for fuel and generation, so we turn to the API.</p><p><a href="https://www.eia.gov/opendata/">The EIA API</a> is pretty easy to use, particularly since a few excellent humans have already written python wrappers for it. After you’ve gotten your API key, you can pip or conda install EIA_python, the package I’m using here. The “process_eia.py” file contains the code to pull generation for every year available for each plant. The list of calls is constructed from a plant+fuel combo list I made from the EIA form data. I also fold in lat and lon data from EIA-860 so that we can have one file with plants, fuels, and their coordinates, which are necessary to plot/map them.</p><p>To change the state output, you can change the second line below to either filter to a different state or remove the filter and pull data for every plant in the country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*vgshKV4zjUU0srAJlNW8hw.png" /></figure><p>I do have one error exception in this API handling, which should only trigger for plants that are in my underlying data, but haven&#39;t been built yet and therefore don’t exist in the fuel/generation API data even though they have a plant code and location.</p><h3>A bit on the geography portion</h3><p>The gis_data folder has a shapefile of counties in Virginia. You’ll want a shapefile of whatever geography you’re working with. If you don’t know what a shapefile is, you can download QGIS for free and start messing around with GIS. Most data-heavy federal offices/departments will have a bunch of shapefiles (BLS, <a href="https://www.eia.gov/maps/layer_info-m.php">EIA</a>, <a href="https://www.census.gov/geographies/mapping-files/time-series/geo/carto-boundary-file.html">Census</a>) available for free, as will your state and often your local town or county these days with <a href="https://gisdata-arlgis.opendata.arcgis.com/">open data portals</a>.</p><p>There are two potential pain areas with this code, and the shapefile/GIS management is one of them. It’s very easy to mess up the cartographic projections piece and end up with a blank or broken map. Fortunately, you will get an error from geopandas in the console (although the code <em>will</em> keep running), so typically you know what’s gone wrong. Despite knowing your crs is messed up, it’s still easy to lose track of what coordinate system everything is in. It’s easiest to handle it up front with every new piece of geographic data you create or import. You can check the crs on an imported shapefile with “.crs”.</p><p>In the example below, I’m importing a shapefile of Virginia counties, putting it into the GeoDataframe format, checking the crs, and then converting it to the project crs I want to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*OByEDAGBJgp6aYc4avREVA.png" /></figure><p>EPSG:32618 is actually not a great crs for Virginia, but I was re-purposing this code from something else, and since I knew it worked just kept using that. An easy upgrade would be to change the crs to something more locally relevant.</p><h3>Setting up the environment</h3><p>The other major potential issue is just the geopandas working environment. Geopandas can be a bit finicky. Sometimes I have no issue with my local installation in a WinPython folder that I installed with “pip install geopandas”, but other times it kills the kernel in odd ways. The safest and msot reliable way I’ve found to execute different geopandas code is to set up a specific environment with an Anaconda installation. <a href="https://geopandas.org/install.html">I used the instructions in the geopandas docs</a> and that has worked great so far. One thing to be aware of is that I do use a few additional libraries/packages (ImageMagick, ffmpeg) for the gif processing, and you’ll want to make sure those are installed in the geo_env you set up so that they’re accessible to the environment.</p><p>You might not need to set up an environment. This piece of code has been fine for me to run in just my regular installation, but when I do something more complicated, like a <a href="https://twitter.com/ConnorWaldoch/status/1279159266788737024?s=20">grouping of generation into a hex overlay</a> it crashes 100% of the time unless it’s in the prepared geopandas environment.</p><h3>The Main Loop</h3><p>The main loop consists of lines ~50–110 (at the time of writing this), and it sets up the data structures to be plotted inside the matplotlib animation function. I iterate over the years in the generation data, then get the plants that existed in that year, then get the fuels used at that plant in that year, and then use those values to create the underlying geopandas GeoDataFrame to be plotted. Each year-plant-fuel combination is appended to a list, “circlesArray”. This contains a slice of the geographic data for the point in question, which has been transformed into a polygon circle through use of the buffer function from geopandas, and then values pulled in from other datasets, such as the color of the bubble (based on fuel type).</p><p>All of the color values are being divided by 255, because when I manually chose fuel type colors originally I wrote them as 1–255 values instead of 0–1. The first 3 values are rgb, and the 4th is alpha, or transparency. I usually include a bit of transparency so the underlying borders can be seen, as well as other bubbles.</p><h3>Plot Visualization</h3><p>This part really depends on your mindset: what you’re trying to convey and what you think looks good. You can completely modify how the circles are sized, their colors, transparencies, whether or not they have a border, etc. The circle size is one of the trickiest things to get right, and may require some manual examination of the underlying data along with a lot of iteration. Typically, there are 2 ways to determine the size:</p><p>1). Apply one formula to everything, e.g., size = generation*1000 2). Decide breakpoints and manually classify, e.g., if generation &gt; 10 then size is 1000, generation &gt; 20 size is 2000, etc.</p><p>The range here is really big, so I end up doing both. From my perspective, I want to show every plant, even if they didn’t run, to show that they exist. I classify the first few generation ranges, then apply a formula to everything over 250,000 MWh.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*wUGu9WpgRdjfVNPdmAhkZg.png" /></figure><p>Earlier in the code I use a normalization function from matplotlib to set up the “normBubble” function to take in a value and return a new one from 0–1. Now, realistically, it’s important to realize that the vast majority of generation is coming from a very small number of plants compared to the amount actually in operation, but scaling those to their “true” size relative to everything else looks pretty terrible, so I’m making a trade-off between accuracy and legibility. I prefer someone looking at this map to be aware of the variety and location of different generation resources in addition to parsing “Wow, the nukes generate a lot!”</p><p>I manually create the legend for the graph as well by appending patch objects with the fuel colors inside the animate function. I’ve commented out some of the patches because Virginia doesn’t have certain power plants, like wind or geothermal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*rMFy8KprMj6oOyYfTfQ8LQ.png" /></figure><h3>That’s it</h3><p>If you’re confused or curious, feel free to <a href="https://twitter.com/ConnorWaldoch">hit me up on twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cfaf9f2866a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/fuel-through-time-eia-api-geopandas-3cfaf9f2866a">Fuel Through Time, EIA API+Geopandas</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HHI at the Movies Redux: Disney Ascendant and sad BoxOfficeMojo]]></title>
            <link>https://medium.com/keeping-stock/hhi-at-the-movies-redux-disney-ascendant-and-sad-boxofficemojo-af0f6ea4ce58?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0f6ea4ce58</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[disney]]></category>
            <category><![CDATA[market-concentration]]></category>
            <category><![CDATA[box-office]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Sun, 12 Jan 2020 00:17:35 GMT</pubDate>
            <atom:updated>2020-07-17T19:26:56.519Z</atom:updated>
            <content:encoded><![CDATA[<h3>HHI at the Movies Redux: Disney Ascendant and Sad BoxOfficeMojo</h3><p>I’m finally revisiting one of the only things I’ve ever posted that somehow continues to get some views each week more than two years after I wrote it, <a href="https://medium.com/@waldoch/herfindahl-hirschman-at-the-movies-b0430acf7cee">Herfindahl-Hirschman at the movies</a>.</p><p>The deal took a lot longer to go through than it seemed at the time. Announced in late 2017, it wasn’t fully on the books until March 2019. Not an eternity in the world companies this large inhabit, but a bit slower than people were anticipating for a successful deal that faced limited scrutiny at the time.</p><p>In the two years since I originally wrote that post, BoxOfficeMojo, an old internet workhorse, was finally crushed and ground to dust by Amazon after 11 years. From that dust a decidedly 2010s redesign has arisen, replete with “clean” design and a lot of data and other information locked away behind the IMDBPro paywall. This sucks, <a href="https://www.thewrap.com/box-office-mojo-criticized-after-redesign-that-includes-a-paywall-for-some-data/">and it only happened last October</a>. Fortunately there’s still enough there for me to do this analysis, but I was hoping to extend things a bit and generate some fun additional figures (mergers by genre, director, etc.). In fact, my original plan for this update was to wrap everything in a <a href="https://plot.ly/dash/">dash framework</a> and deploy it via AWS. I might still do that eventually, but my original use case isn’t totally viable since I don’t plan on subscribing to IMDBPro.</p><h3>Code Updates</h3><p>With the redesign came one nice feature. All movies in a year are on a single page. Originally I was pulling the total number of release in a year from somewhere else, and then iterating over the number of pages provided by:</p><blockquote>(number of releases)/(rows per page) = number of pages per year</blockquote><p>It wasn’t the most efficient way of doing things, but it was the first thing I thought of (I tend to do the coding and writing for anything like this in a single sitting) and it worked.</p><p>This time all it took was:</p><pre>import pandas as pd<br> <br>def moviesGrab_v2(yearStart, yearEnd):<br>    linkBase = r&#39;<a href="https://www.boxofficemojo.com/year/&#39;">https://www.boxofficemojo.com/year/&#39;</a><br>    <br>    for year in range(yearStart, yearEnd+1):      <br>        dfsYear = pd.read_html(linkBase+str(year), header = 0)<br>        <br>        if year == yearStart:<br>            dfAll = dfsYear[0]<br>            dfAll[&#39;Year&#39;] = [year] * len(dfAll)<br>        else:<br>            dfYear = dfsYear[0]<br>            dfYear[&#39;Year&#39;] = [year] * len(dfYear)<br>            dfs = [dfAll, dfYear]<br>            <br>            dfAll = pd.concat(dfs, ignore_index = True)<br>        <br>    return(dfAll)</pre><pre>allYears = moviesGrab_v2(1977, 2019)</pre><pre>allYears.to_csv(&#39;all_year_boxoffice.csv&#39;, index=False)</pre><p>I didn&#39;t have to change the processing and calculation code much, just point things to the correct column names. I also moved the calculation aspect into a function so that I could just call it a few times instead of slowly outputting each unique set of results. <a href="https://github.com/cwaldoch/HHI_at_the_movies">As always, the complete code is on GitHub.</a></p><p>Back to that redesign, I ran into something that briefly confused me. The public table has some hidden columns when you read the HTML directly. The table you pull with pandas.read_html returns 11 columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*uG4Kvrtek-AQ5qPD9yKpqg.png" /></figure><p>But the public page only has 7 columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*MJy1ZHwd28vF65iYOx1iYA.png" /></figure><p>Genre, Budget, Running Time, and Estimated are missing. It’s pretty clear what the first three are, and it sucks that they’re not available here. To some extent I can understand budget, as that could be perceived as more direct business information, but genre and running time are incredibly easy to find elsewhere, including basic IMDB. To be fair, this information is available when clicking on an individual film and viewing its page, but it would be far more convenient if everything was in a single place. Because those columns exist in the HTML, I can only assume this view is available for “Pro” users.</p><p>It wouldn’t be terribly onerous to strip all the links to individual movies and pull their additional data one by one, but it’s annoying! And, more things could go wrong. Instead of making less than 50 calls to the site, it would require thousands. Not much in the grand scheme of things, but who knows what they’re tracking or blocking in this redesign.</p><h3>Results</h3><p>As a quick reminder, the Herfindahl-Hirshman Index (HHI) is a measure of market concentration. From the <a href="https://www.justice.gov/atr/herfindahl-hirschman-index">Department of Justice</a>:</p><blockquote>The term “HHI” means the Herfindahl–Hirschman Index, a commonly accepted measure of market concentration. The HHI is calculated by squaring the market share of each firm competing in the market and then summing the resulting numbers. For example, for a market consisting of four firms with shares of 30, 30, 20, and 20 percent, the HHI is 2,600 (302 + 302 + 202 + 202 = 2,600).</blockquote><blockquote>The HHI takes into account the relative size distribution of the firms in a market. It approaches zero when a market is occupied by a large number of firms of relatively equal size and reaches its maximum of 10,000 points when a market is controlled by a single firm. The HHI increases both as the number of firms in the market decreases and as the disparity in size between those firms increases.</blockquote><p>I’m using box office results as a handy proxy for market share under the assumption that each consumer has a pool of potential movie-going spending. Something I find interesting about this concept is that movie studios can expand their potential market in a number of ways, but it really all comes down to expanding that pool of money, what do people want to see? It’s not unlike other markets, but it feels different due to the potential for extremely individual tastes that, in aggregates seem to have rolled up into a demand for sequels and franchises recently, often based on pre-existing IP. It’s why categorizations like “Original movie that made over $100M domestic” mean something, each movie is doing market research for everything else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NmWPK0qYsyMU70k2-XNxiw.png" /></figure><p>I’ve put four lines on the plot this time around.</p><ol><li>As Listed: Simply taking each Distributor at face vale and calculating HHI as if they were all unique.</li><li>True Owner: I went through and manually classified Distributors to their current ultimate corporate parent. I did this for Distributors that contributed more than $285B of the $286B total box office take over this timeframe. This is why the concentration is higher than my last go around. While I did reclassify some other studios, I wasn’t as thorough and it was more difficult to match the studio abbreviations I had accidentally pulled at the time.</li><li>Small Consolidation: Not really consolidation at all, I just didn’t know how to refer to this. Basically, it groups together studios that are *extremely* clearly related. For example, everything with Sony in the name, or Twentieth Century Fox, Fox Searchlight Pictures, and Fox Atomic.</li><li>Disney Passes: True Owner, but Disney didn’t buy Fox. This really demonstrates what a massive acquisition that was.</li></ol><p>I notice a few interesting trends from this figure. First, the 2000s were relatively competitive. Since I’m not an industry insider or even much more than a casual moviegoer I can only speculate, but it sure seems like the advent and widespread adoption of digital cameras may have spurred on a mini-boom in terms of movie studio diversification. The costs associated with compute time and digital products declined quite a bit over this period. Concentration began declining in the mid 1990s, reaching its nadir in the early 2000s for the Small Consolidation and As Listed calculations. I’m referring to those rather than True Owner because a number of smaller studios that have been acquired in the past ten to fifteen years were still independent at the time.</p><p>However, by the mid-2010s concentration has returned to or exceeded the level in the 1980s and early 1990s. It didn’t stop there though, accelerating dramatically in the latter half of the past decade.</p><p>Returning to the DoJ:</p><blockquote>The agencies generally consider markets in which the HHI is between 1,500 and 2,500 points to be moderately concentrated, and consider markets in which the HHI is in excess of 2,500 points to be highly concentrated. <em>See</em> U.S. Department of Justice &amp; FTC, <a href="https://www.justice.gov/atr/public/guidelines/hmg-2010.html#5c"><em>Horizontal Merger Guidelines</em> § 5.3</a> (2010). Transactions that increase the HHI by more than 200 points in highly concentrated markets are presumed likely to enhance market power under the <em>Horizontal Merger Guidelines</em> issued by the Department of Justice and the Federal Trade Commission. <em>See id.</em></blockquote><p>So it’s a moderately concentrated market, which isn’t a problem on its face. The issue is in the drastic jump in concentration with Disney’s acquisition of Fox, an even higher concentration than I determined last go around thanks to more thorough distributor mapping. If the DoJ was at least examining the 2016 and 2017 box office (the deal was first announced in late 2017) during their review I find it hard to imagine they didn’t see a huge leap in concentration there (841 and 697 points in my results). I bet the Disney lawyers had a great time pointing out that streaming exists and Netflix was spending $$$ on movies, with Amazon and others on track to do the same, and that was just the same as in-theater distribution. Of course, there’s no chance they mentioned their own plans to launch a streaming service anchored not just by traditional Disney, but all of the other IP and #content <a href="https://www.cartoonbrew.com/disney/chart-every-company-that-disney-owns-172130.html">they’ve acquired</a> over the past 20 years.</p><p>The boundaries of this market are tough! To some extent media is going to be interchangeable. Just today, I was trying to decide whether to see Uncut Gems (movie), watch some more of the Mandalorian (streaming TV), or keep playing Nier Automata (videogame). In the end, I decided to finish this post, but along the way I watched a caster I like cover an <a href="https://www.youtube.com/watch?v=mX2Oa26z690&amp;t=279s">Age of Empires 2 game on YouTube</a>, briefly checked in on <a href="https://www.twitch.tv/t90official">his Twitch</a>, and had <a href="https://open.spotify.com/playlist/37i9dQZF1Et9oUhi868BRZ?si=WsInIcjgQweR0L7Ntb0ToQ">Spotify going</a> pretty much the entire time, unless I was <a href="https://open.spotify.com/show/5AvwZVawapvyhJUIx71pdJ">listening </a><a href="https://open.spotify.com/show/07SjDmKb9iliEzpNcN2xGD">to </a><a href="https://open.spotify.com/show/1XxdBLxVohmBYKuinAdrZ9">a </a><a href="https://open.spotify.com/show/6mTel3azvnK8isLs4VujvF">podcast</a>. Suffice it to say, today’s media landscape is fractured in a really neat way (<a href="https://en.wikipedia.org/wiki/2019_merger_of_CBS_and_Viacom">although</a> <a href="https://www.complex.com/pop-culture/2019/11/disney-acquisition-timeline">that’s </a><a href="https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitions_by_Amazon">snapping </a><a href="https://www.washingtonpost.com/nation/2020/01/09/collegehumor-shuts-down/">back</a>) with innumerable sources competing for any one person’s attention.</p><p>Despite the still-somewhat-disparate broad media market, the roughness of this HHI proxy, and the ultimate decision to let the merger proceed it’s hard to see a case for the seeming lack of scrutiny. If the first market share proxy I can come up with produces a dramatic increase in concentration I’d be concerned.</p><h3>A Bit More</h3><p>Beyond that, Fox was one of only 5–6 companies that were huge players. Here’s Disney buying a few of those other studios in place of Fox, all resulting in huge jumps in concentration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBvIS7NWVtCozfnL_15QSg.png" /></figure><p>The other larger studio owner is ViacomCBS. There are only 3 “mid-size” studios in terms of lifetime (1977–2019) box office: LionsGate ($12B), MGM ($9B), and Dreamworks ($9B).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*kS3NUsxywZ_s4RaQgQicOw.png" /><figcaption>Total Box Office Take &gt; $1B (1977–2019)</figcaption></figure><p>Fox would have been 5th in that table, slightly below Sony. Altogether, this seems like just another story in the consolidation that seems to be happening in every sector, at least per the renewed interest in assessing and responding to consolidation across the board (not just in media, but the tech, industrial, and agriculture spaces, as well as slightly more abstract areas like wealth and income).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0f6ea4ce58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/keeping-stock/hhi-at-the-movies-redux-disney-ascendant-and-sad-boxofficemojo-af0f6ea4ce58">HHI at the Movies Redux: Disney Ascendant and sad BoxOfficeMojo</a> was originally published in <a href="https://medium.com/keeping-stock">Keeping Stock</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Links and Bits from the back half of 2018]]></title>
            <link>https://medium.com/@waldoch/links-and-bits-from-the-back-half-of-2018-f7b6a9aaffe?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7b6a9aaffe</guid>
            <category><![CDATA[year-in-review]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[politics]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Mon, 31 Dec 2018 18:36:20 GMT</pubDate>
            <atom:updated>2018-12-31T19:14:05.973Z</atom:updated>
            <content:encoded><![CDATA[<p>This year got *busy*. I started off strong, writing at least 1–2 times a week for the first few months of the year. Then I got engaged, my fiancé finished grad school and got a full-time job, we started house hunting (which was mostly a <a href="https://ggwash.org/view/69885/we-recently-bought-a-house-in-arlington-amazon-make-it-hard">tedious grind in which we were powerless</a>), and eventually entered the endless loop of home+yard renovation and reconfiguration. We also picked a wedding venue/caterer/etc. while living a 7–8 hour drive from the actual location, and have been generally gearing up for that event later on in 2019.</p><p>Around June I started keeping a list in my notes app. A list comprised mostly of interesting links, a few thoughts, the roughest of data sketches, perhaps the ghost of a post title. Typically I didn’t have time to immediately read the entirety of whatever the link was to, I wasn’t ready to learn a cool new tool, or I planned on fleshing out a thought later. Most of the interesting things I saw in the last 6 months didn’t end up on this list (for example, some things that won’t appear are basically any articles I loved in the New Yorker, simply because I tend to consume those on the spot without making a note to read them later), but it still ended up at 99 odds and ends. In end-of-the-year fashion I’ve gone through and looked at each and every one, then categorized them all here.</p><h3>The Categories</h3><ul><li>Tools and Useful/Interesting Data+Reports</li><li>Urban Design, Transit, Housing, etc.</li><li>Interesting Non-Categorized Reads</li><li>Modern Economics Papers, Reports, and Writing</li><li>Historical Economics Papers, Reports, and Writing</li><li>Energy and Environmental</li><li>Trump and Modern Politics</li><li>Internet Miscellany</li><li>Books (That I haven’t read)</li><li>Ideas</li></ul><h3>Tools and Useful/Interesting Data+Reports</h3><ul><li><a href="https://projects.propublica.org/api-docs/campaign-finance/">ProPublica Campaign Finance API</a> — Looks like an invaluable tool for tracking and visualizing this information. I haven&#39;t had a drive to do anything with it yet, but I’m very glad it exists. ProPublica consistently does excellent work.</li><li><a href="https://gijn.org/resources-for-finding-and-using-satellite-images/">A list of resources for finding <strong>and</strong> using satellite imagery.</a></li><li>YouTube: <a href="https://www.youtube.com/watch?v=aZ1G_Q7ovmc">PyViz: Easy Visualization and Exploration for all your Data</a>. A talk from SciPy 2018 on data visualizations via Python, focused on HoloViews.</li><li>Python: <a href="https://samirak93.github.io/analytics/Animation-and-Convex-Hull-in-Bokeh.html">Animation And Convex Hull In Bokeh.</a></li><li><a href="https://eig.org/dci">2018 Distressed Communities Index</a></li><li><a href="https://docs.google.com/document/d/1BfLPJpRtyq4RFtHJoNpvWQjmGnyVkfE2HYoICKOGguA/edit#">Bellingcat’s Online Investigation Toolkit.</a> And <a href="https://twitter.com/AricToler/status/1075853373260279808">a Twitter thread on how to use it.</a></li><li><a href="https://labs.spotify.com/2018/11/15/introducing-chartify-easier-chart-creation-in-python-for-data-scientists/">Chartify: Python Graphing from Spotify.</a></li><li><a href="https://medium.com/s/story/five-unusual-alternatives-to-pie-charts-389cc4676974">Five unusual alternatives to pie charts.</a></li><li><a href="https://research.umn.edu/units/uspatial/news/3d-printing-models-derived-lidar-data">3D Printing Models derived from Lidar Data</a></li></ul><h3>Urban Design, Transit, Housing, etc.</h3><ul><li><a href="http://japanpropertycentral.com/2018/08/japan-property-price-index-for-april-2018/">JAPAN PROPERTY PRICE INDEX FOR APRIL 2018</a>. Tokyo is a hell of a letter better at affordable housing than the biggest American cities.</li><li><a href="https://theoutline.com/post/5807/the-million-dollar-brownstone-that-no-one-owned?zd=1&amp;zi=ma426dzb">The million-dollar brownstone that no one owned.</a> An extremely interesting read. Get ready to learn about titles. Really appreciate these pieces diving more into things in and around the housing bubble and rotten systems that underpin our economy.</li><li><a href="https://www.buildzoom.com/blog/pockets-of-dense-construction-in-a-dormant-suburban-interior">America’s New Metropolitan Landscape: Pockets Of Dense Construction In A Dormant Suburban Interior</a>.</li><li><a href="http://eyeonhousing.org/2018/09/multifamily-units-by-building-size/">Multifamily Units by Building Size</a>, September, 2018 release of 2017 data via the Census Bureau.</li><li><a href="https://link.springer.com/article/10.1007/s11116-018-9923-2">The impact of ride-hailing on vehicles miles traveled</a>. “ When accounting for mode replacement and issues such as driver deadheading, we estimate that ride-hailing leads to approximately 83.5% more VMT than would have been driven had ride-hailing not existed.”</li><li><a href="https://usa.streetsblog.org/2018/11/23/u-s-finally-legalizes-modern-european-style-train-cars/">U.S. Finally Legalizes Modern, European-Style Train Cars.</a> Now municipalities can hopefully starting buying less custom and more standardized (and please dear god cheaper, rail cars).</li><li><a href="https://www.fastcompany.com/90276423/how-to-redesign-cities-to-fight-loneliness">How to redesign cities to fight loneliness.</a></li><li>Twitter: Kate Wagner’s (of McMansion Hell) <a href="https://twitter.com/mcmansionhell/status/1075577356033642497">self-selected best writing of the year.</a></li><li>The Economist: <a href="https://www.economist.com/united-states/2018/12/22/why-american-cities-are-so-weirdly-shaped">Why American cities are so weirdly shaped.</a></li><li><a href="https://www.dezeen.com/2018/12/19/owen-hatherley-moscow-post-soviet-space/">Modern Autocrats have Modern Cities.</a></li><li><a href="https://fivebooks.com/reader-list/a-crash-course-on-understanding-cities-menashedisrael/">A Crash Course in Understanding Cities.</a></li></ul><h3><strong>Interesting Non-Categorized Reads</strong></h3><ul><li>ProPublica: <a href="https://features.propublica.org/liberia/unprotected-more-than-me-katie-meyler-liberia-sexual-exploitation/">Unprotected</a>. “An acclaimed American charity said it was saving some of the world’s most vulnerable girls from sexual exploitation. But from the very beginning, girls were being raped.” <strong>Read the whole thing</strong>.</li><li><a href="https://medium.com/s/trustissues/the-lifespan-of-a-lie-d869212b1f62">The Stanford Prison Experiment was a sham.</a></li><li><a href="https://www.tor.com/2018/07/24/revealing-charlie-jane-anders-the-city-in-the-middle-of-the-night/">A preview excerpt of the next Charlie Jane Anders book</a>, <em>The City in The Middle of the Night</em>, publishing in February, 2019.</li><li>The Times: <a href="https://www.thetimes.co.uk/article/britain-s-role-in-the-rohingya-tragedy-pdmmkwzc7">Britain’s role in the Rohingya tragedy.</a></li><li>NY Times: <a href="https://www.nytimes.com/2018/09/29/upshot/mini-recession-2016-little-known-big-impact.html">The Invisible Recession of 2016.</a></li><li>The Nation: <a href="https://www.thenation.com/article/who-owns-kafka/">Who Owns Kafka?</a> On the complicated path of his papers and ideas and who owns or even has a right to any of it.</li><li><a href="http://Universal Basic Income Is Silicon Valley’s Latest Scam">Universal Basic Income is Silicon Valley’s Latest Scam.</a></li><li>New Yorker: <a href="https://www.newyorker.com/magazine/1993/04/05/secrets-of-the-magus/amp">Secrets of the Magus</a>. 25 years old. Still a great article.</li><li>NY Times: <a href="https://www.nytimes.com/2018/11/04/world/middleeast/mckinsey-bcg-booz-allen-saudi-khashoggi.html">On American consultancies and Autocrats</a> (Saudi Arabia in this case, the China article came later in the year).</li><li><a href="http://Kenji López-Alt on Where to Eat in San Mateo">Kenji López-Alt on Where to Eat in San Mateo</a>. Always trust and love Kenji.</li><li><a href="https://eurasianet.org/a-journey-along-georgias-strategic-stream">A journey along Georgia’s strategic stream</a>. The country, not the state. History, resources, and politics.</li><li>New Yorker: <a href="https://www.newyorker.com/magazine/2019/01/07/how-mark-burnett-resurrected-donald-trump-as-an-icon-of-american-success">How Mark Burnett made Trump.</a></li><li><a href="https://autotranslucence.wordpress.com/2018/03/30/becoming-a-magician/">On a leap in competitive body-painting and innovation shifts</a> driven by perpendicular thinking.</li><li><a href="https://syonyk.blogspot.com/2018/05/why-typical-home-solar-setup-does-not-work-off-grid.html">A thoroughly useful grasp of home solar and its interactions with the grid.</a></li><li>Wired: <a href="https://www.wired.com/story/why-hard-escape-amazons-long-reach/">Everything Amazon owns and controls.</a></li><li><a href="https://medium.com/future-crunch/99-good-news-stories-you-probably-didnt-hear-about-in-2018-cc3c65f8ebd0">99 Good News Stories You Probably Didn’t Hear About in 2018</a></li></ul><h3>Modern Economics Papers, Reports, and Writing</h3><ul><li>NBER: <a href="https://www.nber.org/papers/w24717">The Economic Limits of Bitcoin and the Blockchain</a></li><li>Course Syllabus for Scott Ashworth’s <a href="http://home.uchicago.edu/~sashwort/syllabus_next">“Philosophical Foundations of Public Policy”</a>, includes links to the readings.</li><li>NBER: <a href="https://www.nber.org/papers/w25018">Difference-in-Differences with Variation in Treatment Timing</a></li><li><a href="https://kelley.iu.edu/dlubensk/docs/Learning%20from%20Speculating.pdf">Learning from Speculating and the No Trade Theorem.</a> “ We develop an overlapping generations model in which rational wealth-maximizing speculators with a common prior trade a common value asset based only on private information. . . In a single period snapshot the no trade theorem would ensue, and observed trade is attributed to younger cohorts who appear as noise traders by entering unprofitable trades.”</li><li>Brad DeLong: <a href="https://www.bradford-delong.com/2018/09/resources-on-the-kinds-of-uses-of-math-i-am-trying-to-use-in-econ-101b-this-fall.html">Resources on the math</a> he’s trying to use in teaching Econ 101.</li><li>NBER: <a href="https://www.nicolasgendroncarrier.com/_pdf/Gendron-Carrier_etal_WP_2018.pdf">Subways and Urban Air Pollution.</a> “ We find that particulate concentrations drop by 4% in a 10km radius disk surrounding a city center following a subway system opening.”</li><li>NBER: <a href="https://www.nber.org/papers/w25066.pdf">Diverging Trends in National and Local Concentration.</a> “. . .we present evidence that the positive trend observed in national product-market concentration between 1990 and 2014 becomes a negative trend when we focus on measures of local concentration.”</li><li><a href="https://twq.elliott.gwu.edu/sites/g/files/zaxdzs2121/f/downloads/41-3%20Brunnermeier%20et%20al.pdf">Beijing’s Bismarckian Ghosts: How Great Powers Compete Economically</a>.</li><li>Twitter: <a href="https://twitter.com/kelleher_/status/1049398956747309056">A thread on the debate/discussion of discounting</a> between Bill Nordhaus and Nicholas Stern.</li><li>Twitter: <a href="https://twitter.com/m_clem/status/1075850063669268480">Yet Another Mariel Boatlift Study and thread.</a></li><li><a href="http://jwmason.org/slackwire/in-jacobin-a-demystifying-decade-for-economics/">A Demystifying Decade for Economics (J.W. Mason</a>). On the divergence and evolution of macro economics between high minded academic theory and actual economics-in-practice as utilized by central bankers and such).</li><li><a href="https://academic.oup.com/ser/article-abstract/16/1/61/4731616">Financialization as calculative practice: the rise of structured finance and the cultural and calculative transformation of credit rating agencies</a>. “ This specific culture of credit risk assessment enabled the emergence and exponential growth of structured finance markets, fueling a culture of high-risk investments. This article contributes to the literature on financialization by arguing that financialization occurs both at the level of organizational culture and calculative practices.”</li><li><a href="https://scholar.princeton.edu/sites/default/files/davidcho/files/jmp-david-cho.pdf">The Labor Market Effects of Demand Shocks: Firm-Level Evidence from the Recovery Act</a>. “I find that companies which experienced these demand shocks responded by increasing both employment and wages relative to their counterparts. Furthermore, the magnitudes of these changes suggest that the labor supply to an individual firm is relatively inelastic, even in a deep recession, and provide evidence of monopsonistic wage-setting in U.S. labor markets.”</li><li>Prospect: <a href="https://www.prospectmagazine.co.uk/magazine/wrong-financial-crisis">How economists predicted the wrong financial crisis</a>. A review of <em>Crashed</em>, by Adam Tooze, one of my favorite books of the year. Worth a read if you’re thinking about checking out the book.</li><li>Barron’s: <a href="https://www.barrons.com/articles/how-the-u-s-saved-the-world-from-financial-ruin-1533328064">How the U.S. Saved the World From Financial Ruin</a>. more thoughts on <em>Crashed</em>.</li><li><a href="https://adamtooze.com/2018/08/19/framing-crashed-4-the-turkish-crisis-the-missing-chapter/">The “missing” chapter from <em>Crashed</em>, on Turkey.</a></li></ul><h3>Historical Economics Papers, Reports, and Writing</h3><ul><li>VOX EU: <a href="https://voxeu.org/article/trade-and-growth-iron-age">Trade and Growth in the Iron Age.</a></li><li>A <a href="http://glineq.blogspot.com/2018/09/the-americas-armed-trade-and-cheap.html">very late review</a> of “The Great Divergence”<em>. </em>Which is on the divergence between West and East around 1750–1800.</li><li>Wikipedia: <a href="https://en.wikipedia.org/wiki/Onion_Futures_Act">The Onion Futures Act.</a> See Also: Volatility</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-z5RIM6YYTDjyBDFzpXjGg.png" /></figure><ul><li><a href="https://economic-historian.com/2018/07/financial-instability-and-the-panic-of-1857/">Financial Instability and the Panic of 1857</a>. I’m always interested in historical financial crashes and panics. Some feel so arbitrary now, while others seem to reveal universal human truths. Spinning out the lessons into modernity is the tricky bit.</li><li><a href="https://www.cambridge.org/core/journals/journal-of-global-history/article/ben-franklins-ghost-world-peace-american-slavery-and-the-global-politics-of-information-before-the-universal-postal-union/723CD19CD759ECE20C6C87187EDFEBF3">Ben Franklin’s ghost: world peace, American slavery, and the global politics of information before the Universal Postal Union</a>. “ This article examines two movements of the 1850s that sought change through global communications: using mail steamers to resettle American free blacks in Africa and reducing international postage rates to such a low rate that increased communications would prevent war. These two nearly simultaneous histories suggest that the evolving concept of the nation-state deserves further investigation as an element at the conjuncture of global communications and social reform.”</li><li><a href="http://www.lse.ac.uk/Economic-History/Assets/Documents/WorkingPapers/Economic-History/2018/WP275.pdf">Technologies of money in the Middle Ages: The ‘Principles of Minting’.</a></li><li><a href="https://www.cambridge.org/core/services/aop-cambridge-core/content/view/F5FBC37E7F0124DBD30D550E98C42AEB/S0003598X18000224a.pdf/viking_age_tar_production_and_outland_exploitation.pdf">Viking Age tar production and outland exploitation</a>. AN interesting look at the archaeological excavation of prehistoric tar pits and the development of trading networks as Scandinavians used and produced more tar in order to facilitate their sailing.</li><li>Twitter: <a href="https://twitter.com/Undercoverhist/status/1078806395426869249">The story of a century of economist and engineer intermingling at Stanford.</a></li></ul><h3>Energy and Environmental</h3><ul><li><a href="https://policyintegrity.org/publications/detail/capacity-markets-and-externalities">Capacity Markets and Externalities</a>. Relevant to my job and my interests, but probably not for most people. On the interaction between capacity markets and state policies.</li><li>Twitter: <a href="https://twitter.com/noahqk/status/1042902954355769344">On Carbon Pricing Models.</a></li><li>Twitter: <a href="https://twitter.com/JesseJenkins/status/1043162431013486594">An overview of current carbon pricing</a> relative to required prices.</li><li><a href="https://insideclimatenews.org/news/20092018/hurricane-florence-solar-panel-energy-resilience-extreme-weather-damage-wind-flooding">Solar Energy Largely Unscathed by Hurricane Florence’s Wind and Rain.</a></li><li>The Guardian: <a href="https://www.theguardian.com/environment/2018/sep/27/orca-apocalypse-half-of-killer-whales-doomed-to-die-from-pollution">Orca Apocalypse</a>, pollution dooms half of orca population.</li><li>Twitter: <a href="https://twitter.com/DrRamBio/status/1010900831640489984">Climate Models via Mario</a></li><li><a href="https://www.gao.gov/assets/690/688811.pdf">Four Regions Use Capacity Markets to Help Ensure Adequate Resources, but FERC Has Not Fully Assessed Their Performance</a>. A GAO report from a year ago that remains relevant and informative.</li><li><a href="https://b8f65cb373b1b7b15feb-c70d8ead6ced550b4d987d7c03fcdd1d.ssl.cf3.rackcdn.com/cms/reports/documents/000/002/327/original/Carbon-Majors-Report-2017.pdf?1499691240">The Carbon Majors Database</a>. And, specifically, the companies whose business it is.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HHe1bO1WNgiHuh9e" /></figure><h3>Trump and Modern Politics</h3><ul><li>Bloomberg, on M&amp;A: <a href="https://www.bloomberg.com/opinion/articles/2018-07-12/is-the-justice-department-ready-for-prime-time-this-time">Is the Justice Department Ready for Prime Time This Time?</a></li><li>Politico: <a href="https://www.politico.com/story/2018/08/04/campaigns-dark-money-groups-donors-anonymous-ruling-762440">Judge’s ruling invalidates FEC regulation allowing anonymous donations to ‘dark money’ groups.</a> This seemed really important to me at the time, but as a great example of how crazy news is at the moment, I’ve completely forgotten to follow up.</li><li>Twitter: <a href="https://twitter.com/ndrew_lawrence/status/1032058517425991680">The rambling narcissist.</a></li><li>Dissent Mag: <a href="https://www.dissentmagazine.org/online_articles/trump-year-one-tragedy-farce-right-wing-media">When the Farce is the Tragedy</a>. “I am arguing that the danger did not start with Donald Trump, but was building while he was still a national joke. Nor will the danger of a democracy-threatening crisis end when he leaves office. The 2016 Republican primary campaign clearly demonstrated that the GOP base, informed and egged on by the media machine, has the power to deliver the nomination to whoever can best appeal to its worst instincts.”</li><li>NYRB: <a href="https://www.nybooks.com/articles/2018/06/28/hitlers-rise-it-can-happen-here/">It Can Happen Here</a>. One of those great, winding New York Review of Books double review in which the author (Cass Sunstein) weaves together interesting threads from these books on the slow, then sudden Nazi-ification of ordinary German life with disintegration of democratic norms today.</li><li>Twitter: Noah Smith on<a href="https://twitter.com/Noahpinion/status/1034853593482899456"> the failure of three conservative pillars</a> in the early 2000s.</li><li>Twitter: <a href="https://twitter.com/ddayen/status/1050582489725587457">Senator Warren explains</a> why John Kelly called her an, “impolite arrogant woman”.</li><li>The Nation: <a href="https://www.thenation.com/article/thinking-conservative-part-five-epistemology-and-empathy/">Thinking Like a Conservative (Part Five): Epistemology and Empathy</a> (Rick Perlstein). “Have you ever noticed how many conservatives cannot believe a sane, sincere, intelligent person could disagree with them? That’s a central component of the right-wing mind.”</li><li><a href="https://hmmdaily.com/2018/10/30/these-are-the-bad-times/">These Are the Bad Times.</a> “ Try Turkey. Turkey still has elections. It also locks up hundreds of thousands of people in prison for their political positions or their social status. It went through an attempted coup and cracked down with martial law. Living under Turkish conditions — an authoritarian democracy, but certainly not the Nazi regime — would be an awful fate for Americans. We got a meaningful glimpse of the possibilities when the Turkish president’s security detail jumped off and beat up peaceful protesters on American soil last year. The Trump administration <a href="https://www.voanews.com/a/united-states-dismisses-charges-erdogan-bodyguards/4311950.html">dropped the charges</a> against most of them later on.”</li></ul><h3>Internet Miscellany</h3><ul><li>Twitter: <a href="https://twitter.com/mikamckinnon/status/1030188006966214660">A thread on licking rocks from a field geophysicist</a>.</li><li><a href="https://twitter.com/ChillAndCoolGuy/status/1032369946267770881">This tweet.</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*QP4mT7HjHhFwZLvWBioLVg.png" /></figure><ul><li><a href="https://twitter.com/florianederer/status/1034150257570054146">Another Tweet.</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*5q7QHk4cL6AiXTIGI-R1SQ.png" /></figure><ul><li>Twitter: <a href="https://twitter.com/AgentTinsley/status/1038619677688135686">Men in tennis arguing without loss.</a></li><li>Twitter: <a href="https://twitter.com/iam_somov/status/1047889761791332352">An amazing shot/sequence</a> from Showtime’s “Kidding”.</li><li><a href="http://beatnikkids.com/2012/07/books-for-boys-6-secret-knowledge-of.html">THE SECRET KNOWLEDGE OF GROWN-UPS</a>. Simply kept for posterior as the random blog post that reminded me of a book I loved from my childhood that had been entirely memory-holed until the internet reminded me.</li><li>Twitter: <a href="https://twitter.com/MaxFagin/status/1058062583511404545">On redefining the kilogram.</a> I know some people that worked on this a few years ago. ALways seemed interesting, and it led to some fun conversations, ME: “Where are you headed this weekend?” FRIEND:“Oh, off to DC for a conference on redefining one of our basic underpinnings of the universe around us.”</li><li>Twitter: <a href="https://twitter.com/reedkavner/status/1075108410406899712">On the speed of social media evolution and TikTok.</a></li></ul><h3>Books (That I haven’t read)</h3><ul><li>Amazon: <a href="https://www.amazon.com/Good-Mad-Revolutionary-Power-Womens/dp/1501181793/ref=tmm_hrd_swatch_0?_encoding=UTF8&amp;qid=1536717714&amp;sr=1-1">Good and Mad: The Revolutionary Power of Women’s Anger</a>. I haven’t gotten around to this yet, but I know my partner liked it.</li><li>Harper’s: <a href="https://harpers.org/blog/2018/08/chapo-trap-house-guide-to-revolution-book-review/">A review of the Chapo book</a>. I can’t stand listening to Chapo, but this review was more positive than I had anticipated, so I’m curious to see where it all goes.</li><li>Amazon: <a href="https://www.amazon.com/Gift-Global-Talent-Migration-Business/dp/1503605027">The Gift of Global Talent: How Migration Shapes Business, Economy &amp; Society</a>. Another book I haven’t read, but which appears interesting.</li><li>Amazon: <a href="https://www.amazon.com/Souls-Yellow-Folk-Essays/dp/0393241742">The Souls of Yellow Folk: Essays</a>, by Wesley Yang. I haven’t read this, but I’ve seen general acclaim, specifically from those who would more personally connect with the author’s background.</li><li>Amazon: <a href="https://www.amazon.com/Open-versus-Closed-Personality-Redistribution-ebook/dp/B01MR3D7QR">Open versus Closed: Personality, Identity, and the Politics of Redistribution</a></li><li><a href="https://www.elle.com/culture/books/g24525524/best-comfort-reading-books/">23 Authors on their Solace Reading for Difficult Times.</a></li></ul><h3>Ideas</h3><ul><li>Yelp data voter index</li><li>The shockingly narrow pipeline for Supreme Court Justices and the institution’s positions as a largely reactionary body.</li><li>Does the brief increase in crime/murder correlate with internal urban displacement due to the high point of a reversed migration pattern?</li><li>List of things that last longer than they used to: Steady improvement — Light bulbs (but not originals), Hard drives-&gt;SSDs, but all less than magnetic tapes Cars? Hvac? Divergence (High and low ends, Walmart and rei, no sears or kohl’s) — Clothes? Shoes? Shortened obsolesence — Phones? Although maybe we just use them more now. Appliances in general I guess? Houses are the shrugging emoji in this thought I guess, so disparate.</li><li>3D printing LiDAR, but no color 3D printers in the feasible cost range yet :(</li><li>People are stupid, SUV sales, fuel economy, oil prices, etc.</li><li>home sales, sqft squares, total prices, compare regions</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7b6a9aaffe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[America’s Oil Imports]]></title>
            <link>https://medium.com/@waldoch/americas-oil-imports-2290f8e77d76?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/2290f8e77d76</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[oil]]></category>
            <category><![CDATA[geopolitics]]></category>
            <category><![CDATA[fossil-fuels]]></category>
            <category><![CDATA[energy]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 12:55:29 GMT</pubDate>
            <atom:updated>2018-08-17T12:55:29.205Z</atom:updated>
            <content:encoded><![CDATA[<p>We are not utterly dependent on foreign oil. We particularly don’t rely on foreign oil from nominally hostile countries. The former has been true both relatively recently and in the past, the latter has almost always been the case since the US began relying on petroleum to move and produce. The singular period in which oil from elsewhere was both truly critical to the US and not generally available was the 70s. This time was the product of an emerging global market with a limited set of suppliers and a concentration of demand. Since then, supply has diversified and OPEC’s sway has dwindled.</p><p>This is all not to say that oil hasn’t occasionally been expensive (commonly experienced directly by most Americans “at the pump”), but access has been consistent and diverse enough for a long enough period of time that the last ~20 years of “foreign oil” grandstanding should have been considered political malpractice outside of relatively small pockets of time. Fossil fuel resource exploration and extraction also follows boom and bust cycles naturally as low prices tend to suppress investment in future supply, but that is certainly not the underpinning of the fear in American politics around oil. Rather, I believe this meme has its roots in those oil shocks of the 70s and the phenomenon of generational memory. But first, data.</p><h3>Data+Python</h3><p>The EIA has a <a href="https://www.eia.gov/dnav/pet/pet_move_impcus_a2_nus_ep00_im0_mbbl_m.htm">nice database</a> of U.S. petroleum imports stretching back to 1981, which begins to differentiate different import sources in 1993. There are many columns, with long, annoying names. Fortunately, they’re also standardized so with a little Python string splitting action I can get the regional or country names I want. I threw in US Production too to make things more clear.</p><pre>import pandas as pd<br>import pdb<br>import numpy as np<br>import matplotlib.pyplot as plt<br></pre><pre>df = pd.read_csv(&#39;oil_imports.csv&#39;)</pre><pre>colList = list(df.columns)<br>colList2 = colList[3:]</pre><pre>actualNames = [x.split(&#39; of &#39;)[0].split(&#39; from &#39;)[1] for x in colList2]<br>   <br>columns = colList[:3]<br>columns.extend(actualNames)<br>colDict = dict(zip(colList, columns))</pre><pre>df.rename(columns = colDict, inplace=True)<br>df.fillna(0)</pre><pre>for x in df.columns:<br>    print(x)</pre><pre>sumCats = [&#39;All Imports&#39;, &#39;US Production&#39;,<br>    &#39;Non-OPEC Countries&#39;,<br>    &#39;Persian Gulf Countries&#39;,<br>    &#39;OPEC Countries&#39;]<br>    <br>df[&#39;dates&#39;] = pd.to_datetime(df[&#39;Date&#39;])<br>for x in df.columns:<br>    <br>    if x in sumCats:       <br>        plt.plot(df[x], label = x)<br>        <br>xTicks = list(range(0,len(df),30))        <br>xTickLabels = [df[&#39;Date&#39;].values[x] for x in xTicks]       <br>        <br>plt.xticks(xTicks, xTickLabels, rotation=70)<br>plt.title(r&#39;US Oil Info 1/1981-11/2017&#39;)<br>plt.ylabel(&#39;Thousands of Barrels&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()</pre><h3>Graphs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*7c555RZR2kWn2KUj.jpg" /></figure><p>Unfortunately, this doesn’t go back far enough. In fact, it’s somewhat tantalizing to end when it does for the OPEC vs. Non-OPEC comparison. We need to go further. An older and more useful EIA table can be <a href="https://www.eia.gov/totalenergy/data/annual/showtext.php?t=ptb0507">found here</a>. While there’s no immediately clear way to download the data directly, if you graph it you’ll be given an option to grab the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/0*i9-loBxIjEjdRAA7.jpg" /></figure><p>I had to do a bit of interpolating daily barrels out to monthly values to fit the format I was doing (hence the steps), but overall, this graph is suddenly far more informative. Both times the US encountered an “energy crisis” real or somewhat invented (1970s vs 1990), OPEC had an outsized share of our imports. One more note, this graph doesn’t tell the whole story. The US has a large position in terms of the refining and production of various petroleum products, and not all oil is created equal, with various uses for different grades. Additionally, oil is a fungible, global commodity. All of that is to say that not all of the oil produced in and imported to the US is being consumed here.</p><h3>Generations and Memory</h3><p>I have a theory that the ability for this thread to continually find its way into modern American politics is the result of the true oil shocks occurring when our current political class was in its formative years.To start, I need some ages. <a href="http://online.wsj.com/public/resources/documents/info-CONGRESS_AGES_1009.html">This WSJ infographic</a> was useful to quickly pull the ages I was looking for.</p><p>In 2001, the average House age was ~55, and ~59 in the Senate. By 2011, legislators had aged up to 57 in the House, and 62 in the senate. For the first oil crisis in 1973, those 2001 legislators were on average in their mid 20s to early 20s, while the 2011 crop would have been in their late teens and mid 20s. While the pathways towards adult hood have become more disparate recently, personalities and opinions tend to cohere fairly early in life. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3792649/">Recent research has also indicated that those living under greater stress tend to move into adulthood earlier</a>. Given the relative largess of many legislators and the lack of intergenerational economic movement in America it could be reasonable to surmise a slightly later identity coherence for our average legislator, pushing out the age of adulthood “acquisition”. Taken together, it doesn’t seem terribly risky to guess that these major political events shaped the positions of modern legislators well past both middle age and relevance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2290f8e77d76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Metro Works]]></title>
            <link>https://medium.com/@waldoch/how-metro-works-52c048b61bb5?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/52c048b61bb5</guid>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[metro]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[transit]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 12:54:38 GMT</pubDate>
            <atom:updated>2018-08-17T12:54:38.045Z</atom:updated>
            <content:encoded><![CDATA[<p>Yesterday (back in February of 2018, when I originally wrote this) I watched my train that should show up at 6:34, pull up and leave at 6:31 from outside the station before I could get to it.</p><p>Today I got to the station at 6:30 on the dot, and it said the train was 4 minutes out. The board said 4 minutes for several more minutes, then 2, then 1, then back to 2, and eventually arrival. By the time it did arrive it was 6:37, and as we sped away from the station I noticed my phone tick over to 6:38.</p><p>To recap, the train showed up 6 minutes later than the day before, and 3 minutes later than when you would expect it to show up based on train arrivals from WMATA’s trip scheduler. Unfortunately I can’t get that corroborating information from a timetable because it doesn’t exist, they don’t publish one at all.</p><p>The interval between trains at this time of day is supposed to be 8 minutes. Each of the last two days the trains has been off of the mark by 3 minutes (in both directions!), or 37.5% of the total ideal headway. I also have about an 8 minute walk from home to the metro, and a tight schedule between waking up and walking out the door. As I began commuting by train I left earlier and earlier until I hit upon a time where it is nearly impossible to not end up on a set of trains that will get me to work. I leave not only early enough to make sure I get on the train before the train I probably need, but even earlier to try and deal with that train’s capricious timing. This is all a bit confusing in text form, so here’s a graph to try and make sense of it.</p><p>I used <a href="http://omz-software.com/pythonista/">Pythonista</a> on my phone to do this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*iqB8sFw3Ag9R0NR_.PNG" /></figure><h3>Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*P9Yd3DVYwsTd4rpv.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52c048b61bb5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ground Source Heat Pumps are not Geothermal]]></title>
            <link>https://medium.com/@waldoch/ground-source-heat-pumps-are-not-geothermal-a7285ba7625?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7285ba7625</guid>
            <category><![CDATA[geothermal-power]]></category>
            <category><![CDATA[home]]></category>
            <category><![CDATA[hvac]]></category>
            <category><![CDATA[energy]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 12:52:37 GMT</pubDate>
            <atom:updated>2018-08-17T12:52:37.587Z</atom:updated>
            <content:encoded><![CDATA[<p>I just wanted to clear up this misconception I hear quite a bit in the home energy and efficiency space when people are talking about ground source heat pumps (GSHPs). GSHPs rely on the temperature of the soil a few meters down. The temperature at that depth is roughly the average annual air temperature for the area in which the installation exists. This is also why the type of soil matters and why regionally GSHPs have different efficiencies. Soil conductivity even plays a part, and relatively recent glacial till (such as in the upper midwest) is going to perform very differently than the high desert of Utah. On the other hand, we have geothermal. It should be evident from considering the term geothermal briefly that it refers to something different. Specifically, the heat (thermal) of the earth (geo) itself. This can be close to the surface in rare instances (think hot springs or volcanoes), but is also a fact of life as you go deeper into the earth’s crust.</p><h3>A Look at Geothermal</h3><p><a href="https://energy.mit.edu/wp-content/uploads/2006/11/MITEI-The-Future-of-Geothermal-Energy.pdf">MIT and INL produced an excellent report on this resource</a> over a decade ago (the real money maps start on page 62). In most cases, geothermal resources are looked at from a largescale engineering perspective, so that even when this resource is close enough to the surface to be efficient for home heating it tends to be a district or municipality-level resource. I mean, look at the costs associated with geothermal wells surveyed in the paper, they’re well outside the range of an individual homeowner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/0*Eodoz5fcymlxnECr.png" /></figure><p>Finally, these maps of ground surface temperature and then the temperature at a depth of 3.5km are also useful for additional context. Additionally, the ground surface temperatures show why GSHPs are so efficient (and only fractionally as efficient during the winter) in their summertime cooling applications across the midwest and great plains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*YJvVYdx2uRdNc1nX.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*abep7F-jtt3WaeRl.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7285ba7625" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Global Market for Gas]]></title>
            <link>https://medium.com/@waldoch/a-global-market-for-gas-3006d64fefc4?source=rss-3f5a9a48ccba------2</link>
            <guid isPermaLink="false">https://medium.com/p/3006d64fefc4</guid>
            <category><![CDATA[natural-gas]]></category>
            <category><![CDATA[geopolitics]]></category>
            <category><![CDATA[energy]]></category>
            <category><![CDATA[lng]]></category>
            <dc:creator><![CDATA[Connor Waldoch]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 12:49:17 GMT</pubDate>
            <atom:updated>2018-08-17T12:49:17.069Z</atom:updated>
            <content:encoded><![CDATA[<p>One issue that I’m currently trying to untangle over some portion of my commuting time is the basic logic around the formation of a global natural gas market. On one side, the difficulty of gas transportation implies a high friction against the formation of such a market, but recent (1–2 years) LNG pricing has appeared to be comprised almost entirely of resource cost + transport cost, implying a high liquidity (global!) market. So that’s it, we’re done, right?</p><p>Well, not really. There’s an understanding at the moment that we might be in a period of over supply. Up and midstream firms in the US (and here I’m conflating oil and gas due to their commingled presence in many operations and companies) have repeatedly weathered oversupply and financial stressors to continue drilling, fracing, and producing. This is a transition time, and some analysts see supply being consumed by long term contracts and additional demand; investment inherently constrained by falling prices initially beget from the new (liquid or not) global supply. The ultimate fate of existing nukes in developed nations and how quickly developing countries are able to transition away from fossils fuels (and foreign dependence) both play a role, although industrial and residential demand cannot be discounted as long term sinks as well.</p><h3>Politics</h3><p>Finally, there’s the politics of it all. To some extent Russia has managed to keep pieces of the former USSR on a leash (along with pieces of the EU in an at least slightly more subtle process) with its pipelines and extraction expertise. In that context, shale has been the major fracture point in terms of American-Russo relations of the last decade. An ascendant American oil and gas production and export regime may only challenge Russian hegemony at the margins, but when your economy and influence is so concentrated in a particular sector, loss of marginal control is a deeply worrying prospect. Additionally, America ‘s expertise in the field isn’t limited to a national company at the state’s beck and call (i.e., Gazprom), but rather a diverse portfolio of profit seeking enterprises which will be more than happy to lend their (expensive) expertise to whomever is willing to pay.</p><h3>The Future</h3><p>None of this is to count out the old hands of LNG export either, from Australia and Qatar on down to Trinidad and Tobago. Competition between these countries (and the national and international firms paying to develop new terminals and producing new trains at existing ones) could generate a cycle of production and competition that at the very least continues to promote the appearance of a global market at least into the 2020s. At which point, having had what appeared to be a global market for more than 5 years, perhaps it will simply exist despite today’s skeptics. Conversely, if demand collapses or transportation and infrastructure developers fail to secure the permits and financing to support further lock-in the global market will stall out and separate into the regional fiefdoms that previously defined global natural gas.</p><p>Either way, if current price signals suggest the global market is real there’s no reason to not participate in the moment. Plus, if you believe its collapse is imminent, based on the backlog of tankers and terminals you should have ample opportunity to profit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3006d64fefc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>