<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Molly Graber on Medium]]></title>
        <description><![CDATA[Stories by Molly Graber on Medium]]></description>
        <link>https://medium.com/@mgraber?source=rss-760b26638399------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*MTbJAr4Y-xT-16VACuX1Dg.jpeg</url>
            <title>Stories by Molly Graber on Medium</title>
            <link>https://medium.com/@mgraber?source=rss-760b26638399------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:48:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mgraber/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Meet the Engineers: Molly Graber]]></title>
            <link>https://medium.com/nyc-planning-digital/meet-the-engineers-molly-graber-27f4b2bada9c?source=rss-760b26638399------2</link>
            <guid isPermaLink="false">https://medium.com/p/27f4b2bada9c</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[people]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[city-planning]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Molly Graber]]></dc:creator>
            <pubDate>Fri, 20 Aug 2021 19:00:27 GMT</pubDate>
            <atom:updated>2021-08-20T19:00:27.190Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yySACEKHsnfJA05slLze2A.png" /></figure><p><em>Molly Graber, a Colorado native, has been with the Data Engineering team at NYC Planning for almost 2 years. Coming from a background in Geography and Statistics, Molly is the main contributor to core data products including Community Profiles and Population FactFinder. Come learn about how she grew as a data engineer and how her work is leaving deep impact on NYC OpenData.</em></p><h3><strong>How I came to the Data Engineering team</strong></h3><p>I came to NYC generally, and to NYC’s Department of City Planning specifically, after completing a masters degree in geography from the University of Colorado. In my graduate work, I focused on using computational techniques to understand and mitigate uncertainty in small-area American Community Survey data. I was fortunate to work in a <a href="https://www.colorado.edu/rocky-mountain-research-data-center/">Federal Statistical Data Research Center</a>, a secure environment where I could run analyses on disaggregated Census responses to see how spatial enumeration units (Census blocks, tracts, etc.) affect the data that informs allocation of billions of dollars in federal funding.</p><p>I came to NYC with a goal of using my technical background to improve data quality (and ideally decision-making, by implication), a general interest in demography and urban landscapes, and a strong desire to apply my training to contribute to my new, local community. My path through grad school and before (I’d studied quantitative economics, also at CU) was generally guided by a fascination with how our societies — small and large — make decisions with uncertain information. Spatial data particularly fascinated me, since an undergrad foray into GIS made me realize how adding a spatial component makes data uncertainty even more perplexing!</p><p>I landed in the Department of City Planning’s Data Engineering team. It was a good fit, where I use technical skills to produce high quality data used by planners, DCP’s economic and demographic researchers, city analysts in other agencies, and the public.</p><h3><strong>What my days look like</strong></h3><blockquote>“As a member of a small team, I’m able to contribute to our data workflows from start-to-finish… I’m in regular communication with my coworkers. Data Engineering is a very collaborative team, especially considering how much of our days are spent writing code on our own.”</blockquote><p>My days as a data engineer are primarily spent either writing code in python, SQL, or bash, or doing “detective” work to identify inconsistencies in our data products. I particularly enjoy the investigative part of my days, where I’m able to trace data anomalies backwards to find where in our logic (or our source data) they originated. This type of problem-solving is both creative and systematic, and it improves the quality of data we produce. Throughout all of these activities, I’m in regular communication with my coworkers.</p><p><strong>Building a data product, start-to-finish</strong></p><p>As a member of a small team, I’m able to contribute to our data workflows from start-to-finish. The early stages of a project typically involve scoping out the requirements for a given data product, based on the needs of the data user and most valuable characteristics of our source information. This is typically iterative, and includes drafts of both schema and logic. As we identify the work requirements, we plan out specific tasks to tackle in code and organize these as GitHub issues. We’re very organized, which I suspect is one reason our small-but-mighty team has accomplished so much, even in the midst of a pandemic!</p><p>I’m also able to determine how we’ll “gather the ingredients.” What is the best source data for answering a given question (the key questions of our products’ end-users)? Where can we most consistently get this data? How can we retrieve it in a way that is automated and ensures we have the most up-to-date information possible? What are the shortcomings of our source data, and how will these affect our final products?</p><p>Next comes writing the necessary logic, typically in Postgres, for combining our raw data into a set of tables meeting our previously-scoped objectives. This portion of my work is a little less technology-heavy, and a little more logic-heavy. My most trusty tools are databases where I can test out queries, <a href="https://medium.com/nyc-planning-digital/github-actions-in-action-9f980bee91f2">GitHub actions workflows</a> where I can quickly rerun entire pipelines for testing purposes — I think of these as my sandbox — and the invaluable code reviews of my coworkers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R88XxcdWvR1y4mCWum2vSg.png" /><figcaption>Streamlit application for PLUTO version-to-version QAQC</figcaption></figure><p>I’m also able to brainstorm ways of checking the quality of our resulting datasets. As someone passionate about data quality, I love this aspect of my work. Sometimes, QAQC processes are ones Data Engineering uses to identify bugs in our own logic. These might take the form of SQL queries flagging edge-cases or outliers. Sometimes, we use similar queries to create tables flagging records that potentially need review by the domain expert, which we then incorporate as manual corrections. These corrections allow for record-specific exceptions to our logic. Other times, we use QAQC tables to identify potential improvements to source data, including the files underlying <a href="https://medium.com/nyc-planning-digital/geosupport-%EF%B8%8Fpython-a094a2d30fbe">Geosupport</a> (NYC’s geocoding software). Not all of our QAQC processes take the form of tables, however. We’ve also created Streamlit applications to aid in domain-expert review, with visualizations showing version-to-version differences and various summary statistics.</p><p>Data delivery is a fun and varied aspect of our work. We strongly believe that making good quality data includes making data that is easily accessible for its intended audience. We’re constantly improving ways of delivering our data to users, largely guided by their particular needs and skillsets. Often our data ends up hosted as CSVs in DigitalOcean with download links directly in a GitHub repository’s ReadMe. Sometimes, these hosted datasets are automatically synced with tables in Carto for Planning Labs’ (the team in charge of many of DCP’s data exploration applications) “staging” or “testing” applications. Some of our datasets go to our GIS team for final processing prior to publication on <a href="https://www1.nyc.gov/site/planning/data-maps/open-data.page">Bytes of the Big Apple</a> and <a href="https://opendata.cityofnewyork.us/">NYC Open Data</a>. We’ve even created ways for hosting our data on Sharepoint, as much of city government work happens in Microsoft ecosystems. Most recently, we’ve been exploring hosting data on Google Cloud Platform for immediate integration with analysis and visualization tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5t_0H5_tOUFJK6qGblp5g.png" /><figcaption>Automated workflow execution via GitHub Actions</figcaption></figure><p>Tying it all together, we also spend our days refining automation and execution of these steps. Over the past year or so, <a href="https://medium.com/nyc-planning-digital/github-actions-in-action-9f980bee91f2">we’ve relied heavily on GitHub actions</a>. This has let us produce data products at a more frequent cadence, and even to off-load some of the execution of our processes to other teams.</p><p><strong>I “talk” a lot!</strong></p><p>Data Engineering is a very collaborative team, especially considering how much of our days are spent writing code on our own. We’re in fairly constant communication with each other (these days, in the form of Teams chats and GitHub issue comments) and heavily rely on each other for brainstorming potential causes of bugs and ideas for process improvements. When solving a problem, multiple perspectives are usually better than one, and we learn from each other. Our highly collaborative culture means we don’t accidentally duplicate work or let things fall through the cracks by assuming someone else is working on a task. We highly value the maintainability of everything we create. Collaboration and open communication help us prevent knowledge from getting siloed in a single person.</p><p>Our chattiness extends beyond our mighty team of three. We’re also in regular communication with the business owners of our data products. We reach out to make sure that our logic matches what they’d intended, give them opportunities to refine their desires for a product, and keep them up-to-date about our progress and timeline. Since many of our datasets feed into downstream analyses, we know others rely on our ability to accurately communicate our progress, timeline, and work capacity.</p><p>We also communicate our work to the agency and public. As a team that primarily produces behind-the-scenes data (rather than front-end applications), we often need to explicitly show off our work for it to be seen. We’ve done this by blogging, writing public-facing documentation, and (our favorite) giving presentations and demos. Our audiences are other DCP teams, other agencies, and the general public. These showcases are crucial to our continued success as a team. We’re able to draw more people to use our data, as well as keep in-the-loop about new use-cases that pop up.</p><h3><strong>Why DCP’s Data Engineering team is great</strong></h3><blockquote>“I’m able to work with cutting-edge technologies and techniques typical of the private sector, but within a city agency… I’m surrounded by experts in various domains (planners, demographers, economists, urban designers, lawyers), and am able to learn from their expertise.”</blockquote><p>Working for DCP’s Data Engineering team is truly unique. I’m able to work with cutting-edge technologies and techniques typical of the private sector, but within a city agency. I appreciate that the code I develop is placed in a larger context. Many users of my work are within my own organization. The data we create and the quality improvements we bring have direct impact on decision-making for the city where I live and work. I’m able to indirectly support some of the most basic and fundamental city services — ones I rely on myself! My job also fuses the old with the new. While the tools I use on a day-to-day basis are typically modern and rapidly changing, I’m able to add my contributions to legacy data that have existed in some form since before digitization, and likely will continue to exist for years to come.</p><p>I especially appreciate that I am a tech employee in an organization that is predominately not staffed by people with tech backgrounds. I love improving my abilities to communicate technical concepts to audiences with other skillsets. I’m also surrounded by experts in various domains (planners, demographers, economists, urban designers, lawyers), and am able to learn from their expertise.</p><p>The best part of my role in Data Engineering is the people I work with. I have incredibly talented coworkers who expose me to new tools or concepts daily. I work under supportive leadership and have lots of room to grow within my position.</p><h3><strong>What I’ve learned over the past few years</strong></h3><blockquote>“We’re able to pivot and explore new tools fairly quickly, which allows for a fast pace of skill development… While technical skill acquisition is probably most measurable, I’ve picked up project management skills as well. I’ve also learned a lot about NYC!”</blockquote><p>Over my time with Data Engineering, I’ve been thrilled at how quickly I’ve been able to hone new technical skills. Within the past year and a half, I’ve vastly improved my python, SQL, bash, Docker, and AWS abilities. I’ve learned postGIS, GitHub actions, and how to integrate automated workflows with platforms such as Microsoft and Carto. I’ve also been able to work in Google Cloud Platform (a relatively new addition to our toolkit). We’re able to pivot and explore new tools fairly quickly, which allows for a fast pace of skill development.</p><p>While technical skill acquisition is probably most measurable, I’ve picked up project management skills as well. By taking on our latest enhancement effort for the<a href="https://medium.com/nyc-planning-digital/introducing-dcps-housing-database-dcp-s-latest-open-data-product-b581aee97a51"> Developments Database</a>, I’ve learned how to scope product improvements with business owners, determining relative difficulty and value of various changes, break larger tasks into actionable items, translate business requests into technical GitHub issues, and communicate our progress throughout the duration of the enhancement period.</p><p>I’ve also learned a lot about NYC! Working closely with the Geographic Research Unit, I’ve learned about many of the city’s geographical quirks. In working on processing Census and American Community District data for Population FactFinder, I’ve learned about NYC’s demography and why counting people in the city is such a challenge. I’ve learned about the zoning process and how both neighborhood studies and lot-level zoning actions ultimately affect what can get built where. I’ve learned about the building permitting process and how this data can valuably suggest where future New Yorkers will live. In interactions with my DCP colleagues, as well as others working for the City, I’ve learned the ways in which data supports the essential functions of a city agency. I’ve also learned of the insufficiency of data in guiding decision-making, and how community engagement and relationships are also key.</p><h3><strong>Dreams for the future</strong></h3><blockquote>“Going forward, we aim to focus on creating a larger data ecosystem.”</blockquote><p>As a team, Data Engineering has big dreams for our future. We’ve gotten good at producing high quality data in a way that is transparent, efficient, and automated. Going forward, we aim to focus on creating a larger data ecosystem. This means using platforms such as Google Cloud Platform to allow for easy data discovery, a searchable data catalogue, integrated metadata, and direct access to computational resources for exploratory data analysis and visualization. We also hope to add value to our data by communicating “best practices” and dataset literacy, particularly when working with uncertain data such as small-area ACS data. We’re also hoping to not only build data, but a community of data users. We’re exploring ways of allowing city analysts and the public to share analysis ideas, processing tips, and feedback with us and each other. In short, the future of Data Engineering holds opportunity for learning new tools and improving the context and quality of our work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27f4b2bada9c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nyc-planning-digital/meet-the-engineers-molly-graber-27f4b2bada9c">Meet the Engineers: Molly Graber</a> was originally published in <a href="https://medium.com/nyc-planning-digital">NYC Planning Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proximity-constrained siting]]></title>
            <link>https://medium.com/nyc-planning-digital/proximity-constrained-siting-ddc7b4fa03dc?source=rss-760b26638399------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddc7b4fa03dc</guid>
            <category><![CDATA[civictech]]></category>
            <category><![CDATA[urban-planning]]></category>
            <category><![CDATA[nyc]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Molly Graber]]></dc:creator>
            <pubDate>Fri, 13 Mar 2020 17:26:16 GMT</pubDate>
            <atom:updated>2020-03-13T17:26:16.517Z</atom:updated>
            <content:encoded><![CDATA[<p>Walk down a commercial street in NYC, and you will likely see a diverse mixture of establishments lining the sidewalk. However they happened to pop up, your dinner spot might share a block with a laundromat, florist, or bakery. While this mix of uses is frequently organic, the city also identifies some establishment-types it would rather not have clustered. A block full of high-end clothing stores might give a neighborhood its special character. A block full of nothing but low-price liquor stores would give a neighborhood a very different special character. Occasionally, policy-makers decide some types of “character” should not exist in NYC, and use proximity restrictions to break up clusters of establishment-types.</p><p>For example, let’s say I strongly believe there are detrimental effects caused by clusters of donut shops — sprinkles littered on the sidewalks, sugar-high children, local residents’ diets ruined... These effects compound the more donut shops I have in a given area. To ameliorate this problem, I impose a restriction that no donut shop can exist within 500ft of another donut shop. This way, NYC donut-makers can theoretically still make a living, but a problematic cluster is broken up.</p><p>Proximity restrictions like this one are an interesting siting problem, whether theoretical or actual (as of yet, NYC does not have an Agency of Fried Confections opening city-run donut shops). Not every lot is “eligible” to have a donut shop. As soon as a lot gets a shop, all of the lots within a 500ft radius are no longer available for future shops. Given these restrictions, some placements of shops on lots will lead to a greater number of total possible donut shops than others.</p><blockquote>When the locations of establishments on city lots have proximity restrictions, what is the greatest number of establishments we can legally fit within the city?</blockquote><p>NYC Planning’s Data Engineering team was recently tasked with answering this question. We began with spatial data for city lots — a subset of the <a href="https://www1.nyc.gov/site/planning/data-maps/open-data/dwn-pluto-mappluto.page">PLUTO</a> dataset. Our first step was to buffer each of these lots to 500ft, creating data that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*PqCvM4BgXZWkTRXr.png" /><figcaption>Input data: Eligible portions of PLUTO lots (darker purple) buffered to 500ft (lighter purple)</figcaption></figure><p>These buffers can tell us whether any given pair of lots can <em>both</em> have an establishment on them. Let’s look at a simplified example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/0*xkx_AxXcbkxLeweD" /><figcaption>City “lots” are numbered squares. The lines represent buffers.</figcaption></figure><p>If the buffer for a lot overlaps with another lot, then we cannot have establishments on both of them, because they are within 500ft of each other. In this case, we can have an establishment on block 1 or block 2, but not on both. That would lead to a cluster! We can represent this in a matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*FHJGbN3-M-LD22V1qtrWTA.png" /><figcaption>The “possibility matrix” for our simple example</figcaption></figure><p>If an establishment exists on lot 0, we cannot place an establishment on lot 1. They are too close. To represent this, we put a 0 in the 0,1 position of our matrix. We can, however, place an establishment on lot 2, so we place a 1 in the 0,2 position of the matrix. Using this matrix, we’ve transformed a complicated spatial problem with distance calculations in to simple zeros and ones.</p><p>We can use this matrix to efficiently calculate all of the possible arrangements of establishments amongst these lots. If you are interested in how this is done, keep reading!</p><h4>Using the matrix to find all possible shop arrangements</h4><p>The first step is to hypothetically assign an establishment to a lot. For simplicity, we will assign it to the first indexed lot, lot 0. Referring back to our matrix, we can know all of the lots that can <em>also</em> have an establishment. Look at the first row. This row tells us where we can put a new shop, given that there is already a shop on lot 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*pTlvOKxXRdE9-FnHVi9upQ.png" /><figcaption>We have a shop on lot 0. Where can the next shop go?</figcaption></figure><p>Remember, the “1s” mean there <em>can</em> be a shop placed on that lot. In this case, we can place a shop on lot 2 or lot 3. We cannot place a shop on lot 1, because it is too close. The “ — ” in the first slot shows that we already have a shop on lot 0.</p><p>Let’s place the next shop on lot 2. Take a look at our simple picture above. The buffer for lot 0 does not overlap with lot 2. This is a “legal” shop placement. Can we fit in any more shops?</p><p>This is where things get interesting. We can’t just make a decision based on distance from lot 2. We also have to account for lot 0, our initial shop. To do this, we can multiply all of the values in the 0 row of our matrix by the 2 row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*64GQdhu6ip1C12WUhms0hg.png" /><figcaption>We have shops on lots 0 and 2. Where can the next shop go?</figcaption></figure><p>In this case, the only 1 left in the row is associated with lot 3. We can place a shop on lot 3. Can we squeeze in one more shop? The picture above should intuitively tell you “no.” But let’s check the matrix. We now have to make sure the next shop is far away from <em>three </em>existing shops. To represent this, we multiply rows 0, 2, and 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*wFUHD3S95mYi-jXsWIKDdQ.png" /><figcaption>We now have shops on lots 0, 2, and 3. There are no more possible shop locations.</figcaption></figure><p>There are no ones. This means all of the other lots are either too close (have zeros), or already have shops on them (have -). So a possible arrangement of shops in the city is lots 0, 2, and 3 with a grand total of three possible lots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*Jae8ggdcqerPKCp2" /><figcaption>Our first solution</figcaption></figure><h4>What about the other arrangements?</h4><p>Take a look back at the simple example we’re working with. We could have arranged shops differently. If we had instead started by placing a shop on lot 1, rather than lot 0, we would have found a different, equally legal arrangement. It would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/0*ezHGOzVkCoK1r-EK" /><figcaption>Another possible arrangement of shops on lots</figcaption></figure><p>Lots 1 and 3 are far enough away to both have shops. We can’t fit any more shops into this scenario, because both lots 0 and 2 are too close to an existing shop.</p><p>To make sure we find <em>all</em> possible arrangements of shops on lots, we repeat the matrix-based process we walked through above. Each time, we chose a different lot on which to place our first shop. Sometimes, this gives us the same group of lots, just identified in a different order. This isn’t <em>really</em> a new solution, so we treat these as a single arrangement.</p><p>From here, we can take inventory of all of the possible arrangements of shops.</p><h4>The best-case scenario</h4><p>If you’re someone who really likes donuts (and likes having a variety of places to buy them), the second solution is <em>not</em> the optimal solution. Given this arrangement, the <strong>maximum possible shops<em> </em></strong>is two. We call this the <strong>worst-case scenario</strong>. As it turns out, our first example was the <strong>best-case scenario, </strong>with a total of three lots that could be the future home of a shop.</p><p>Not only do we know how many shops could exist in the best-possible scenario, we also know how many ways there are obtaining that best-case scenario. We can even go back to the spatial data, PLUTO, and make a map of each arrangement!</p><p>If you’d like to dig into the code, you can find our algorithm in <a href="https://github.com/NYCPlanning/aue">this repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddc7b4fa03dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nyc-planning-digital/proximity-constrained-siting-ddc7b4fa03dc">Proximity-constrained siting</a> was originally published in <a href="https://medium.com/nyc-planning-digital">NYC Planning Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>