<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Unearth - Medium]]></title>
        <description><![CDATA[Critigen serves non-government organizations, not-for-profits, private and public donors to implement technology solutions that improve the efficiency and effectiveness by which our partner organizations achieve their missions. - Medium]]></description>
        <link>https://medium.com/critigenopensource?source=rss----e73230f14c7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Unearth - Medium</title>
            <link>https://medium.com/critigenopensource?source=rss----e73230f14c7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:00:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/critigenopensource" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing: A New Python Client for the MapRoulette API]]></title>
            <link>https://medium.com/critigenopensource/introducing-a-new-python-client-for-the-maproulette-api-f1c42789f86b?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/f1c42789f86b</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Matthew Manley]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 16:01:02 GMT</pubDate>
            <atom:updated>2020-07-20T16:38:05.336Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new way to interact with MapRoulette</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0OJKfTbj951d1_PjZo2H9w.png" /></figure><p>MapRoulette is a powerful micro-tasking platform that provides professional and community mappers tools to improve the quality and coverage of OpenStreetMap. While most users are familiar with <a href="http://maproulette.org/">maproulette.org</a>, the publicly available MapRoulette API enables users to build custom tools that utilize MapRoulette data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7n0vK4EDVcFVMJpOTR6BdA.png" /><figcaption>The MapRoulette API <a href="https://maproulette.org/docs/swagger-ui/index.html?url=/assets/swagger.json#/">documentation page.</a></figcaption></figure><p>Finding the right endpoint for your workflow can be challenging. The MapRoulette <a href="https://github.com/osmlab/maproulette-java-client">Java Client</a> helps abstract some of the must-have Project, Challenge, and Task management functionality into a single Java library.<br>We wanted to provide a similar experience for the growing Python community. That’s where our latest Python Client steps into the spotlight. The Critigen OSM team is excited to introduce the new <a href="https://github.com/osmlab/maproulette-python-client">MapRoulette Python Client</a>!</p><p>We decided to build a Python Client with two goals in mind:</p><ol><li>Empower our team and the OpenStreetMap community to build custom tools to help manage and gain insights from MapRoulette data.</li><li>Reduce code redundancy and improve the overall experience with the MapRoulette API.</li></ol><p>The new API Client is essentially a wrapper for the <a href="https://requests.readthedocs.io/">Requests</a> library that supports several common use-cases. In just two months since our <a href="https://github.com/osmlab/maproulette-python-client/releases/tag/v1.0.0">first release</a>, we’ve managed to eliminate duplicate code while maintaining core functionality.</p><p>This article introduces some of the Python Client’s features. Let’s walk through some examples.</p><h3>Creating a Configuration Object</h3><p>The Configuration object controls how we connect to the MapRoulette server. You can pass a number of parameters including <em>hostname</em>, <em>protocol</em>, <em>api_key</em>, and <em>certs</em>. A common parameter to set would be the <em>api_key</em> as many API calls require passing an API key in the request headers. While the default hostname is <em>maproulette.org</em>, this parameter is configurable because the MapRoulette application can be deployed in different locations. You can also control whether to use HTTPS or HTTP protocols. If the instance of MapRoulette you’re trying to connect to requires client-side certs, you can pass those when creating the Configuration object. In this example I’ll stick to the default hostname, but pass my custom API key (intentionally removed):</p><pre>import maproulette<br>config = maproulette.Configuration(api_key=&#39;{YOUR_API_KEY}&#39;)</pre><p>That’s it! Now you have a Configuration object that you can use to make API calls.</p><h3>Creating API Objects</h3><p>API objects are another fundamental part of the Python Client. At the time of writing, you have access to four objects containing different API calls: Project, Challenge, Task, and User. Each object requires a Configuration object as an argument. For example, if you want access to the Project methods, create a Project API object.</p><pre>project_api = maproulette.Project(config)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*C3GVtwHyxfu5RDlgpR7YKg.gif" /><figcaption>By creating a Project API object, you can access all of the Project-related methods within the Python Client.</figcaption></figure><p>If you want access to the Challenge methods, create a Challenge object.</p><pre>challenge_api = maproulette.Challenge(config)</pre><h3>Using Models</h3><p>Models are a blueprint for constructing custom MapRoulette objects. As of <a href="https://github.com/osmlab/maproulette-python-client/releases/tag/v1.2.1">Version 1.2.1</a>, we support Models for Tasks, Challenges, Projects, and Priority rules. Use the ChallengeModel to add properties to a new Challenge. For example, you can create a ChallengeModel with one of the required properties — <em>name.</em></p><pre>challenge_model = maproulette.ChallengeModel(name=&#39;My Test Challenge&#39;)</pre><p>And easily add more properties.</p><pre>challenge_model.description = &quot;A description of my new challenge&quot;<br>challenge_model.instruction = &quot;Do something&quot;</pre><p>Priority rules are commonly used properties for Challenges. These allow you to construct a set of Priority values for Tasks within a Challenge, indicating which Tasks are most important to resolve. A Challenge can have up to three rules<em> </em>— high, medium, and low — and each rule can be made up of several sub-rules joined by <em>AND/OR </em>logic. I find the PriorityRuleModel especially useful due to the layers of complexity involved in correctly formatting a Priority rule for the API. See the following example.</p><p><strong>Goal</strong>: Create a PriorityRuleModel object with two sub-rules constituting a single high Priority rule for a Challenge.</p><pre>rule_1 = maproulette.PriorityRule(<br>    priority_value=&#39;highway.footway&#39;,<br>    priority_type=maproulette.priority_rule.Types.STRING,<br>    priority_operator=maproulette.priority_rule.StringOperators.EQUAL<br>)<br>rule_2 = maproulette.PriorityRule(<br>    priority_value=&#39;highway.pedestrian&#39;,<br>    priority_type=maproulette.priority_rule.Types.STRING,<br>    priority_operator=maproulette.priority_rule.StringOperators.EQUAL<br>)<br>high_priority_rule = maproulette.PriorityRuleModel(<br>    condition=maproulette.priority_rule.Conditions.OR, <br>    rules=[rule_1, rule_2]<br>).to_json()</pre><p>The PriorityRuleModel creates the following JSON structure.</p><pre>{<br>    &quot;condition&quot;: &quot;OR&quot;,<br>    &quot;rules&quot;: [<br>        {<br>            &quot;value&quot;: &quot;highway.footway&quot;,<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;operator&quot;: &quot;equal&quot;<br>        },<br>        {<br>            &quot;value&quot;: &quot;highway.pedestrian&quot;,<br>            &quot;type&quot;: &quot;string&quot;,<br>            &quot;operator&quot;: &quot;equal&quot;<br>        }<br>    ]<br>}</pre><p>This example signifies the power of Models! The PriorityRuleModel ensures the proper format is created for the MapRoulette API. Now you can assign this PriorityRuleModel as the ChallengeModel’s <em>high_priority_rule</em> property.</p><pre>challenge_model.high_priority_rule = high_priority_rule</pre><p>For simplicity, you can use an Overpass query to define the Tasks within the Challenge. For example, this query selects two ways — one that should get assigned a high Priority value based on its <em>highway</em> tag and one that shouldn’t.</p><pre>query = &quot;way(id:612034979,91581586);\nout body geom qt;&quot;<br>challenge_model.overpassQL = query</pre><p>Now create your Challenge!</p><pre>import json<br>print(json.dumps(api.create_challenge(challenge_data), indent=4, sort_keys=True))</pre><p>The <em>create_challenge</em> method returns a JSON object of the newly created Challenge and its associated properties.</p><pre>{<br>    &quot;data&quot;: {<br>        &quot;blurb&quot;: &quot;&quot;,<br>        &quot;checkinComment&quot;: &quot;&quot;,<br>        &quot;checkinSource&quot;: &quot;&quot;,<br>        &quot;cooperativeType&quot;: 0,<br>        &quot;created&quot;: &quot;2020-06-26T19:28:07.511Z&quot;,<br>        &quot;customBasemap&quot;: &quot;&quot;,<br>        &quot;dataOriginDate&quot;: &quot;2020-06-26T19:28:07.514Z&quot;,<br>        &quot;defaultBasemap&quot;: -1,<br>        &quot;defaultBasemapId&quot;: &quot;&quot;,<br>        &quot;defaultPriority&quot;: 0,<br>        &quot;defaultZoom&quot;: 13,<br>        &quot;deleted&quot;: false,<br>        &quot;description&quot;: &quot;A description of my new challenge&quot;,<br>        &quot;difficulty&quot;: 2,<br>        &quot;enabled&quot;: true,<br>        &quot;exportableProperties&quot;: &quot;&quot;,<br>        &quot;featured&quot;: false,<br>        &quot;highPriorityRule&quot;: {},<br>        &quot;id&quot;: 13814,<br>        &quot;infoLink&quot;: &quot;&quot;,<br>        &quot;instruction&quot;: &quot;Do something&quot;,<br>        &quot;lastTaskRefresh&quot;: &quot;2020-06-26T19:28:07.514Z&quot;,<br>        &quot;lowPriorityRule&quot;: {},<br>        &quot;maxZoom&quot;: 19,<br>        &quot;mediumPriorityRule&quot;: {<br>            &quot;condition&quot;: &quot;OR&quot;,<br>            &quot;rules&quot;: [<br>                {<br>                    &quot;operator&quot;: &quot;equal&quot;,<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;value&quot;: &quot;highway.footway&quot;<br>                },<br>                {<br>                    &quot;operator&quot;: &quot;equal&quot;,<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;value&quot;: &quot;highway.pedestrian&quot;<br>                }<br>            ]<br>        },<br>        &quot;minZoom&quot;: 1,<br>        &quot;modified&quot;: &quot;2020-06-26T19:28:07.518Z&quot;,<br>        &quot;name&quot;: &quot;My Test Challenge 1&quot;,<br>        &quot;osmIdProperty&quot;: &quot;&quot;,<br>        &quot;overpassQL&quot;: &quot;way(id:612034979,91581586);\nout body geom qt;&quot;,<br>        &quot;owner&quot;: 1814383,<br>        &quot;parent&quot;: 3006,<br>        &quot;popularity&quot;: 1593199687,<br>        &quot;remoteGeoJson&quot;: &quot;&quot;,<br>        &quot;requiresLocal&quot;: false,<br>        &quot;status&quot;: 1,<br>        &quot;statusMessage&quot;: &quot;&quot;,<br>        &quot;tags&quot;: [],<br>        &quot;updateTasks&quot;: false,<br>        &quot;virtualParents&quot;: []<br>    },<br>    &quot;status&quot;: 201<br>}</pre><p>Let’s open this <a href="https://maproulette.org/browse/challenges/13814">test Challenge</a> in MapRoulette! You’ll notice that there are two Tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PS1fX6UC5d79EOFq0X-Z1w.png" /><figcaption>The Challenge’s overview page in MapRoulette.</figcaption></figure><p>As expected, one Task has a high Priority value, while the other has a medium value (the project’s default Priority).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJ6eyCpYIJogFndzm89OLw.png" /><figcaption>Notice, each Task has a Priority value assigned.</figcaption></figure><h3>Conclusion</h3><p>These are just a few features in the MapRoulette Python Client that we’re excited to introduce, and hopefully, help users streamline their workflows. See something we should add? Submit an <a href="https://github.com/osmlab/maproulette-python-client/issues">issue</a>! Or better yet, contribute!</p><p>Remember MapRoulette is the game of chance where everyone is a winner. Users get happy by getting mappy!</p><p><em>Helpful links</em></p><p>· <a href="https://github.com/osmlab/maproulette-python-client">Github repo</a></p><p>· <a href="https://pypi.org/project/maproulette/">PyPI Package</a></p><p>· <a href="https://maproulette-python-client.readthedocs.io/en/latest/">Documentation</a></p><p>·<a href="https://github.com/osmlab/maproulette-python-client/tree/dev/examples"> Examples</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1c42789f86b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/introducing-a-new-python-client-for-the-maproulette-api-f1c42789f86b">Introducing: A New Python Client for the MapRoulette API</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving OpenStreetMap Data Quality with Atlas Checks]]></title>
            <link>https://medium.com/critigenopensource/improving-openstreetmap-data-quality-with-atlas-checks-774a5831a34a?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/774a5831a34a</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[atlas-check]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Sayana Saithu]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 17:49:23 GMT</pubDate>
            <atom:updated>2020-07-03T18:03:58.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pyq6q2WiQdfV409Xdhz93g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@justjohnl?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">John Lockwood</a> on <a href="https://unsplash.com/photos/FcLq69V7Rsc">Unsplash</a></figcaption></figure><p>The <a href="https://en.wikipedia.org/wiki/OpenStreetMap">OpenStreetMap</a> (OSM) project, a free, editable map of the world, has been widely used by organizations, NGOs, humanitarian groups, and the scientific research community as a data provider for turn-by-turn navigation, disaster and pandemic response, education, and more. The data is community-driven, so any registered user can add, update, or delete map data. As a result, OSM end-users are consistently looking to improve the quality and integrity of the basemap. The open-source community has developed several quality assurance tools (Osmose, OSMCha, KeepRight) to ensure new edits withhold the highest standards. In this post, I’ll review the open-source quality assurance tool <a href="https://github.com/osmlab/atlas-checks">Atlas Checks</a>.</p><h3>What are Atlas Checks?</h3><p>The Atlas Checks framework provides quality assurance tools for programmatically detecting OSM mapping errors. Each Atlas Check is an algorithm that defines and flags map errors with individual instructions describing the problem and how to fix them.</p><blockquote>“An Atlas Check is simply an algorithm that you can write that will find and flag issues in the OSM Basemap data through the use of Atlas, an in memory graph that represents the underlying OSM Basemap data.” <a href="https://github.com/osmlab/atlas-checks/blob/dev/docs/dev.md">Atlas Checks Guide</a></blockquote><p>As of June 2020, there are a total of <a href="https://github.com/osmlab/atlas-checks/blob/dev/docs/available_checks.md">63 available Atlas Checks</a> that detect map errors! The following categories broadly classify each check:</p><ol><li>Geometry checks</li><li>Tag checks</li></ol><p><strong>Geometry-based </strong>Checks flag map errors related to the shape or construction of OSM features. Polylines that cross buildings, features that self intersect, duplicate geometries, and roads that cannot be entered or exited are great examples — all of which can present navigation problems. The <a href="https://github.com/osmlab/atlas-checks/blob/dev/src/main/java/org/openstreetmap/atlas/checks/validation/linear/edges/FloatingEdgeCheck.java">FloatingEdgeCheck</a> flags such streets that are disconnected from road networks, making them non-navigable. <a href="https://github.com/osmlab/atlas-checks/blob/dev/docs/checks/roundaboutConnectorCheck.md">RoundaboutConnectorCheck</a> is another geometry related check that flags roads connected to roundabouts with extremely sharp angles, making it impossible to turn.</p><p><strong>Tag-based</strong> checks identify errors related to <a href="https://wiki.openstreetmap.org/wiki/Tags">OSM tags</a>, which are key-value attributes used to describe features of the core OSM elements. For example, an OSM Way tagged <em>highway=residential</em> describes the<em> </em>highway classification. Tag-based checks can identify features with missing tags, invalid tag combinations, or invalid tag values based on <a href="https://wiki.openstreetmap.org/wiki/Main_Page">OSM Wiki</a> or <a href="https://taginfo.openstreetmap.org/">TagInfo</a>. For example, <a href="https://github.com/osmlab/atlas-checks/blob/dev/src/main/java/org/openstreetmap/atlas/checks/validation/tag/HighwayToFerryTagCheck.java">HighwayToFerryTagCheck</a> flags OSM Ways which invalidly contain both <em>highway=*</em> and <em>route=ferry</em> tags. The OSM Wiki states that all <em>route=ferry</em> tags must also include a <em>ferry</em> tag. The value of the <em>ferry</em> tag is similar to <em>highway</em> types like <em>primary</em>, <em>secondary,</em> etc. For example, the Way in Figure 1 has a <em>route=ferry</em> tag and a <em>highway=path</em> tag but is missing a <em>ferry</em> tag. Hence this Way will be flagged. The check’s instruction directs the editor to remove the <em>highway</em> tag and replace it with the <em>ferry=path</em> tag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ld1z0FAoidz9oAOuIVjMxw.png" /><figcaption><strong>Figure 1: OSM Way (</strong><a href="https://www.openstreetmap.org/way/339440691"><strong>339440691)</strong></a><strong> with route=ferry tag and missing a <em>ferry=*</em> Tag.</strong></figcaption></figure><h3>Data Model Comparison: Atlas vs. OSM</h3><p>The Atlas project provides helpful tools (country-slicing, way-sectioning, tag filtering, etc.) for efficiently representing OSM data in-memory. Although the data model is a graph network optimized for routing, it stays true to the core OSM elements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*m-hRviL3I1jLoXKQQzpKnA.png" /><figcaption>OSM vs. Atlas Data Model.</figcaption></figure><ul><li>Nodes — point features stored as coordinates</li><li>Ways — ordered lists of Nodes forming open or closed lines</li><li>Relations — a collection of Nodes and Ways as members</li><li>Tags — represent key-value pairs as descriptors</li></ul><p>Atlas converts navigable OSM Ways and Nodes to Edges (e.g., roads, ferry routes) and Nodes (start/end of Edges and connections). All non-navigable features in OSM are modeled as Atlas Lines (e.g., power lines), Areas (e.g., parks, lakes), Points (e.g., trees, monuments), or Relations (e.g., bus lines). In addition to this, all the tags and geometries from OSM remain intact in Atlas.</p><h3>Atlas Checks Data Flow</h3><p>The Atlas Checks framework provides an <a href="https://github.com/osmlab/atlas-checks/blob/dev/src/main/java/org/openstreetmap/atlas/checks/distributed/ShardedIntegrityChecksSparkJob.java">Apache Spark job</a> that allows users to execute checks on a cluster. The input Atlas data provides partitioned country files (or shards) to process in parallel. Each Spark executor loads a set of Atlas shards in-memory, executes a configurable list of checks, and writes the output data to the provided file system (e.g., HDFS, S3, Cloud Storage).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnjeayZAXeEnmKLMeiVoOg.png" /><figcaption><strong>Figure 2: Atlas Checks Data Flow.</strong></figcaption></figure><p>Figure 2 represents the flow of data through the Atlas Checks framework. The Planet OSM data in the form of PBF files are first converted into Atlas files using the Atlas API. This conversion includes operations like country slicing, way sectioning, sharding, etc. (check out the <a href="https://github.com/osmlab/atlas">Atlas project</a> to learn more about these processes).</p><p>Once the PBF to Atlas generation is complete, we can use these files as the input to the next step. The Atlas Checks Spark job outputs line-delimited GeoJSON files for easy integration with editing tools like MapRoulette and JOSM. Optionally, the Atlas Checks project provides tools to upload data to Postgres, allowing users to leverage the power of SQL and PostGIS for spatial queries and analysis.</p><h3>Check Development</h3><p>Atlas Checks are written in Java and require the following tools for development and usage:</p><ul><li><a href="https://adoptopenjdk.net/?variant=openjdk11&amp;jvmVariant=hotspot">Java 11</a></li><li><a href="https://gradle.org/install/">Gradle</a></li><li><a href="https://git-scm.com/downloads">Git Command Line Tools</a></li></ul><p>An Atlas Check’s source code can be split into three sections:</p><ol><li>Initialize</li><li>Validate</li><li>Generate</li></ol><p>I’ll explain each of these using the <a href="https://github.com/osmlab/atlas-checks/blob/dev/src/main/java/org/openstreetmap/atlas/checks/validation/tag/HighwayToFerryTagCheck.java">HighwayToFerryTagCheck</a> as an example. As mentioned previously, the requirement of the check is to flag Edges with both a <em>highway</em> and <em>route=ferry</em> tag.</p><ol><li><strong>Initialize</strong></li></ol><p>Each check is initialized as the configuration is passed through the constructor. Here, class variables are set to be used throughout the check. In this case, the value of <em>minimumHighwayType </em>is derived from the configuration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0e9c1a7fbaa7feef0f2ad3b905c99d4/href">https://medium.com/media/c0e9c1a7fbaa7feef0f2ad3b905c99d4/href</a></iframe><p>The Spark Job is powered by a configuration file that allows users to apply and tweak global and individual settings for each run. These settings include OSM tag filters, MultiPolygon in and exclude filters, MapRoulette API configurations, and more. For more details on the checks config, check out the <a href="https://github.com/osmlab/atlas-checks/blob/dev/docs/configuration.md">documentation</a>.</p><p><strong>2. Validate</strong></p><p>During this stage, Atlas objects that don’t meet the conditions defined in the <em>validCheckForObject </em>function are filtered out. In our example, only Edges with a <em>route=ferry</em> tag and a <em>highway</em> classification greater than <em>this.minimumHighwayType</em> will be valid for the check.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb9e76ec1f4cc562476ba2feb6c703c5/href">https://medium.com/media/eb9e76ec1f4cc562476ba2feb6c703c5/href</a></iframe><p><strong>3. Generate</strong></p><p>During the Generate stage, Atlas objects that pass the Validate stage are further evaluated for map errors in the <em>flag() </em>method. Objects meeting the conditions of the flag method are flagged and given appropriate instructions. Once the check has completed, a line delimited GeoJSON is generated where each line represents a Flag object. Each flag contains information about its respective check, an editing instruction, and attribute information about the flagged Atlas object(s).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c995e6ced6ec12842196dcf86686eacf/href">https://medium.com/media/c995e6ced6ec12842196dcf86686eacf/href</a></iframe><h3>Atlas Checks Analysis</h3><p>Although Atlas Checks’ logic is based on specific OSM standards and rules, the flags produced may contain false positives due to various factors. For example, a geometry-based check, <a href="https://github.com/osmlab/atlas-checks/blob/dev/src/main/java/org/openstreetmap/atlas/checks/validation/intersections/LineCrossingWaterBodyCheck.java">LineCrossingWaterBodyCheck</a>, flags OSM Ways that cross waterbodies invalidly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OPfhfhjrHnAtPdV-Fl5WA.png" /><figcaption><strong>Figure 3: An </strong><a href="https://www.openstreetmap.org/way/318165685"><strong>OSM Way</strong></a><strong> (ferry route) intersecting a water body.</strong></figcaption></figure><p>In Figure 3, an <a href="https://www.openstreetmap.org/way/318165685">OSM Way</a> with <em>route=ferry</em> tag crosses a river. It was flagged erroneously by the LineCrossingWaterBodyCheck as an invalid crossing. To eliminate potential false positives, we recommend analyzing a sample of each checks’ results across unique markets around the world. The benchmark for accuracy is 90%!</p><h3>What’s Next?</h3><p>As the demand for quality assurance increases, the Atlas Checks project will continue to evolve with hopes to be adopted as the industry standard.</p><p>Check out the repository for <a href="https://github.com/osmlab/atlas-checks/blob/dev/docs/available_checks.md">new checks</a>, and create a GitHub<a href="https://github.com/osmlab/atlas-checks/issues/new?assignees=&amp;labels=new+integrity+check&amp;template=new-integrity-check.md&amp;title=%5BNew+Check%5D"> issue</a> if you have ideas for new use cases or improved functionality!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=774a5831a34a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/improving-openstreetmap-data-quality-with-atlas-checks-774a5831a34a">Improving OpenStreetMap Data Quality with Atlas Checks</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Earth Day 2020]]></title>
            <link>https://medium.com/critigenopensource/earth-day-2020-b965f174f9db?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/b965f174f9db</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[earth-day]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[climate-change]]></category>
            <dc:creator><![CDATA[Alex Sweeney]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 16:58:19 GMT</pubDate>
            <atom:updated>2020-04-17T17:05:04.105Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to celebrate in the time of COVID-19</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5aL-6Dl1UFC98j6eBJsGQ.jpeg" /><figcaption><em>Photo by </em><a href="https://unsplash.com/@nypl?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>The New York Public Library</em></a><em> on </em><a href="https://unsplash.com/s/photos/earth?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></figcaption></figure><p>April 22, 2020 — the 50th anniversary of the first <a href="https://www.earthday.org/history/">Earth Day</a> — will be a global day of science, education, and advocacy to promote the environment. This year’s theme of climate action is especially important, even as our lives are overshadowed by the COVID-19 outbreak. <br> <br>Despite the challenges and limitations presented by COVID-19, there are still many ways to get involved and make a difference. We’ve put together a list of opportunities that cover climate action, virtual science, geography, and ways to appreciate the earth from your own neighborhood.</p><h3><strong>Climate Action</strong></h3><p>Learn how to <a href="https://350.org/climate-strikes-in-the-time-of-coronavirus/">organize and protest</a> during a global pandemic; hear from writers, activists, and scientists during weekly webinars from the <a href="https://globalclimatestrike.net/press-release-talks-for-future-weekly-webinars-launched/">Global Climate Strike</a> movement; or take part in the <a href="https://www.earthday.org/campaign/act-on-climate-change/"><em>Act On Climate Change Campaign</em></a>. Want to learn more about the basics of climate change? Organizations like the <a href="https://www.ipcc.ch/">IPCC</a>, <a href="https://climate.nasa.gov/">NASA</a>, and <a href="https://scied.ucar.edu/climate">UCAR</a> have great sites that walk through the science.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMj6OdBPpz7psF22FIhpgQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">NASA</a> on <a href="https://unsplash.com/s/photos/satellites?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3><strong>Virtual Science and Education</strong></h3><p>Earth Day has its own citizen science project, <a href="https://earthchallenge2020.earthday.org/">Earth Challenge 2020</a>. It’s not the only project out there though. You can also<a href="https://www.climateprediction.net/"> help model future climates</a> with your home computer, pick from a variety of <a href="https://www.zooniverse.org/">natural science projects </a>that can be done from home, or contribute environmental themes on OpenStreetMap through <a href="https://osm.earth/">OSM Earth</a>. Your local museum, university, city, or county might be hosting a virtual earth day celebration. For example the Smithsonian Museum in Washington, D.C. is doing a <a href="https://earthoptimism.si.edu/2020-summit/">virtual summit</a>, so be sure to check out local museums and government websites.</p><h3><strong>Stay Home &amp; Stay Healthy</strong></h3><p>Find some COVID friendly suggestions from the <a href="https://www.earthday.org/11-actions-for-the-planet-during-a-pandemic/">Earth Day</a> organization, along with other ways you can <a href="https://www.un.org/en/actnow/index.shtml">live</a> more environmentally friendly and <a href="https://www.earthday.org/earth-day-challenge/">take action daily</a>. Learn how to identify the <a href="https://www.nwf.org/nativePlantFinder/plants">plants</a>, <a href="https://www.audubon.org/birding/backyard">birds</a>, and <a href="https://www.insectidentification.org/">insects</a> that live in your area during walks, or cozy up and watch <a href="https://medium.com/uncclearn/the-7-most-epic-climate-change-documentaries-abd52e3ddf64">documentaries</a> on climate change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQ_SulyXsYKYhYcrkmOJOQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@asya_tes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Asya Tes</a> on <a href="https://unsplash.com/s/photos/computers?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>What are your plans for Earth Day 2020? Tell us how you’re celebrating our precious orb on Twitter @osmquality or in the comments below!</p><p><em>By: </em><a href="https://medium.com/u/7a0d2cdeb8fb"><em>Ana Ordonez</em></a><em> &amp; Alex Sweeney</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b965f174f9db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/earth-day-2020-b965f174f9db">Earth Day 2020</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to run a Java 11 Spark Job on Google Cloud Dataproc]]></title>
            <link>https://medium.com/critigenopensource/how-to-run-java-11-on-your-gcp-dataproc-spark-cluster-8472f5fbaaf8?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/8472f5fbaaf8</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[dataproc]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[D Baah]]></dc:creator>
            <pubDate>Thu, 09 Apr 2020 16:02:17 GMT</pubDate>
            <atom:updated>2020-06-30T05:58:10.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JP6e1u6QywSyYrLBXojyyA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mitchel3uo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mitchell Luo</a> on <a href="https://unsplash.com/s/photos/google-cloud?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Managing Java &amp; Spark dependencies can be tough. We recently migrated one of our open source projects to Java 11 — a large feat that came with some roadblocks and headaches. Luckily, installing OpenJDK 11 was the easy part. In this tutorial, we’ll show you how to set up your Google Cloud Platform Dataproc Spark jobs to run software compiled in Java 11.</p><h3>Introducing GCP Dataproc</h3><p>Dataproc is a fairly new addition to the Google Cloud Platform (GCP). It offers everything you need to execute and manage large Apache Spark and Hadoop data processing jobs in the cloud. This includes services like on-demand ephemeral clusters, autoscaling, rapid cluster creation, out-of-the-box GCP service integration (Cloud SQL, Cloud Storage, etc.), and much more.<br> <br>In this article, we’ll show you how to do the following:</p><ol><li>Create a Custom Image</li><li>Create a Cluster <em>from</em> your custom image</li><li>Run a Java 11 Spark Job on your cluster</li></ol><p>Before we begin, let’s first review some GCP terminology:</p><ul><li><strong>Cluster</strong> — A cluster is a combination of master and worker machines used to distribute data processing.</li><li><strong>Image</strong> — Predefined virtual machine configurations used as instructions to launch new instances.</li><li><strong>Instance</strong> — A virtual machine hosted on Google’s infrastructure.</li><li><strong>Cluster Properties</strong> — Configuration properties for open source tools (hive, spark, yarn, presto, etc.) installed on each cluster instance.</li></ul><h3>1. Creating a Custom Image</h3><p>Although GCP Dataproc offers managed public images for both Debian and Ubuntu operating systems, none of them come pre-configured with Java 11. We’ll use the GCP Compute Engine custom image feature to spin up an image with open JDK 11 installed.<br> <br>Before we get started, make sure you have the google cloud sdk installed and configured on your machine. For installation instructions, please visit: h<a href="https://cloud.google.com/sdk/docs/quickstarts">ttps://cloud.google.com/sdk/docs/quickstarts.</a><br> <br>First, let’s clone this very useful custom-images repo. Here, we’ll execute a python script that automates the custom image generation workflow.</p><pre>git clone <a href="https://github.com/GoogleCloudDataproc/custom-images">https://github.com/GoogleCloudDataproc/custom-images</a></pre><p>Create the following shell script &amp; name it java-11.sh</p><pre><strong>#!/usr/bin/env bash</strong></pre><pre>echo &quot;Installing custom packages...&quot;</pre><pre>sudo apt-get -y update<br>sudo apt install -y openjdk-11-jdk<br>export JAVA_HOME=&quot;/usr/lib/jvm/java-11-openjdk-amd64&quot;export PATH=$PATH:$JAVA_HOME/bin</pre><pre>echo $JAVA_HOME<br>java -version</pre><pre>echo &quot;Successfully installed java 11 packages.&quot;</pre><p>Execute the generate_custom_image.py with the following arguments:</p><ul><li><strong>image-name</strong>: the name of your image (must be unique)</li><li><strong>dataproc-version</strong>:<strong> </strong>see <a href="https://cloud.google.com/dataproc/docs/concepts/versioning/overview">https://cloud.google.com/dataproc/docs/concepts/versioning/overview</a> for more details</li><li><strong>customization-script</strong>: path to our java-11.sh script</li><li><strong>zone</strong>: GCP region/zone - <a href="https://cloud.google.com/compute/docs/regions-zones">https://cloud.google.com/compute/docs/regions-zones</a></li><li><strong>gcs-bucket</strong>: GCP Storage bucket URL</li></ul><pre>python3 generate_custom_image.py \<br>    --image-name &quot;custom-ubuntu-java11&quot; \<br>    --dataproc-version &quot;1.4.24-ubuntu18&quot; \<br>    --customization-script java-11.sh \<br>    --zone &quot;us-west1-a&quot; \<br>    --gcs-bucket &quot;gs://my-gcp-storage-bucket&quot; \<br>    --shutdown-instance-timer-sec 500</pre><p>Coffee break ☕️ — this will take a few minutes. Once complete, quickly verify that your custom image exists.</p><pre>gcloud compute images list --filter=&quot;name=(&#39;custom-ubuntu-java11&#39;)&quot;</pre><p>Awesome! We now have a custom image we can use to create a Dataproc cluster.</p><h3>2. Creating a Cluster</h3><p>Let’s create a cluster. GCP does an awesome job of providing the Developer community with an abundance of tools for interacting with their various APIs. Dataproc has client libraries available in various programming languages (<a href="https://github.com/googleapis/google-api-nodejs-client">Node.js</a>, <a href="https://github.com/googleapis/google-api-python-client">Python</a>, <a href="https://github.com/googleapis/google-cloud-java">Java</a>, <a href="https://github.com/googleapis/google-cloud-go">Go</a>, <a href="https://developers.google.com/api-client-library">and more</a>), as well as via the gcloud cli. Since we haven’t fully automated this workflow, we’ll use the GCP console UI.<br> <br>Go to the <a href="https://console.cloud.google.com/dataproc/clustersAdd">Create Cluster</a> page in your GCP console.</p><p>After configuring resources for your master and cluster nodes, scroll towards the bottom of the page and select <strong>“Advanced options”.</strong></p><ol><li>Set your cluster image as “custom-ubuntu-java11”.</li></ol><figure><img alt="Pick your image" src="https://cdn-images-1.medium.com/proxy/1*MXQygj2lhnztwa3VmKz82g.png" /></figure><figure><img alt="Select custom image tab" src="https://cdn-images-1.medium.com/max/1024/1*MzsuVtIGq2cTdQsdK6rqLA.png" /></figure><p>2. Add the following cluster properties. Again, these are custom properties used to overwrite the default Spark config.</p><figure><img alt="Cluster property example" src="https://cdn-images-1.medium.com/max/1024/1*PyvBW_5hhAulgbK-LvtmwQ.png" /></figure><p><strong>spark-env:JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64</strong><br><strong>spark:spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64</strong></p><p>This sets the default JAVA_HOME <a href="https://spark.apache.org/docs/latest/configuration.html#environment-variables">environment variable</a> for the Spark driver and <a href="https://spark.apache.org/docs/latest/configuration.html#runtime-environment">executors</a>.</p><p>At this point, we’re good to go — select “Create” and watch your cluster provision.</p><h3>3. Submit Spark Job</h3><p>Now that we have everything set up, this last part should be quick. In the Daraproc console, open the <a href="https://console.cloud.google.com/dataproc/jobsSubmit">Job Submission</a> page and select the cluster we created above.<br> <br>First, make sure you’ve set the <strong>Job Type</strong> to Spark. Next, add your main class, program arguments, and the path to your jar file (in this example, I uploaded a shaded jar to a <a href="https://cloud.google.com/storage/docs/json_api/v1/buckets">Cloud Storage bucket</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpZvCoEGESfKEUNRm3uFJQ.png" /></figure><p>You now have a Spark job running Java 11 software! If you run into a <em>ClassNotFound</em> or <em>NoSuchMethodError</em> Exception, there may be a dependency conflict between your jar and the software installed on the cluster —which can be resolved by <a href="https://cloud.google.com/blog/products/data-analytics/managing-java-dependencies-apache-spark-applications-cloud-dataproc">building a shaded jar</a>.<br> <br>Now that we have the process down for creating clusters with the right software, we’re all set up for automation with workflow orchestration tools like <a href="https://github.com/apache/airflow">Apache Airflow</a>, <a href="https://github.com/Netflix/metaflow">Metaflow</a>, <a href="https://github.com/spotify/luigi">Luigi</a>, and many more. Enjoy!</p><p><em>Leave a comment below if you have additional questions!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8472f5fbaaf8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/how-to-run-java-11-on-your-gcp-dataproc-spark-cluster-8472f5fbaaf8">How to run a Java 11 Spark Job on Google Cloud Dataproc</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collecting Metrics, Going Mobile & More]]></title>
            <link>https://medium.com/critigenopensource/collecting-metrics-going-mobile-more-ffedb0e70d2?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/ffedb0e70d2</guid>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Matthew Manley]]></dc:creator>
            <pubDate>Wed, 01 Apr 2020 01:11:11 GMT</pubDate>
            <atom:updated>2020-04-10T21:29:00.780Z</atom:updated>
            <content:encoded><![CDATA[<h4>Insights from a MapRoulette Meetup</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*If9B8m4jfWg8Qb0mO8wXHA.png" /><figcaption>MapRoulette hosts challenges that are focused on improving data quality in OSM. Users can search for challenges based on a number of parameters — location, theme, difficulty, etc.</figcaption></figure><p><a href="https://maproulette.org/">MapRoulette</a> has become the go-to micro-tasking tool for OSM, thanks in large part to its diverse user base ranging from local experts to corporate organized mapping teams and everyone in between. Since its inception, MapRoulette has been a powerful tool for improving data quality in OSM — one task at a time. Users are served tasks they can complete in the OSM editor of their choice. To date, over 500,000 tasks have been completed in MapRoulette! Critigen’s Open Data &amp; Development team regularly uses MapRoulette to publish <a href="https://github.com/osmlab/atlas-checks">Atlas-Checks</a> results as well as online mapathons in an effort to contribute to OSM. <br> <br> A colleague and I recently attended a MapRoulette meet-up in Seattle. There were around 25 people who attended from different backgrounds — all representing important stakeholders for MapRoulette. The general theme of the day was slated to be “MapRoulette Metrics” but a wide range of topics were discussed.</p><h3>Collecting Metrics from MapRoulette</h3><p>Metrics have been a hot topic within the MapRoulette community. There has been a push, led primarily by corporate mapping teams, to engage in a broader discussion about how metrics can be collected from MapRoulette. The conversation at the meetup revolved around two types of metrics: <em>changeset metrics</em> and <em>task status metrics</em>.</p><h4>Changeset Metrics</h4><p>Examples of changeset metrics include total kilometers of road added per task, number of nodes created per task, or time spent editing per task. In these cases the metrics are based on information that can be found in OSM via the changeset ID. The current mechanism MapRoulette uses to link each task to its respective changeset ID is imperfect, and needs to be enhanced for greater accuracy and reliability.</p><h4>Task Status Metrics</h4><p>The second type of metrics is based purely within the MapRoulette environment. Each task has a status that can be assigned to one of the following seven values:</p><ul><li>0 — Created</li><li>1 — Fixed</li><li>2 — False Positive</li><li>3 — Skipped</li><li>4 — Deleted</li><li>5 — Already Fixed</li><li>6 — Too Hard</li></ul><p>Below is an example of how these task statuses are surfaced to the user within the MapRoulette UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*avxslwHYPi8SrLrJZD5KUg.png" /></figure><p>These statuses and any changes to them are recorded in the task history. Task histories can be aggregated and used to gather metrics on challenges or projects. This type of metric can be obtained easily using the <a href="https://maproulette.org/docs/swagger-ui/index.html?url=/assets/swagger.json#/">MapRoulette API.</a></p><h3>Quick Fixes</h3><p>Beyond our discussion of metrics, another topic was a new functionality in MapRoulette called “quick fixes”. These are tasks that can be completed by the user without ever leaving MapRoulette! This type of task can be an efficient way to clean-up OSM data, but the tasks must be very simple (i.e. a yes or no question). Below is an example of a quick fix challenge where the user is given a set of tasks that may benefit from a “lanes = 1” tag. The user can either accept, reject, or skip each task. If they accept, that tag will be updated directly in OSM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkMA-7ivp1msKtBvUTNGyQ.png" /></figure><p>So far the only testing that has been done has involved simple tag changes. There is some interest in expanding into geometry-based quick fixes, but this is still being tested. Our discussion revolved around the best way to roll out quick fixes and what they would be most useful for.</p><h3>Mobile</h3><p>A good amount of time was spent discussing the potential for a MapRoulette mobile app. One cool idea was to allow users to create tasks in the field, that armchair mappers could work on in the usual desktop format. Another idea was to serve tasks that require field validation in quick-fix format so a mobile user could complete them within the app. There was an emphasis on determining the value that the app would provide and how that would compliment the current MapRoulette workflow. In any case, app development is a large project and the consensus was to start small.</p><h3>User Groups</h3><p>MapRoulette has functionality to track project progress on the task level. However, it’s not easy to understand the diversity of users that are accessing your project. For example, is there one super-user completing all the tasks or multiple users? Introducing user groups may help in collecting user metrics. In order to harvest any user related information, user groups would have to agree on some tracking or monitoring mechanisms. Corporate use-cases include managing the editing/review process with editors. A more community-aligned use-case would be allowing users to join a user group based on their interests (i.e. address mappers or reviewers). MapRoulette could suggest challenges that are relevant to the user’s interests if this information was available.</p><h3>Community Engagement</h3><p>Martijn van Exel, MapRoulette’s creator, was present at the meeting. He continues to be integral in guiding MapRoulette’s development as an important open source mapping tool. An important piece to this is the continued engagement of the OSM community with the main priorities of retaining avid MR users, and attracting new mappers. There was a spirited discussion about the potential for further gamification of MapRoulette and how to do more outreach through <a href="https://twitter.com/maproulette">Twitter</a>, Slack, and mapathons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*H_RyeWjcBW76fKgUt5tvUA.png" /></figure><h3>Summary</h3><p>Overall, this was an inspiring meeting to attend and witness so much personal and organizational investment in MapRoulette. It’s clear that it remains an important asset to the larger OSM community. There’s a lot of excitement around its continued development, but the project is in need of engineers and developers to engage and begin contributing. This will be key, as any development of the project is limited by personnel and time constraints. Critigen continues to contribute to MapRoulette by regularly pushing challenges, and we recently started working on a <a href="https://github.com/osmlab/maproulette-python-client">Python client for the MapRoulette API </a>that we’re super excited to roll out in the near future.</p><p>Want to contribute to OSM? Feel free to contribute to our <a href="https://maproulette.org/browse/projects/31237">GeoWeek Virtual Mapathon Project</a> that we <a href="https://medium.com/critigenopensource/improving-osm-data-one-virtual-mapathon-at-a-time-cb64e3445f69">blogged</a> about earlier this year!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffedb0e70d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/collecting-metrics-going-mobile-more-ffedb0e70d2">Collecting Metrics, Going Mobile &amp; More</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving OSM Data — One Virtual Mapathon at a Time]]></title>
            <link>https://medium.com/critigenopensource/improving-osm-data-one-virtual-mapathon-at-a-time-cb64e3445f69?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb64e3445f69</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[maproulette]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[mapathon]]></category>
            <dc:creator><![CDATA[Muning Monica Brandeis]]></dc:creator>
            <pubDate>Wed, 19 Feb 2020 17:19:05 GMT</pubDate>
            <atom:updated>2020-02-19T17:19:05.301Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving OSM Data — One Virtual Mapathon at a Time</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*llLRL6_IAH4_HSPcU_RcbQ.png" /></figure><blockquote>Critigen’s Open Data &amp; Development team hosted an online mapathon during Geography Awareness Week to contribute to OpenStreetMap. Unlike traditional mapathons that typically digitize new features, our team tested a new approach, a different user interface, and a focus on correcting map errors.</blockquote><h3>Mapathons 101</h3><p>Traditionally, mapathons are organized to bring communities together to map environments important to their locality. Today, new tools and technology have expanded the scope and lowered barriers for participation — making it easier for mappers to contribute various themes and features. Mapping can refer to either tracing something using aerial photos (like a building), or correcting the digitization of something that’s malformed and/or against mapping standards (improperly mapped roundabout). <br> <br>OpenStreetMap (OSM) is a collaborative project that aims to create a free, editable, open map of the world. It provides an excellent interface for communities around the globe to contribute their local knowledge and map their own communities. Hosting an OSM mapathon provides immediate feedback and support to these communities. To contribute our efforts to Geography Awareness week, Critigen created a virtual OSM mapathon. In the following sections, we’ll walk through the conceptual arrangement and planning of a mapathon, Critigen’s experiment, and observations from our inaugural virtual mapathon.</p><h3>Getting Started</h3><p>Hosting a successful mapathon is similar to hosting a focus group; a clear purpose and focused theme, effective marketing and communication, engagement of participants with similar interests, and inclusive event space are all essential ingredients. In addition, having facilitator(s) that are familiar with the OSM data model and mapping best practices is key. Here are some tips to assist you in your mapathon planning:</p><ul><li><strong>The purpose should be simple.</strong> What do you want to accomplish with your participants? Examples include: digitizing parks, squaring and closing buildings, or realigning roads using aerial photos or satellite imagery.</li><li><strong>The complexity of your theme should match the participants’ knowledge and skills.</strong> If you’re hosting the event for a large group, consider providing training materials beforehand to mitigate potential barriers to participation.</li><li><strong>If hosting a mapathon in person, it’s important to select an open but dedicated space.</strong> The participants should feel relaxed and free to ask questions and engage with other participants.</li><li><strong>Target the duration of the event to be between one and four hours.</strong> If the duration is longer than one hour, consider multiple sessions to provide more participant interaction.</li><li><strong>The facilitator plays a pivotal role.</strong> Facilitators should be able to answer all types of questions participants may have, as well as control the agenda and schedule. This is particularly important if there is a training session before the mapathon or the length is over two hours. Specifically, the facilitator should know the contents of what will be mapped in detail, be able to provide directions on different edits, and most importantly, transfer their knowledge to the participants with ease.</li></ul><h3>Experimental Mapathon: New approach, platform, and mapping content</h3><p>To celebrate Geography Awareness week at Critigen, we organized a company-wide mapathon with a twist: 100% virtual. The planning and preparations were slightly different than conventional mapathon. Here’s how we set it up:</p><ul><li>Uploaded MapRoulette Challenges for 24 US and 2 UK cities based on staff locale.</li><li>Created seven training videos to educate participants on how to edit different map errors.</li><li>Posted MapRoulette Challenges including both geometry and tagging errors.</li><li>Broadcasted the event via email and Slack.</li><li>Kept the mapathon open for a few months to allow users to find time to participate and collected data on users’ interaction and activities.</li><li>Hosted an in-person mapathon as a control group by using the same MapRoulette challenges.</li></ul><h3>Lessons Learned</h3><p>Our virtual mapathon experiment was a hit from the beginning. In the first two months, we collected more than 800 edits from 27 different contributors (25% of participants were new mappers!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZM2PhVQmdpPSLsDfxjjWA.png" /><figcaption>Completion Stats for Our Mapathon</figcaption></figure><p>Three main takeaways emerged from our experiment:</p><ul><li><strong>Keep it simple</strong>. Most edits were simple geometry fixes because it required the least amount of research. For attribution-based map errors, the challenge difficulty was heavily dependent on resource availability. For example, without great street-side imagery, correcting and validating speed limit tags is virtually impossible. Mapathon organizers should evaluate the feasibility of edits when creating tasks that rely on different resources. In addition, providing fix scenarios could help facilitate edits. For instance, when flagging an address point that’s missing a road name, providing nearby street names for mappers to select from may be an easier editing task than having them research surrounding roads to find the correct name. Keeping tasks simple and small allows mappers to get into a rhythm and really crank through some fixes!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TuU02G6rWCzMjVOfbbZTOw.png" /><figcaption>Example editing task for the Spiky Building Check</figcaption></figure><ul><li><strong>New theme: correcting map errors.</strong> The locality is always important in conventional mapathons so that participants can feel engaged and contribute their local knowledge to the map. Our experiment populated 26 cities based on where Critigen staff are located. However, we observed that participants preferred working on editing tasks they were interested in (e.g. feature types or attributes), rather than focusing on making edits in specific cities. We noted that locality was a minor factor to consider when developing tasks, particularly when correcting map errors was the event’s theme. Over time, as OpenStreetMap is becoming “complete”, we’ll all need to focus on perfecting map quality and finding some smaller scale changes and fixes.</li><li><strong>Communication is critical.</strong> Our control group had more conversations and exchanged ideas on how to fix difficult map errors, enabling our group to fix tasks that were not as straightforward as others. While participants in the control group work with OSM data every day, a knowledge gap was present when it came to editing features. Scheduling remote office hours and/or working with key, local stakeholders may mitigate communication and knowledge gaps that exist.</li></ul><h3>It’s Your Turn!</h3><p>We enjoyed hosting a virtual mapathon, here are some ideas to make the most of your experience:</p><ul><li>Host simple editing tasks and make sure all resources (aerial and/or street-view) are available (by reviewing sample tasks).</li><li>Since cities/regions are not as critical as they would be in a conventional mapathon, it’s a good idea to generate tasks for an area that require many fixes (a critical density of issues).</li><li>Have dedicated remote “virtual office hours” for volunteer Q &amp; A and discussions. Send out reminder emails to help promote and engage more people.</li><li>Reward participants by sharing their efforts and achievement with the community. A leadership board for personal achievement is used in most platforms nowadays, but we think community recognition may be the next stage of rewards.</li><li>If planning to host a virtual mapathon routinely, consider combining with some dashboard metrics to monitor and analyze the project process over time.</li></ul><p>Mapping new features can be fun, but correcting map errors makes OSM better. By sharing our experience, we hope to inspire more mapathons to focus on improving OSM data quality in the future!</p><p><em>Youtube Training Videos</em></p><ul><li><a href="https://youtu.be/wSXtvtWG-qs">Atlas Checks and Map Errors Introduction</a></li><li><a href="https://youtu.be/OkEzlfEOR_0">MapRoulette Introduction</a></li><li><a href="https://youtu.be/pdxDFYjf7Nk">OSM Editing Introduction</a></li><li>Editing Demo— <a href="https://youtu.be/_enPipnNEHY">Address Point Match Check</a></li><li>Editing Demo— <a href="https://youtu.be/ATFEXvsIyBc">Building Road Intersection Check</a></li><li>Editing Demo— <a href="https://youtu.be/lkA1NgDRoNM">Spiky Building Check</a></li><li>Editing Demo — <a href="https://youtu.be/WLI9ILTERRs">Unusual Layer Tag Check</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb64e3445f69" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/improving-osm-data-one-virtual-mapathon-at-a-time-cb64e3445f69">Improving OSM Data — One Virtual Mapathon at a Time</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An approach to integrating MapboxGL in React / Redux.]]></title>
            <link>https://medium.com/critigenopensource/an-approach-to-integrating-mapboxgl-in-react-redux-b50d82bc0ed0?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/b50d82bc0ed0</guid>
            <category><![CDATA[mapbox]]></category>
            <category><![CDATA[redux]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Cole Bessee]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 18:15:04 GMT</pubDate>
            <atom:updated>2019-12-20T18:15:03.952Z</atom:updated>
            <content:encoded><![CDATA[<h3>An approach to integrating MapboxGL in React / Redux</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hv5kzTztO3sQaM3YB2iQkg.png" /></figure><p>As a geospatial consulting company, we’ve encountered just about every permutation of popular front-end frameworks and mapping libraries. From Backbone with EsriJS to modern Angular with Leaflet, we’ve taken on projects with all sorts of front-end tech stacks. These modern front-end stacks have really improved our efficiency as developers and make it easy to on-board new developers to our projects. Recently, we’ve been using React as a common framework across many of our projects. Likewise, Mapbox has been emerging as a developer favorite for performant map based data visualizations in the browser. React and Mapbox easily pair together for simple use cases and visualizations. However, challenges arise as the complexity of the application grows. The biggest challenge is how to logically organize code within the class structure defined by your framework of choice. In the case of React, we have a front-end JS framework that leverages JSX to allow developers to be more descriptive in their code and abstracts away the need for developers to directly modify the DOM. Meanwhile, mapping libraries like Mapbox provide developers with fine tuned control over the map visualization. In a map based application this means a large chunk of your code will end up in a single “Map” component that listens to user input from various other components to manage and style the map object. Unfortunately that can lead to a Map component with thousands of lines of code and convoluted, difficult to manage integration points with the rest of your application. <br> <br>Recently, we were tasked with modernizing and enhancing an existing Mapbox based web application. The goal was to bring an improved, documented structure to the codebase that would allow developers to more rapidly add features. Naturally, we reached for React as our framework of choice to bring some structure. In this post, we’ll share the strategy we used to integrate Mapbox GL with React/Redux.<br> <br>This project uses the following architecture:</p><ul><li>React with class based components</li><li>Redux, using an action/reducer pattern, for global state management</li><li>MapBoxGL JS for mapping visualizations</li></ul><p>It’s best if you’re familiar with these patterns and concepts individually before we look into the code and structure associated with integrating them in a more seamless way. Specifically, we’ll look at the component and file structure, how to share the map instance using the React Context API, and useful reducers for modifying stored map state.</p><h3>The Application</h3><p>Before diving into specific code, let’s take a look at the basic component hierarchy of our application. The application features a single map instance and a side panel of UI controls that manage different map visualization experiences. Each of these experiences is represented as a single tab in a vertical accordion such that only one accordion tab and therefore one set of data visualizations can be enabled at a time. As each accordion expands, additional UI controls are exposed that allow the user to further customize that map experience. A handful of other UI controls exist outside of the accordion that control persistent data points and can be visualized in any of the map experiences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*_jSxlYmrnOLdX272tl2vrA.png" /><figcaption>The Application and Component Hierarchy</figcaption></figure><h3>Component and File Structure</h3><p>For this project, a standardized approach was taken to each component. This approach encapsulates both rendering the UI for controlling the map as well as defining any associated data layers. Examples of components that follow this structure in the above graphic are the map experience controls and the controls for persistent data. For each of these, the component folder is comprised of three files: a JS file which defines the react component class, a sass file for styling the component’s UI, and a layer definitions file. The layer definitions file exports methods for adding the necessary sources and layers to the map in order to support the visualization of data that is controlled by that React component UI. Below are the important aspects of the component file structure.</p><pre>components/<br>  map/<br>    sidepanel/<br>      PopulationPanel/<br>        PopulationLayers.js<br>        PopulationPanel.js<br>        PopulationPanel.scss<br>redux/<br>  actions/ <br>  reducers/</pre><p>For this post we’ll be focusing on one of the map experience controls, the PopulationPanel component as an example case. The following layer definition file provides functions for adding a handful of population related data sources and layers to the map.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19f8e595c9e97e0efc55381e92fba54a/href">https://medium.com/media/19f8e595c9e97e0efc55381e92fba54a/href</a></iframe><h3>Sharing the Map Instance</h3><p>Now we have functions defined that help us add the data controlled by our component to the map. The ultimate goal is to keep the Map component in charge of defining the basic details and let the various UI components take charge of the management of each layer they control. To do so, we need to provide a reference from our map object to each of our map experience components so they can add/remove data and event listeners to the map. Originally, we thought we might use our existing Redux store to share the map instance across the application. However, this turned out to be far too heavy of an object that caused redux to throw warnings and lag behind. Instead, we leveraged React’s context API to share the map instance created in the map component with the various components in the SidePanel. <br> <br> First, we simply create a context and export it for use. We store this file in a context folder at the root of our project.</p><pre>import React from &#39;react&#39;;<br><br>const MapContext = React.createContext();<br><br>export default MapContext;</pre><p>Then, in the map component, we define a provider for the MapContext and supply it a value of the map instance to share with all of its children. It’s important that any component that needs access to this value is a child of the provider.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f3163cec340198658aa7cf88cca9b24/href">https://medium.com/media/0f3163cec340198658aa7cf88cca9b24/href</a></iframe><p>The SidePanel and all of its children are now children of the MapContext Provider. Any of these components can now import the context object, set their own context type to the context object, and extract the value from the components context property for use. Using a population component as an example, we can now add the population sources and layers to the map inside our component by using the functions we defined in the adjacent layer definition file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b680370e202c2711f35463499095438b/href">https://medium.com/media/b680370e202c2711f35463499095438b/href</a></iframe><h3>Mapbox Style and the Redux Store</h3><p>Now that we have the ability to define and add map layers in each component, our next step is to manipulate those map layers based on the UI state of our map experience components. In the MapBox GL JS API, the map visualization is defined by the <a href="https://docs.mapbox.com/mapbox-gl-js/style-spec/">MapBox style object</a>. This object contains definitions for each single data layer available to the map: whether it’s visible, how to filter the data, and directions for how to style the layer. The style of the map needs to be both visible to the map component and available to the other UI components for editing. To do so, we add the map style object to the redux store then, provide actions that each component can use to update specific parts of the style object while the Map component subscribes to this style via props and manages updating the map. The following are some actions and associated reducer cases for common MapBox layer, source, and style updates.</p><h3>Actions</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ad4f4dbeaf3102a156a4f62dafa3a7e/href">https://medium.com/media/9ad4f4dbeaf3102a156a4f62dafa3a7e/href</a></iframe><h3>Reducer</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b57a056d2efe39631fc05c91df15000/href">https://medium.com/media/5b57a056d2efe39631fc05c91df15000/href</a></iframe><p>In the map component, we want to set the initial state of style in the redux store and watch for any changes to update the map accordingly. To start, we set up a simple map instance and provide it to the MapContext.Provider as a value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/552a973065707fb6ce2450e24ca89156/href">https://medium.com/media/552a973065707fb6ce2450e24ca89156/href</a></iframe><p>Next, we use redux-connect to connect to the store and provide access to actions via props. The only action we need here is setStyle.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe503c90a92d8943b958cdd302444076/href">https://medium.com/media/fe503c90a92d8943b958cdd302444076/href</a></iframe><p>To initialize our stored map style, we call the setStyle action after the map loads and pass in the current style object of the map, which we can retrieve by using the getStyle MapBox API method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d98c1969d70f38c98ca3111e38089925/href">https://medium.com/media/d98c1969d70f38c98ca3111e38089925/href</a></iframe><p>Finally, using the componentDidUpdate lifecycle method the map component can observe any changes to the style in the store and update itself using the <a href="https://docs.mapbox.com/mapbox-gl-js/api/#map#setstyle">setStyle API method</a>. By default this method will intelligently diff the current map style and only apply the necessary updates rather than rebuilding the map from scratch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7beb11d398820c87d0b5d89763d855c/href">https://medium.com/media/a7beb11d398820c87d0b5d89763d855c/href</a></iframe><p>The map style object is now part of the redux store, the map component is listening to changes, and updating the map instance to match. Now we can use our redux actions within our PopulationPanel component to change the map visualization based on user interaction with the UI. Earlier we created a source and layer in PopulationLayers.js. Now, we can define two more layers (MB_STATEPOPULATION and MB_COUNTYPOPULATION), however we only want one of these three layers to be visible at a time. To achieve this, we can wire up a set of radio buttons and use the setViz action to toggle the visibility of each layer.<br><br>First, we set up our internal state in the component constructor to keep track of which radio button is active.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fdb553fa0d138cda82d524091d2e9b8/href">https://medium.com/media/1fdb553fa0d138cda82d524091d2e9b8/href</a></iframe><p>Then we define a set of helper functions for adding and removing the layers when each radio button is selected.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75651eb3fbcfd9cfd1b98c29a1bb7740/href">https://medium.com/media/75651eb3fbcfd9cfd1b98c29a1bb7740/href</a></iframe><p>Next, we call these functions in a new function that we can bind to our radio buttons onChange event.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a528a0b38538e4d9ae2dffc307052435/href">https://medium.com/media/a528a0b38538e4d9ae2dffc307052435/href</a></iframe><p>Leaving us with this final PopulationPanel Component.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6806631dc76e6afa99541ad255ab734c/href">https://medium.com/media/6806631dc76e6afa99541ad255ab734c/href</a></iframe><h3>Conclusion</h3><p>Integrating a modern front-end framework like React with mapping libraries such as MapBox GL JS gives developers the ability to create data rich map visualization applications. Often times, a map centric application can be difficult to manage as the large amount of map related logic causes the main map component to grow to unmaintainable size. Fortunately, we can minimize the growth that occurs in the map component by using the redux store to share and modify the map state, and the React context API to share the map instance across components. This allows us to adopt a component structure that allows individual components to be in charge of their own UI as well as the associated map data, visualizations and interactions.</p><p><em>Resources</em></p><ul><li><a href="https://docs.mapbox.com/mapbox-gl-js/api/">Mapbox GL JS</a></li><li><a href="https://reactjs.org/">React</a></li><li><a href="https://redux.js.org/">Redux</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b50d82bc0ed0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/an-approach-to-integrating-mapboxgl-in-react-redux-b50d82bc0ed0">An approach to integrating MapboxGL in React / Redux.</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenStreetMap Quality Measurement]]></title>
            <link>https://medium.com/critigenopensource/openstreetmap-quality-measurement-7b208ed868b8?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/7b208ed868b8</guid>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Tianyi Ren]]></dc:creator>
            <pubDate>Mon, 25 Nov 2019 23:33:12 GMT</pubDate>
            <atom:updated>2019-11-25T23:33:12.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CuwEuqulAAwe8YfOZA2Uyw.png" /></figure><blockquote>For the second straight year, Critigen’s Open Data &amp; Development team was well represented at the SOTM US conference. In our OSM series, we’ll highlight each topic in detail, reveal our findings, and bring forward important questions to the OSM community.</blockquote><p><strong><em>Authors: Tianyi Ren, Monica Brandeis</em></strong></p><p>As participation and coverage increases, data quality continues to be a key focus area within the OpenStreetMap(OSM) community. Both researchers and communities have adopted various approaches to monitor OSM’s road network data quality through aspects such as data accuracy, consistency, and completeness <em>(Ather 2009)</em>. Rule-based validation is another critical approach that targets both data accuracy and consistency — ultimately aiming to validate data by flagging well-defined mapping errors.</p><p>There are various existing <a href="https://wiki.openstreetmap.org/wiki/Quality_assurance">quality assurance tools</a> for rule-based validation, such as Keep Right, OSMOSE, and Atlas Checks. These tools capture mapping errors based on topology rules and tagging schemas on <a href="https://wiki.openstreetmap.org/wiki/Main_Page">OSM Wikipedia</a>. Atlas Checks in particular focuses heavily on flagging road network mapping errors and fits our needs in evaluating OSM road network quality.</p><h3>Rule-Based validation Using Atlas Checks</h3><p><a href="https://github.com/osmlab/atlas-checks">Atlas Checks</a> is an open source quality assurance framework that flags various mapping errors. Leveraging the <a href="https://github.com/osmlab/atlas">Atlas API</a>, Atlas Checks can convert an OSM PBF file into a data format that efficiently represents OSM as a navigable network consisting of Nodes and Edges. As of today, there are over 40 Atlas Checks available that accurately detect attribution and geometry based errors. Each individual check looks to solve a specific issue with OSM data. See the following examples:</p><ul><li>AddressStreetNameCheck is an attribute based check that targets Nodes that contain street names which do not match the nearby road network.</li><li>BuildingRoadIntersectionCheck is a geometry based check that flags OSM road features that overlap with buildings.</li></ul><p>Atlas Checks can optionally output both GeoJSON and MapRoulette Challenges — empowering users to fix each problematic feature in their favorite editor.</p><h3>Map Quality Measurement (MQM) Tool</h3><p>While current rule-based validation tools generate feature based results to precisely flag mapping errors, fixing all these issues can be time consuming and inefficient. To preserve feature level detail and prioritize initial mapping efforts, there’s a need for a tool to identify priority mapping areas based on the density of the mapping errors.</p><p>In 2018, we developed the <a href="https://github.com/spatialdev/osmquality">MQM (Map Quality Measurement)</a> tool to fill this need. MQM uses Atlas Checks results to generate a vector grid layer that clusters mapping errors in a user defined area. The following steps describe its process. First, it creates a bounding box using a boundary file from a user defined area. It then uses the K-D tree algorithm to repeatedly split this bounding box in half while counting mapping errors within each grid. This continues until the majority of grids contain a relatively small amount of errors. Users can set the termination threshold of the splitting process to be when a certain percentage of grids contain less than a certain amount of errors. For instance, in our project the grid splitting ends when 90% of the grids contain less than 10 mapping errors, leaving the remaining 10% of the grids as error hot-spots. Finally, the hot-spot grids are rendered based on the counts of mapping errors within them, resulting in what we call the ‘MQM layer’. <strong>Figure 1, </strong>below, illustrates the grid generation process for the city of Minneapolis in the United States, and <strong>Figure 2 </strong>shows the MQM layer created using the final grids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7XBtsNeVD0Q89D-Vj8iJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Z3HSA573qMaiFSLP0s_Rw.png" /></figure><h3>The MQM Web App</h3><p>The MQM tool helps better assess map data quality, especially for large datasets. Utilizing the tool, we conducted a case study to measure the OSM road data quality in 51 major U.S. cities (the most populated city in each state), and created a <a href="https://osmquality.io/">web app</a> to host the results. In this web app, users can view mapping error hot-spots in these cities, weight the results by usage level, and check out the cities’ rankings based on their overall OSM road data quality. More details can be found in our <a href="https://2019.stateofthemap.us/program/fri/map-quality-measurement-2-0-a-map-quality-collaboration.html">State of the Map US 2019 conference talk.</a></p><h4>Visualize Mapping Error Hot-Spots Using MQM</h4><p>To retrieve OSM road data for these 51 cities, we pulled the full OSM historic PBF files and filtered the data using city boundary files. We ran the data through 11 Atlas Checks that are focused on road network issues, and used the output GeoJSONs to generate the MQM layer for each city. Finally, we rendered the MQM layers on the web app, overlaying the actual city boundaries for better visualization. The MQM tool also generates fundamental statistics such as total OSM road feature counts, MQM grid size, mapping errors by type (geometry vs. attribute), among others, which we also included in the ‘Quick Stats’ panel on the web app to better assist in the planning of mapping activities. <strong>Figure 3, </strong>below, shows the user interface of the <a href="http://osmquality.io/">MQM app</a>. On the left is the MQM layer for the city of Minneapolis, and on the right is the Quick Stats panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2a6kvKoF3yuPV2tQMcfxGg.png" /></figure><h4>Weight MQM Results by Usage</h4><p>Besides viewing the MQM results directly on the app, users can also explore weighting them by different usage factors. We chose population and car ownership as two factors to represent usage, and derived census-level data from the U.S. Census Bureau’s 2017 ACS 5-year reports. In order to generate results weighted by usage, we first converted the data into raster layers and overlaid them onto the MQM grids layer. We then used zonal statistics to calculate the mean value of the raster cells within each grid, and normalized the values to generate ‘census layers’. Finally, we combined these ‘census layers’ with the MQM layer and assigned the highest weight (0.7 out of 1.0) to the MQM layer. <strong>Figure 4, </strong>below, illustrates the complete workflow. This allows users the ability to select and view the MQM results weighted by population or car ownership within each city. <strong>Figure 5 </strong>provides an example of the MQM results weighted by population in the city of Detroit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Qr5ahmfLGpG0r8SRW7zwbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hROZJUO-41XrFA0gkNxBHg.png" /></figure><h4>Rank Cities Based on OSM Data Quality</h4><p>Using MQM, we generated road mapping error distributions for 51 major U.S. cities, and ranked them based on their overall OSM road data quality. The overall data quality in each city is measured by the percentage of road features that have mapping errors. The lower the percentage rate, the higher the ranking. With OSM data constantly being updated, we plan to conduct this study annually to test the MQM tool and monitor digitization and editing efforts in the OSM community year to year.</p><h3>MQM in 2020</h3><p>We think MQM is a useful tool for campaign coordinators or project managers to strategically plan out mapping activities on OSM. With that in mind, we’re continuing to enhance and refine MQM so it’s more applicable to a broader audience. One of our goals is to allow more types of input data so that other rule-based validation results can be processed by MQM. We’re also actively seeking opportunities to integrate MQM into other mapping tools, such as the HOT Tasking Manager, to assist mapping efforts across the OSM community, and further improving our data quality evaluation process and metrics. Please reach out to us for any comments and questions and stay tuned for MQM in 2020!</p><h4>References</h4><p><em>Ather, A.(2009). A Quality Analysis of OpenStreetMap Data. MEng Thesis, London, University College London.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b208ed868b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/openstreetmap-quality-measurement-7b208ed868b8">OpenStreetMap Quality Measurement</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HOT Summit & State of the Map 2019]]></title>
            <link>https://medium.com/critigenopensource/hot-summit-state-of-the-map-2019-933d4c9e8b92?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/933d4c9e8b92</guid>
            <category><![CDATA[state-of-the-map]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <dc:creator><![CDATA[Stephen Cerqueira]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 20:52:11 GMT</pubDate>
            <atom:updated>2019-10-14T21:25:15.645Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*d83Vm-2atpjDDVvI.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*lVvGlyIWZeBAPiH2folxiw.png" /></figure><p>Critigen’s Open Data &amp; Development (OD&amp;D) were well represented in the small university town of Heidelberg, Germany for back-to-back HOT Summit and State of the Map 2019 global conferences. Five consecutive days of talks and presentations provided a peek into the entire gamut of the OSM community, which is growing with each passing year in both size as well as in the diversity of OSM data manipulation and application. As my colleague Todd <a href="https://medium.com/critigenopensource/state-of-the-map-2018-milan-transit-rocks-europe-is-an-oven-damn-the-mosquitoes-80544929021d">pointed out with last year’s State of the Map conference in Milan</a>, attendance at these world conferences continues to highlight the give-and-take relationship between the passionate small-scale contributors that spearheaded this grassroots organization and the large corporate influencers which are accelerating OSM’s growth on the global scale. As the stage was set for the brightest minds in open source mapping to collaborate, here are our impressions from these two conferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zcC-ytXua3OPQ7FgalQ9rA.png" /></figure><p><strong>The robot mappers are not going anywhere…</strong></p><p>The most obvious takeaway from the conferences is the increasing interjection of artificial intelligence and machine learning into OSM. Commercial community members are using their resources to develop applied automation using AI and ML techniques that maximize efficiency of OSM data creation, error/vandalism detection and validation.</p><ul><li>Microsoft has been working with HOT to train an algorithm that is used to generate new building predictions. By combining Microsoft’s <a href="https://github.com/hotosm/ml-enabler">ml-enabler</a> service with the <a href="https://tasks.hotosm.org/">HOT Tasking Manager</a>, the two are working hand-in-hand to populate new OSM building data for underrepresented communities in Tanzania and Uganda.</li><li>DataPink is also targeting OSM buildings through their deep learning tooling. <a href="https://github.com/datapink/robosat.pink">RoboSat.pink</a> is a tool that allows for building validation and analysis by comparing two building imagery datasets while also training a data model to predict OSM building locations on larger swaths of land using a smaller land subset.</li><li>The most production-ready AI-inspired OSM tool comes from Facebook, whose <a href="https://mapwith.ai/">RapID editor</a> can greatly assist editors by converting satellite imagery into predicted features - in particular road segments. RapID is also integrated with iD Editor and the Tasking Manager, so the possibility of logging into the Tasking Manager to complete tasks where roads and buildings are essentially already drawn for you without a click of your mouse is a tantalizing thought bringing much promise to the development of the OSM map…</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qatRW-xBChuhNUhbtLSFCA.jpeg" /></figure><p><strong>Integrity is everything.</strong></p><p>A subject that the Critigen OD&amp;D team has a keen interest in is improving the methodology and tooling for OSM data integrity. The State of the Map conference introduced a number of interesting tools and processes providing quality assurance and control to OSM data.</p><ul><li>Critigen’s very own Quincy Morgan promoted the huge upgrades to be seen in iD Editor v3, including expanded functionality to filter what types of features a user wants to edit and keeping selections intact through multiple zoom levels.</li><li>We had a chance to speak with the likes of <a href="https://wiki.openstreetmap.org/wiki/User:Mvexel">Martijn van Exel</a> of TomTom and HOT’s <a href="https://www.hotosm.org/people/wille-marcel/">Wille Marcel</a>, maintainers of <a href="https://wiki.openstreetmap.org/wiki/MapRoulette">MapRoulette</a> and <a href="https://wiki.openstreetmap.org/wiki/OSMCha">OSMCha</a> respectively, via small town hall type talks to provide feedback on how to further improve their tools.</li><li>Mapillary is using their extensive infrastructure in photo mapping to train an algorithm that produces street level 3D modeling to validate OSM data. What was even more exciting to see was Mapillary’s tooling being used by a community near and dear to our hearts. The Seattle-based OD&amp;D team was delighted to hear about Mapillary’s partnership with students from the Geography department at the University of Washington on an experiment to validate OSM data using dense Mapillary imagery data. Go dawgs!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*iYZe6HOllpknc0WAKbXokg.png" /></figure><p><strong>Data is only useful if you apply it.</strong></p><p>The potential of OSM data in real world applications is limitless. I firmly believe we have only scratched the surface of what can be done with OSM. Here are some intriguing applications of OSM data presented at the conferences.</p><ul><li><a href="https://www.graphhopper.com/">GraphHopper</a> is a Directions API that markets at efficient route planning and navigability. It boasts a quick, memory efficient Java routing engine that can used for a wide array of spatial analyses: simple routing with elevation considerations, navigability impact of automobiles based on construction closures, analyzing the most optimal roads for cyclists, and the classic “where to put a new …?” spatial analysis: fire station edition.</li><li><a href="https://wiki.openstreetmap.org/wiki/Hikar">Hikar</a> is a small scale augmented reality Android application geared towards helping hikers and walkers alike traverse through trails and similar natural pathways. The application will obtain your location through location and sensor APIs before overlaying OSM data onto what you see before you. The application provides augmented signposts with tips, tricks, and warnings about the trails you are on; the community can now request for signposts to be implemented on POIs seen on their hike/walk.</li><li><a href="https://vrmap.kairo.at/">WebVR</a> is a platform to project OSM data in virtual reality. The implementation of the data in VR is limited at the moment, but there appears to be some hope that this model is scalable. Imagine the implications of having an OSM basemap or having data generated from OSM be the map standard in one of the hottest tech industries…</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLFaC1NpFxlx1rxdYmF2xQ.jpeg" /></figure><p><strong>We’re all in this together!</strong></p><p>Open source mapping through OSM continues to be led by Western countries and corporations. The vast majority of presentations given at the State of the Map conference were by representatives of American or European companies. Heidelberg is the 8th European city to host the State of the Map world conference, of which there have been only 12 (and one of which was held in the United States). What makes this year’s scheduling of conferences interesting is that this is only the second time in the HOT Summit’s young history (this year was only the 5th ever HOT Summit) that the HOT Summit and State of the Map world conference were held back-to-back in the same city (the only other time being Brussels in 2016). The logistics of attending the two conferences at separate times and locations is extremely unfavorable for most, which is unfortunate because HOT Summit bestowed much needed insight into how OSM is being used in communities outside of Europe and the United States.</p><p>There are amazing humanitarian efforts ongoing in local African, Latin American, Pacific Island, and South Asian communities using OSM for a variety of humanitarian efforts: mapathons to map out local communities in preparation for or in the wake of a natural disaster, analyses of current OSM data for social and environmental equality, and in general, promoting underrepresented voices and communities through OSM. Getting this two-for-one opportunity to attend both conferences gives greater context on how we can all use OSM for good despite the differences in our individual circumstances. OSM is set out to create and distribute geographic data for the whole world. Therefore, it is imperative that we <a href="https://2019.stateofthemap.org/">bridge the map</a> to showcase how OSM is truly a global entity. With that being said, we were excited to hear during the closing ceremonies of this year’s conference that next year’s State of the Map world conference will be held for the first time in Africa! We cannot wait to see what has progressed in the OSM realm when Critigen takes on Cape Town, South Africa next year!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VG_jA3eZJTZ3WWGZqVLHyg.jpeg" /><figcaption>Todd, Stephen, Kim, Daniel (left to right)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=933d4c9e8b92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/hot-summit-state-of-the-map-2019-933d4c9e8b92">HOT Summit &amp; State of the Map 2019</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OSM Water: How well are Minnesota’s water features mapped?]]></title>
            <link>https://medium.com/critigenopensource/osm-water-how-well-are-minnesotas-water-features-mapped-7c3c91aef604?source=rss----e73230f14c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/7c3c91aef604</guid>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[openstreetmap]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Matthew Manley]]></dc:creator>
            <pubDate>Wed, 02 Oct 2019 19:38:54 GMT</pubDate>
            <atom:updated>2019-10-02T19:38:53.964Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fIwDXeh96xOSUaRdqUC-_A.jpeg" /></figure><blockquote>For the second straight year, Critigen’s Open Data &amp; Development team was well represented at the SOTM US conference. In our OSM series, we’ll describe our presentations in-depth, reveal our findings, and bring forward important questions to the OSM community.</blockquote><p><strong><em>Authors: Matthew Manley, Alex Sweeney</em></strong></p><h3>What’s going on with natural=water?</h3><p>With so much focus on routing and road features, our team was looking to concentrate on an unexplored corner of OSM. We’ve spent the past few months evaluating the quality of water features in OSM, and wanted to dive deeper into the origin of water data in Minnesota; specifically looking at community-generated data versus externally-sourced bulk imports.<br> <br>Bulk imports have been a contentious issue within the OSM community. In many ways bulk imports conflict with the core ethos of OSM: empowering local mappers to digitize their local communities. Despite this, automated imports are responsible for a substantial amount of data in OSM, and have come from datasets such as: TIGER, Bing Maps’ building footprints, and the National Hydrography Dataset (NHD) among others. Figure 1 illustrates the effect of two key imports in Minnesota.</p><figure><img alt="Figure 1. Effect of bulk imports on highway and water feature counts in MN." src="https://cdn-images-1.medium.com/max/1024/1*45koorQ7l44CZyv3HstBAQ.png" /><figcaption><em>Figure 1. Effect of bulk imports on highway and water feature counts in MN.</em></figcaption></figure><p>Immediately apparent from this graph are large jumps in highway and water feature counts that correspond to TIGER and NHD bulk imports. Together they represent massive feature creation events in Minnesota, contributing lots of data over very short periods. In this way, bulk imports significantly contribute to “map completeness” in OSM. But how active is the OSM community in editing these bulk imported features? And to which degree has the data “moved beyond” the import? Examining how and if these features have changed since ingest can provide insight into how active the OSM community is in maintaining and updating features after bulk imports.</p><p>That is where we decided to focus our study: identifying how OSM water data evolves from the bulk import phase to its current state. To narrow our geographic scope, we used Minnesota as our case study given that SOTM US convened in Minneapolis this year (it also doesn’t hurt that it’s known as the “land of 10,000 lakes”!).</p><h3>Methodology</h3><p><strong>Inland water definition</strong></p><p>In order to define “inland water” we turned to the OSM basemap itself. A <a href="https://github.com/gravitystorm/openstreetmap-carto/blob/master/water.mss">CartoCSS Style Sheet</a> definition controls how features are rendered on OSM. We selected the key value pairs that are symbolized as water and used those to build our inland water definition. You can see the tags we ended up with in the snippet below.</p><pre>landuse=basin,reservoir<br>natural=spring,water,wetland<br>waterway=canal,ditch,drain,stream,river,riverbank<br>wetland=tidalflat,reedbed</pre><p><strong>Time-Slices</strong></p><p>Since we wanted to analyze inland water features over time, we captured three “time-slices” of OSM data derived from the full OSM history PBF file using <a href="https://osmcode.org/osmium-tool/">Osmium Tool</a>. For those that aren’t familiar, Osmium Tool is an awesome command line tool for manipulating and extracting OSM data. The first step in creating these slices was soft-clipping the history file to the Minnesota boundary. We filtered the PBFs to only include features that fit our definition of inland water and generated three time-slices for our analysis:</p><ul><li>Pre-NHD: 2009/09/30</li><li>Post-NHD: 2011/09/30</li><li>Current state: 2019/04/07</li></ul><p><strong>Change Analysis</strong></p><p>In order to find geometric changes of inland water features, we uploaded our PBF time-slices to a PostgreSQL database using <a href="https://gdal.org/programs/ogr2ogr.html">ogr2ogr</a> and created custom queries to classify changes. Since we were primarily interested in what happened to features after NHD imports were completed, we compared the 2011 and 2019 time-slices. We used PostGIS and SQL functions to classify each feature into one of five categories. Figure 2 shows the breakdown in these classifications.</p><figure><img alt="Figure 2. Change classifications." src="https://cdn-images-1.medium.com/max/1024/1*s0geoP2wMq88kevsnoooAg.png" /><figcaption><em>Figure 2. Change classifications.</em></figcaption></figure><p>No Change</p><p>Using unique OSM identifiers (ID), we inner joined features from the 2011 and 2019 time-slices and compared their geometries using the PostGIS function <a href="http://postgis.net/docs/ST_Equals.html">ST_Equals</a>. If the geometries didn’t change between time-slices we categorized it as <strong><em>no change.</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3272cf68d17789bea6804dbe320947f9/href">https://medium.com/media/3272cf68d17789bea6804dbe320947f9/href</a></iframe><p>Change</p><p>Conversely, to isolate features that<strong> did</strong> change we used an inner join to find everything excluded from the no change category; creating a mutually exclusive set. From here we bucketed features into five different categories.</p><p><em>i. Change in Geometry</em></p><p>To find features with a <strong>change in geometry</strong> we isolated those with the same OSM ID in 2011 and 2019 but with differing geometries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7a40cb4d37c39d523f75001e4cb9ce3/href">https://medium.com/media/e7a40cb4d37c39d523f75001e4cb9ce3/href</a></iframe><p><em>ii. Replaced</em></p><p>If an OSM ID wasn’t found in both time-slices, we checked to see if there was a spatial representation for that feature in 2011 and 2019 but having differing OSM IDs. We accomplished this by using a combination of the PostGIS functions: <a href="https://postgis.net/docs/ST_Intersects.html">ST_Intersects</a> and <a href="https://postgis.net/docs/ST_Touches.html">ST_Touches</a>, to find coincident features that overlapped each other. For example, if an OSM ID was only present in 2011 and intersected but didn’t just touch a feature in 2019, we determined that feature to be <strong>replaced</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fba79e299d1c2f0d036362f33076b0e/href">https://medium.com/media/5fba79e299d1c2f0d036362f33076b0e/href</a></iframe><p><em>iii. Deleted</em></p><p>We identified <strong>deletions</strong> by isolating OSM IDs that existed only in 2011 and weren’t replaced by a feature in 2019.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27b9d4c1598fff7fbb5331a99fef44b6/href">https://medium.com/media/27b9d4c1598fff7fbb5331a99fef44b6/href</a></iframe><p><em>iv. Added</em></p><p>Similarly, we found <strong>additions</strong> by isolating OSM IDs that existed only in 2019 and were not replacements for features in 2011.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a690052d4dfa13365df4c703ab26589/href">https://medium.com/media/4a690052d4dfa13365df4c703ab26589/href</a></iframe><h3>Findings</h3><p><strong>The State of Water in Minnesota</strong></p><p>In terms of the composition of OSM water data in Minnesota, we found that the overwhelming majority of data was imported from NHD: 84% of polygons and 98% of lines. This speaks to the importance of bulk imports more generally. If NHD hadn’t been imported, it’s unlikely that similar amounts of data could be generated by the community in such a short amount of time.</p><p><strong>Identifying Change In Bulk Imports</strong></p><p>By leveraging inner joins and PostGIS functions, we found that only 15% of features imported from NHD had <strong>changed</strong> between 2011 and 2019. The remaining 85% of features were left unchanged. We successfully identified four different types of change: changes in geometry, replacements, deletions, and additions. Figure 4 illustrates this breakdown. In examining our results we found several examples of changes to NHD data that added to the overall accuracy of the map. We isolated six examples of this and presented them in a slider visualization <a href="https://water.osmquality.io/">here</a>. These cases illustrate the OSM community’s contribution to validating and editing bulk imported data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*zaGG14zsMW0j6VVoSgDXpw.png" /><figcaption><em>Figure 4. Changes in NHD water features between 2011 and 2019</em></figcaption></figure><h3>Conclusions &amp; Next Steps</h3><p>Bulk imports can be a huge asset for the OSM community, as long as they‘re done in a careful and mindful manner. The community should be ready to address any inaccuracies that are introduced into OSM via the import of external data. The examples we presented in our slider visualization indicate that users are indeed interacting with and improving bulk-imported NHD data. Continuing to encourage this type of editing would be beneficial to the community as a whole as these edits are in themselves a form of validation.<br> <br>However, we were intrigued by the low rate of change for the bulk-imported features from 2011–2019. This may suggest that even though bulk imports can drastically improve map completeness, the rate at which the community interacts with these features may be quite low. What we weren’t able to answer was — “Why?”. Here are three possibilities:</p><ul><li>Perhaps the imported data is accurate and doesn’t require editing. This may be the case for some features, but we identified some inaccuracies within our analysis.</li><li>Another possibility is that bulk imports truly do decrease editing rates due to a psychological component: if users see a relatively complete map they <em>may</em> be discouraged from editing.</li><li>Or a simpler explanation: the rates of editing water features may be low in general. This would need to be examined in contexts beyond Minnesota, but is nevertheless a possibility. One of the reasons we chose to analyze water was due to our perception that water features are often forgotten.</li></ul><p>Along with answering the above question of why editing rates are low, there are many other interesting directions to pursue in the future. For one, we’d like to see how updated NHD data stacks up to OSM. Just how much has NHD changed since import and is there an opportunity for refreshing the data? We also concede that this analysis doesn’t take into account changes in tags, and future work should integrate a tagging time-series analysis.</p><h3>Community Feedback</h3><p>The community feedback we received was positive and a lively Q&amp;A session followed our presentation at SOTM US. A couple of themes emerged from this discussion and primarily centered on how replacements were classified, how NHD features were mapped to OSM tags, and the feedback validation loop between external data sources and OSM. For more insight into our analyses take a look at our <a href="https://github.com/spatialdev/mn-water-2019">github repo</a> for the SQL behind it. As for NHD features and how they were mapped to OSM tags, <a href="https://wiki.openstreetmap.org/wiki/National_Hydrography_Dataset">this resource</a> describes which key pairs were used upon ingest.</p><p>We believe that the feedback loop between external data sources and the OSM community is a compelling topic for further investigation.</p><blockquote>Can we establish a communication channel between third party resources and OSM, and if so, how? What’s the best way to combine rich datasets and an active community?</blockquote><p>Up next, we’ll look to explore the differences between this year&#39;s NHD data refresh and what’s live in the OSM database.</p><p><em>Stay tuned &amp; follow Critigen’s Unearth publication for the second part of our OSM series. We’re also live on twitter </em><a href="https://twitter.com/osmquality"><em>@osmquality</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c3c91aef604" width="1" height="1" alt=""><hr><p><a href="https://medium.com/critigenopensource/osm-water-how-well-are-minnesotas-water-features-mapped-7c3c91aef604">OSM Water: How well are Minnesota’s water features mapped?</a> was originally published in <a href="https://medium.com/critigenopensource">Unearth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>