<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohammed Ayub on Medium]]></title>
        <description><![CDATA[Stories by Mohammed Ayub on Medium]]></description>
        <link>https://medium.com/@mohammedayub44?source=rss-a46b66a846fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MCizYgDHj9CtRj-x4L1Rsg.jpeg</url>
            <title>Stories by Mohammed Ayub on Medium</title>
            <link>https://medium.com/@mohammedayub44?source=rss-a46b66a846fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:14:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohammedayub44/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Batch Geo-coding with Pelias…]]></title>
            <link>https://medium.com/@mohammedayub44/batch-geo-coding-with-pelias-5b920734922f?source=rss-a46b66a846fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b920734922f</guid>
            <category><![CDATA[geocoding]]></category>
            <category><![CDATA[pelias]]></category>
            <category><![CDATA[nfirs]]></category>
            <category><![CDATA[location-intelligence]]></category>
            <dc:creator><![CDATA[Mohammed Ayub]]></dc:creator>
            <pubDate>Sun, 13 Oct 2019 19:44:34 GMT</pubDate>
            <atom:updated>2019-10-13T19:44:34.179Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*bxgsdkJvj2u5mp-Hpg7KlA.gif" /></figure><p>In this blog post, let’s talk about Batch Geo-coding big data sets using a Pelias Instance setup for North America region and mapping the results. This is a follow up post to my first one which goes through the <a href="https://medium.com/@mohammedayub44/yet-another-awesome-geo-coder-open-sourced-spatial-search-c295970a8226">entire Pelias setup process</a>.</p><p>Most of the time, people search for one address at a time and that works great with a Front-End User Interface and a search bar (very much like what Google/Bing/Uber Maps have). However, sometimes you run into use cases where you download a file with addresses from an open source site or your colleague sends you a customer data file with addresses which need to be mapped. Having such functionality that can take these spatial address files, Geo-code and then export them in a usable format (like CSV/TXT to create maps) becomes critical especially if you’re doing like 20 million addresses at a time (Yes 20 million ! not kidding).</p><h3>Batching :-</h3><p>First let’s check if your Pelias services are up and running on the instance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yhaa0sNKoorIErI5EiRB7A.png" /></figure><p>We can see that search &amp; reverse search are running at localhost:4000</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*wbMvnTaijHMatmKGYrEQqg.jpeg" /></figure><p>Secondly, following this <a href="https://www.dropbox.com/s/ymvu63ijk12wx3y/test_pelias2.py?dl=0">python script</a> which will help you upload a file, get lat-long and output the results into a folder. It will create a results folder which contains your output files grouped by state.</p><p>Batch Geo-coding is as simple as running the above script by passing below arguments :</p><ul><li>Input File (CSV Preferred) Containing the Addresses (-s)</li><li>Name of Output File (-o)</li><li>(Optional) Name of Full Address column, if exists (-f)</li></ul><p>Let’s break down the script to see what’s happening –</p><ul><li>It reads the CSV file in chunks (thanks to pandas chunksize parameter)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*A52ABqgga9-cETkjDZQyMw.png" /></figure><ul><li>Throws errors if provide full address column or other address columns are not present, otherwise, if check pass then concatenates all address fields into one combined column.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6iBYA5bLvjCQMJGUyXTEBg.png" /></figure><ul><li>For each chunk, group by State and then query the Pelias search API for each address.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*-QfPYuZWsg49LK8pcttsXQ.png" /></figure><ul><li>To store the results, we initialize all returned columns from Pelias as lists, I did this because it’s easier to append them to a Data Frame and then convert to CSV.</li><li>Parse &amp; Append returned matched address components into respective variables (had to do checks on each variable, as for some bad addresses with some address elements missing there were no matched results.)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbsRvbgR-Uy-Ob_y5GmIlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*P2X6yBmnjmeXAJL_VaPrJA.png" /></figure><ul><li>Write the processed file/chunk to disk as CSV.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*Tfwq_C0EXBMEUw2THoCPaQ.png" /></figure><p><strong>Run Time Stats –</strong></p><p>Using general purpose AWS instance of m5a.2xlarge with 8CPUs gives</p><p>1 Million Addresses — ~2–3 hrs.</p><p>25 Million Addresses — ~48–56 hrs.</p><h3><strong>Mapping the Results :-</strong></h3><p>Data set I used here is related to <a href="https://www.usfa.fema.gov/data/nfirs/">Fire Incidents across the United States</a> which have lot of variations in their address quality. Interesting bit on this is very few people in US have visualized such data on a National Scale to find trends. Geo-coding such data sets open ups lot of new arenas for data modernization and exploration techniques.</p><p>Below are some basic maps that I pulled together after Geo-coding about 25 million incidents for the year 2016.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTGmdR5yJV4rq0MYh94HMw.png" /><figcaption>Figure 1 — Heat Map of Incident Distribution in Massachusetts</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xLABMAJpfJc-7osocOqqiQ.png" /><figcaption>Figure 2 — Heat Map of Incident Distribution in Kansas</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*hTBjEFA_FsKO3VfgxqdFlg.png" /><figcaption>Figure 3 — % Distribution of Incident Count by State</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*ST4XAEzVWHhfZabwLOXMEg.png" /><figcaption>Figure 4 — Word Cloud of Top Cities (Size = Count of Fire Incidents)</figcaption></figure><blockquote><strong>Note</strong>: There are a lot of caveats around with making sense of these results. Like Geo-coding errors which are likely to happen due to the quality of the free form addresses fields. Also, some fire departments might report more incidents than others just because of their diligent data practices and reporting resources available.</blockquote><p>Some helpful Links to Understand Fire Data — <a href="https://iafcevents.webex.com/ec3300/eventcenter/recording/recordAction.do?siteurl=iafcevents&amp;theAction=poprecord&amp;path=pop_program_info&amp;recordID=13744442&amp;internalRecordTicket=4832534b0000000465543e7b13ec18cf2f4da5cc677cff9c58bcf8c23028139c1f66c3361ffae785">Fire Data in a serious State</a> by Sara Wood</p><blockquote>Common Errors you might run into –</blockquote><p>1) Incorrect CSV format — you might have to change the params of the read_csv to adjust to your CSV file.</p><p>2) Memory Error — change the value of your chunksize to a smaller one or increasing your instance memory.</p><p>(Stay tuned for more updates…)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b920734922f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yet Another Awesome Geo-coder — Open Sourced Spatial Search]]></title>
            <link>https://medium.com/@mohammedayub44/yet-another-awesome-geo-coder-open-sourced-spatial-search-c295970a8226?source=rss-a46b66a846fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/c295970a8226</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[pelias]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[geocoding]]></category>
            <dc:creator><![CDATA[Mohammed Ayub]]></dc:creator>
            <pubDate>Wed, 26 Jun 2019 21:22:07 GMT</pubDate>
            <atom:updated>2019-09-25T15:07:32.956Z</atom:updated>
            <content:encoded><![CDATA[<h3>Yet Another Awesome Geo-coder — Open Sourced Spatial Search</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UEG5GwJRPeBwxN87W86GGw.png" /></figure><p>What if you collect very large amounts of spatial information such as millions of addresses annual that should be used for geographic analysis but are skeptical of using commercial geo-coders (like Google, HERE, ESRI etc.) or are discouraged by the slow speed of many open source like PostGIS TIGER Geo-coder. There’s a good, if still imperfect, alternative to these options in the works called as “<strong>Pelias</strong>”. Initially started by some amazing people from <a href="https://www.mapzen.com/">Mapzen</a> before that company folded, Pelias now a part of <a href="https://www.linuxfoundation.org/press-release/2019/01/mapzen-open-source-data-and-software-for-real-time-mapping-applications-to-become-a-linux-foundation-project/">Linux Foundation Project</a>.</p><p>Geo-coding at its roots is searching through known addresses or geographies (called ‘shapefiles’, in geeky terms) to find the best match (a Latitude/ Longitude) for a given input address. Pelias makes this possible by indexing globally shared spatial data from various sources into a database called <strong>Elastisearch</strong> which basically allows for spatial search on steroids !!</p><p>You might have come across previous medium posts from Postzen team about <a href="https://medium.com/postzen/setting-up-a-pelias-instance-1dbc740b0b69">setting up Pelias Instance</a> and the <a href="https://medium.com/postzen/hosted-services-are-hard-and-so-can-you-1420c7037c20">progress the project</a> has made since its inception in early 2015 (or before). If your wondering why write another post, there are couple of reasons. Firstly — the entire platform can now be run on top of docker containers that involves much simpler steps to setup than before. The Docker approach raises a host of new questions, and this blog will hopefully address these issues and provide some solutions. It’s a bit of a lengthy read, but it will be worth your time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*0blHMgiDRquKXUIeKF9D4w.jpeg" /></figure><p>The easiest (and the recommended) way to setup Pelias currently is through Docker containers. In early 2017, Pelias decided to break up its API modules into different micro-services and multiple GitHub repositories for different data import pipelines, which makes it look like a daunting task to understand and set one up. By using the new Docker setup you don’t have to worry about most of the gory details of setting-up each micro-service individually or running into serious issues while downloading data for specific regions of interest.</p><blockquote>Getting the prerequisites out of the way !!!</blockquote><p><strong>System Requirements:</strong><br><strong>1. Spin up the EC2 instance —</strong> <br>Since Pelias Geo-coder supports from multiple countries, the specs here might vary on how much data you need to download and what region you are interested in. (At-least 8 GB and 300 GB storage recommended).<br>I have used this AMI (ami-0653e888ec96eab9b) Ubuntu 16.04 /SSD Volume Type with m5a.xlarge and 300GB of storage. (Scale Up/Down as necessary)</p><p><strong>(Optional)</strong> Creating a Cloud9 Environment for development — You don’t necessarily require this section. As an alternative you can just SSH from any available client (such as Putty etc.)</p><p>Install NodeJs on your created instance for Cloud9 — <a href="https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/">https://tecadmin.net/install-latest-nodejs-npm-on-ubuntu/</a>.</p><blockquote>Install NodeJS on Instance</blockquote><pre>curl -sL <a href="https://deb.nodesource.com/setup_10.x">https://deb.nodesource.com/setup_10.x</a> | sudo -E bash -<br>sudo apt-get install nodejs<br>node -v<br>npm -v<br>which node <br>#Copy this path for later (usually it’s /usr/bin/node)<br>sudo mkdir cloud9 <br>#Optionally Set the working directory for Cloud9 (Default would be /home/ubuntu if not set)</pre><p>In your AWS Management Console, search for Cloud9</p><ul><li>Select Create Environment</li><li>Enter name and description</li><li>Select Connect to remote server (SSH) option.</li><li>Enter Username and Host Name of instance created in previous step.</li><li>Copy the Public SSH key into the Ubuntu Instance.</li><li>Add the NodeJS binary path which we copied earlier.</li><li>Click Next and Continue. It should open up new Cloud9 tab which will ask you to install a few requirements before you start using it. It will take some time to install the Cloud9 IDE in your environment.</li><li>When it prompts “Installation Completed” Click Finish.<br> You should be presented now with the Welcome Tab.</li></ul><p><strong>2. Install docker and docker-compose </strong><br>(There is always a DigitalOcean Blog for these kind of things, however below steps worked for me)</p><blockquote>Install Docker and Docker Compose</blockquote><pre>curl -fsSL <a href="https://download.docker.com/linux/ubuntu/gpg">https://download.docker.com/linux/ubuntu/gpg</a> | sudo apt-key add -<br>sudo add-apt-repository “deb [arch=amd64] <a href="https://download.docker.com/linux/ubuntu">https://download.docker.com/linux/ubuntu</a> $(lsb_release -cs) stable”<br>sudo apt-get update<br>apt-cache policy docker-ce<br>sudo apt-get install -y docker-ce<br>sudo systemctl status docker<br>sudo docker info<br>sudo docker run hello-world<br>sudo curl -L <a href="https://github.com/docker/compose/releases/download/1.18.0/docker-compose-%60uname">https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname</a> -s`-`uname -m` -o /usr/local/bin/docker-compose<br>sudo chmod +x /usr/local/bin/docker-compose<br>docker-compose — version</pre><p>— — — This brings us to the end of Prerequisites for Pelias — — —</p><blockquote>Running the Automated Build…</blockquote><p>Now we will step through the automated build provided by the Pelias team. Instead of running all commands at once we will step through the download,prepare and import steps separately for each data source to see what is happening at each step. You can also run the download, prepare and import step for all data sources in parallel using pelias download all, pelias prepare all, pelias import all statements.</p><p>Will be setting this up for Portland Metro Area. However, the steps remain same for other example projects provide. The DATA_DIR in .env file and the pelias.json data source attributes change accordingly.</p><blockquote>Create directories for the code and data files</blockquote><pre>sudo mkdir /code /data<br>sudo chown 1000:1000 /code /data</pre><blockquote>Clone the repository</blockquote><pre>cd /code<br>git clone <a href="https://github.com/pelias/docker.git">https://github.com/pelias/docker.git</a><br>cd docker<br>sudo ln -s “$(pwd)/pelias” /usr/local/bin/pelias<br>which pelias</pre><p>Every time you run Pelias CLI command, it will look for .env file and a couple of variables to see if they are set. You will see this file in each of the example project folders. <em>Be sure to navigate to the respective folder to run the Pelias CLI commands.</em></p><pre># Change working directory<br>cd projects/portland-metro</pre><pre># Configure environment<br>sed -i ‘/DATA_DIR/d’ .env<br>echo ‘DATA_DIR=/data’ &gt;&gt; .env</pre><pre># Check all the environment variables are set correctly<br>sudo pelias system env</pre><p>Above commands are part of the <a href="https://github.com/pelias/docker#generic-build-workflow">helper script</a> the team has created to manage all Pelias docker images (similar to docker style commands).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*JE-4VwqwGfTJQKQAIsTw6A.png" /></figure><p><strong>Step 1. Pull down all the required images</strong></p><blockquote>Pull all Pelias Images</blockquote><pre>sudo docker image ls <br>sudo pelias compose pull</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*GHxnZqjDduUwG8mM6M0Vug.png" /><figcaption>Before running the pull command</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*CSE616lSgG2xvf22RcKMAQ.png" /><figcaption>After running the pull command</figcaption></figure><p>As you can see it has pulled down images for the following -<br><strong>Data Sources</strong> — <em>openstreetmap, whosonfirst, openaddresses, polylines, transit</em><br><strong>CSV Importer</strong> (eg. import Geonames CSV files) — <em>csv-importer</em><br><strong>Elasticsearch Database </strong>— <em>elasticsearch </em><br><strong>Performance and Accuracy settings for Pelias </strong>— <em>schema</em> (from the code it looks like it contains tuning parameters for accuracy like how to traverse the geographical hierarchy etc.)<br><strong>TravisCI based Statistical Machine Translation Test Suite for Pelias system </strong>— <em>fuzzy-tester</em><br><strong>Point in Polygon Service based on WhosOnFirst data </strong>— <em>pip-service</em><br><strong>Location Approximation Module </strong>(for non-exact address matches) — <em>interpolation</em><br><strong>NLP Parser to detect geographic based input texts </strong>— <em>placeholder</em><br><strong>Address Pre-processing module </strong>— <em>libpostal-service</em><br><strong>API Module </strong>to run HTTP requests like search,reverse,autocomplete operations — <em>api</em></p><blockquote><strong>Note on Pelias.json</strong> → All the downloads and imports of data into Elasticsearch is driven by this JSON file. You should see this in every project folder. All the images that were downloaded in the previous step have certain configurations that are set in this JSON file. Like Index setting for Elasticsearch, API Endpoint URL’s, Data source URL’s and path where to store them etc.</blockquote><p><strong>Step 2: Start the Elasticsearch database service (sequence is important)</strong></p><blockquote>Start the Elasticsearch Service</blockquote><pre>sudo pelias elastic start<br>sudo pelias compose ps<br>sudo pelias elastic wait<br>sudo pelias elastic create</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*wGCAHVRtA-p7KMXTtD3zfQ.png" /><figcaption>Starting Elasticsearch Service</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*YFQAXxpUN_6qtoW1kspiqQ.png" /><figcaption>Creating pelias index to add documents</figcaption></figure><p><strong>Step 3: Download the required data from each source (sequence is important)</strong></p><blockquote>Download WhosonFirst DATA</blockquote><pre>sudo pelias download wof</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*M57lo0WnHqYGT8-p_5GuMw.png" /><figcaption>Config for WOF Download in pelias.json</figcaption></figure><p>Looking at the JSON file you can see we are downloading two places with ID’s ‘85688513’ and ‘85688623’. You can use this <a href="https://spelunker.whosonfirst.org/">Spelunker </a>tool to get more place ID’s you want to download the data from. You can use the search bar on the top right or just add ‘/id/&lt;numeric_id&gt;’ in the URL to get the details. We are storing all data in ‘/data/whosonfirst’ location. See all available JSON options <a href="https://github.com/pelias/whosonfirst#configuration">here</a>.<strong> </strong>This creates a SQLite database to store all the data. <strong>(Optional) </strong>You can install sqlite3 and query the databases to verify the data downloaded.</p><blockquote>(optional) Install sqlite3</blockquote><pre>sudo apt install sqlite3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*ziUChA18ap1EmtUCwcPfww.png" /><figcaption>Querying the WOF Downloaded Data</figcaption></figure><blockquote>Download OpenAddresses DATA</blockquote><pre>sudo pelias download oa</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*gToijOSy0KEpwpCWKWmwiQ.png" /><figcaption>Config for OA Downloads in pelias.json</figcaption></figure><p>You can see we are downloading 7 csv files and storing them in location ‘/data/openaddresses’. Right now its bit of a painful process of figuring out the right csv’s for diffirent regions. Thankfully the team has made available the csv’s for custom regions already. See all CSV File Listing <a href="http://results.openaddresses.io/?runs=all#runs">here</a>. See all possible configurations <a href="https://github.com/pelias/openaddresses#configuration">here</a>.</p><p>How OpenAddresses talks to WhosOnFirst data to get more info for each record using Admin Lookup service (like region, neighborhood, continent etc.) read about it <a href="https://github.com/pelias/openaddresses#admin-lookup">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tdAvx2NLT3CxnaouV1KEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7c5pUy7SgKO3tDI6-9pPw.png" /><figcaption>Downloaded files</figcaption></figure><blockquote>Download Open Street Map (OSM) DATA</blockquote><pre>sudo pelias download osm</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*eWOYoa86xjKn3ugqOKjBZg.png" /><figcaption>Config for OSM Downloads in pelias.json</figcaption></figure><p>There are two sources of getting OSM Data <a href="https://planet.openstreetmap.org/">OSM Apache Server</a> or <a href="https://download.geofabrik.de/">EU Server</a> . As you can see this config takes ‘.pbf’ files. If you run into issues downloading huge .pbf files (like the full planet build or North America build, either increase the hardware specs or download the state-wise .pbf files). See all possible configurations <a href="https://github.com/pelias/openstreetmap#configuration-settings">here</a>.</p><p>How OpenStreetMap talks to WhosOnFirst data to get more info for each record using Admin Lookup service (like region, neighborhood, continent etc.) read about it <a href="https://github.com/pelias/openstreetmap#administrative-hierarchy-lookup">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kmz9Kw_ttoF4E27W9yIHtQ.png" /><figcaption>OSM Download</figcaption></figure><blockquote>Download TIGER Data</blockquote><pre>sudo pelias download tiger</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*iXz9MaISJZv8NZPNGyNMLQ.png" /><figcaption>Config for TIGER Downloads in pelias.json</figcaption></figure><p>Tiger data is used for interpolation service. Mainly for US region, as specified we are downloading 7 US county files which contain address block ranges for interpolation. [<strong>Note: If no state values are specified then entire US address feature files are downloaded</strong>]. It downloads the zip files and extracts them as shape-files into separate folders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*Hzs5i21jBT8F12VnF8dKKQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zmL4yEX-Qzf-TEJ2XjzVQ.png" /></figure><blockquote>Download Transit Data</blockquote><pre>sudo pelias download transit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*X9LZnsEG5DottBvOBWlnxw.png" /></figure><p>It downloads common landmarks,stops and street intersections. Its just a bunch of CSV files made available from different sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5LdcIbZIllHpLjYCCAGcEw.png" /></figure><p><strong>Step 4: Prepare the downloaded data to be ingested into Elasticsearch (sequence is important)</strong></p><blockquote>Prepare Polyline data</blockquote><pre>pelias prepare polylines</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*dL9FSPDhkxPpDNT-neuy0A.png" /></figure><p>This will check for atleast one .pbf files in your ‘/data/openstreetmap’ and extract road network from it into polyline format. If you face issues preparing huge PBF files into polyline format, use <a href="https://github.com/pelias/polylines/wiki/Generating-polylines-from-Valhalla">Valhalla</a> to do so as recommended. <br>See all available configurations <a href="https://github.com/pelias/polylines#configuration">here</a>.</p><p>How Polylines talks to WhosOnFirst data to get more info for each record using Admin Lookup service (like region, neighborhood, continent etc.) read about it <a href="https://github.com/pelias/polylines#administrative-hierarchy-lookup">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*bTaexUfZM5POldaH09qMhA.png" /></figure><p>You might see info messages like ‘found 0 refs for way xxxxxx’. These are not error messages and you can safely ignore <a href="https://github.com/pelias/openstreetmap/issues/491">them</a>.</p><blockquote>Prepare Interpolation</blockquote><pre>pelias prepare interpolation</pre><p>It prepares data available from Polyline Extract, OA, OSM, Tiger into two sqlite databases. One called street.db and other as address.db. Read more about the workflow process <a href="https://github.com/pelias/interpolation#workflow">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*CxElQIf2nDdBlGwBHoHlDA.png" /></figure><blockquote>Prepare Placeholder</blockquote><pre>pelias prepare placeholder</pre><p>The following best describes it —</p><blockquote>“Placeholder supports searching on and retrieving tokens in different languages and also offers support for synonyms and abbreviations.<br>The engine includes a rudimentary language detection algorithm which attempts to detect right-to-left languages and languages which write their addresses in major-to-minor format. It will then reverse the tokens to re-order them in to minor-to-major ordering.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*Qse2B2hWfpi_Rm-XdWnNdw.png" /></figure><p>It extracts all the GeoJSON’s present in WhosOnFirst data folders as wof.extract file and then loads them as sqlite database record.</p><p><strong>Step 5: Start the import into Elasticsearch (sequence is important)</strong></p><blockquote>Import WOF Data</blockquote><pre>sudo pelias import wof</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*uhlcHkGkZViUWTV3vdrTVg.png" /></figure><p>You might get some connection errors on Elasticsearch bulk parse. It will retry to parse records and finish import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2jioKz-VQL7XT-DhqzIRPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*VkvzB0Ezfa3VyO2WyhtaFQ.png" /></figure><blockquote>Import OpenAddresses Data</blockquote><pre>sudo pelias import oa</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7mB9BIWGLyqH-OVhgqonA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R9g4-1QJTC4cCifONMGT0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*QNtEW5Q6XLHZMg3IA8YwTw.png" /></figure><p>As you can see it uses the ‘wof-admin-lookup’ and ‘wof-pip-service:master’ in the background</p><blockquote>Import OpenStreetMap Data</blockquote><pre>sudo pelias import osm</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKeHI25bQVdW43MeCYdo-w.png" /></figure><blockquote>Import Polyline Data</blockquote><pre>sudo pelias import polylines</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfH-bhP5t8eq1GZMsT5bCw.png" /></figure><blockquote>Import Transit Data</blockquote><pre>sudo pelias import transit</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBnIhT6kOAiDBBl_dYFlZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*eq4TXPP2LkztSg-IwZ4kug.png" /></figure><p><strong>Step 6: Spin up the API services</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*zaE683oleMk1ZdMrcZVOeg.png" /><figcaption>Config for API Service Endpoints in pelias.json</figcaption></figure><pre>sudo pelias compose up<br>sudo pelias compose ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*QP7R5yhr7crXIp2XabZhJA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*toIOr7HQP1JT8ZjHv2_jFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47reG1u2g9nZiHP14J30mg.png" /><figcaption>docker ps output</figcaption></figure><p><strong>Step 7: Run the tests (optional)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*7RVdYQ8s15QM62d2yTuLxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*QQI8dYmxzrbdCGEGyG-SrQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*UfcD_JzIYxfRxaTxSlZr9w.png" /></figure><p>90% of the test run fine so we are okay. Please see the test run strategy deployed by the team <a href="https://github.com/pelias/fuzzy-tester">Here </a>and <a href="https://github.com/pelias/acceptance-tests/issues/109">Here</a>.</p><p><strong>Step 8: Running some examples to test the API’s locally</strong><br>To learn more about which Endpoint to use for different tasks, refer to<a href="https://github.com/pelias/documentation/blob/master/search-workflows.md"> this page.</a></p><p>Here are some examples I tried:</p><blockquote>Search Endpoint</blockquote><p>curl -G ‘http://localhost:4000/v1/search’ --data-urlencode “text=1901 Main St”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ni2iOJvpM53CvVHyTlz58A.png" /></figure><blockquote>Reverse Geo-Coding Endpoint</blockquote><p>curl -G ‘http://localhost:4000/v1/reverse’ --data-urlencode “point.lat=45.433467 point.lon=-122.650095”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vdvw3gtDayQ2PwM_wSh6qw.png" /></figure><blockquote>Point-in-Polygon Service Endpoint</blockquote><p>curl http://localhost:4200/-122.650095/45.533467</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oudUhr2q6-4PqmDMVlzLzw.png" /></figure><p>Hoping this was helpful. Happy Geo-coding !</p><p>Feel free to post any comments,questions, concerns and emotional outbursts.</p><blockquote>(Coming Up Next …) Running Batch Geo-coding using Pelias</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c295970a8226" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>