<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Uli Strötz on Medium]]></title>
        <description><![CDATA[Stories by Uli Strötz on Medium]]></description>
        <link>https://medium.com/@ustroetz?source=rss-92c5417f7d67------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*HrfsBBvWhxjFOcXh.jpeg</url>
            <title>Stories by Uli Strötz on Medium</title>
            <link>https://medium.com/@ustroetz?source=rss-92c5417f7d67------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:38:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ustroetz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Orchestrating scheduled data batch jobs on AWS]]></title>
            <link>https://medium.com/tech-blog-door2door/orchestrating-scheduled-data-batch-jobs-on-aws-ee45f940696f?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee45f940696f</guid>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[aws-batch]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Tue, 11 Apr 2017 16:47:21 GMT</pubDate>
            <atom:updated>2017-04-19T12:07:23.351Z</atom:updated>
            <content:encoded><![CDATA[<p>For one of our latest products, we are analyzing various public transport data and combining it with movement data from different sources. It is a classical <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL process</a>, which, in our case, consists of three batch jobs: Analyzer, Aggregator, and Deployer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkitTvHPHDMYXnJ2kghBfQ.png" /></figure><p>The problem we are trying to solve can be summarized in these two requirements:</p><ul><li>Orchestrate the three batch jobs in a way that their interdependencies are respected</li><li>Schedule the three batch jobs to run once a day</li></ul><p>The following describes our approach to the problem.</p><p>AWS offers a variety of options to create such a data processing pipeline. Our first idea was to create three AWS Lambda functions and orchestrate them with <a href="https://aws.amazon.com/step-functions/">AWS Step Functions</a>. AWS Step Functions is super useful to visualise your processing steps and orchestrate them accordingly. The major downside was the extensively discussed 5 minute timeout limit of Lambda.</p><p>We ended up using the new <a href="https://aws.amazon.com/batch/">AWS Batch</a> to run our three applications. AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. We pushed all three applications to a repository on EC2 container service, set up an AWS Batch compute environment and job queue for the entire project, and created a job definition for the three batch jobs. Now we are ready to execute all three of them from the AWS web interface or CLI. The big advantage of AWS Batch is, that it lets you specify job dependencies based on the job ID. This solves our first requirement.</p><p>The last missing piece was to schedule them to automatically run once a day. We thought AWS would handle this for us too and we didn’t have to worry about. The product is advertised as:</p><blockquote>AWS Batch plans, schedules, and executes your batch computing workloads …</blockquote><p>So we expected there was a configurable cron to schedule the jobs. But there we couldn’t find a way to schedule the Batch jobs. The only information the docs provide is this short paragraph describing job scheduling:</p><p><a href="http://docs.aws.amazon.com/batch/latest/userguide/job_scheduling.html">Job Scheduling - AWS Batch</a></p><p>Reaching out to the AWS Support center to ask how to run these scheduled jobs only gave us the answer that we can setup a Lambda function to schedule and orchestrate them. We wanted to avoid having another piece of code to maintain. Nevertheless, this was the only feasible option to make them work together. So we tried to keep it as simple as possible in a basic Python script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/157b6b4982f8815f0fe7931ff7570897/href">https://medium.com/media/157b6b4982f8815f0fe7931ff7570897/href</a></iframe><p>Lambda offers very simple scheduled Events (via Cloudwatch). That’s the feature we were expecting out of AWS Batch to make it a complete service. Nevertheless, the simple feature to specify a depending job when submitting a new one makes the service quite useful for our use case. We are looking forward to more features and more regions for AWS Batch.</p><p>The complete architecture to orchestrate our scheduled data batch jobs on a daily basis with Lambda is illustrated here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ekFen9vGNkvvWNulQhWFZA.png" /></figure><p>The above setup fulfills our two requirements. The jobs run sequentially by specifying job dependencies in AWS Batch. AWS Lambda allows us to trigger the process once a day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee45f940696f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-blog-door2door/orchestrating-scheduled-data-batch-jobs-on-aws-ee45f940696f">Orchestrating scheduled data batch jobs on AWS</a> was originally published in <a href="https://medium.com/tech-blog-door2door">door2door Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimising rides to the airport through data analysis]]></title>
            <link>https://medium.com/door2door-blog/optimising-rides-to-the-airport-through-data-analysis-c24e54f5a5fa?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/c24e54f5a5fa</guid>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[airports]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[public-transit]]></category>
            <category><![CDATA[door2door]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Thu, 10 Nov 2016 15:45:11 GMT</pubDate>
            <atom:updated>2017-02-16T16:37:22.315Z</atom:updated>
            <content:encoded><![CDATA[<p>Here at door2door we analyse how supply and demand works in public transport on a daily basis.</p><p>When we talk about supply in public transport, we are usually asking where transport agencies offer rides. We want to know how well passengers can efficiently move from A to B, and what bus or train lines are available at any given time.</p><p>Just as important is the question of whether the transport provided matches that demand. We collect demand information from our <a href="https://www.allyapp.com/">ally mobility app</a>, which acts as a sensor in the city. Interestingly, the transit system does not always match demand.</p><p>Let’s take a more concrete example. Let’s say you want to go to the main airport in Berlin, Tegel Airport.</p><h3><strong>Analysing supply to Tegel Airport</strong></h3><p>At the moment there are only two ways to reach Tegel directly: by car or by four bus lines.</p><p>By analysing the supply data we can identify what areas in Berlin can reach Tegel Airport within 15 minutes by public transport (the dark red area in the map below). As you see, a very small part of the city can reach the airport within 15 minutes. We can extend that question and ask who can reach the airport within 30 minutes (the light red area). A much bigger area of the city, but still only half the city can reach its airport within 30 minutes. This is especially an issue in East Berlin where no one can reach their main airport within 30 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5JNwoe8MFKkQCf0-_c91Kg.gif" /><figcaption>Public Transport Supply to Tegel Airport</figcaption></figure><p>Next we can ask what direct connections (meaning no transfers) there are to the airport. Transfers are one of the biggest obstacles in public transport. The lines in black show all four direct bus connections to the airport. Unlike most metropolitan cities, Berlin has no train or metro connections to its main airport.</p><p>This analysis very clearly demonstrates how Berlin’s transit system reflects the historical divide of the city: all direct lines run from the west.</p><p>This supply analysis already tells us a whole lot about Berlin’s transit system, but it really gets interesting when we compare this to where Berlin’s current demand is.</p><h3>Analysing demand to Tegel Airport</h3><p>The heat map below shows where the highest demand to Tegel airport in the city is. The darker the heat map, the higher demand is. As you can see there is particularly high demand in northeast Berlin (Mitte and Prenzlauer Berg), but also in southeast Berlin (the Kreuzkölln area).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcsuwZKmpdNP_eso-EqScQ.png" /><figcaption>Demand to Tegel Airport</figcaption></figure><p>What you also probably noticed is that there is a huge mismatch between the supply and demand. In Kreuzkölln (southeast Berlin), the demand is high, but many factors including that there are no direct connections means that it takes more than 30 minutes to reach the airport.</p><p>What becomes abundantly clear from this map is that even in one of the best connected cities in the world, there are significant gaps between what Public Transit authorities can currently provide and the demand of the city’s citizens.</p><p>We have been working on a solution for exactly this kind of issue. These gaps can be filled through <a href="https://en.wikipedia.org/wiki/Demand_responsive_transport">demand responsive transpor</a>t, like our <a href="https://blog.door2door.io/the-transportation-revolution-is-here-allygator-shuttle-43234653407f#.yzsnotnqc">allygator shuttle</a>. A transport system that responds directly to the ongoing demand of its riders. In other words, when there is high demand to get the airport in, for example, southeast Berlin, buses are rerouted to that area to take riders to the airport.</p><p>It is only through this data analysis that we can learn where the system can be improved with new services.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c24e54f5a5fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/door2door-blog/optimising-rides-to-the-airport-through-data-analysis-c24e54f5a5fa">Optimising rides to the airport through data analysis</a> was originally published in <a href="https://medium.com/door2door-blog">door2door Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eric visits Door2Door!]]></title>
            <link>https://medium.com/door2door-blog/eric-visits-door2door-9f541f56a7fc?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f541f56a7fc</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[mapbox]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Fri, 18 Sep 2015 14:08:52 GMT</pubDate>
            <atom:updated>2016-08-02T13:09:02.080Z</atom:updated>
            <content:encoded><![CDATA[<p>This week we had the pleasure of hosting a masterclass by Eric Gunderson, the CEO of Mapbox, here at Door2Door HQ. Mapbox is one of the hottest startups in the geo field and just raised $52.6M in their most recent funding round. Eric brought the cutting edge valley feeling to our Berlin office and we could definitely feel the excitement coursing through his veins at the thought of the amazing opportunities in the geo industry. Both Door2Door and Mapbox are heavily involved in the OpenStreetMap community and from the get go Eric told us how much he likes and appreciates our aggressively open stance with regards to the data we generate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i8egD7344XwPvXCG.jpg" /><figcaption>Eric Gunderson, CEO of Mapbox</figcaption></figure><p>He also shared his love for our Apple Watch version of ally mobility app— which is in fact the first app to make use of Mapbox Directions API for this device! From here he talked about our current state: one in which we are constantly surrounded by sensors that produce massive amounts of location based data. Making sense of this data is where we come in and as geo startups have a tremendous opportunity to utilize all this information and build something great out of it.</p><p>Mapbox exemplifies this through their Directions API, and over the next few months their Berlin based routing team will focus on integrating real time data into their routing. However to achieve this quite a lot of restructuring is necessary in OSRM; building the graph alone currently takes 6+ hours of work. So in order to incorporate real time data this approach isn’t effective. Thus in order to get this data they recently launched <a href="http://opentraffic.io/">OpenTraffic</a>, with partners such as the World Bank and Mapzen, but we mustn’t forget the massive responisbility that comes with the collection of this type of data and therefore the caution and server security that accompanies it.</p><p>Towards the end he gave us some interesting insights into the daily operations of Mapbox itself. For instance: how they prefer Slack over Hipchat and how they exclusively operate on Github as a management tool — giving us lots of interesting tips to learn from!</p><p>At Door2Door we are really excited to have such an inspiring and progressive partner in the industry and are really looking forward to working together in the future!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f541f56a7fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/door2door-blog/eric-visits-door2door-9f541f56a7fc">Eric visits Door2Door!</a> was originally published in <a href="https://medium.com/door2door-blog">door2door Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we use Mapbox to improve on ally routes]]></title>
            <link>https://medium.com/tech-blog-door2door/how-we-use-mapbox-to-improve-on-ally-routes-5d13887b8ea0?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d13887b8ea0</guid>
            <category><![CDATA[ally-mobility-app]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[door2door]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[maps]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Fri, 17 Jul 2015 11:28:20 GMT</pubDate>
            <atom:updated>2016-10-20T15:29:19.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*584x2xa7DQ3Z6J7ZqZOF9Q.jpeg" /></figure><p>Working closely together with our community is an essential part of our daily business. And making this work a reciprocal process is just as important. By combining user contributions with our ongoing route refinement, we’re very excited to continuously improve the ally routing all over the world. For this we partnered up with the San Francisco based startup <a href="https://www.mapbox.com/">Mapbox</a> — an open source mapping company deeply involved in <a href="http://www.openstreetmap.org/">OpenStreetMap</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*6Vj_SvSYAP5OfjXkfF5Gpg.gif" /></figure><p>Using Mapbox’s directions API gives us the ability to create together with you the best routing app for your city. Whenever you see a route you don’t like, you think is wrong or is missing something, just make use of our feedback feature. Our routing team will immediately look into the problem you submitted and try to fix it. Next time you or someone else is looking up that route it will already look better.</p><p>The directions API is using OpenStreetMap as a datasource. <a href="https://blog.door2door.io/ally-contributes-to-open-street-map-30fcfac9769b#.yrxjoiysk">A while ago</a>, we wrote how we at Door2Door already improve OpenStreetMap — a collaborative world map everyone can edit and improve, just like Wikipedia. This is where you as part of our community come in: When you tell us about a missing street, we add it to OpenStreetMap and it will be reflected in the routing the next time.</p><p>Also, Mapbox is currently investing a ton of energy in its directions API and the backend open source project OSRM. So stay tuned — it will get better every day. And together with you we can make it the best routing for your city, no matter if you use car, bike, foot or public transport.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d13887b8ea0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-blog-door2door/how-we-use-mapbox-to-improve-on-ally-routes-5d13887b8ea0">How we use Mapbox to improve on ally routes</a> was originally published in <a href="https://medium.com/tech-blog-door2door">door2door Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Helping to Put Dar es Salaam on the Map]]></title>
            <link>https://medium.com/tech-blog-door2door/helping-to-put-dar-es-salaam-on-the-map-4d302e54e84c?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d302e54e84c</guid>
            <category><![CDATA[trackyourcity]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Thu, 16 Jul 2015 10:56:47 GMT</pubDate>
            <atom:updated>2016-10-04T15:44:59.639Z</atom:updated>
            <content:encoded><![CDATA[<p>At Door2Door we love OpenStreetMap and so continuously endeavour to improve the quality of OSM worldwide when we can. This time we had the chance to partner up with Technische Universität München (TUM) and organize a Mapathon, as part of a bigger project to help make Dar es Salaam more resilient for natural hazards.</p><p>In recent months, Dar es Salaam has sadly experienced a rash of deadly flash floods and so the motivation behind the mapathon was to generate map data to be used to develop an early warning system to alert residents of flooding in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VBR-23kOvTzgnT8B.jpeg" /></figure><p>The organized Mapathon is part of a long term consortium Disaster Risk Reduction programme with the World Bank, Humanitarian OpenStreetMap Team (HOT), local universities and various international Red Cross chapters consequently ally is delighted to be able to contribute alongside such great organisations.</p><p>The city was split up into smaller sections by the <a href="http://tasks.hotosm.org/project/1083">HOT OSM tasking manager</a>. As well as 140 students mapping locally in Dar es Salaam, ally mapped together with students from Technical University Munich remotely using <a href="https://developmentseed.org/blog/2015/07/06/drone-imagery/">amazing drone imagery</a>. The students will now continue throughout the week to improve the map data.</p><p>We were really excited to contribute towards making Dar es Salaam more resilient for future floods, especially as this marks the first step of our mapping and data collection in Dar es Salaam… next stop Dala dalas! Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d302e54e84c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-blog-door2door/helping-to-put-dar-es-salaam-on-the-map-4d302e54e84c">Helping to Put Dar es Salaam on the Map</a> was originally published in <a href="https://medium.com/tech-blog-door2door">door2door Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ally contributes to Open Street Map]]></title>
            <link>https://medium.com/tech-blog-door2door/ally-contributes-to-open-street-map-30fcfac9769b?source=rss-92c5417f7d67------2</link>
            <guid isPermaLink="false">https://medium.com/p/30fcfac9769b</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[door2door]]></category>
            <category><![CDATA[ally-mobility-app]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Uli Strötz]]></dc:creator>
            <pubDate>Mon, 11 May 2015 15:10:33 GMT</pubDate>
            <atom:updated>2016-10-20T15:30:04.596Z</atom:updated>
            <content:encoded><![CDATA[<p>The ally story began as allryder, a transit app for Germany, Switzerland and the UK. Once established in major cities like London and Berlin, we began to turn our attention to transit in emerging markets, where getting around is trickier and deploying the city demands far more than integrating an API!</p><p>As the first step along this journey, we built <em>Minibus by allryder</em> which launched in Istanbul at the end of 2014. The simple app aimed to connect the dots of informal modes of transport in Istanbul — meaning hundreds of (so far) unrecorded minibus routes.</p><p>This project proposed very different and new challenges to the developers; unlike in London where the transport agency’s API could be used to map routes within the ally app, the minibus routes in Istanbul were completely unrecorded. Not only this, but many roads and streets in Istanbul are inaccurately mapped or have parts missing altogether.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*Wj6HM38_IjT5pSgc.png" /></figure><p>The solution? Crowdsourcing routes! The outcome? Improved OpenStreetMap data and the first accurate minibus routing app for the people of Istanbul. We were lucky to have a super enthusiastic team who helped us crowdsource over 300 minibus routes! After sourcing the data, the routes were post-processed by matching them to the actual roads the buses went on, using OpenStreetMap as a reference dataset.</p><p>OpenStreetMap is the Wikipedia of maps, built by more than two million volunteers from around the world. Whereas some parts of the world are mapped to the tiniest detail (down to every single tree!), some are mapped very rudimentarily, with only small parts of streets existing. What sets the project apart from other free online map services, namely the ubiquitous Google Maps- is that contributors can fix or add to the map if they find any inaccuracies or something missing. Through crowdsourcing in this way, OpenStreetMap becomes more and more accurate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-nFsUlzKlRcnGrt8.png" /></figure><p>From the results found whilst crowdsourcing the Istanbul minibus routes, the team were at the same time able to make big improvements to OpenStreetMap. If a minibus travelled along a road that had not yet been mapped, the backend team used an algorithm to create a small subset of that bus route that had not been mapped. This was then uploaded to OpenStreetMap, using satellite imagery to confirm the road before being finally added to the OpenStreetMap database. ally was also able to fix some wrongly assigned one-ways, missing links between highways and other attributes, thus improving the data quality on OpenStreetMap this way too.</p><p>Improving the data quality of the streets in Istanbul is just the first step for ally’s OpenStreetMap data contributions. Using lessons from Istanbul we have been able to add to OpenStreetMap whilst integrating other cities with informal modes of transport. Most recently this was through Colectivo routes in Santiago! With our backend developers working on integrating more and more cities and modes of informal transport, we plan to make more and more additions to OpenStreetMap, for example in Mexico City and Manila… all to the greater good of an open map!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30fcfac9769b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-blog-door2door/ally-contributes-to-open-street-map-30fcfac9769b">ally contributes to Open Street Map</a> was originally published in <a href="https://medium.com/tech-blog-door2door">door2door Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>