<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by James Graham on Medium]]></title>
        <description><![CDATA[Stories by James Graham on Medium]]></description>
        <link>https://medium.com/@jgraham0325?source=rss-e6d2feb1d727------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9bgXNwYJSVTZI1sIdvWbQA.png</url>
            <title>Stories by James Graham on Medium</title>
            <link>https://medium.com/@jgraham0325?source=rss-e6d2feb1d727------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:51:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jgraham0325/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Azure Data Factory: Dynamic feature branch testing]]></title>
            <link>https://medium.com/@jgraham0325/azure-data-factory-dynamic-feature-branch-testing-b8e0ce45aae8?source=rss-e6d2feb1d727------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8e0ce45aae8</guid>
            <dc:creator><![CDATA[James Graham]]></dc:creator>
            <pubDate>Tue, 07 Dec 2021 22:30:33 GMT</pubDate>
            <atom:updated>2021-12-07T22:30:33.516Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This blog describes how to effectively build, run and automatically test Azure Data Factory Pipelines on feature branches before merging them into the main branch in Git.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bUCTxNJYPLn6Ua7R" /><figcaption>High-level flow for development</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uSaaKVsuEIZD_rZq" /><figcaption>GIT branching flow</figcaption></figure><h3>What’s the problem it’s trying to solve?</h3><p>Since Azure Data Factory (ADF) is entirely cloud based, it’s not possible to run automated tests against it locally. ADF can be run from the web based GUI whilst on a feature branch but this isn’t accesible to automated tests, typically written in Python.</p><p>There are 2 main reasons for implementing this pipeline:</p><ol><li>Automatically test pull requests before merging into main branch</li></ol><ul><li>Keeps the main branch ready to release to higher environments at any time</li><li>Avoids blockages in developing features</li><li>Avoids contention between developers</li><li>Reduces risk of bugs making it into higher environments</li></ul><p>2. Ability to spin up an instance of ADF feature branch code on demand</p><ul><li>Allows QA engineers (or developers) to write tests (Python) that interact with a Data Factory in Azure. This means tests can be written earlier in the process, before code is merged into the main branch</li></ul><h3>Implementation</h3><p>To get this working in your own environment:</p><ol><li>Fork the <a href="https://dev.azure.com/jgraham03250857/jgraham0325/_git/DynamicDataFactoryPoC">sample code repo</a> to your own Azure Devops repo or Github repo</li><li>Follow the 6 steps in the <a href="https://dev.azure.com/jgraham03250857/jgraham0325/_git/DynamicDataFactoryPoC?path=/README.md&amp;_a=preview">README</a></li></ol><h3>Continuous Integration Pipeline explanation:</h3><p>The <a href="https://dev.azure.com/jgraham03250857/jgraham0325/_git/DynamicDataFactoryPoC?path=/azure-pipelines.yml">Azure Devops pipeline</a> (aka the CI pipeline) is written in yaml and orchestrates the process of building, testing and deploying the ADF code.</p><p>It runs the following tasks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*pP7EZJn7OScG7lUsHSxCyw.png" /></figure><p>CI pipeline code:</p><p>See <a href="https://dev.azure.com/jgraham03250857/jgraham0325/_git/DynamicDataFactoryPoC?path=/azure-pipelines.yml">here for the full repo</a>, including helper functions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c4430c6d4fdfd48c89ac15e1b43b738/href">https://medium.com/media/5c4430c6d4fdfd48c89ac15e1b43b738/href</a></iframe><h3>Running the pipeline</h3><p>Every time you save in the ADF web GUI, it creates a commit on your feature branch so it’s likely to be excessive to run the CI pipeline for every commit. Instead, Azure Devops can be configured to run the CI pipeline when a pull request is created, the code is likely to be more stable at this point. See docs for creating a <a href="https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops">branch policy in Azure Devops</a>.</p><p>The CI pipeline can also be run manually on demand. The default behaviour is to delete the temporary ADF instance used for testing but there is a check box that can be unticked to retain the ADF instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*nlGJvIxXF1G4WFRM8EkbMA.png" /><figcaption>Running the CI pipeline on demand</figcaption></figure><h3>Azure Data Factory tests</h3><p>Tests are automatically run as part of the Azure Devops pipeline. Python is used for testing that the new ADF pipeline exists. This is a very simple test to demonstrate the concept but in reality the tests should:</p><ul><li>Set up test input data</li><li>Run the ADF pipeline</li><li>Confirm the output is as expected</li></ul><p>The test results appear on the pipeline execution page. An example is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vti07uCwIqMJvTncDjoT6g.png" /></figure><h3>References</h3><ul><li><a href="https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery-improvements">Microsoft article for automatic ADF publish</a></li><li><a href="https://github.com/Azure-Samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors">Azure modern data warehouse samples</a></li><li><a href="https://mrpaulandrew.com/2019/12/18/best-practices-for-implementing-azure-data-factory/">Azure Data Factory best practices</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8e0ce45aae8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualising Streaming Geospatial Data]]></title>
            <link>https://medium.com/@jgraham0325/visualising-streaming-geospatial-data-60566b7cd881?source=rss-e6d2feb1d727------2</link>
            <guid isPermaLink="false">https://medium.com/p/60566b7cd881</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[leaflet]]></category>
            <dc:creator><![CDATA[James Graham]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 15:58:23 GMT</pubDate>
            <atom:updated>2019-04-18T21:09:27.477Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F330370173%3Fapp_id%3D122963&amp;dntp=1&amp;url=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F330370173&amp;image=http%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F775402744_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/79c55aac1b6128633fd95c464c12a9d6/href">https://medium.com/media/79c55aac1b6128633fd95c464c12a9d6/href</a></iframe><p>I recently worked on a project that involved streaming credit card transaction data and classifying it based on risk probability. Building upon this, I wanted to explore the options for visualising the data. I decided to focus on the geographical aspect since it’s a key component when trying to identify fraudulent transactions… and I’m a bit of a map nerd.</p><p><strong>Code</strong>: <a href="https://github.com/jgraham0325/streaming-visualization">https://github.com/jgraham0325/streaming-visualization</a></p><p><strong>Live Demo</strong>: <a href="https://streaming-visualisation.appspot.com/">https://streaming-visualisation.appspot.com/</a></p><h3>Use Cases</h3><p>There are many reasons why being able to visualise streaming geospatial data could solve real-world problems. Essentially these boil down to scenarios where it’s important to make a decision soon after an event has occurred. It could be a person who’s responsible for making the decision or a machine learning algorithm that automates the process.</p><p><strong>Some examples:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*RlrySSPVMuwk-_mItWZq7Q.png" /><figcaption><a href="http://www.waze.com/">Waze hazard</a></figcaption></figure><p><strong>Connected Car</strong>: With the increasing amount of sensors in cars and their access to the internet, it’s possible to alert a driver to hazards on the road before they become a danger. E.g. a tree that’s just been blown down on the road ahead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*cki8v7pNf3KrBmISRUtq4A.png" /><figcaption><a href="https://azure.microsoft.com/en-gb/features/iot-accelerators/connected-factory/">Azure IoT connected factory</a></figcaption></figure><p><strong>Internet of Things</strong>: Predictive maintenance can be enhanced by visualising the location of potential problems and finding the closest supply of replacement parts. It can also identify patterns or clusters that otherwise wouldn’t be obvious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*fyZaR8FeJ8aVw6sugpGFTQ.png" /><figcaption><a href="https://www.flightradar24.com/">FlightRadar24</a></figcaption></figure><p><strong>Flight Tracking:</strong> Showing real-time flight information can help predict delays, handle irregular operations, and analyse route efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*MxzBJuDSsb4BI7psGuEwrg.png" /><figcaption><a href="http://www.weatherbug.com/">Real-time fire hazards (Weatherbug)</a></figcaption></figure><p><strong>Disaster relief: </strong>Collecting actionable GIS data before and after hazardous events, like fires, can help people to avoid dangerous situations.</p><h3>Technology Choices</h3><p>There are a number of commercial products that offer the ability to display geospatial data in near real-time.</p><p><strong>Commercial products include</strong></p><ul><li><a href="https://www.esri.com/arcgis-blog/products/arcgis-pro/real-time/real-time-data-arcgis-pro/">ArcGIS</a>: A desktop or cloud based product by ESRI, who have been a dominant force in commercial mapping applications for decades. It’s powerful but the licence costs are expensive.</li><li><a href="https://cesiumjs.org/demos/">Cesium</a>: Particularly good at visualising 3D data. It’s used by sites such as FlightRadar24, which is accessed 45 million times a month. Billed monthly based on storage and usage.</li><li><a href="https://www.kinetica.com/products/streaming-analytics/">Kinetica</a>: Unique selling point is the combination of GIS with AI/ML.</li><li><a href="https://www.zoomdata.com/master-class/streaming-analytics-and-internet-things/geospatial-data-visualization-demo/">Zoomdata</a>: Well known for streaming visualisation capabilities but not particularly for GIS. Able to show data in a wide variety of graphs and combine them with plots on a map.</li></ul><p>There are also a number of JavaScript APIs that can be used, for a more DIY approach.</p><p><strong>Javascript APIs</strong></p><ul><li><a href="https://leafletjs.com/">Leaflet.js</a>: simple, open source and offers a good library of plugins (including Mapbox JS).</li><li><a href="https://openlayers.org/">OpenLayers</a>: powerful, open source but more complex than others.</li><li><a href="https://docs.mapbox.com/mapbox-gl-js/api/">Mapbox GL</a>: good for displaying complex data layers using WebGL.</li><li><a href="https://developers.google.com/maps/documentation/javascript/tutorial">Google Maps JS API</a>: easy to use but requires a paid licence past a threshold.</li><li><a href="https://threejs.org/">Three.js</a>: Uses WebGL to create 3D graphics in a web browser.</li></ul><p><strong>Decision</strong>: Use Leaflet.js since it’s easy to use, flexible and doesn’t incur any licence costs. A Node.js server is used along with the Socket.io library to push real-time events to the map in the client’s browser. I also implemented some Three.js visualisations for comparison.</p><h3>High-level architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*IyNO1uzR0MJACGAqPHFqEQ.png" /></figure><h3>Performance</h3><p>Showing hundreds of thousands of points on a map at once is technically challenging and arguably not very useful. People find it difficult to interpret this data in its raw format.</p><p>To address these issues, points are typically aggregated using heatmaps or clusters of points. Through some trial and error, I’ve found that the performance of these layers can vary widely depending on their implementation. Filters are also useful and in the case of credit card transactions, only high-risk transactions are shown. This brings the typical number of credit card transactions down from 8.9M/hr (based on UK average of 924M a year) to 840/hr (based on average fraud rate of 0.08% and assuming 9/10 high-risk transactions are false positives)</p><p>As part of the PoC, the following layers were implemented:</p><ul><li><a href="https://codepen.io/qkevinto/pen/EVGrGq">Three.js (WebGL)</a>: This is the most resource intensive option since it’s rendering a 3D map of the world. The only way to make it performant was to limit the number of points displayed simultaneously to less than 100. Using a 2D WebGL map allowed for thousands of points to be displayed but the resolution was too low to be useful in practice.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9e-_FDx-tBwLZ_O9o6PTw.png" /><figcaption>3D WebGL heatmap using three.js. Flashes of light indicate a new transaction</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6F72EdrdrlQo3PiPOHouQ.png" /><figcaption>2D WebGL heatmap using three.js</figcaption></figure><ul><li><a href="https://github.com/Leaflet/Leaflet.heat">Leaflet.heat plugin</a>: This was able to download and render over 10K points in less than 1 sec. It’s useful for identifying hot spots but doesn’t allow the user to zoom down to see the individual points, which is often essential for identifying potential fraud.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxkKI5Enn4-8VlBR6_P1sQ.png" /><figcaption>Heatmap with 5,000 high-risk transactions.</figcaption></figure><ul><li><a href="https://github.com/Leaflet/Leaflet.markercluster">Leaflet MarkerCluster plugin</a>: This is the most commonly used plugin for grouping points that are close together to make them manageable on screen. However, it took 2–3 sec to render 10K points and didn’t work well with streaming data since each time a data point was added, the layer needed to be refreshed. I replaced this with the PruneCluster implementation mentioned below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D16XULpA1SY4AATohNgBQg.png" /><figcaption>Clustered points of high-risk transactions using Leaflet Marker Cluster plugin</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0O4GhnajSXkMpe-VtE_fiw.png" /><figcaption>Example of 15 high-risk transactions in a single location. With MarkerCluster, new data would cause the “spider” to contract to a single point due to the whole layer refreshing</figcaption></figure><ul><li><a href="https://github.com/SINTEF-9012/PruneCluster">Leaflet PruneCluster plugin</a>: This was found to be the most performant solution and worked well with streaming data. It’s also designed for IoT applications in mind, where existing points could frequently change location. Cosmetically it’s very similar to the Marker Cluster plugin.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBcZrj5qQiPQ4fJTyHCEGQ.png" /></figure><p>The performance stats for PruneCluster are shown below - performed on a modern laptop with the Chrome 38 browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nv2HHIvqg9npYFuSJUYCOw.png" /><figcaption>Indicative performance of rendering points on map, based on Leaflet <a href="https://github.com/SINTEF-9012/PruneCluster">PruneCluster plugin</a></figcaption></figure><h3>Challenges &amp; Lessons learned</h3><ul><li><strong>Event cache: </strong>Redis Pub/Sub makes it simple to push new events to the client but doesn’t give the option of retrieving recent previous events when a client first connects. Using a <a href="https://www.infoworld.com/article/3212768/how-to-use-redis-for-real-time-stream-processing.html?page=2">sorted set in Redis</a> or the <a href="https://github.com/RedisLabsModules/RedisTimeSeries">Time Series Module</a> could allow this but adds additional complexity. For this PoC, a simple cache is maintained on the server in a Javascript array that allows newly connected clients to load previous events based on a maximum threshold.</li><li><strong>Historical analysis</strong>: A slider would need to be introduced to control the period of time displayed. This is reasonably easy to implement with JQuery and Leaflet.js.</li><li><strong>Street view</strong>: Is a useful tool for investigating potentially risky areas. Whilst the Google Maps API has the best integration with this feature, it’s possible to build this into almost any browser based map as shown by this PoC.</li></ul><h3>Conclusion</h3><p>Visualising data geospatially can unlock valuable insights that would otherwise be missed. With minimal effort and a generous open source community it’s possible to create powerful visualisations without spending a penny!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60566b7cd881" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>