<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Julian Ford on Medium]]></title>
        <description><![CDATA[Stories by Julian Ford on Medium]]></description>
        <link>https://medium.com/@julian-ford?source=rss-4a4add3367f4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*h1qVOeQonkVVNE7ZPRjncA.jpeg</url>
            <title>Stories by Julian Ford on Medium</title>
            <link>https://medium.com/@julian-ford?source=rss-4a4add3367f4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:21:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@julian-ford/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I’m Not Sold on AI Video for Teaching yet. But I Tried It Anyway.]]></title>
            <link>https://julian-ford.medium.com/im-not-sold-on-ai-video-for-teaching-yet-but-i-tried-it-anyway-f036b1ba4c07?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f036b1ba4c07</guid>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Thu, 09 Jul 2026 15:11:39 GMT</pubDate>
            <atom:updated>2026-07-09T19:57:15.056Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m sceptical about a lot of what gets sold to schools. Every few years there’s a new thing that’s going to “transform learning,” and most of it ends up as an expensive login nobody uses by February. So when the talk turned to AI-generated video, my instinct was to fold my arms.</p><p>But I had a specific, small problem I couldn’t shake, and it made me curious enough to actually try.</p><h3>The problem was a field trip</h3><p>We were going to a site — the details don’t matter, could be a castle, a coastline, a factory. And I knew, from experience, that half the class would arrive having thought about it for exactly zero seconds. They’d tumble off the bus, blink at the thing, and spend the first hour catching up on context they should have walked in with.</p><p>What I wanted wasn’t a lesson. I’ve got plenty of those. I wanted a <em>hook</em>. Ninety seconds, something they might actually watch on the bus on the way there, that made them curious before they arrived. Not “here are the facts you’ll be tested on” — more “here’s why this place is strange and worth your attention.”</p><p>That kind of short, punchy, made-for-this-exact-moment clip is something I’d never make myself. I don’t have the hours, the editing skills, or frankly the inclination to learn video production on top of everything else.</p><h3>So what happened when I tried it</h3><p>Genuinely useful, in a narrow way.</p><p>I could describe the idea for a short reel and get something back quickly — a rough, watchable clip that set a mood and posed a question. For the bus-ride hook, that’s honestly about the right level. It doesn’t need to be a documentary. It needs to make a fifteen-year-old look up from their phone for a minute and think “huh.”</p><p>And there’s something to the <em>repeatability</em> of it. Once I’d figured out the shape of a good hook, I could imagine making one for every trip, every new topic, without it eating my Sunday. That’s the part that made me stop folding my arms quite so tightly.</p><h3>But — and this is a big but — it is not the answer</h3><p>Teachers are already using ChatGPT. Of course they are. And students are too. I don’t think AI slop is going to solve teachers problems entirely — but you can’t ignore AI EdTech completely either.</p><p>It doesn’t know my class. It can’t read the room, can’t tell that a clip landed flat, can’t follow a student’s question down an unexpected and brilliant rabbit hole. It has no idea <em>why</em> a place matters to the kids in front of me, only that it’s supposed to look interesting. It produces something plausible, not something wise.</p><p>And a hook is just a hook. The clip on the bus does nothing on its own. All the actual learning still happens where it always has — in the conversation on the coach, the argument at the site, the thing a student notices that I didn’t, the debrief where it finally clicks. If the video ever tried to <em>be</em> the lesson, it would fail, and it would deserve to.</p><p>There’s also a quieter worry. It’s easy, with a tool that makes flashy things fast, to mistake engagement for understanding. A slick clip that gets a “whoa” is not the same as a student who can explain the idea back to you. I’d want to watch myself carefully there.</p><h3>Where I’ve landed (for now)</h3><p>Curious. Still unconvinced it’s a revolution. Fairly convinced it’s a useful, narrow tool — a way to make the small, disposable, get-them-interested content that I’d never otherwise make, so I can spend my actual energy on the parts of teaching that matter and that no machine can touch.</p><p>A ninety-second reel to watch on the bus. Not a teacher. Not a lesson. Just a spark, cheaply and quickly made, pointed at the right moment.</p><p>If it stays in that lane — a helper for the bits around the edges, with me firmly deciding what’s worth showing and why — then fine. I’m in, cautiously. The second anyone tells me it can replace the teaching itself, I’ll be folding my arms again.</p><p><em>I ended up poking at this properly during a hackathon, building a little tool on to see how far the “quick hook” idea could go. The building taught me as much about the limits as the possibilities — which is roughly where a sceptical educator ought to end up.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f036b1ba4c07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing Alexa skills for EdTech]]></title>
            <link>https://julian-ford.medium.com/developing-alexa-skills-for-edtech-b1bac46acb05?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1bac46acb05</guid>
            <category><![CDATA[edtech]]></category>
            <category><![CDATA[edtech-tools]]></category>
            <category><![CDATA[alexa]]></category>
            <category><![CDATA[alexa-skills]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Thu, 10 Oct 2024 16:53:08 GMT</pubDate>
            <atom:updated>2024-11-09T11:42:34.244Z</atom:updated>
            <content:encoded><![CDATA[<p>The Amazon Alexa has been around, according to my research for over ten years now — and the little low profile devices are a common feature in households everywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*715_kkS_l40GXeKb" /><figcaption>Photo by <a href="https://unsplash.com/@hckmstrrahul?utm_source=medium&amp;utm_medium=referral">Rahul Chakraborty</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the Summer of 2018, Amazon launched a competition asking developers to build and submit Alexa skills to their eco-system. Hungry for an outsize hoodie, I sat down (with my school-age daughter) and had a go at putting something together. In this story, I’ll explain how we came up with an idea and developed it into a complete Alexa skill.</p><p>I work in EdTech and that was a natural place to start, so after a few conversations here’s what we came up with as a pre-requisites for the skill:</p><ul><li>It should be simple and fun to use</li><li>It should have some educational value</li><li>It should be easy to code</li><li>It should be very cheap (ideally free) to host</li><li>It should have some graphical feedback (given that the Echo Show had just launched)</li></ul><p>In retrospect, I can see that was quite an ambitious list especially as my coding skills are very limited — however, inspired by my daughter’s fascination, we decided to persevere.</p><p>I started by looking at some sample applications provided by Amazon — particularly in the trivia space. This would certainly help fit with most of the criteria we’d set. Having reviewed the development documents I could see that the biggest challenge would then be training and recognizing “utterances” — those kind of generalizations that have to be kettled into some sort of intent with the Alexa skill itself. In this area we then had a rather brilliant idea and that was to make sure that the utterances were <strong>almost 100% numbers</strong> — and that led us to the path of building <a href="https://www.amazon.co.uk/Stockeroo-Times-Table/dp/B07J4VSH9X"><strong>Times Table Test</strong></a>.</p><p>Times Table Test is quite simply a skill that tests (presumably) school-children on their maths times-tables (eg 2 x 2=4, 3 x 2=6). Times tables were taught by rote when I was at school and we got tested on them every day — for better or worse. Now, I’m not sure if that is still a thing on the UK Education Curriculum but I figure it’s still a thing somewhere… so the next task was to figure out some kind of design for the skill.</p><p>This is what we came up with…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1yc-Cyr-nB02YtG3hQ9vA.png" /><figcaption>Times Table Test Design Note</figcaption></figure><p>When it came to developing the skill I leaned pretty heavily on the Trivia Sample App. They can be found here — <a href="https://github.com/alexa-samples">https://github.com/alexa-samples</a></p><p>The basic <strong>Node </strong>template has the main code body contained in the<em> index.js</em> file and contains:</p><ul><li>Set up constants for the skill settings for example the WELCOME_MESSAGE and the HELP_MESSAGE</li><li>Code to populate an array (12) with the questions and and answers for the test</li><li>A function to check if the response is valid (usually a number)</li><li>A function to handle the User Guess using a generate a Response string response</li><li>A function to Check if it is the last Question and offer an “end game” series of options</li></ul><p>There were a couple of areas where the code become a little bit tricky. In particular the requirement to shuffle the questions rather than asking them in numeric sequence and also to make the game flow, the introduction to the question had commentary stored in an array as follows…</p><blockquote>const randomNext=[‘Thank you. Here is the next question. ‘,’Next question. ‘,’OK. Here\’s the next one. ‘];</blockquote><p>…and then a intro line for the questions was selected at random. This was an interesting quirk of skill building which was trying to make the the process not sound entirely robotic using the same phrasing over and over.</p><p>A late addition was the inclusion of <strong>cardRenderer </strong>which added visual output if you’re using a device like an Echo Show with visual output.</p><blockquote>this.response.cardRenderer(cardText, ‘Question Number ‘ + currentQuestionIndex, imageObj);</blockquote><p>I actually recorded a (rather low quality) <a href="https://www.youtube.com/watch?v=5dupv_7FA5I">video </a>of the finished product that you can view on You Tube.</p><p>Hosting charges are zero because the skill uses an AWS Lambda function and sits comfortably within the free tier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KfbCkeXaBMglTzYdsXBtLA.png" /><figcaption>Amazon Free Tier</figcaption></figure><p>I published the skill up after reading the documents, got my hoodie and I never really thought a lot more about it….but… I have been surprised to find that the skill gets a pretty substantial amount of usage. Somewhere between 10 and 50 unique customers using it each week — and about 15OO sessions in the last 12 months with a respectable return rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q8tzFPgi9m1qBBtD0NmT9A.png" /><figcaption>Last 12 months usage stats</figcaption></figure><p>Perhaps those few hours spent putting it together helped a child (or a stressed parent) learn this most basic of EdTech skills? Hope so !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1bac46acb05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Putting a UI on our Data Quality System]]></title>
            <link>https://julian-ford.medium.com/putting-a-ui-on-our-data-quality-system-26fdcdae60cc?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/26fdcdae60cc</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-dictionary]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 15:42:28 GMT</pubDate>
            <atom:updated>2024-10-11T08:51:52.921Z</atom:updated>
            <content:encoded><![CDATA[<p>Our initial deployment of a Data Quality system was in headless mode. However the reality of interacting with business users and domain experts (eg data stewards) required us to provide an intuitive and user friendly dashboard view of the Data Quality estate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*x5lMSmV7ovraEVMTZPjP9Q.png" /><figcaption>Data Quality System UI Design</figcaption></figure><p>After consultation with the end-users we built a fairly simple UI that provided an alerting and administrative dashboard.</p><p>This UI allowed a user to :</p><ul><li><strong>Dashboard: </strong>Investigate any high priority items. Take a look at the count and individual instances of exceptions and who they had been sent to for potential resolution</li><li><strong>Rule Detail:</strong> Look at a DQ Rule in detail including who authored it, who recipients of exceptions are, setting a priority and potentially disabling it</li><li><strong>Rule Output</strong> : I added an embedded graphical view of the rule output with particular focus on the history of the counts generated and whether that data was trending up-or-down</li><li><strong>Audit :</strong> We added a full record of current and historic records that had been flagged and also added the facility to <a href="https://medium.com/@julian-ford/ignoring-records-in-a-data-quality-system-22d4bed23a6c">ignore a record</a> based on whether it was in fact valid.</li></ul><p>This is the final article that covers how we have developed and used our Data Quality system over the last 10 years or so.</p><p>The Data Quality System is actually part of a more complete Data Management Solution that we use to administer</p><ul><li>Data Governance Events</li><li>A complete business Data Dictionary</li><li>Lineage between system interfaces</li><li>Database Meta-Data and change tracking</li></ul><p>We’ll explore the full solution in later articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26fdcdae60cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Expanding the Data Quality System to monitor for Fraud]]></title>
            <link>https://julian-ford.medium.com/expanding-the-data-quality-system-to-monitor-for-fraud-a06f3da87701?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a06f3da87701</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 15:05:29 GMT</pubDate>
            <atom:updated>2024-10-11T08:48:56.739Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ve previously discussed how our Data Quality System was built out of a need to identify and monitor occasional recurrent data issues in our LOB systems. At the time of writing we have about 400 rules that check our data environment daily. The bulk of these rules return zero exceptions and save cluttering up some poor Data Steward’s inbox but many data issues remain persistently repetitive.</p><p>One thing I am grateful for is the safety-net of an automated check that runs in the background of our system, patiently searching for data errors that if not corrected will come back and bite us one way or the other. We are particularly sensitive to the accuracy of our MIS reporting — and the DQ System has proved itself a great asset.</p><p>Over the years we have had issues like commission records not having been generated correctly in the CRM because of oddities in the underlying data. We’ve been able initially to use SQL to search for and identify those records and then simply add that SQL the DQ Engine as a perpetual check on the quality of our data.</p><p>Another example of where we have been able to apply Rules is in the area of Fraud Detection. So while a rule will generally simply check for the presence or absence of a record, a more sophisticated rule could check whether a value falls out of a range based on mean, median or standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*-TUjgaPt1wvgKjNZJO7wow.png" /><figcaption>Introducing Anti-Fraud Monitoring</figcaption></figure><p>In certain circumstances, we might want to look at a service being hit a disproportionate amount of times or a report or series of reports or exports from a report being run by a user. In this case, rather than looking at the CRM itself, we might search the log files for the last 24 hours and compare activities to the mean or daily average for anything suspicious.</p><p>As with the previous example, the appeal of this approach is that once locked into the system, the DQ Engine is providing an increasingly sophisticated early warning to improper use.</p><p>Finally <a href="https://medium.com/@julian-ford/putting-a-ui-on-our-data-quality-system-26fdcdae60cc">we’ll put a UI on</a> our headless Data Governed DQ and Anti-Fraud Framework to help our business users get the most out of the system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a06f3da87701" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing Data Governance and a Data Quality System together]]></title>
            <link>https://julian-ford.medium.com/bringing-data-governance-and-data-quality-together-8a56d44c9722?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a56d44c9722</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 14:45:42 GMT</pubDate>
            <atom:updated>2024-10-11T08:46:59.492Z</atom:updated>
            <content:encoded><![CDATA[<p>At the same time that we introduced the Data Quality System (part of a wider Data Management System) we also convened a Global Data Governance Group.</p><p>In the UK, GDPR was starting to come up on our radar as an area that required intense focus and so working with stakeholders from across the organization and the newly appointed DPO, we started to meet on a monthly basis to discuss salient data issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*eII7Rhe8abDceY--XXcoPg.png" /></figure><p>Fairly central to that initiative was the need to bring Data Management and Data Governance together based on the principles of DMBOK.</p><p>The way we integrated the system was as follows :</p><ul><li>We added a section to the Data Management System (Events) that allowed us to discuss and log any issues related to Data Quality or Data Management</li><li>In some of those cases we then added a DQ Rule (or rules) that allowed us to note the issue that had been raised and track the effectiveness of our DQ Rule and any associated action points raised during the session</li></ul><p>So for example, off the back of a Data Governance meeting we might have recognized the need to accurately record the DOB of a customer. We could add some rules to check that the DOB wasn’t in the future (that happens quite a lot!) and that the DOB was outside a particular range — for example &gt; 100 Years ago (remember that we have our <a href="https://medium.com/@julian-ford/ignoring-records-in-a-data-quality-system-22d4bed23a6c">Ignore feature</a> now for situations where a false positive might be raised).</p><p>Using some composite reports — this is now done in Power BI — we were then able to bring together a complete picture of :</p><ul><li>The DG issue item under discussion</li><li>The action plan to resolve it</li><li>The rules created to monitor the improvement</li><li>Trendlines to support that improvement</li><li>An audit of specific recurring incidents that require a further follow up</li></ul><p>In addition to these tactical playbacks, we also had a whole section of the Data Governance agenda dedicated to any DQ Rules that were tagged as High Priority and were trending upwards.</p><p>Next priority on the list for the Data Governance group was <a href="https://medium.com/@julian-ford/expanding-the-data-quality-system-to-monitor-for-fraud-a06f3da87701">tackling fraud</a>…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a56d44c9722" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ignoring Records in a Data Quality System]]></title>
            <link>https://julian-ford.medium.com/ignoring-records-in-a-data-quality-system-22d4bed23a6c?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/22d4bed23a6c</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 14:28:00 GMT</pubDate>
            <atom:updated>2024-10-10T08:23:36.630Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://julian-ford.medium.com/dynamic-routing-in-a-dq-system-aa38ea88790a">Previously </a>we talked about developing our Data Quality Rules and introducing a DQ Framework that controlled how we executed them. Version 1.0 of the Data Management system got off to a flying start, but we were soon confronted with more complex scenarios.</p><p>In the previous article we discussed how we introduced <strong>dynamic </strong>routing to allow federated data stewardship. The next challenge turned out to be the need to flag false positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*dg7Yr0o5PzUIVUHu3MZguQ.png" /></figure><p>The first example of where we found a requirement to exclude a record from our exception reporting was in the case of looking out for <strong>Test </strong>records in our <strong>Production </strong>environment.</p><p>Due to some poor practice in the use of the CRM we saw an increasing amount of Test records creeping into our production environment. There were test records for suppliers, customers, products and opportunities.</p><p>Together with the Data Governance group we created a policy on Test records appearing in Production and embarked on a program of activity to get these stamped out. We also introduced some new rules to check for Test records — at a very basic level this simply meant looking for the word “Test” in any supplier account.</p><p>The issue with this (as we soon discovered) is that plenty of legitimate suppliers have the word “test” in their name — think Acme Testing Corporation. The basic SQL rules we had would continuously return this record as an exception.</p><p>There was a reasonably simple solution to the problem — given that we were already tracking each exception by it’s unique key — and that was to allow the data steward to click on the link in the email and flag that particular record to “be ignored”.</p><p>Now going back to the four steps in our original framework we find that we have an additional step to close out any records that should be ignored</p><ol><li>Find all active DQ rules</li><li>Execute the rule using dynamic SQL</li><li>Collate any output and send it to the steward attached to the rule</li><li><strong>Remove any records that have been flagged as “ignore”</strong></li><li>Write the records and information (date stamp, #exceptions) to the log file</li></ol><p>In a slightly later phase, we added a full UI to the system that allows users to audit who has flagged which records to be ignored and they can be adjusted appropriately. We’ll talk about the UI in another article.</p><p>Next up though we’ll talk about <a href="https://medium.com/@julian-ford/bringing-data-governance-and-data-quality-together-8a56d44c9722">Data Governance</a>…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d4bed23a6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Routing in a Data Quality System]]></title>
            <link>https://julian-ford.medium.com/dynamic-routing-in-a-dq-system-aa38ea88790a?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa38ea88790a</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 14:07:05 GMT</pubDate>
            <atom:updated>2024-10-10T08:25:37.839Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://julian-ford.medium.com/developing-a-data-quality-system-598bb1e21925">previous article</a> we talked about a simple framework for detecting errors in a reporting layer and distributing those back to data stewards for attention and correction.</p><p>That solution worked well in the early days of the roll-out however we soon faced a request for dynamic routing.</p><p>For example a DQ Developer might want to create a single rule looking for duplicate customers but need to re-route the responsibility for rectifying the issue to different data stewards and different operational hubs based on the source of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*bnkdGey8VvywjPpt2L5NNg.png" /></figure><p>To meet this requirement, we introduced an (optional) third column to the SQL DQ rule output. So before where had the record primary key and the record description we added <strong>record_email</strong>. When populated, the DQ framework knows to redirect the DQ exception to the email recipient in the 3rd column instead of (or as well as) those tagged for receipt on the rule.</p><p>To use our previous DQ #1 rule we could add a 3rd statement as follows :</p><p><em>— Return Duplicate Customer Records —</em></p><p><em>— (added dynamic routing) —</em></p><p><em>select crm_id as record_pk, crm_name as record_description, operation_name + “@mydqdatacorp.com” as </em><strong><em>record_email</em></strong></p><p><em>from customers</em></p><p><em>group by crm_id, crm_name, operation_name</em></p><p><em>having count (*) &gt; 1</em></p><p>In the <a href="https://medium.com/@julian-ford/ignoring-records-in-a-data-quality-system-22d4bed23a6c">next part</a> we’ll look at Ignoring false-positive records.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa38ea88790a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing a Data Quality System]]></title>
            <link>https://julian-ford.medium.com/developing-a-data-quality-system-598bb1e21925?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/598bb1e21925</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 13:52:24 GMT</pubDate>
            <atom:updated>2024-10-11T09:34:17.658Z</atom:updated>
            <content:encoded><![CDATA[<p>In a <a href="https://medium.com/@julian-ford/i-developed-a-data-quality-system-ten-years-ago-and-its-still-going-strong-b558f32d04f1">previous story</a> I talked about the need to develop a Data Quality system. In this story we’ll look at the approach I took. The vision I had for the system was a set of re-usable SQL based “cartridges” (aka Rules) that would be loaded into a Data Quality framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*jAqARG_VhkyLGPDKIi2sFA.png" /></figure><p>Any records returned from a SQL rule / cartridge would be sent to the to: and cc: addresses attached to the rule. The framework would keep a log of when the rule was run, how many exceptions were detected and emailed out. That way we could see whether the impact of our wider Data Quality strategy was having an effect.</p><p>If the SQL rule returned zero rows — then nothing was sent. That was — obviously — the ultimate objective to get the rules consistently returning no data exceptions.</p><p>The format of the SQL rules had to always be consistent and return <strong>two specifically named columns</strong>.</p><ul><li><strong>record_pk </strong>: the unique reference for the exception record</li><li><strong>record_description </strong>: a user-friendly description of the exception</li></ul><p>For example SQL DQ Rule #1 might have the following …</p><blockquote>— Return Duplicate Customer Records —</blockquote><blockquote>select crm_id as record_pk, crm_name as record_description</blockquote><blockquote>from customers</blockquote><blockquote>group by crm_id, crm_name</blockquote><blockquote>having count (*) &gt; 1</blockquote><p>The framework runs the following steps at 6AM daily</p><ol><li>Find all active DQ rules</li><li>Execute the rule using dynamic SQL</li><li>Collate any output and send it to the steward attached to the rule</li><li>Write the records and information (date stamp, #count, exceptions) to the log file</li></ol><p>That’s it. We were up and running and that particular de-dupe rule still runs 365 days a year 10 years later.</p><p>I had estimated that we might have 10 or so rules set up and running in the first 2 or 3 months, but in reality we had well over 100 (and there are 400+ rules in the system as of today).</p><p>Inevitably there was a demand for increasingly sophisticated rules and for a more customisable DQ Framework.</p><p><a href="https://medium.com/@julian-ford/dynamic-routing-in-a-dq-system-aa38ea88790a">Here’s </a>how we addressed it…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=598bb1e21925" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I developed a Data Quality System ten years ago and it’s still going strong]]></title>
            <link>https://julian-ford.medium.com/i-developed-a-data-quality-system-ten-years-ago-and-its-still-going-strong-b558f32d04f1?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b558f32d04f1</guid>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 13:30:40 GMT</pubDate>
            <atom:updated>2024-10-09T09:35:52.373Z</atom:updated>
            <content:encoded><![CDATA[<p>In September 2014 I was facing significant Data Quality issues at my place of work. As Head of Data it was my responsibility to make sure that data coming out of our systems and into our management information was complete and and of a high quality. It wasn’t.</p><p>Some examples of recurrent issues from the CRM</p><ul><li>Duplicate entries for our Supplier Agents</li><li>Incomplete Customer Information</li><li>Customer records with implausible data (eg a DOB in the future)</li></ul><p>I was tasked with setting up a Data Management System to intercept these kinds of data issues but the system had to go beyond Data Quality and into the realm of Data Management. Data Quality (was) and always will be the short-term priority.</p><p>I had a look at a number of solutions on the market at the time (for example Microsoft MDS) but the options weren’t a good fit for my organization. In particular there was a business requirement to</p><ul><li>Allow custom Data Quality rules to be added as the business evolved</li><li>Target data stewards with rule exceptions on a daily basis but also allow for dynamic routing of exceptions to different recipients</li><li>Allow some exceptions to be tagged as Ignored</li><li>Log all runs and exceptions</li><li>Tie DQ rules back to actions laid down by the Data Governance council which allowed for monitoring and continuous improvement</li></ul><p>Ultimately the DQ System became part of a complete Data Management System including a Data Dictionary, Data Governance, Data Flow mapping and Metdata Management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQ17AG3Y4jKakdHw5C7lSA.png" /><figcaption>Complete Data Management Solution</figcaption></figure><p>Let’s <a href="https://medium.com/@julian-ford/developing-a-data-quality-system-598bb1e21925">take a look</a> at how we solved the DQ problem first…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b558f32d04f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying Images using Custom Vision and COVID-19 XRay Data]]></title>
            <link>https://julian-ford.medium.com/classifying-images-using-custom-vision-and-covid-19-xray-data-63f46a3d4349?source=rss-4a4add3367f4------2</link>
            <guid isPermaLink="false">https://medium.com/p/63f46a3d4349</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[x-rays]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[coronavirus]]></category>
            <dc:creator><![CDATA[Julian Ford]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 12:32:54 GMT</pubDate>
            <atom:updated>2020-03-28T14:43:19.274Z</atom:updated>
            <content:encoded><![CDATA[<h3>Classifying X-Ray Images using Custom Vision — in less than 5 minutes</h3><p>Building a working model in less than 5 minutes — code free</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KVJXAlcxygsTJlEF" /></figure><p>I’ve been playing around with Microsoft’s Custom Vision <a href="https://www.customvision.ai/">service</a> for a few months now, looking at how I can use it to quickly tag documents in our work archives without the need for human intervention. Custom Vision is a service (with a <a href="https://azure.microsoft.com/en-gb/pricing/details/cognitive-services/custom-vision-service/">free tier</a>) from Microsoft that allows you to create an image classifier without any code. The service is free for 5,000 training images and 10,000 predictions a month.</p><p>The basic steps are to upload a set of training images, label them and then let the training algorithm work its magic. Once the algorithm is complete you can upload an image for testing — and if all looks good — deploy via API.</p><p>In the last few weeks, we’ve started to see the medical community open up some data sets to the AI community in the battle against the Coronavirus. Now, I am <strong>neither </strong>a professional data scientist or clinician, but when I saw that Dr Joseph Paul Cohen of the University of Montreal had made a set of chest x-rays <a href="https://github.com/ieee8023/covid-chestxray-dataset">available</a> for this type of analysis I thought it would be an interesting potential task for the Custom Vision technology. Since it is cheap and quick to deploy, I thought why not load up the training image sets and see if there was any value in the results? Dr Cohen has a pretty unequivocal statement on his Git site <strong>“Note: please do not claim diagnostic performance of a model without a clinical study!” — so please treat this as an exercise in learning Custom Vision only.</strong></p><p><strong>Setting Up</strong></p><p>The first steps are to create a new project in Custom Vision. I went with the default settings including setting the Domain to General. Once the Project is provisioned, you are presented with a (pleasantly intuitive) workspace where you can start uploading and training your data.</p><p>To load the images it is a simply a case of locating them from the four training folders</p><ul><li>covid (60 images)</li><li>normal (70 images)</li><li>pneumonia_bac (70 images)</li><li>pneumonia_vir (70 images)</li></ul><p>When you select each set to upload, Custom Vision presents you with a dialog box that allows you to tag the set in one go. I simply added the folder label to each set as I uploaded it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*vMhsqf_0gE4pek1WuxQtBQ.png" /></figure><p>Once you’ve uploaded and tagged the full set of training images, you’re ready to train the model. A couple of options are presented when you hit the big green “Train” button. The Quick option runs very quickly (less than a minute). The Advanced option is intended for “challenging and fine grained data-sets”. I think it is safe to say that these images qualify for the latter option. If you go for the Advanced training option you’ll need to set a “compute training budget” that can be between 1 and 24 hours and will notify you by email on completion.</p><p><strong>Findings</strong></p><p>Custom Vision presents its results using conventional <a href="https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall">Precision </a>and <a href="https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall">Recall </a>scores and a measure called AP which is classed as a summary of Precision and Recall at different thresholds. These results are available for the overall model and for the performance of each individual Tag. There’s no fine tuning — for example the option to override the thresholds.</p><p>Mindful of the advice from the provider of the data — I won’t provide specifics of the results. For the Quick Train option they are pretty average. Certainly not something to base a clinical diagnosis on. However, I did find a significant rise in the accuracy of the classifications once I switched on the Advanced options (I set the compute budget to 1 hour). On the 2nd iteration all three of the metrics were 80%+</p><p>Thank you for reading my article. I hope you learned something about Custom Vision and will be inspired to use this free service for the greater good.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63f46a3d4349" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>