<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark Mayo on Medium]]></title>
        <description><![CDATA[Stories by Mark Mayo on Medium]]></description>
        <link>https://medium.com/@markmayo?source=rss-1238eda3a746------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*KNhdW7fQi8dSYBIG.</url>
            <title>Stories by Mark Mayo on Medium</title>
            <link>https://medium.com/@markmayo?source=rss-1238eda3a746------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:15:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@markmayo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[“There are zero bugs in the software”]]></title>
            <link>https://itnext.io/there-are-zero-bugs-in-the-software-8e50d02d298e?source=rss-1238eda3a746------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e50d02d298e</guid>
            <category><![CDATA[quality]]></category>
            <category><![CDATA[quality-software]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[quality-control]]></category>
            <dc:creator><![CDATA[Mark Mayo]]></dc:creator>
            <pubDate>Tue, 22 Feb 2022 00:27:44 GMT</pubDate>
            <atom:updated>2022-02-22T08:06:29.813Z</atom:updated>
            <content:encoded><![CDATA[<p>Zero defects is a fallacy I used to believe in. Then the real world killed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OS1aWH0NXYJh4lEF8VIIlw.jpeg" /></figure><p>Back in university days, I remember meeting a friend who was working at a local software company. He said they had a release coming up (waterfall days), and they’d release with x,xxx bugs.</p><p>I was shocked. Appalled. How could you release software with bugs in it? Bugs you knew about, bugs you have documented, but bugs you’ve decided “meh, we can ship anyway”. Those poor customers!</p><p>Years later, maybe I’ve become more cynical, or more of a realist, but there are many ways to show that no commercial software is defect free. And that can be a good thing.</p><p><strong>Cost-benefit of fixing bugs</strong></p><p>In the ‘real world’ you need to make money. Obviously, customers want features from you, and those (especially newer) features likely have issues/defects. Most developers/QAs would love to fix those. But it comes down to a cost/benefit analysis. Do you spend 6 months fixing all the bugs you know of (and possibly introduce more), or do you get those new features out, earn money from customers, and find out what bugs they actually encounter and need fixed.</p><p>Now obviously, getting the customers to ‘test’ your software is less than ideal, and if it’s got glaringly obvious defects you’re probably going to lose customers and money. So we fix the top defects. The glaringly obvious or dangerous. The Critical vs the Important. And it’s a balancing game — after a release/regression test — is the software ‘good enough’? Do we have a back up plan — a patch, or a hotfix that we can release through a support process, or a workaround that support techs can help with?</p><p><strong>How do you know you have zero bugs? Who fixes the fixes?</strong></p><p>So you’ve tested. Your whole team has tested. You’ve found a ton of bugs. Let’s say you fix them. Now how do you know you’ve not introduced more? Usually you’ll run a smoke test or a regression suite of tests to check critical areas. Ok, you may find a couple and you fix those? How do you know you’ve not made more issues with those fixes? This can go on forever, but the point is clear — your fixing of issues can also create new issues you may not know about.</p><p><strong>How do you know you’ve found all the defects?</strong></p><p>Similarly to the above, you’ve presumably run a set of regression tests. Ideally the devs ran some unit tests too. Great! Are they enough? Do they test EVERYTHING? <br>Let’s say they’re fairly comprehensive. Great — do they check spelling within the app as well? What about UI tests — do you test every browser? Every combination of firmware with device under test? Every possible screen resolution? There’s always more to test, and as a result, always more bugs that could be found. Running static analysis yet? That’ll find a bunch more.</p><p><strong>Aha, but I could prove it!</strong></p><p>There are some lovely theoretical papers I took at uni where you mathematically showed that a function was correct. It was horrendous and took forever for a simple method. It’s perhaps possible to prove something works, but that also assumes no defects in your proof…</p><p><strong>Speaking of</strong>, <strong>no code is an island</strong></p><p>One bug we had at an engineering firm with boards getting thinner and thinner was literally alpha particles coming in and ‘bit flipping’. This may seem like a stretch, but <a href="https://en.wikipedia.org/wiki/Soft_error">soft errors</a> are a problem, and have caused <a href="https://www.businessinsider.com.au/cosmic-rays-harm-computers-smartphones-2019-7">problems in national elections and car manufacturers</a>! Their code may be perfect, but if bits are flipping and they don’t detect it, you’ve got yourself a new defect in your software that wasn’t even your code’s fault, as such.</p><p>Similarly, your code doesn’t run on its own. Odds are it’s sitting on top of an OS, which is using driver code, and you may be transmitting data, or using third party libraries, all of which could cause or have issues that bubble up into a problem in your code.</p><p><strong>The user will never do that</strong></p><p>Years ago we had to deal with a defect with air traffic control software. There was a drawing tool that the ATCs could use to circle planes, draw lines between them and so on. It’d been used for years, without problem, planes all over the screen, lines everywhere. But it was just a simple line drawing utility, nobody ever used it for more than that, right?</p><p>Fast forward to Christmas, where one bored controller decided to draw a reindeer on his screen late at night. The sheer number of lines connected caused an overflow in the graphics library and brought down their system. A scenario never believed possible and highly improbable.</p><p><strong>What is a defect?</strong></p><p>Finally, some defects are unfortunately subjective. After all, it is up to the end user / customer to determine quality of software, and if they consider something a defect, even if you don’t, there’s a disconnect, a flaw in requirements, or a problem with the user experience flow. So even if you think, even if you believe you are at zero defects, the fact no large scale apps in the various app stores have perfect 5-star reviews shows you — something upsets some customer somewhere, every time.</p><p><strong>The good news!</strong></p><p>However, this is not meant to be depressing, cautionary or a warning. It’s actually to provide solace — perfect can be the enemy of good. Don’t aim for perfect — aim to produce great software, every day, quality software that makes customers happy. Just remember, some of them won’t be, and that sometimes - that is still good enough.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e50d02d298e" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/there-are-zero-bugs-in-the-software-8e50d02d298e">“There are zero bugs in the software”</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Simple Home Automation Tweaks That Save Us Hours and Money Every Week]]></title>
            <link>https://markmayo.medium.com/3-simple-home-automation-tweaks-that-save-us-hours-and-money-every-week-16094a6cfaba?source=rss-1238eda3a746------2</link>
            <guid isPermaLink="false">https://medium.com/p/16094a6cfaba</guid>
            <category><![CDATA[robotics-automation]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[home-automation]]></category>
            <category><![CDATA[software-test-automation]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Mark Mayo]]></dc:creator>
            <pubDate>Tue, 31 Aug 2021 05:40:28 GMT</pubDate>
            <atom:updated>2021-08-31T09:24:01.106Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*xw41mx7BpKkQK5yB.jpg" /><figcaption>Automation can save time and money</figcaption></figure><p>Home automation tech gets better and better, and while I do <a href="https://www.linkedin.com/in/mark-mayo-4581582/">software test automation</a> in many of my contracts, I really enjoy playing around with new tech and automation. I was restricted a bit when renting, but now that we’ve bought a house there’s more opportunity to save us hours of effort and often money as well.</p><p><strong>Our robot Rosie</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*hEOrK5737uL7wa5Q.jpg" /><figcaption>Rosie cleans better than I ever will</figcaption></figure><p>This one surprised me when I asked my wife what she wanted for her birthday a couple of years back, and she answered “a mop and a vacuum cleaner”. Sensing a trap I cautiously asked her to clarify, but she was already onto telling me how she’d seen a robot vacuum. Some research and watching fascinating videos of little circular robots cleaning up dirt, we ordered a Roborock S5 online (they’re up to S7 now) and dubbed it Rosie. At the time we were in a two bedroom unit — we now live in a four bedroom house, we both hate cleaning, and well, Rosie takes care of the vacuuming. We also have two dogs that shed hair like nothing else, and Rosie hunts it out — she’ll clean under the bed or the couch, places we’d never vacuum ourselves! <br>If you’re looking yourself, the S7 is out now, and many new variants have self-emptying models as well, to save yourself even more time ;) Incidentally, robot window washers seem to be the next thing we’re looking into…</p><p><strong>Google Home integrations</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*DzvmQxjy46_zpCQR.jpg" /><figcaption>Connecting apps makes them more powerful</figcaption></figure><p>We use Google Home every day to ask about the weather etc, but find it especially useful when cooking — ‘Google, set a timer for 10 minutes’. It’s simple but useful. However, Google also integrates into other apps. For example, we use <a href="https://www.ourgroceries.com/">Our Groceries</a> — a shopping list app, and can just say “Hey Google, add milk to Our Groceries” — it connects through to the app, and adds the item. Yes, you can use Google’s own lists, but since both my wife and I use OurGroceries, it just makes sense.</p><p><strong>Click and Collect Shopping</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*a2rRnxy4xUk0nwSa.jpg" /><figcaption>We eat healthier too</figcaption></figure><p>This is also when it gets better using a shopping list like above. We have our regular weekly shopping items saved in our online supermarket account. Then we open Our Groceries and add anything from there. Takes just a few minutes to do our shopping, we’re not tempted by any instore offers of chocolate or chips or other wasteful spending that isn’t good for us. Then we click ‘on our way’ and go grab the pre-packaged shopping from the supermarket. Never need to enter the store!</p><p>Sidenote: I’m halfway through a script that already adds some stuff from OurGroceries to our supermarket cart automatically, which should make the shopping experience even more seamless for us.</p><p><strong>Heating and Other Appliances</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*6zF1rgKTxtW57mJA.jpg" /><figcaption>Heating? You got it! Lights, yes!</figcaption></figure><p>This depends on your energy plan, but smart switches are amazing, as are IR transmitters. They’re dirt cheap on AliExpress, and can be connected to Google Home/Alexa etc.</p><p>So on our energy plan, we get an hour ‘free’ a day. So you can bet that the second that hour starts, especially mid winter, every heater in the house starts blasting, our electric blankets turn on (if we’re home) and we get an alert to start the dishwasher (because sadly we don’t have a way to press ‘start’ from Google Home, yet. We’re looking at an EV next, and I can guarantee I’ll be setting it up to charge on low-electricity cost hours, and always in the hour of free energy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16094a6cfaba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Kepler GL to Visualise over 35,000 Earthquakes near Christchurch, New Zealand]]></title>
            <link>https://itnext.io/using-kepler-gl-to-visualise-over-35-000-earthquakes-near-christchurch-new-zealand-d207a760e000?source=rss-1238eda3a746------2</link>
            <guid isPermaLink="false">https://medium.com/p/d207a760e000</guid>
            <category><![CDATA[earthquake]]></category>
            <category><![CDATA[keplergl]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[christchurch]]></category>
            <category><![CDATA[new-zealand]]></category>
            <dc:creator><![CDATA[Mark Mayo]]></dc:creator>
            <pubDate>Thu, 27 May 2021 05:13:05 GMT</pubDate>
            <atom:updated>2021-05-27T05:47:26.760Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a step-by-step tutorial showing how I used Kepler GL to visualise 35,000 earthquakes over a period of 14 or so years in or around Christchurch, New Zealand.</p><p>From their site, “<em>Kepler</em>.<em>gl</em> is a data agnostic, WebGL empowered, high-performance web application for geospatial analytic visualizations.”. To me, that’s a way to view data in unique or interesting ways, which gets my attention. So I sat down for an evening to play, to see if I could visualise the earthquakes from New Zealand — specifically Christchurch and Kaikoura in Canterbury, over the last 10 years. For those who are unaware, <a href="https://en.wikipedia.org/wiki/2011_Christchurch_earthquake">Christchurch had some devastating earthquakes in 2010–2011</a>, followed by thousands of aftershocks, and Kaikoura up the coast had <a href="https://en.wikipedia.org/wiki/2016_Kaik%C5%8Dura_earthquake">their own brutal shake in 2016</a>.</p><p><strong>First, the tech:</strong></p><p>— Built with Deck.gl, Kepler.gl utilizes WebGL to render large datasets quickly and efficiently.</p><p>— You can easily drag and drop a dataset, add filters, apply scales, and do aggregation on the fly.</p><p>Built on React &amp; Redux, Kepler.gl can be embedded inside your own mapping applications.</p><p><strong>Getting Started</strong></p><p>Data, we need data. To get all the earthquake data, <a href="https://geonet.org.nz/">Geonet</a> is a government provided data source of earthquakes (as well as volcanic warnings and other such alerts). They’ve helpfully provided a <a href="https://quakesearch.geonet.org.nz/">Quake Search</a>, which allows you to produce CSV files of a particular region, period and all the shakes in that time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Metx2l4rUCU9Bu_k8XaETA.png" /><figcaption>using Quake Search to filter Canterbury quakes</figcaption></figure><p>This provided a download link for a CSV file, the sample contents of which look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3V6u9KFg4mUj_i1zPBZ_Q.png" /><figcaption>so many earthquakes</figcaption></figure><p>Uploading the data</p><p>Clicking ‘Get Started’ on Kepler takes you to <a href="https://kepler.gl/demo">this page</a>, which allows you to load data for the visualisation. This is where we load the CSV file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Ao-fitr6sYyqH0qdLJonA.png" /><figcaption>data time</figcaption></figure><p>Note that this all happens client-side — your data isn’t shared — they’re careful to say ‘load’ and not ‘upload’.</p><p>Now that it’s loaded in, you’ll be presented with a map of…California. This is slightly annoying until you realise that the data isn’t really being used yet, it’s just loaded in. So to prepare, we’ll move the map to New Zealand, and set the base map to Satellite, so it’s not just a dark map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A4uapj-SeAM3MsdplojBBg.png" /><figcaption>spying on hobbits from space</figcaption></figure><p>Under the layers tab, we can see the csv file is loaded, and has 31k+ rows (think, each of those is a shake!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*BhFZeo3BbI5v-hd0G2AW7A.png" /></figure><p><strong>Layering up</strong></p><p>At this stage, we have a map, we have data loaded, now we want the data on the map. As a layer. So we click “Add Layer”, give it a name (“shakes”), and choose a basic point type. Kepler GL now helpfully asks for you to map fields from the csv file to their own Latitude, Longitude and Altitude fields. The fields from the csv file are conveniently shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*967PgrtiaVtjhTAxQGeHLg.png" /></figure><p><strong>Detour.</strong></p><p>At this point you may notice other Basic types and want a play. For example, hex bins gives us essentially a heatmap of the quakes immediately:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*GIVm5QspOmUkG7PsZaS98Q.png" /></figure><p>But we’ll stick with the basic point types. It’s worth noting though while we’re looking at the above picture — it looks like some hard lines for boundaries — but that’s because from Geonet I requested ONLY quakes from Canterbury, so they stop at the borders with Otago, West Coast, Marlborough etc. It’s also cool to see how far out at sea they can be detected!</p><p>So back to points, which looks a little simpler:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*AiIO0AUyAuCB-KHmnwYRhA.png" /></figure><p><strong>Time for Time!</strong></p><p>Great! But that doesn’t really show us when it happened, just…where. So we need time added somehow. This is where ‘Filters’ come in. Under the ‘Filters’ add a Filter, choose the ‘origin time’ and … wow, now we have a timeline!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyyRh7BlXezI-UcKuw9WQQ.png" /><figcaption>even at this point, the two big events are very visible on the timeline</figcaption></figure><p>You can clearly see the main events time-wise, with the fading aftershocks over time.</p><p><strong>Moving through time</strong></p><p>This is nifty, and if you look at the timeline you can see two bars. You can move those to set a time window width:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKOPPnkurd4p_t4MqFid8Q.png" /></figure><p>And then hit play to see it all animating as time plays!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*du8dwKzQJ3899ZYjnQn-jg.gif" /></figure><p>That’s cool, but it’s hard to tell the earthquakes apart. Which was the ‘big’ September quake? Where did the Kaikoura swam start?</p><p><strong>Colouring based on magnitude</strong></p><p>This is where the custom features of points come in. You can specify the colour to change based on a field — for this I’ll choose magnitude, and the radius to adjust based on this field too, although because of the wide variety (some ‘big’ shakes are too deep to feel), I’ll use the magntitude type here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*DtPMFlt3y2SJ3tlgvze7Hg.png" /><figcaption>ooh, colours!</figcaption></figure><p>Time to hit play again and see how that looks!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4FbBQKsFORaIYK836pAl-g.gif" /><figcaption>Animated, with colours and relative radius adjustments</figcaption></figure><p>And there we have it. The whole process took me a couple of hours because I was playing with all the settings and writing this up. But, if you know what you’re doing, you can animate over 10 years of earthquake data in under 5 minutes, and look good doing it!</p><p><strong>Final thoughts</strong></p><p>This is a powerful platform that I’ve only just begun to play with, but there are so many opportunities for beautiful data to easily show huge amounts of information clearly and cleanly. Take a look at the arc graph of commutes in England and Wales, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*HfCDvrzET4qKdG0EOQV-pg.png" /></figure><p>If your city/town’s council is like mine, they probably have some public data that is just sitting there, waiting to be visualised. Why not give it a try now?</p><p><em>Note: The author is not associated with Kepler GL or GeoNet, but does appreciate their content very much.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d207a760e000" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/using-kepler-gl-to-visualise-over-35-000-earthquakes-near-christchurch-new-zealand-d207a760e000">Using Kepler GL to Visualise over 35,000 Earthquakes near Christchurch, New Zealand</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s a Trap! Avoid Focusing on Vanity Metrics in Software Testing]]></title>
            <link>https://itnext.io/its-a-trap-avoid-focusing-on-vanity-metrics-in-software-testing-7a627c7848?source=rss-1238eda3a746------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a627c7848</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[vanity-metrics]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[quality-software]]></category>
            <dc:creator><![CDATA[Mark Mayo]]></dc:creator>
            <pubDate>Sat, 24 Apr 2021 04:52:55 GMT</pubDate>
            <atom:updated>2021-04-26T07:05:51.632Z</atom:updated>
            <content:encoded><![CDATA[<p>Several years ago, a new PM came onboard a team I was on and introduced me to the term “Vanity Metrics”. At the time we were excitedly watching the number of downloads of our new app, and she bluntly said that didn’t matter, and that really, conversion and retention of users were far more important.</p><p>Now obviously you can’t get to conversion and retention without those initial downloads, and briefly being #1 on the app store certainly helped, but it did make me more aware of how some numbers really are ‘vanity’ and don’t contribute towards your team’s goal of shippable quality. Customers don’t care how many downloads you had in the past hour. Customers care if it does what they want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cD0JhxPvjtJ2zT_bd7A2eQ.jpeg" /><figcaption>Vanity metrics — not always a true reflection of quality</figcaption></figure><p>As a Quality Engineer / Test Lead / QA Lead or whatever we call ourselves these days (vanity name?), time and experience has also shown me a number of metrics within testing/quality that, when it comes down to it, are purely (or nearly purely) for vanity.</p><p><strong>Test Coverage Percentage</strong></p><p>In your latest software project, be it unit tests, integration or even automated UI tests — do you measure test coverage? Don’t misunderstand me, coverage is GOOD, it’s useful to write tests. However, striving for what one manager claimed to me was an industry standard of 80%, and another wanted 100% on everything is, frankly, not that ‘useful’. You can have 100% of coverage and still not find bugs. Take this code sample:</p><pre>double divide (double a, double b)<br>{<br>    return a / b;<br>}</pre><p>We can write a unit test that passes 100% coverage of this. That doesn’t mean it’s bug free though. If we don’t test edge cases, such as where b is 0.0, it’s immediately evident that it may not be caught.</p><p><strong>Total Number of Bugs</strong></p><p>What’s the ideal number of bugs? Zero! But if you’ve worked on any software project, you (hopefully) are aware that’s next to impossible. Consider the quality triangle of time, cost and quality. Pick two. If you want high quality, and zero bugs is REALLY high quality, it’s either going to take forever or cost a fortune.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VZNIOlGTEQxU_4Pbq3CjQ.jpeg" /><figcaption>Can you spot the bug?</figcaption></figure><p>Now that doesn’t mean it’s not a leading indicator. If the number of bugs found is increasing over time, especially those of high severity, it’s worth investigating if they’re being found in a particularly complex module that perhaps needs further attention. Or if affecting a particularly valuable customer, it needs to be considered. Similarly if the cost risk of bugs is high (think a pace-maker or defibrillator). But at some point you need to make money, and ship software. And there will be bugs still in it. The only way to avoid more bugs is to not ship more software, and that’s not a solution.</p><p>One further risk is behavioural change. I’ve been at a company where the Project Manager literally told us to stop raising bugs until the release was out, so that it wouldn’t get delayed any more. Then he was congratulated for an on-time release, and the engineering team was chastised for the resultant bugs that were discovered…</p><p><strong>Total Number of Tests</strong></p><p>I’ve worked at a company where there was a team of 20 manual testers executing 4,000 test cases every release. And this was on a media set top box, the tests were NOT generally short. Rapid releases were not possible.</p><p>The “solution” was to hire a team of test automation engineers, to automate the lot.</p><p>The downside? Most of those tests were frankly, useless. Many were nested inside other tests, or were incredibly, incredibly unlikely to fail. The amount of effort invested, versus having a set of smoke tests of critical features was very arguably a misuse of time, for devices that could be updated over the net in the event there was an issue. Furthermore, there were 15–20 different variants of the boxes, but aside from screen resolution and boot up sequences, there was no huge case for running all tests on all devices. The <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto Principle</a> is always something to consider here.</p><p>Along those lines, say you have 100 or 1000 tests. If they’re not the right tests — testing the critical pieces, or the pieces most likely to break, they’re far less useful than 5–10 smoke tests that test critical functionality.</p><p><strong>Test Pass Rate</strong></p><p>Like the total number of tests, test pass rate -while an indicator, and certainly a useful guide to keep an eye on trends, should not be used to say “we have no bugs since 100% of our tests pass”. While tests suddenly dropping to 70% pass rate should raise eyebrows and cause concern, 100% pass of tests that don’t test the critical stuff is again, not as useful as it could be.</p><p><strong>Static Analysis Metrics</strong></p><pre>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <br>Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)</pre><p>Static analysis tools are wonderful. They find issues automatically, they can help with coding style conformity, and at a company where we methodically fixed 16,000 warnings in our C code base, we found a lot of critical bugs that were being missed.</p><p>However, that warm, happy feeling when pylint scores your code as 10/10? That’s nice, but again, correctly written, coding-standards adherent code doesn’t mean that code is doing the right thing. It’s another tool in our tool belt, but shouldn’t be used to convey ‘quality’ in terms of delivering the correct behaviour in the product.</p><p><a href="https://tim.blog/2009/05/19/vanity-metrics-vs-actionable-metrics/">Eric Reis posits</a> that “the only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions”. Vanity metrics look good, but do they help move your business forward? Look instead for actionable metrics —ones that indicate where you should focus your testing, where you can add value and quality to the product.</p><p><strong>Examples of Actionable Metrics in Testing</strong></p><p>Recently changed code — if all the changes in this release are in module x, there’s potentially little benefit to testing module y that hasn’t been touched since 2015 and has no inter-dependencies. Perhaps target your testing, or new quality initiatives — at module x.</p><p>Customer issues — if feature Y is causing the helpdesk 90% of the issues — focus on feature Y. Find out where the issues are, the commonalities, and attempt to break down the biggest problems and what may be causing them. Accelerating the shipping of quality product by reducing these issues will reduce the customer complaints, and increase customer happiness. Customer happiness IS quality!</p><p>Remember — if you can measure it, you can manage/improve it! Take those actionable metrics and start working on them now, and late at night after your next commit, when you see that perfect linting score, take a moment, smile, and realise while that it doesn’t matter, it’s still nice to see good news.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a627c7848" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/its-a-trap-avoid-focusing-on-vanity-metrics-in-software-testing-7a627c7848">It’s a Trap! Avoid Focusing on Vanity Metrics in Software Testing</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>