<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ian Makgill on Medium]]></title>
        <description><![CDATA[Stories by Ian Makgill on Medium]]></description>
        <link>https://medium.com/@ianmakgill?source=rss-46fe835343c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xDSEkXryND7WV-sr.jpeg</url>
            <title>Stories by Ian Makgill on Medium</title>
            <link>https://medium.com/@ianmakgill?source=rss-46fe835343c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:06:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ianmakgill/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Opening up tender notice data]]></title>
            <link>https://medium.com/understanding-spend/openopps-opening-up-tender-notice-data-a1a42cd3357d?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1a42cd3357d</guid>
            <category><![CDATA[selling-to-government]]></category>
            <category><![CDATA[open-contracting]]></category>
            <category><![CDATA[tenders]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Mon, 25 Apr 2016 11:07:13 GMT</pubDate>
            <atom:updated>2016-07-13T13:40:07.627Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKAuC3EbDJfYqNRvTqM1vQ.jpeg" /></figure><p>Our new project is to open up tender notice data. There are already a bunch of services offering aggregated tender notices, but we feel many of them are missing a crucial point: that this data will increasingly become open.</p><p>Read more about our project on our blog: <a href="https://openopps.com/blog/post/10/opening-up-tender-notice-data/">https://openopps.com/blog/post/10/opening-up-tender-notice-data/</a>.</p><p>If you’d like to sign up for our beta test, or just to find out a bit more about what we’re doing, please get in touch via the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1a42cd3357d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/openopps-opening-up-tender-notice-data-a1a42cd3357d">Opening up tender notice data</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reviewing IT outsourcing contracts in Government.]]></title>
            <link>https://medium.com/understanding-spend/reviewing-it-outsourcing-contracts-in-government-5d0fc6f05556?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d0fc6f05556</guid>
            <category><![CDATA[outsourcing]]></category>
            <category><![CDATA[news]]></category>
            <category><![CDATA[transparency]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Fri, 15 Jan 2016 18:15:30 GMT</pubDate>
            <atom:updated>2016-06-06T11:22:06.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYLkBbXyCrPAAHQjxCPxvg.png" /><figcaption>The Spend Network “Teapot” diagram of how procurement data should be published.</figcaption></figure><p>Computer Weekly have <a href="http://www.computerweekly.com/news/4500270961/Government-reviews-IT-contract-strategy-as-opaque-outsourcing-deals-set-to-expire?platform=hootsuite">reported</a> that the Cabinet Office is planning to review the long-term IT outsourcing deals let by Government between 1995–2008, a background document seen by Computer Weekly states:</p><p><em>“The opaque nature of service delivery hides the fact that many [of these contracts] are not good value for money.”</em></p><p>We’ve analysed this market <a href="https://medium.com/understanding-spend/spend-network-reveals-trends-in-government-it-spending-7a06df6442a#.ey4i143nv">here</a>; despite public announcements that the market needed reforming and the cuts to spending the market for outsourced IT services is still growing.</p><p>This market, along with many others, would benefit from greater transparency of contracting, not just details of who won what but also how a contract progresses over time, including amendments to the contract and the spend going to the supplier against that contract.</p><p>A genuinely transparent view of contracts would have given all of us clear sight of the contracts that were not delivering and would also have made it clear which suppliers performed the best, whilst highlighting the suppliers most likely to fail.</p><p>Which is why the work of <a href="http://www.open-contracting.org/">www.open-contracting.org</a> is so important, it makes it very easy to publish meaningful data on contracting, we’ve written about open contracting <a href="https://medium.com/understanding-spend/why-the-open-contracting-standard-matters-2f5c6672ccbf#.3f6gg59z8">here</a>.</p><p>Opaque purchasing obstructs fair competition and rewards failure, true transparency encourages competition and reduces uncertainty in tenders. If Government wants to save money, they should start by declaring that the details of all contracts over £10m should be published using the open contracting data standard.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d0fc6f05556" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/reviewing-it-outsourcing-contracts-in-government-5d0fc6f05556">Reviewing IT outsourcing contracts in Government.</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why the Open Contracting Standard matters.]]></title>
            <link>https://medium.com/understanding-spend/why-the-open-contracting-standard-matters-2f5c6672ccbf?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f5c6672ccbf</guid>
            <category><![CDATA[contracts]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[open-contracting]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Mon, 11 Jan 2016 17:04:04 GMT</pubDate>
            <atom:updated>2016-07-13T13:47:41.353Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LwLluoqyyNosfERDbXEUrQ.jpeg" /><figcaption>Unfinished Monorail Suspension Bridge, Malaysia (<a href="https://www.flickr.com/groups/urbex-malaysia/">https://www.flickr.com/groups/urbex-malaysia/</a>)</figcaption></figure><p><a href="http://www.open-contracting.org/">Open Contracting</a>, a project to encourage the open publication of public sector contracting data, is establishing a data standard for contracting and procurement data (<a href="http://standard.open-contracting.org/">http://standard.open-contracting.org/</a>). Much of this work has been done by the prolific and highly talented <a href="https://twitter.com/timdavies">Tim Davies</a>, who is breathing life into the standard.</p><p>As more and more public money is directed to companies, it becomes increasingly critical to ensure this money actually meets the needs of citizens; better data has the chance to deliver new insights into the trillions of dollars spent by governments every year.</p><p>Our blog sets out four reasons why we’re strong advocates of the standard: <a href="https://openopps.com/blog/post/11/why-the-open-contracting-data-standard-matters/">https://openopps.com/blog/post/11/why-the-open-contracting-data-standard-matters/</a>.</p><p><strong>Update</strong>: details of the Open Contracting Partnership’s funding arrangements are here: <a href="http://www.open-contracting.org/our_transparency">http://www.open-contracting.org/our_transparency</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f5c6672ccbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/why-the-open-contracting-standard-matters-2f5c6672ccbf">Why the Open Contracting Standard matters.</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why is it so hard to find public sector tender opportunities?]]></title>
            <link>https://medium.com/understanding-spend/why-is-it-so-hard-to-find-public-sector-tender-opportunities-d19777913f30?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d19777913f30</guid>
            <category><![CDATA[procurement]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[selling-to-government]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Mon, 11 Jan 2016 15:06:16 GMT</pubDate>
            <atom:updated>2016-07-13T13:42:05.499Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*TA_eJiXWj7n-ILTs3eXeAA.jpeg" /><figcaption>US Navy Archives Personnel, Bess Glenn in foreground, August 14, 1942. (<a href="https://catalog.archives.gov/id/3493249">catalog.archives.gov/id/3493249</a>)</figcaption></figure><p>Why is it so hard to find the right public sector tender opportunity for your business? We’ve been looking at some of the reasons and have come up with <a href="https://openopps.com/blog/post/12/why-is-it-so-hard-to-find-public-sector-opportunities/">these six key points.</a></p><p>In the meanwhile, we’d love to hear about what you’d like to see that could help you when you’re looking for tenders. Either comment below or get in touch directly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d19777913f30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/why-is-it-so-hard-to-find-public-sector-tender-opportunities-d19777913f30">Why is it so hard to find public sector tender opportunities?</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spend data and personal information — a guide for publishers]]></title>
            <link>https://medium.com/understanding-spend/spend-data-and-personal-information-a-guide-for-publishers-37650bf4bc53?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/37650bf4bc53</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:21:22 GMT</pubDate>
            <atom:updated>2015-12-23T16:22:37.737Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/199/1*n89RLf4cjvkR5SLqFt-7bA.jpeg" /><figcaption>“Aclu-v-ashcroft-redacted” by Transferred from en.wikipedia; transferred to Commons by User:Innotata using CommonsHelper.. Licensed under Public Domain via Wikimedia Commons — <a href="https://commons.wikimedia.org/wiki/File:Aclu-v-ashcroft-redacted.jpg#/media/File:Aclu-v-ashcroft-redacted.jpg">https://commons.wikimedia.org/wiki/File:Aclu-v-ashcroft-redacted.jpg#/media/File:Aclu-v-ashcroft-redacted.jpg</a></figcaption></figure><p>First published: 22nd August 2013</p><p>In a broad sense, the law is quite simple: you can’t publish anything that might identify an individual. Complying with the law is less straight forward. It would be nice if we could just search our output files for all the occurrences of “Mr.”, “Mrs.” or “Miss” and redact accordingly, but personal data is often quite difficult to find and successfully repressing the data requires diligent checking and good organisational practices.</p><p>To complicate matters, many companies and organisations use personal names as their identifiers; there are nearly 1,700 companies in the Companies House register that include “Mr” in their names, and there are still more companies with titles that could be confused with personal names, for example “Lisa Watkins Ltd” would likely trigger a high score in any automatic matching process.</p><p>On top of that, there’s the work of Doctors, Barristers and other professionals; these individuals aren’t incorporated but any transactions made to these individuals may well have been placed in return for commercial services. Even if these transactions were commercial, there is a chance that the data could be used to identify an individual, and therefore this data should be suppressed.</p><h3>What to look for in spending data</h3><p>The primary source of personal data is in the name field from the transaction. Ensuring that this data has been cleansed is likely to ensure that most of your potential breaches have been resolved, however, sometimes transactions can include privacy sensitive information in the “description” field of the invoice which could include name, case file or social security number. For this reason, all columns in the dataset should be analysed.</p><p>Columns to pay special attention to:</p><ul><li>Name</li><li>Address</li><li>Narrative / description</li><li>Department</li><li>Category</li></ul><h3>Identifying names</h3><p>There are a number of typical indicators that the payment is made to an individual:</p><ul><li>Use of “Mr”, “Mrs”, “Miss” etc at the start of the name field. Remember to also look for Military and other monikers, a list of these abbreviations can be found here: <a href="http://www.codeproject.com/Questions/262876/Titles-or-Salutation-list">http://www.codeproject.com/Questions/262876/Titles-or-Salutation-list</a></li><li>Use of an initial followed by a name, e.g. “D. Harrison”</li><li>Payment is not associated with an invoice</li><li>The payment instruction details specify a refund or specifics such as “Direct Payment”</li></ul><p>It is possible to use a procedure called “pattern matching” that can highlight any items in a database that match a certain pattern of characters. Using these routines, it is possible to highlight entries that may include personal name data.</p><h3>How to address the problem</h3><p>The most important field in spending data is the “supplier name”, as this will most likely contain the most valuable personal data, but publishers need to be aware of the potential for identities to be triangulated from additional data, such as narratives and transaction descriptions. It is therefore necessary to review all fields in a dataset.</p><h3>Step 1: Flagging at source</h3><p>Evaluating the entries in the “supplier name” field to assess whether the data includes an individual’s name is an inefficient and largely ineffective method for flagging personal data breaches. The most effective method of suppressing publication of this data is to ensure that personal data is flagged as such when payments are made. Every Department in the Council will have a legitimate reason for issuing payments to an individual, so it is advisable to establish an organisation-wide protocol for flagging payments to individuals. Most Councils have a standard procedure for co-ordinating payments that includes raising a purchase order (PO). Users that generate POs for payments to individuals should simply append a predetermined code to the recipients’ name, which can then in turn be picked up by the IT department so that the data can be suppressed before publication.</p><h3>Step 2: Monitoring the Supplier Database</h3><p>The data used in payment reports will originate from the organisation’s finance system, which includes a record of suppliers to whom payments are made. To prevent fraud, there is normally a strict procedure for adding suppliers to this database. This procedure should include a requirement that any personal data is flagged for later suppression, effectively creating a second filter to prevent personal data breaches. It is important to note that this procedure should not form the primary prevention mechanism, as the name in the supplier field is often simply not enough to identify whether a payment is for an individual or not, however, this step should be used in order to flag any payments that appear to be to an individual.</p><h3>Step 3: Pre publication reviews</h3><p>All data that is to be published should go through a two-stage pre-publication review. The first part should include an automated review of the data, where a script is used to select entries that look like they may include personal data. The scripts should be capable of screening for the following:</p><ol><li>Pre-determined flags that show that a payment is to an individual</li><li>Common pattern matches used in names (e.g. “Miss”)</li><li>Names of payees that are known to the Council, (e.g. they have been identified as personal payments before)</li><li>Any specific funding codes that are likely to indicate that a payment is going to an individual (e.g. Social Care Direct Payments).</li></ol><p>Once data has been selected, it should be reviewed manually to confirm whether the data provides sufficient information to identify an individual. There is no need to manually review data that has already been flagged as an individual by the Department making the payment, or has been identified as an individual through previous work to prevent breaches. Data that has been flagged because it triggered a pattern match or through a funding code should be checked manually.</p><p>Once a data line has been identified as a payment to an individual, then the key pieces of text should be stored as a record that allows the Council to suppress that data in the future, (see step 3 above) and for use in the automated flagging procedures for ensuing months.</p><p>A further, manual review needs to be undertaken to ensure that personal payments are not missed. Typically, payments to individuals will involve small sums (relative to the amount paid to companies) in a small number of transactions. Therefore, ordering the data by the lowest value transactions and then looking through the payment lines to try and identify any payments to individuals. Care should be taken to ensure that reviewers are aware of the potential for foreign names to appear in the text and steps should be taken to ensure that a foreign language review is undertaken where necessary. Although this work sounds onerous, in actuality it is a very small task; we have reviewed two years of transactions by over 30 organisations in just two days, a regular monthly review should occupy just minutes of staff time, not hours.</p><h3>Step 4: Removing data</h3><p>You should never delete whole rows of data, instead you should overwrite any data that might constitute a data breach. In particular, there should be no reason for removing either date or value fields from a transaction as these cannot be used to identify an individual. Additional data such as department and narrative information should only be overwritten if it contains data that could identify an individual. Councils should also take steps to detail why the data has been censored. For example, it would be suitable to replace a person’s name with the following “Redacted to comply with the Data Protection Act”. Providing this additional information gives the data user a good understanding of the nature of the underlying data and advises data consumers that the Council is undertaking its role as a data producer responsibly.</p><h3>What NOT to do: Examples of overly restrictive disclosure procedures</h3><p>Data privacy issues should never act as justification for avoiding public disclosure. A good example of this might be the suppression of spend data to a Barrister, on the grounds that the data could be used to identify the individual. Whilst it is right to suppress personal data, the Barrister will be the member of a Chamber of Barristers and any transaction could be allocated to the Chambers rather than to the individual Barrister. The same applies to payments to Doctors, payments should be allocated to the Doctor’s practice, not to the individual Doctor concerned.</p><p>We have noticed a worrying trend of Councils to redact data on the basis that the payment was made to a “sensitive supplier”, or that the transaction was “commercially sensitive”. The LGA cites the issues of arbitration, commercial confidence and transactions relating to the underwriting of debt as suitable reasons to redact spend information (see below).</p><blockquote><em>No Examples of transactions that may be excluded from publication Reason Redacted or Excluded</em></blockquote><blockquote><em>1 Salary payments to staff (including bonuses), except when published under the senior salary scheme. These will be published separately Personal information protected by the Data Protection Act Excluded</em></blockquote><blockquote><em>2 Pension contributions (excluding service charge) and National Insurance Contributions Personal information protected by the Data Protection Act Excluded</em></blockquote><blockquote><em>3 Severance payments Personal information protected by the Data Protection Act Excluded</em></blockquote><blockquote><em>4 Payments to individuals from legal process — compensation payments, legal settlements, fraud payments Personal information protected by the Data Protection Act Redacted</em></blockquote><blockquote><em>5 Competition prizes — where a normal part of operations Personal information protected by the Data Protection Act Redacted</em></blockquote><blockquote><em>6 Settlements made with companies as an arbitration which is conditional on confidentiality Commercial-in-confidence — exempt under FOI Redacted</em></blockquote><blockquote><em>7 Potential betrayal of a commercial confidence, or prejudice to a legitimate commercial interest Very rare and will need to be justified Excluded</em></blockquote><blockquote><em>8 Transactions relating to the financing or underwriting of debt e.g. purchase of credit default swaps Outside the definition of expenditure for this purpose Excluded</em></blockquote><blockquote><em>9 Provisions or promises to pay not yet realised Excluded</em></blockquote><p>If Councils are to be genuinely open about their activities, there should be no suppression of payments to any commercial entity listed in Companies House (or any other Companies register). Individual companies can choose to protect their tender submissions and other commercial data that they send to the Council from release under the Freedom of Information Act, however, there really should be no position where the simple fact of a payment is seen as commercially sensitive. Whilst the suppression of data on payments to companies as part of settlements is a common legal practice, Councils should do their utmost to resist activity which is so protectionist and undemocratic. Perhaps financing payments should be recorded separately, but it is hard to justify their exclusion from open publication.</p><h3>Privacy for individuals providing commercial services</h3><p>In 2010, the European Court of Justice (ECJ) ruled that mandatory disclosure of names of German recipients from EU farm subsidies amounted to a breach of their privacy. The ECJ decision has not been overturned since and has led to a substantial decrease in access to overall EU spending, due to the large share that the Common Agricultural Programme (CAP) occupy within the total budget. Whilst we recognise the need to protect individuals in receipt of non-commercial payments from Government, e.g. Housing Benefit or Pensions payments, Spend Network believes that all commercial payments by Government bodies should be published openly. If individuals who receive monies from Government in exchange for services, or as part of a grant for a commercial enterprise need to remain anonymous, they can always choose to reject that payment. However, much like the Barristers, these farms are commercial entities and the farms should be named on each transaction.</p><p>The world is just getting used to the existence of open spending data, but as the data attracts increased usage Governments will come under greater pressure to create dependable, consistent and accurate datasets. Now is the time to ensure that your data is correctly presented, free of data that may breach regulations and can be used by organisations like spendnetwork.com.</p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/spend-data-and-personal-information-a-guide-for-publishers.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37650bf4bc53" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/spend-data-and-personal-information-a-guide-for-publishers-37650bf4bc53">Spend data and personal information — a guide for publishers</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How accurate is our data?]]></title>
            <link>https://medium.com/understanding-spend/how-accurate-is-our-data-a506327fba6a?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a506327fba6a</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[spend-analysis]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:15:11 GMT</pubDate>
            <atom:updated>2015-12-23T15:49:03.928Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*h-LzpW_MXyE_wScZY-NxJg.png" /><figcaption>A sample of Spend Network’s data on North Somerset</figcaption></figure><p>First published: 29th of November 2013</p><p>Our data is actually a perfect representation of the data published by government; the only problem is that sometimes that data isn’t as good as it should be. So how good is our data really? We’d say its pretty good actually. Is it perfect? No. Could it be better? Without a doubt.</p><p>So where are the problems and what do we do to resolve them? Below, we list some of the problems we see with data and what we do to curate better data:</p><h3>Erroneous transactions</h3><p>In some cases we simply have erroneous transactions, something that you instinctively know is wrong. In one case, we found a transaction with Poundland (a high street retailer where every item is priced at £1) for £16,000. On other occasions, the data can be less obvious, but once you start to investigate, something just looks wrong.</p><p>The problem with these data elements is that in order to find them you really have to see the data visually, to check it and evaluate it so that you can spot anomalies. That takes time and effort and we can never hope to find all of the errors in data published by Government, but we’ll always be looking and when we see something that doesn’t look right, we’ll go back and check it.</p><h3>Poorly labelled transactions</h3><p>In some cases, we face perfectly valid transactions that have just been poorly labelled. A good example is a number of payments made by HMRC to a supplier simply called “Aspire”. This is in fact the HMRC’s new IT upgrade programme being run by CapGemini, which accounts for £3.7bn. The problem is that we ended up matching these payments to a small charity called ‘Aspire 2’ simply because it was the best matched organisation in our database. Our routine checks identified that this didn’t look like a strong match and we had to research the project to find the right match.</p><p>Again, these transactions are numerous and weeding them out requires manual interventions into the data as well as a good knowledge of Government purchasing.</p><h3>Personal data</h3><p>In some circumstances we see personal data creeping into the feeds from Government. Obviously this is a serious matter, so we go out of our way to make sure we’re suppressing this data, even if Government is not. We’ve developed routines to spot names and we follow this up with some additional manual matching to try and suppress all of the names we see in data.</p><h3>Redacted data</h3><p>Finally, we face redacted data, and frankly there is far too much of it. In the case where a payment is being made to a named individual, it makes sense to redact this data, however, this should not be an excuse for redacting supplier names. Where supplier name is also the name of a company then it is fine to use that data, as it is a reference to a company not to an individual. Where a payment is to an individual professional, e.g. a Doctor, or a Barrister, or a taxi driver, then it is right to suppress their name, but a clear indication should be given as to why the data is redacted, and it should also be possible to give an indication of the nature of the payment. In the case of a payment to a Doctor, it is reasonable to allocate the payment to the Doctor’s surgery for example.</p><h3>Conclusion</h3><p>At the risk of channelling Donald Rumsfeld (never a comfortable thing to do) we face known errors, unknown errors and entries where it isn’t possible to know if there’s an error. We use matching, cleansing and checking to ensure that our data is as clean as it can be, to create a usable tool for research and intelligence.</p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/how-accurate-is-our-data.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a506327fba6a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/how-accurate-is-our-data-a506327fba6a">How accurate is our data?</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On data curators]]></title>
            <link>https://medium.com/understanding-spend/on-data-curators-ea657544f975?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea657544f975</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:12:20 GMT</pubDate>
            <atom:updated>2015-12-23T15:53:57.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*nA7FRp1J5zpvZ2p70Q8wFA.jpeg" /><figcaption>“NDOC magnetic tape library” by NODC — <a href="http://www.nodc.noaa.gov/General/anniversary50.html">http://www.nodc.noaa.gov/General/anniversary50.html</a> (via email from NOAA webmaster). Licensed under Public Domain via Wikimedia Commons — <a href="https://commons.wikimedia.org/wiki/File:NDOC_magnetic_tape_library.jpg#/media/File:NDOC_magnetic_tape_library.jpg">https://commons.wikimedia.org/wiki/File:NDOC_magnetic_tape_library.jpg#/media/File:NDOC_magnetic_tape_library.jpg</a></figcaption></figure><p>First published: 31st December 2013</p><p>Someone recently referred to the role of Spend Network (and other open data businesses) as data curators.</p><p>It is quite an elegant term for what we do.</p><p>We are a business, we exist to make money. We take a raw material (openly published government data) and manipulate it so that it can be used for insights and intelligence. We sell that insight to government and suppliers so that they can make better decisions about buying and selling.</p><p>In order for us to sell data it has to reach a professional standard that means it can be analysed and evaluated. Poor data causes problems, so we try to find the errors and weed them out. So, when Tower Hamlets spends three times more than average one month, we look to see why. When the same Council publishes transactions with Badenoch &amp; Clarke for 23 months out of 24, it also pricks our curiosity. In both cases we’ve emailed the Council and asked them to check their records.</p><p>By using the data ‘in anger’ we spot anomalies, we find errors and we challenge publishers to put mistakes right. This work is necessary for our clients, but it also helps to make publishers more aware of their responsibilities when publishing data. Hopefully, knowing that they’re being watched will improve the quality of the data being published for everyone.</p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/on-data-curators.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea657544f975" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/on-data-curators-ea657544f975">On data curators</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open data institute lecture]]></title>
            <link>https://medium.com/understanding-spend/open-data-institute-lecture-8be28be58672?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8be28be58672</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[procurement]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:09:00 GMT</pubDate>
            <atom:updated>2015-12-23T14:57:42.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*RKd3_xI2GjOC7yr9hHs7Vw.png" /></figure><p>First published: 18 January 2014 |Tags <a href="http://spendnetwork.github.io/blog/tag/odi.html">ODI</a></p><p>Here’s the slides and the <a href="https://soundcloud.com/theodi/odi-fridays-can-open-data?utm_source=feedburner&amp;utm_medium=feed&amp;utm_campaign=Feed%3A+odifridaylunchtimelectures+%28Open+Data+Institute%3A+Friday+Lunchtime+Lectures%29&amp;utm_content=FeedBurner">sound cloud page</a> from our <a href="http://www.theodi.org/">ODI</a> Lunchtime lecture.</p><p><a href="http://www.scribd.com/doc/201904681/Can-open-data-improve-government-procurement">Can open data improve government procurement?</a> by <a href="http://www.scribd.com/OpenDataInstitute">Open Data Institute</a></p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/open-data-institute-lecture.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8be28be58672" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/open-data-institute-lecture-8be28be58672">Open data institute lecture</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[London recruitment data]]></title>
            <link>https://medium.com/understanding-spend/london-recruitment-data-3fbce385d0c3?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fbce385d0c3</guid>
            <category><![CDATA[recruitment]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[government]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:06:05 GMT</pubDate>
            <atom:updated>2015-12-23T15:57:15.979Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*O7yb43QlbWNKU9nlvdyn5w.png" /><figcaption>London boroughs’ spending on agency staff for FY 2011–2012 and FY 2012–2013</figcaption></figure><p>First published: 18th January 2014</p><p>An updated version of our recruitment data for London (FY 2012–13).</p><p>In our eyes, the most interesting chart is the growth chart on slide 3. Despite the need for double digit budget reductions in Local Government, the demand for temporary staffing is grew by more than 4% during the year.</p><p>If anyone is interested in this work, we can complete this analysis for the whole of England across central and local government.</p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/london-recruitment-data.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fbce385d0c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/london-recruitment-data-3fbce385d0c3">London recruitment data</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spend Network API]]></title>
            <link>https://medium.com/understanding-spend/spend-network-api-c3dd0b45c62e?source=rss-46fe835343c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3dd0b45c62e</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Ian Makgill]]></dc:creator>
            <pubDate>Tue, 22 Dec 2015 08:05:05 GMT</pubDate>
            <atom:updated>2015-12-23T13:12:00.358Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s66y0YoMDzKAcGt3tmF5ew.png" /><figcaption>Spend Network’s API: <a href="http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov">http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov</a></figcaption></figure><p>First published: 18th February 2014</p><p>The Spend Network database contains records of transactions (that is spending) between UK public sector organisations and their suppliers. The primary fields in each transaction record describe: a) transaction date, b) buyer, c) supplier, and d) amount spent.</p><p>In its current version, the Spend Network API enables developers to list and to view transactions for a single buyer. There is also a “discovery” resource which returns a list of the UK buyers for which Spend Network currently holds transactions.</p><h3>VIEWING THE API</h3><p>The API data can be browsed in HTML format or in JSON format. When viewed in HTML format it becomes simple to browse and to explore the structure of the data. The data contains hyperlinks that can be used to page through large sets of records. To see an example, load the following page in a web browser:</p><p><a href="http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov">http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov</a></p><p>To see any API resource in JSON format, simply append “?format=json” to the URL. As an example, the above URL is converted to JSON in the following line.</p><p><a href="http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov/?format=json">http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov/?format=json</a></p><h3>RESOURCE 1 : RETRIEVE A LIST OF BUYERS</h3><p><a href="http://www.spendnetwork.com/api/entities/">http://www.spendnetwork.com/api/entities/</a></p><p>This provides a list of records, each of which has 3 fields which are described below:</p><p>ENTITY_ID: Spend Network’s identifier of a buyer. e.g: DCM048_DFCMAS_gov</p><p>ENTITY_NAME: Buyer Name. e.g. DEPARTMENT FOR CULTURE, MEDIA AND SPORT</p><p>TRANSACTIONS: This is a link to the API page that gives the transactions for this buyer. e.g. <a href="http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov/">http://www.spendnetwork.com/api/spend/entity/DCM048_DFCMAS_gov/</a></p><h3>RESOURCE 2: RETRIEVE TRANSACTIONS FOR A SELECTED BUYER</h3><p>It is necessary to append the ENTITY_ID. ENTITY_ID can be discovered by using RESOURCE 1 (as in the previous section).</p><p><a href="http://www.spendnetwork.com/api/spend/entity/">http://www.spendnetwork.com/api/spend/entity/</a></p><p>This resource provides a list of records, each of which has five fields which are described below.</p><p>TRANS_ID: Spend Network’s unique ID for this transaction. E.g. DCM048_DFCMAS_gov_2011_04_36</p><p>AMOUNT_EXECUTED: The value of the published transaction.</p><p>SUPPLIER_SOURCE_STRING: The name of the supplier published in the spend statement.</p><p>DATE_COMPLETED: The published date string.</p><p>FULL_DETAILS: This is a hyperlink to the ‘detail’ record for this transaction. An example is: <a href="http://www.spendnetwork.com/api/spend/detail/DCM048_DFCMAS_gov_2011_04_36/">http://www.spendnetwork.com/api/spend/detail/DCM048_DFCMAS_gov_2011_04_36/</a></p><h3>RESOURCE 3: RETRIEVE A SINGLE TRANSACTION RECORD IN DETAIL</h3><p>The URL to access this resource is given below. It is necessary to append the TRANS_ID key for the record required. Obtaining a TRANS_ID is described in detail in the section describing RESOURCE 2.</p><p><a href="http://www.spendnetwork.com/api/spend/detail/">http://www.spendnetwork.com/api/spend/detail/</a></p><p>This resource returns a record comprising key value pairs. The record comprises all of the data made available by in the raw data that is uploaded in the Spend Network database.</p><p>An example record can be seen by opening the following page: <a href="http://www.spendnetwork.com/api/spend/detail/DEC066_DOEACC_gov_2011_04_102">http://www.spendnetwork.com/api/spend/detail/DEC066_DOEACC_gov_2011_04_102</a></p><p><em>Originally published at </em><a href="http://spendnetwork.github.io/blog/spend-network-api.html"><em>spendnetwork.github.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3dd0b45c62e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/understanding-spend/spend-network-api-c3dd0b45c62e">Spend Network API</a> was originally published in <a href="https://medium.com/understanding-spend">Understanding spend</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>