<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ministry of Justice Digital &amp; Technology on Medium]]></title>
        <description><![CDATA[Stories by Ministry of Justice Digital &amp; Technology on Medium]]></description>
        <link>https://medium.com/@justiceDandT?source=rss-a6654cbd45a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*F-U2494K2cBzFgMtvs3lQQ.jpeg</url>
            <title>Stories by Ministry of Justice Digital &amp;amp; Technology on Medium</title>
            <link>https://medium.com/@justiceDandT?source=rss-a6654cbd45a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:57:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@justiceDandT/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Moving towards Continuous Delivery]]></title>
            <link>https://medium.com/just-tech/moving-towards-continuous-delivery-8200131350fa?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8200131350fa</guid>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[legacy-code]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Tue, 14 Jan 2020 08:53:54 GMT</pubDate>
            <atom:updated>2020-01-14T08:53:54.676Z</atom:updated>
            <content:encoded><![CDATA[<p>by Bethan Palmer and Mat Moore (Software Development Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcjI9ASxo9cvQ6HgEOtZYA.jpeg" /></figure><p>In LAA digital, we are moving towards Continuous Delivery as part of modernising CCMS (Client and Cost Management System). CCMS is one of several <a href="https://mojdigital.blog.gov.uk/2018/11/22/announcing-the-application-modernisation-team1/">legacy services we are modernising in the MOJ</a>. So far we have been focusing mainly on one part of CCMS, the Provider User Interface.</p><blockquote>Continuous Delivery is the ability to get changes of all types — including new features, configuration changes, bug fixes and experiments — into production, or into the hands of users, <em>safely</em> and <em>quickly</em> in a <em>sustainable</em> way. <a href="https://continuousdelivery.com/">https://continuousdelivery.com/</a></blockquote><p>Historically, the deployment process for CCMS has involved a lot of steps, and a lot of different people need to be involved throughout the process to get a change from development to production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2r-rXfkc_EsYi_hr" /></figure><p>When a release process is complex and time consuming, releases happen infrequently. Because they happen infrequently they are risky as they contain a lot of changes. And because they are risky they tend to happen even less often. So there becomes a cycle of infrequent releases which is difficult to break out of. We wanted to avoid this situation by moving towards frequent deployments to production.</p><h3>Things that we want to change</h3><p>The bottlenecks in the deployment process were:</p><ul><li>Deployments needed to be planned, and required approval from various people to proceed</li><li>We needed to do a lot of manual testing, and our case workers or the live support team also manually smoke tested each deployment to make sure we hadn’t broken anything</li><li>We needed to run separate builds for each environment we deploy to, instead of building a single artefact and deploying it to each environment in turn</li><li>We had to develop the code in a shared development environment, which was firewalled off from the internet and difficult to work on</li></ul><p>We are working to remove these bottlenecks so that we can deploy to production quickly and easily. Our end goal is to promote changes from the development environment through to production in less than 15 minutes.</p><h3>What we’ve done so far</h3><p>Below are some of the changes we’ve made which are helping us move towards Continuous Delivery.</p><h4>Updated the unit tests</h4><p>There were already some existing unit tests, but several of them contained hardcoded paths to configuration files we didn’t have access to. We re-wrote some of these tests, removed some of them, and wrote some new unit tests to increase our test coverage.</p><h4>Added CircleCI (continuous integration)</h4><p>We added an integration with CircleCI, which runs our unit tests every time we create a Pull Request in GitHub. It also checks our code’s dependencies for security vulnerabilities every night. Eventually we want to automate deployments from here too.</p><h4>Written some end to end tests</h4><p>We’ve written some browser tests that automate someone clicking through the application. Running these gives us confidence that our changes haven’t broken any of the basic functionality.</p><h4>Improved the application build process</h4><p>We have replaced the old Ant build with a new Gradle build. Ant and Gradle are both build automation tools for Java applications, but Gradle has several advantages over Ant. For example with Gradle we are able to delete checked in dependencies and use remote ones, making it much easier to upgrade them. Gradle is a simpler tool for developers to use and the build process is quicker, which makes developing code changes easier.</p><h4>Created a Docker/Tomcat build we can run locally</h4><p>We are now able to build the application and deploy it to Tomcat running in a Docker container. This means we can develop and test our changes locally instead of relying on the shared development environment, which is very difficult to use. This gives us a much quicker feedback loop when doing development work.</p><h4>Spiked out what our ideal deployment pipeline into AWS might look like</h4><p>We did a <a href="https://github.com/ministryofjustice/deployment-pipeline-hello-world">spike to demonstrate how our end goal might work with a simple Hello World application</a>. This involved creating a pipeline in CircleCI that builds the app, tests it, and deploys it to multiple environments in AWS with zero downtime. We also want to look at spinning up ephemeral environments that get deleted after a certain amount of time. Another department in the MoJ, the Office of the Public Guardian, had already created a similar deployment pipeline so we were able to use their work as a guideline for ours.</p><h3>Next steps</h3><h4>Migrate to AWS</h4><p>CCMS could not be included in <a href="https://medium.com/just-tech/reducing-cost-and-latency-of-change-for-legacy-services-5ab85a0f339f">our previous migration to AWS</a>. Now that we have a better understanding of the process, we intend to repeat the same steps to move CCMS out of the current data centre and deploy into public cloud. This will reduce the cost of running the service, but will also allow us to run fully automated deployments. This enables us to make changes more quickly and safely.</p><h4>Increase our test coverage</h4><p>Our test coverage is still very low (6% for the Provider User Interface). We want to increase this so that we have less reliance on manual testing, and can be confident we are releasing changes safely.</p><h4>Adopt Continuous Deployment</h4><p>When we have achieved Continuous Delivery, the next step is to move to Continuous Deployment (where the release is automated from CircleCI rather than a manual process, and there is zero downtime).</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital?source=post_page---------------------------">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/?source=post_page---------------------------">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/?source=post_page---------------------------">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology?source=post_page---------------------------"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8200131350fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/moving-towards-continuous-delivery-8200131350fa">Moving towards Continuous Delivery</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing Disparate Data Sources Under Control with Good Metadata]]></title>
            <link>https://medium.com/just-tech/bringing-disparate-data-sources-under-control-with-good-metadata-f70c9c57bd13?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f70c9c57bd13</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 07:38:44 GMT</pubDate>
            <atom:updated>2019-09-03T07:38:44.918Z</atom:updated>
            <content:encoded><![CDATA[<p>by Robin Linacre, Karik Isichei, George Kelly and Adam Booker (Data Engineering Team)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOA621efiIAoKecOEOJueQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@srd844?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Stephen Dawson</a> on <a href="https://unsplash.com/search/photos/data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>As a data engineering team, we are responsible for harnessing data and making it simple to use for our analysts. We process data from a plethora of sources, from decades-old database systems through to microservices on our Cloud Platform. Our data needs to be quickly delivered to live webapps, and analysts who use R, Python, and Spark. This variety means we need to find a good tradeoff between consistency and flexibility in data processing.</p><p>The more we work on this problem, the more we understand the importance of tools that standardise our data processing, transforming data sources from arcane, unreliable formats into dependable, reusable commodities.</p><p>Central to this effort is the development of a <a href="https://github.com/moj-analytical-services/metadata_schema">standard for machine readable metadata</a>, and the realisation that once this exists, it is useful in almost every stage of the data pipeline:</p><ul><li>The metadata can serve as specification for data providers, who can easily use our <a href="https://github.com/moj-analytical-services/data_linter">open source library</a> to check conformance of their data against the spec, knowing we will be applying exactly the same checks.</li><li>At data ingestion, we can automate multiple checks of whether the incoming data conforms to the metadata, yielding detailed web reports of rows which failed the checks. See <a href="https://mybinder.org/v2/gh/moj-analytical-services/data_linter_demo/master?filepath=index.ipynb">here</a> for a simple demo.</li><li>During data processing, we can automatically harmonise the wide variety of incoming column data types (string, float, int etc.) from different database systems into a common set.</li><li>We can automatically convert our system-agnostic metadata into the format required by specific data storage solutions, automating the process of setting up databases. This makes it trivial, for instance, to generate the code needed to setup an <a href="https://github.com/moj-analytical-services/etl_manager/blame/master/README.md#L187">AWS Athena Database</a>.</li><li>The metadata can be automatically added to a central, searchable data catalogue, enabling data discoverability. We have developed <a href="https://github.com/moj-analytical-services/metadata_vis">an open source GUI</a> on top of our metadata to enable easy searching and automated SQL query generation.</li><li>Since the metadata for a particular table is just a json file, it can be placed under version control. The metadata is a necessary part of our ETL code, and so it lives in the same repository, meaning metadata stays in sync with the code.</li><li>Finally, authoring metadata is simple and fast. Taking inspiration from the <a href="https://jsonschema.net/">jsonschema generator</a> we can automatically generate a first draft of metadata from existing datasets, and use of our <a href="https://github.com/moj-analytical-services/metadata_schema">metadata schema </a>enables <a href="https://code.visualstudio.com/docs/languages/json#_json-schemas-and-settings">autocompletion</a> of manual edits in text editors like VS Code.</li></ul><p><strong>A simple example</strong></p><p>We have developed an online, interactive demo of some of our tools, which you can find <a href="https://mybinder.org/v2/gh/moj-analytical-services/data_linter_demo/master?filepath=index.ipynb">here</a>. This notebook demonstrates how we can:</p><ul><li>Produce a validation report of a dataset which validates successfully against a metadata schema</li><li>Produce a validation report of a dataset which fails to validate</li><li>Auto-create a draft metadata schema from an existing dataset</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f70c9c57bd13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/bringing-disparate-data-sources-under-control-with-good-metadata-f70c9c57bd13">Bringing Disparate Data Sources Under Control with Good Metadata</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Organising Variables in Terraform]]></title>
            <link>https://medium.com/just-tech/organising-variables-in-terraform-dacbdac5a295?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/dacbdac5a295</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Thu, 22 Aug 2019 08:51:21 GMT</pubDate>
            <atom:updated>2019-08-22T08:51:21.611Z</atom:updated>
            <content:encoded><![CDATA[<p>by Andrew Pearce (Web Operations Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ijx4dhkbnW5S-ZLTASVk1w.jpeg" /></figure><p>I use Terraform to provision AWS infrastructure for an application. The app has 3 environments - development, preproduction and production - that have the same configuration.</p><p>Terraform workspaces are used to isolate the resources and to map certain values, such as what AWS account to provision to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*x7VTHNT_4S3M3jDl" /></figure><h3><strong>Variable definitions using tfvars</strong></h3><p>Variable definitions let you set values for lots of variables at once. They can read in automatically when you run a terraform apply or terraform plan instead of individually providing them with -var=.</p><p>You can provide values in Terraform’s syntax with terraform.tfvars files, or in JSON format with terraform.tfvars.json.</p><p>terraform.tfvars.json files are how I provide values specific to each environment.</p><p>I’ve been doing this for a while, but I recently found a new way to organise, define and access these values thanks to some improvements in the underlying Terraform language.</p><h3><strong>My old way of organising variables</strong></h3><p>I used to define maps for each variable. Each map held values for dev, preprod and prod:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uqaePv5qIHMmWNp-" /></figure><p>My tfvars.json had a lot of repeating dev, preprod, prod stanzas:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WWnoc8ADoN0id2jX" /></figure><p>I would then access my values for each environment using a lookup function, matching the name of the environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hmF6ODORYm46ie_A" /></figure><p>This does have the advantage of being able to supply a default when the lookup fails to return a match.</p><h3><strong>My new way of organising variables</strong></h3><p>Now I am using objects to define many values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zDiK-aW19ejBvgUO" /></figure><p>Variables can now be organised by environment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1-q1mZnBwusxxV9a" /></figure><p>Values can now be accessed from the object using the environment name, which here is available in terraform.workspace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mMJ3PaFy6yk0YDCY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RMG5rNXDupfDVFDD" /></figure><p>I still have a need for variable maps, and the two can live alongside each other in the terraform.tfvars.json file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F1zRlFI6q8jxW-ld" /></figure><h3><strong>Why I prefer this</strong></h3><p>For my purposes, this is much neater and easier to read. All the parameters for each environment are grouped together. Adding another environment would be as easy as copying the block and updating as necessary instead of adding a new line to each variable map.</p><p>I can’t speak to the performance improvements for terraform, but it has made my work faster.</p><p>You can read a full implementation of this <a href="https://github.com/ministryofjustice/opg-lpa/tree/master/terraform/terraform_environment">on GitHub</a>.</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital?source=post_page---------------------------">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/?source=post_page---------------------------">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/?source=post_page---------------------------">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology?source=post_page---------------------------"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dacbdac5a295" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/organising-variables-in-terraform-dacbdac5a295">Organising Variables in Terraform</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use multi-stage Docker builds]]></title>
            <link>https://medium.com/just-tech/how-to-use-multi-stage-docker-builds-a112d1ec8a58?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a112d1ec8a58</guid>
            <category><![CDATA[php]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[webops]]></category>
            <category><![CDATA[composer]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Fri, 26 Jul 2019 07:23:04 GMT</pubDate>
            <atom:updated>2019-07-26T07:23:04.245Z</atom:updated>
            <content:encoded><![CDATA[<p>by Tom Withers (Software Development Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0kYzCMqZiJ4mYwWR" /><figcaption>Photo by <a href="https://unsplash.com/@guibolduc?utm_source=medium&amp;utm_medium=referral">Guillaume Bolduc</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Docker has revolutionized the way software engineers build applications; small maintainable pieces of software that can be easily scaled to meet demand. But with new technology, there are now more things to take into consideration. One such issue that arose with Docker is image size. Traditionally, to get an efficient docker image, you would use some complex shell commands within the Dockerfile to keep the layers as small as possible.</p><p>In Docker 17.05, multi-stage builds were introduced and this means we can now make docker images even smaller. Below, I’m going to go over an example of how to do this with a Symfony app and composer.</p><p>With multi-stage builds, you can use multiple FROM commands within a single Dockerfile. Each FROM command can use a different base and symbolises a new stage in the build. The most useful thing is that you can selectively copy artefacts from one stage to another leaving everything that you don’t need behind.</p><p>So let’s create a new Dockerfile and add our first FROM command. This will be using composer as a base image, so we can get all of our composer dependencies. We can also name the different stages within the build by adding AS &lt;NAME&gt; to the FROM command.</p><pre>FROM composer AS composer</pre><p>Now let’s copy over our composer.json and composer.lock files. We’re also going to add a RUN command, which will run a composer install.</p><pre>FROM composer AS composer</pre><pre>COPY composer.json /app</pre><pre>COPY composer.lock /app</pre><pre>RUN composer install</pre><p>So the first part of our multi-stage build is done, this will allow us to install our composer dependencies into the first image and then copy them across into the final image.</p><p>Next, we need to build our final stage:</p><pre>FROM php:7-fpm-alpine</pre><pre>WORKDIR /var/www</pre><pre>COPY ./ /app</pre><pre>COPY — from=composer /app/vendor /app/vendor</pre><p>Here, we start a new build using php7-fpm-alpine as the base image, then we use COPY to copy over the vendor folder into our new image. We leave all the composer gubbins behind, and it’s not saved into our final image.</p><p>The full Dockerfile looks like this:</p><pre>FROM composer AS composer</pre><pre>COPY composer.json /app</pre><pre>COPY composer.lock /app</pre><pre>RUN composer install</pre><pre>FROM php:7-fpm-alpine</pre><pre>WORKDIR /var/www</pre><pre>COPY ./ /app</pre><pre>COPY — from=composer /app/vendor /app/vendor</pre><p>You only need a single Dockerfile. Just run docker build and docker the build process will start.</p><p>The end result of using multi-stage builds is a slim production-ready image without any complexity. I hope you can see the benefits of using multi-stage builds and I would definitely encourage you to try them out!</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital?source=post_page---------------------------">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/?source=post_page---------------------------">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/?source=post_page---------------------------">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology?source=post_page---------------------------"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a112d1ec8a58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/how-to-use-multi-stage-docker-builds-a112d1ec8a58">How to use multi-stage Docker builds</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working Days Calculator]]></title>
            <link>https://medium.com/just-tech/working-days-calculator-262e6637377f?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/262e6637377f</guid>
            <category><![CDATA[date]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[ruby]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Tue, 09 Jul 2019 07:31:31 GMT</pubDate>
            <atom:updated>2019-07-09T07:31:31.639Z</atom:updated>
            <content:encoded><![CDATA[<p>by Rob Nichols (Software Development Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Erpd_TbD76LxPCFU9zVB0g.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@esteejanssens?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Estée Janssens</a> on <a href="https://unsplash.com/search/photos/calendar?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Working out how many working days there are between two dates requires not only knowledge of when weekends fall, but also when bank holidays occur. As bank holidays are moved or created to coincide with key national events (Royal Weddings for example), a simple calculation or static data lookup is not a good long term solution.</p><p>In this article, a solution is described that gathers bank holiday data from a GOV-UK API, and uses that data to calculate working day intervals.</p><h3>A Ruby Solution</h3><p>There are a number of gems that will do working day calculations, but to keep them up-to-date they require a list of bank holidays to be passed into them. They make the calculation simpler, but a system to gather bank holiday data is still required.</p><h3>The API</h3><p>There is a <a href="https://www.gov.uk/bank-holidays.json">GOV-UK API</a> that returns up-to-date bank holiday data.</p><h3>Gem selection</h3><p>There were three main gems considered:</p><ul><li><a href="https://github.com/Intrepidd/working_hours"><strong>working_hours</strong></a></li><li><a href="https://github.com/bokmann/business_time"><strong>business_time</strong></a></li><li><a href="https://github.com/gocardless/business"><strong>business</strong></a></li></ul><p>Of these, <em>business_time</em> was last update two years ago. The other two gems both appear to be more regularly maintained. Both <em>business_time</em> and <em>working_hours</em> monkey patch the core Ruby time objects <em>Time</em>, <em>Date</em> and <em>DateTime</em>. As the scope of usage for this project was limited (there was currently only one calculation to be made), using a gem that didn’t monkey patch core objects was a better fit. That left the <em>business</em> gem — and this was selected.</p><h3>Retrieving the data</h3><p>The first task was to create a service that would retrieve the data from the API. The solution was <em>BankHolidayRetriever</em>, which called the API via Net::HTTP.get_response, parsed the resulting JSON and extracted the array of dates from the required section of the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7844a68615efc21f884bbff72e1aa9b6/href">https://medium.com/media/7844a68615efc21f884bbff72e1aa9b6/href</a></iframe><h3>Storing the data</h3><p>The next task was to store the data to locally cache it. After some consideration as to whether to store the data in memory, or in a memcache, it was decided to use the simplest solution of storing the data in the database via an <em>ActiveRecord Model</em>.</p><p>The model was created with a generator:</p><p>rails g model BankHoliday dates:text</p><p>It was then modified to add serialization of dates to store the array coming from BankHolidayRetriever.dates, some simple validation, and a callback that populated dates on creation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81aeaa371753c89626ad5acfb2080fa6/href">https://medium.com/media/81aeaa371753c89626ad5acfb2080fa6/href</a></iframe><p>A class method BankHoliday.dates was added that would retrieve the latest updated instance of the model and return its dates, or create an instance if none exists.</p><p>Also added to the dates class method call, was a call out to a worker that would update the current data. In this way the bank holiday data will be updated as it is used.</p><h3>The update worker</h3><p>As <em>Sidekiq</em> was available in the current project, a worker was added that will update the Bank Holiday data. Its first task is to see when the last update occurred. If the last update is less than a certain interval old (currently two days), the worker would just close without doing an update.</p><p>If an update is required, the worker uses a new instance of <em>BankHoliday</em> to gather the latest data. It then compares that data with the latest stored data. If the two match, the “updated at” date of the last persisted <em>BankHoliday</em> instance is updated (via touch).</p><p>If the latest stored data does not match the retrieved data, then the new instance is saved and it becomes the latest stored data in the database.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0f34a61efc0550490f3adf85b7929ef/href">https://medium.com/media/f0f34a61efc0550490f3adf85b7929ef/href</a></iframe><p>Note that failure to retrieve the data or save it to the database, raises an exception. When this happens within a worker, this will cause the worker to be put into the retry queue for it to be run again later. In this way short term API and or data errors can be handled without adding additional code.</p><h3>The Calculator</h3><p>With the system now able to return an up-to-date array of bank holiday dates, it was a straightforward task to create a working day calculator using the <em>business</em> gem.</p><p>As this gem accepts the dates as strings, there was no need to convert them to dates first, which simplifies the code.</p><p>At the time of writing all that was required was a way of returning a date a number of days ahead of today. As the <em>business</em> gem provides a number of working day calculations, it will be fairly simple to extend the functionality as required.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/38d26e72793a67b616b95a529d816b5a/href">https://medium.com/media/38d26e72793a67b616b95a529d816b5a/href</a></iframe><h3>The end result</h3><p>With these changes in place, within the app the date ten working days from now can be returned with the call :</p><p>WorkingDayCalculator.working_days_from_now(10)</p><p>Making that call will trigger a worker that will check that the data is up-to-date for the next call.</p><p>The complete set of code with specs, within the context it was used can be seen in the <a href="https://github.com/ministryofjustice/laa-apply-for-legal-aid/pull/593">Pull Request</a>.</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=262e6637377f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/working-days-calculator-262e6637377f">Working Days Calculator</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing cost and latency of change for legacy services]]></title>
            <link>https://medium.com/just-tech/reducing-cost-and-latency-of-change-for-legacy-services-5ab85a0f339f?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ab85a0f339f</guid>
            <category><![CDATA[legacy]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[legacy-systems]]></category>
            <category><![CDATA[application-modernization]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Mon, 17 Jun 2019 11:26:41 GMT</pubDate>
            <atom:updated>2019-06-17T11:26:41.227Z</atom:updated>
            <content:encoded><![CDATA[<p>by <a href="https://twitter.com/jabley">James Abley</a> (Technical Architecture Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xkyr-y0AgzwgqzJIZ2LKDA.jpeg" /></figure><h3>Context</h3><p>Like a lot of parts of government, the <a href="https://www.gov.uk/government/organisations/ministry-of-justice">Ministry of Justice</a> has a large technology estate. Ours includes many things which pre-date <a href="https://www.gov.uk/guidance/digital-and-technology-spend-controls-version-5">GDS spend controls</a>. <a href="https://www.gov.uk/guidance/g-cloud-buyers-guide">How government bought things</a> has <a href="https://www.digitalmarketplace.service.gov.uk/">changed over the last 6 years</a>. But we still have things from before that time. In this article, I’m <a href="https://blog.usejournal.com/toxic-technology-the-growing-legacy-threat-b95ad098a339">calling those things legacy services</a>.</p><p>Many of these legacy services are business-critical core systems without which we could not operate. It would cause massive amounts of pain/frustration if they weren’t available. Our <a href="https://twitter.com/daverog">CTO Dave Rogers</a> has <a href="https://medium.com/@daverog/deep-transformation-a745167befcf">written about some of the problems faced when trying to transform these ageing services</a>.</p><p>Delivering substantive changes to services can take time. But just because something is hard doesn’t mean it’s not worth doing. And if you do it right, you can end up in a much better place, making continuous small improvements.</p><h3>Our problems</h3><p>Some of the properties of these systems are less than desirable. For example, changes are very infrequent. We have services that are only updated once per year. These changes are risky affairs. Downtime with weekend work is the norm. Then there is usually the unofficially accepted post-release cleanup period. This involves further unplanned releases to address problems introduced by the planned release.</p><p>Infrequent releases are not great for the people using the system who may be experiencing frustration. Batching up fixes in this way means we are delaying improving people’s quality of life. That’s quite a blunt way of putting it; it’s meant to be.</p><p>So the cost of change is too large. Changing the services involved the work of many people over a long period of time.</p><p><a href="https://www.amazon.co.uk/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339">The book Accelerate</a> by <a href="https://twitter.com/nicolefv">Dr Nicole Forsgren</a>, <a href="https://twitter.com/jezhumble">Jez Humble</a>, and <a href="https://twitter.com/realgenekim">Gene Kim</a> contains a wealth of evidence (science FTW!) about high-performing technology organisations. It talks about 4 key metrics that are useful to measure when trying to understand Software Delivery Performance:</p><ol><li><strong>Lead time</strong> — the time it takes from a customer making a request to the request being fulfilled</li><li><strong>Deployment Frequency</strong> — this is a proxy for batch size, which is easier to measure and typically has low variability</li><li><strong>Mean Time to Restore (MTTR)</strong> — when a service incident occurs, how quickly is service restored?</li><li><strong>Change Fail Percentage</strong> — what percentage of changes to production fail?</li></ol><p>If we’re doing one release a year, we can see that the average lead time is 6 months. Deployment frequency is annual. The mean time to restore was a week, but sometimes fixes were expedited and done mid-week rather than waiting for another weekend to do a release.</p><p>And the Change Fail Percentage was very high. Maybe not quite 100%, but not far off.</p><h3>Our journey</h3><p>We recently migrated all the services for the Legal Aid Agency from traditional hosting. A business case had established that moving some of the services to the public cloud would give us financial benefits.</p><p>But we also managed to eke out some other benefits as part of the migration. We have made huge improvements to the metrics mentioned above.</p><p>We had some guiding principles for the migration to public cloud:</p><ol><li>We will move out of the current data centre</li><li>We will not provide the service in a way that is any worse than the current service</li><li>Applications delivered into public cloud will have a stable, automated deployment pipeline</li><li>Applications in public cloud should capable of redeployment during normal working hours</li></ol><p>These were enough to get the organisation (and our users!) to a much better place:</p><ul><li>We have got the lead time to the same day, for small changes/fixes.</li><li>The deployment frequency averaged around every 3 days* (previously it was once a year).</li><li>Mean Time To Restore is currently untested, but the lead time and deployment frequency numbers show that it would be resolved on the same day.</li><li>Change Fail Percentage has dropped to 0%.</li></ul><h3>How we did this</h3><h4>Deployment pipeline</h4><p>A deployment pipeline is an automated manifestation of your process for getting changes into an environment. Adopting a deployment pipeline means that <a href="https://twitter.com/jabley/status/1005045771593494528">all changes have to go through version-control</a>. A change happens in version-control and that starts a build. This enhancement to the system in how people worked ensured visibility of changes.</p><p>The application/service gets built once and then deployed everywhere with the relevant configuration for that environment. This gave a nice separation between things that tend to have different rates of change — the application code, and the configuration for each environment.</p><h4>Trunk-based development</h4><p><a href="https://trunkbaseddevelopment.com/">Trunk-based development</a> has helped:</p><ul><li>ensure small batch sizes</li><li>increase the frequency of deployments</li><li>reduce the risk of deployments</li><li>reduce the time spent merging long-lived branches.</li></ul><p>And we can try to break down work so that useful increments can be delivered regularly.</p><p>Stabilisation periods are no longer necessary. Time spent doing difficult merges of long-lived branches that have diverged a long way from the main trunk branch is no longer needed.</p><h4>Adding automated tests</h4><p>Many of the legacy services were created when Test-Driven Development and other practices were not as widespread. They do not have a vast suite of automated tests with a high level of coverage which give us the confidence to know we aren’t breaking things.</p><p>The most valuable investment we decided we could make was adding functional tests which exercised the way our users would typically use the applications. If we could replicate the happy path of how people used the system, we could manage the risk that:</p><ol><li>The migration would not break anything</li><li>We could continue to improve and fix things post-migration</li></ol><p>These sorts of tests are slower to execute, since they typically need a few distributed components, but they gave us the best level of coverage when working with code bases that had not been designed to be testable.</p><p>As part of the deployment pipeline, these tests are a quality gate that runs for every potential change which might go in front of our users.</p><h3>Eliminating waste</h3><p>From <a href="https://www.amazon.co.uk/Lean-Software-Development-Agile-Toolkit/dp/0321150783">a Lean sense</a>, we have also <a href="https://en.wikipedia.org/wiki/Lean_software_development#Eliminate_waste">eliminated various wasteful activities</a>.</p><p>When we had long-lived branches, we used to have a different environment per-branch. People had to track which version was in which environment.</p><p>Now that we’re doing trunk-based development with the ability to deploy to production on the same day, we don’t need to do that. If someone asks what version is in environment X, we always know:</p><ol><li>The deployment pipeline keeps track of it now, rather than a person, so check what the deployment pipeline says is deployed in environment X</li><li>But mostly the answer will be “trunk”</li></ol><p>We have also got rid of several environments; we no longer need to pay for them, or manage them, due to our new ways of working.</p><h3>Conclusion</h3><p>We started the migration wanting to realise the cost-savings available to us from using public cloud. In truth, we know we are still very early on in the journey for this part of the organisation. And it’s not evenly distributed. Some of our colleagues have to work with bits that are still harder than they should be to change and improve.</p><p>But we have achieved lots of small improvements. Small batch sizes:</p><ul><li>reduce risk</li><li>give opportunity for faster feedback and learning opportunities</li><li>are faster to deliver value to the organisation and our users</li></ul><p>That is such a massive difference for the organisation, and something we should be immensely proud of.</p><p>*It was quite high for a while because, for the first time, we had better insight about how the systems were running in production. We could access the logs and see the graphs. We could see the unreported frustrations that our users were tolerating or working around every day. So we fixed them. For most of the teams now, it’s a capability that they can choose to exercise as part of their normal sprint cadence (typically fortnightly).</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ab85a0f339f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/reducing-cost-and-latency-of-change-for-legacy-services-5ab85a0f339f">Reducing cost and latency of change for legacy services</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Unit Testing of PL/SQL]]></title>
            <link>https://medium.com/just-tech/automated-unit-testing-of-pl-sql-1d9d6f77a960?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d9d6f77a960</guid>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[plsql]]></category>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[oracle-pl-sql]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Mon, 10 Jun 2019 13:30:05 GMT</pubDate>
            <atom:updated>2019-06-10T13:30:05.801Z</atom:updated>
            <content:encoded><![CDATA[<p>by Laurence Barea (Software Development Profession)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLzYVS6985vnFNm0mMJC7w.jpeg" /></figure><p>As we moved to the adoption of CI/CD during our infrastructure and application migration to AWS, there was always a question mark surrounding Oracle and PL/SQL releases; we did not have the tools or frameworks available to allow us to deploy Oracle releases via a pipeline, which is not ideal given our goals for deployment automation are concerned.</p><p><a href="https://www.thoughtworks.com/insights/blog/why-test-automation-backbone-continuous-delivery">Testing is backbone of continuous delivery model.</a></p><p>The problem we are having is that although many programming languages out there lend themselves to this process, Oracle databases and its procedural language, PL/SQL, does not.</p><p>Currently, there is a large Oracle footprint within the estate and a correspondingly significant amount of PL/SQL code that sits outside the CI/CD pipeline resulting in Oracle releases having to be deployed manually by DBA’s. This is an issue; delays are created, the release process is made more complex and the involvement of other teams are required.</p><h3>Testing database deployment</h3><p>We considered <a href="https://flywaydb.org/">Flyway by Boxfuse</a> and <a href="https://www.liquibase.org/">Liquibase by Datical.</a> We decided on Liquibase and have been generally pleased with the approach it takes and how it fits in to our workflow. Flyway looked promising but would have cost $3,000 to support our version of Oracle.</p><h3>Testing PL/SQL</h3><p>With regards to PL/SQL we have set up a proof of concept to see a couple of possible solutions in action. Initially, we considered these tools:</p><ul><li>Oracle’s SQL Developer’s built in unit testing</li><li>Code Tester by Quest (Toad)</li><li>utPLSQL, a framework that’s been around for a few years which was originally developed by <a href="https://www.stevenfeuerstein.com/">Steve Feuerstein</a> himself, and</li><li>ruby-plsql-spec, a unit testing framework which is built using Ruby, ruby-plsql (library) and Rspec.</li></ul><p>… and decided on <a href="http://utplsql.org/">utPLSQL</a> and <a href="https://github.com/rsim/ruby-plsql-spec">ruby-plsql-spec</a>. The former because it was more well-established and the latter because of the growing Ruby competency among our teams.</p><h3>The Ruby way</h3><p>The conclusion is ruby-plsql-spec works great! See the <a href="https://docs.google.com/presentation/d/e/2PACX-1vQAed94rI3vk6Sc8zd2h1-nqGBZyhroO38OlgNJBPJit4pUOzbPRwXRypdup3eq-0KzwokTdXH2R79X/pub?start=false&amp;loop=false&amp;delayms=3000&amp;slide=id.p">Unit Testing PL/SQL Procedures From Ruby Code</a> for a more in-depth look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9h13kQzmlYSaJFs1PjleQ.png" /><figcaption>See slides here: <a href="https://docs.google.com/presentation/d/e/2PACX-1vQAed94rI3vk6Sc8zd2h1-nqGBZyhroO38OlgNJBPJit4pUOzbPRwXRypdup3eq-0KzwokTdXH2R79X/pub?start=false&amp;loop=false&amp;delayms=3000&amp;slide=id.p">Unit Testing PL/SQL Procedures From Ruby Code</a></figcaption></figure><h3>The utPLSQL way</h3><p>We deployed utPLSQL onto a development database. Unfortunately we can’t run the database locally at the moment 😞 so we share a single development database hosted in Amazon — there’s a separate piece of work going on to resolve this.</p><p>The initial challenges surrounded taking the automated install apart and doing it manually — this is because AWS RDS does not allow file-system access and it was something we could get on with straight away rather than getting the install scripts to run remotely from the DB in question.</p><p>Working with utPLSQL is very much a manual process; there is no automation — you will have to write all the test code and you will have to maintain it.</p><p>Below is some output generated utPLSQL tests run on the HUB development environment.</p><pre>Between string function</pre><pre>Returns substring from start position to end position [.129 sec] (FAILED - 1)</pre><pre>Returns substring when start position is zero [.002 sec] (FAILED - 2)</pre><pre>Failures:</pre><pre>1) basic_usage</pre><pre>Actual: &#39;234&#39; (varchar2) was expected to equal: &#39;2345&#39; (varchar2 at &quot;HUB.TEST_BETWNSTR&quot;, line 5 ut.expect(betwnstr(&#39;1234567&#39;, 2, 5)).to_equal(&#39;2345&#39;);</pre><pre>2) zero_start_position</pre><pre>Actual: &#39;1234&#39; (varchar2) was expected to equal: &#39;12345&#39; (varchar2 at &quot;HUB.TEST_BETWNSTR&quot;, line 10 ut.expect(betwnstr( &#39;1234567&#39;, 0, 5 ) ).to_equal(&#39;12345&#39;);</pre><pre>Finished in .131936 seconds</pre><pre>2 tests, 2 failed, 0 errored, 0 disabled, 0 warning(s)</pre><h3>Next steps</h3><p>Now that we have got both these frameworks working, we need to start incorporating them into actual work so that we can better understand how best to benefit from them.</p><p>A few points we will be looking to address as we go forwards:</p><ul><li>Testing across multiple databases</li><li>Standing up and tearing down databases as part of the automated testing</li><li>Transaction testing e.g. table data, rollbacks to savepoints etc.</li><li>Complex type testing e.g. associative arrays of nested objects</li><li>Integration with the code pipeline and Flyway DB.</li><li>Exception handling.</li><li>TDD going forwards with regards to the existing code-base.</li></ul><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d9d6f77a960" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/automated-unit-testing-of-pl-sql-1d9d6f77a960">Automated Unit Testing of PL/SQL</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring against cyber security standards]]></title>
            <link>https://medium.com/just-tech/measuring-against-cyber-security-standards-82082c9031a7?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/82082c9031a7</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[information-risk]]></category>
            <category><![CDATA[information-management]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Thu, 06 Jun 2019 11:44:31 GMT</pubDate>
            <atom:updated>2019-06-06T11:44:31.332Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://twitter.com/@JoelGSamuel">Joel Samuel</a> (Cyber Security Consultant Team)</p><p><em>MoJ’s Cyber Security team is led by </em><a href="https://twitter.com/JonPLawrence"><em>Jonathan Lawrence</em></a><em>, Chief Information Security Officer, and is responsible for helping the whole department manage cyber security risks through expert advice/ guidance, incident management/ response and proactive ‘offensive’ penetration testing.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dW9pMZ7kw0Qa8gU7PiS2kg.jpeg" /></figure><h3>Measuring security is hard</h3><p>The relatively simple concept of being sure you have the right security in place to keep systems (and the data within) safe (also known as ‘information assurance’) is easier said than done.</p><p>There are various methodologies for determining risk, measuring controls, value of controls (and so on) and each require a varying level of expertise by the assessor in order to complete them. One size definitely does not fit all when the team is responsible for cyber security in systems ranging from internal IT tools/solutions (laptops, WiFi, Google G-Suite, Microsoft Office 365, Trello, Slack and so on) to case management solutions used for administering over £1 billion a year in legal aid, as well as brand new digital services.</p><h3>External standards</h3><p>There are various external standards that a UK central government organisation is expected to adhere to. One of the key ones for cyber security is the <a href="https://www.gov.uk/government/publications/the-minimum-cyber-security-standard">Minimum Cyber Security Standard</a> (MCSS) that was developed by the <a href="https://www.gov.uk/government/organisations/cabinet-office">Cabinet Office</a> in collaboration with the <a href="https://www.ncsc.gov.uk/">National Cyber Security Centre</a> (NCSC).</p><h3>The Challenge</h3><p>I posed the problem of how in an imperfect cyber security or ‘information assurance’ world does the team measure a diverse range of technology solutions against the MCSS while also making the assessment method simple to use, repeatable, standardised and in a format that would allow us to compare different systems using the same measurements to identify trends horizontally… a tall ask!</p><h3>Design with data</h3><p>The 3rd <a href="https://www.gov.uk/guidance/government-design-principles">government design principle</a> is “<a href="https://www.gov.uk/guidance/government-design-principles#design-with-data">design with data</a>” and, while this is a different kind of ‘design’, I thought this principle matched the challenge really well so I decided to ensure any method created <em>data</em> that could be dynamically turn into <em>information</em>. Free-text is hard to read, analyse and update and is subject to variations in terminology and intention.</p><p><a href="https://twitter.com/warmana">Adrian</a> has <a href="https://medium.com/just-tech/data-or-information-is-there-a-difference-and-does-it-matter-355bb9f4a408">posted about this before</a>.</p><h3>Et voila! Welcome to Standards Assurance Tables</h3><p>I used a bit of math (often quite basic!), <a href="https://support.google.com/docs/answer/3093340?hl=en&amp;ref_topic=9199554">IMPORTRANGE()</a> and a smidge of Google Script to create a template Google Sheet spreadsheet and dubbed it the <strong>Standards Assurance Table</strong>.</p><p>The MOJ believes ‘security’ can work in the open so in addition to publishing it’s <a href="https://github.com/ministryofjustice/itpolicycontent/">IT policies</a>, as part of an open-sourced <a href="https://ministryofjustice.github.io/security-guidance/">cyber security guidance microsite</a>, <strong>the MOJ have published </strong><a href="https://ministryofjustice.github.io/security-guidance/guides/standards-assurance-tables/#standards-assurance-tables"><strong>how the Standard Assurance Tables work and the theory behind them</strong></a>.</p><p>Some information risk management concepts were maintained such as maintaining the use of ‘evidence’ (documentation, drawings and so on — not hearsay) and ‘confidence’ (how well the assessor thinks the evidence demonstrates the target system is meeting the standard) but ultimately ignored existing information risk methodologies for the purposes of this work, to create a clean, simple slate.</p><h3>A horizontal view</h3><p>At the same time an aggregator Google Sheet was created for collated analysis. The aggregator uses even more IMPORTRANGE() to pull all completed Standards Assurance Tables together for horizontal analysis and this works really well given data was kept as data.</p><p>As a result of this work the MOJ has already identified trends (across entirely diverse systems that have never before been compared) and this has directly led how the team are investing time and money to get the most amount of proportional security benefit while responsibly managing public funds.</p><h3>Designed for scale</h3><p>The table wasn’t named the “MCSS Table” because while this work began due to a particular standard I wanted to ensure any method can scale and adapt over time.</p><p>The team are now working with information governance colleagues across the MOJ to expand the <a href="https://ministryofjustice.github.io/security-guidance/guides/standards-assurance-tables/#objectives">objectives</a> beyond the MCSS and build some really powerful data-driven governance and investment.</p><h3>Onwards</h3><p>At the moment the <a href="https://ministryofjustice.github.io/security-guidance/guides/standards-assurance-tables/#standards-assurance-tables">Standards Assurance Tables</a> nor the aggregating Google Sheet analyse or represent change over time to keep the initial scope reasonable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79ec4344dd81ea761f65b1190f1ac36b/href">https://medium.com/media/79ec4344dd81ea761f65b1190f1ac36b/href</a></iframe><p>Each table already has an active Google Script that records changes as they are made. One of the things the team want to do next is leverage this metadata to help measure how systems fair over time, individually and in aggregate — which I think will be quite nifty.</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82082c9031a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/measuring-against-cyber-security-standards-82082c9031a7">Measuring against cyber security standards</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When making a private GitHub repository public, audit the pull requests]]></title>
            <link>https://medium.com/just-tech/when-making-a-private-github-repository-public-audit-the-pull-requests-154200d02f7f?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/154200d02f7f</guid>
            <category><![CDATA[credentials]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Fri, 31 May 2019 15:04:46 GMT</pubDate>
            <atom:updated>2019-05-31T15:04:46.261Z</atom:updated>
            <content:encoded><![CDATA[<p>by Mat Moore (Software Development Profession)</p><p>The GOV.UK Service Manual encourages us to <a href="https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable">make source code open source</a>. This is a great policy because it <a href="https://mojdigital.blog.gov.uk/2017/02/21/why-we-code-in-the-open/">holds us to account and allows code to be reused by other teams</a>. It’s also common for software-as-a-service solutions that work with GitHub to charge extra for private repositories, so open source makes it cheaper and easier for us to deliver software.</p><p>If you make your code open from the start of the project, it’s easy to do it securely. But what about making existing source code open? <a href="https://technology.blog.gov.uk/2018/02/19/how-to-open-up-closed-code/">How to open up closed code</a> by Anna Shipman explores 3 approaches for managing this:</p><ul><li>Cycle all the credentials</li><li>Rewrite the git history to remove sensitive information</li><li>Move the code to a new repository bit by bit</li></ul><p>When my team decided to make one of our private repositories public, we decided to go with option 2 — Rewrite the git history to remove sensitive information — because it would be easier for us than rotating credentials. It turned out this was not enough to avoid disclosing credentials because of the way GitHub references diffs in pull requests.</p><p><a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">GitHub’s own advice</a> states that:</p><blockquote>Once you have pushed a commit to GitHub, you should consider any data it contains to be compromised.</blockquote><h3>What went wrong</h3><p>When the project was originally created, we’d mostly avoided committing credentials to the repository. Instead, we used AWS Parameter Store to manage configuration and credentials for each environment — these get passed into the application as environment variables.</p><p>Unfortunately, the repository did contain one credential. We depend on a shared development database, and we’d committed a password for this in the docker-compose.yml file, which is used to run the application locally.</p><p>To address this we ran <a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">git filter-branch</a> to completely remove the file from the git history.</p><p>We then recreated our docker compose configuration using a separate docker-compose.override.yml file, which is encrypted with <a href="https://github.com/AGWA/git-crypt">git-crypt</a>.</p><p>Unfortunately, the first pull request for this didn’t set up git-crypt correctly, and when reviewing it, I added a comment to point this out. What I didn’t realise is that this comment would persist even after the commit had been rebased and the branch deleted. Whenever code is commented on, GitHub retains the diff and just adds an “outdated” badge if the code is no longer in the repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-YsYhUByLbGRPRjhPn0SaA.png" /></figure><p>This ultimately caused us to leak the credential when we made the repository public. Luckily, we noticed this shortly after and changed the credential.</p><h3>How to avoid this</h3><p>Many automated tools for detecting secrets focus on the git repository itself, rather than the project page on GitHub.</p><p>The problem is that any comment ever made in a code review could potentially expose secrets. In our case the repository was fairly new, so we could have manually audited GitHub issues and pull requests to check for anything sensitive, but this might not always be feasible.</p><p>You could avoid this problem by choosing either of the other two approaches Anna describes in her blog post:</p><ul><li>cycling all the credentials</li><li>starting a new repository entirely.</li></ul><p>You could also combine rewriting history with moving to a fresh repository. This way, you preserve your code’s history, but you get rid of existing issues and pull requests. This means you lose some of the context around the code, but you only need to sanitise the git repository itself before making the repository public.</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=154200d02f7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/when-making-a-private-github-repository-public-audit-the-pull-requests-154200d02f7f">When making a private GitHub repository public, audit the pull requests</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[console.log() needn’t be a slog]]></title>
            <link>https://medium.com/just-tech/console-log-neednt-be-a-slog-cc55e4d9f087?source=rss-a6654cbd45a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc55e4d9f087</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[console-log]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Ministry of Justice Digital & Technology]]></dc:creator>
            <pubDate>Mon, 29 Apr 2019 12:10:46 GMT</pubDate>
            <atom:updated>2019-04-29T12:10:46.984Z</atom:updated>
            <content:encoded><![CDATA[<p>by Will McBrien (Software Development Profession)</p><p>Using console.log() to debug JavaScript on the front-end can be a pain as the default output is not particularly pretty. As lots of output can look stylistically similar, it can be difficult to spot what you’re looking for, which can make debugging harder than it should be.</p><p>The good news is that there are some tricks you can employ to make console.log() much easier to use! Here are some of the ones I have found to be most useful:</p><p><strong>Add some cool styles:</strong></p><p>console.log() can accept a CSS string as a second argument, which can be used to style output. This can help to catch your eye (well if you use some colourful styles):</p><pre>console.log(‘%c &lt; some artful logging &gt;’, ‘color: white; background-color: #31a59f; text-decoration: underline;’);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IjG54DBJmp4QNSZm" /></figure><p>Note the need to add %c before the text else it won’t apply the styles.</p><p>One especially useful way to use this is as <em>before</em> and <em>end</em> markers for non-css console.log() statements to help make it really obvious as to what you’re printing. For instance, say you’re printing a lot of output from a loop, it’s nice to clearly mark where the log begins and ends using this technique:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7773b758d32e3ecbff05d3450480ade/href">https://medium.com/media/c7773b758d32e3ecbff05d3450480ade/href</a></iframe><p>which outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/0*_wpjYqV5An-OwhnZ" /></figure><p>If you use this technique, I recommend creating a snippet in your editor of choice so that you don’t need to spend time thinking of how you want to style things — this also makes it easier to mix and match your styled console.log() output with the default. In Visual Studio Code you can create language specific snippets by using cmd + shift + p and searching for snippets then selecting the language you want to add a snippet to. My snippet for this looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d06e365e6ea49baed18969c51e4d553/href">https://medium.com/media/9d06e365e6ea49baed18969c51e4d553/href</a></iframe><p>You can then use it by selecting from the dropdown auto-completion as you type (i.e. <em>slog</em> or <em>elog </em>respectively). Other editors have similar functionality you can use to add these type of snippets.</p><p><strong>Data tables:</strong></p><p>When you `console.log()` an object it can be annoying to read it’s attributes using the standard log output. You get an output like the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc251c8c80a3a8341f232d2e068ea77f/href">https://medium.com/media/dc251c8c80a3a8341f232d2e068ea77f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*6zgMMnKBt6VZVKCj" /></figure><p>Though this is obviously a really simple example, it is still not that easy to read. Thankfully, there is a useful method that can help with this issue, console.table(), which outputs a nicely formatted tabular representation of a javascript object:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*UdoCo9Spm5XMTIlm" /></figure><p>Which is a really clear thing to view in my opinion.</p><p><strong>Groups:</strong></p><p>If you’re trying to log things from within a nested data structure or loop construct it can be difficult for your eye to spot the log you’re interested in, for instance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f7208c3876002f5e209167b8a929f24/href">https://medium.com/media/1f7208c3876002f5e209167b8a929f24/href</a></iframe><p>Which outputs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/94/0*wnCXltCfcN4LcpLq" /></figure><p>Here everything appears at the in the same vertical column, making it hard to differentiate output at a glance, as you can’t see nesting particularly clearly.</p><p>Instead why not try employing console.group() and console.groupEnd() as a way to differentiate between different nested levels:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5390c65c414d827a6d3d28db54253c50/href">https://medium.com/media/5390c65c414d827a6d3d28db54253c50/href</a></iframe><p>This separates out your output into nested columns, which can be much easier to read:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/0*CXFjC-s8Dx04Vvd9" /></figure><p>Obviously this is another artificially simplified example, but this technique can be useful when outputting complicated nested structures that you would otherwise have to add hacky spaces to the left of console.log() string to get a similar output.</p><p><strong>Another useful approach, use the debugger:</strong></p><p>A more controlled approach is to use the JavaScript built in debugger, this creates a more familiar context if you’re used to pry in Ruby or other ‘stop execution’ debugging techniques. The debugger keyword opens up the browser’s debugging context at the point at which it is declared in your code.</p><p>However, although this approach is more powerful than console.log() I find that sometimes it is easier to just console.log() output if you want quick feedback without context switching into debugging mode, hence why I think the above techniques are still useful tools to have in you toolbox, despite dedicated debuggers being available :-)</p><p>If you enjoyed this article, please feel free to hit the👏 clap button and leave a response below. You also can <a href="https://twitter.com/Justice_Digital">follow us on Twitter</a>, read our <a href="https://mojdigital.blog.gov.uk/">other blog</a> or check us out on <a href="https://www.linkedin.com/company/uk-ministry-of-justice/">LinkedIn</a>.</p><p><strong>If you’d like to come and work with us, please check current vacancies on our </strong><a href="https://jobs.jobvite.com/justicedigitalandtechnology"><strong>job board</strong></a><strong>!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc55e4d9f087" width="1" height="1" alt=""><hr><p><a href="https://medium.com/just-tech/console-log-neednt-be-a-slog-cc55e4d9f087">console.log() needn’t be a slog</a> was originally published in <a href="https://medium.com/just-tech">Just Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>