<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Songkick Tech - Medium]]></title>
        <description><![CDATA[Songkick’s technology blog - Medium]]></description>
        <link>https://medium.com/songkick-dev-blog?source=rss----e0e3c74f91b8---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Songkick Tech - Medium</title>
            <link>https://medium.com/songkick-dev-blog?source=rss----e0e3c74f91b8---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 09:04:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/songkick-dev-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimising platform spend with cluster autoscaling]]></title>
            <link>https://medium.com/songkick-dev-blog/optimising-platform-spend-with-cluster-autoscaling-7ecd4b03252?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/7ecd4b03252</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Eric Escalante]]></dc:creator>
            <pubDate>Fri, 26 May 2023 10:23:33 GMT</pubDate>
            <atom:updated>2023-05-26T10:24:23.437Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post was written originally by </em><a href="https://medium.com/u/e3ae1f38f8e3"><em>Phil Hope</em></a><em> while we worked together in the Platform Team — I just helped with a bit of editing.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*usvs_1g2Dp2z0HXW" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog post, we’ll explore how we leveraged cluster autoscaling capabilities to dynamically allocate resources as needed, resulting in more efficient use of infrastructure and lower costs. Let’s get started!</p><h3><strong>Background</strong></h3><p>Kubernetes workloads at Songkick can be categorised into three types:</p><ul><li><strong>Applications</strong>: Front and Backend Services</li><li><strong>Daemons</strong>: Continuous processes such as campaign subscribers</li><li><strong>Cron Jobs</strong>: Regularly scheduled actions such as backups and data transformations.</li></ul><p>In the case of both application and daemon pods, processes are running continuously. However, the pods launched by cron jobs run specific tasks and are terminated on completion. These tasks include our ETL pipeline (detailed in<a href="https://medium.com/songkick-dev-blog/migrating-our-etl-pipeline-to-luigi-on-a-cloud-8fde921cec51"> Eric’s blog post</a>), event data ingestion from 3rd parties, and scheduled internal maintenance tasks.</p><p>On investigation into our workload performance and spend, it was clear we were faced with two problems: first, we were overspending for portions of the day when no cron jobs were running. Jobs would launch for specific windows in the day, and we would always need to provision the total nodes needed at peak for this. Secondly, on periods of high application demand, pods would compete for resources which meant some cron-related pods would be terminated because of their lower priority.</p><p>After looking into migrating to a serverless approach such as <a href="https://cloud.google.com/run/docs/overview/what-is-cloud-run">CloudRun</a> or a fully automated cluster like <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview">GKE Autopilot</a>, we settled on <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler">GKE Cluster Autoscaling</a> (CA) since it required the least amount of change to our existing deployment and infrastructure codebases, while at the same time giving us flexibility of only paying for what we actually used.</p><h3>Cluster Autoscaling — How does it work?</h3><p>CA adjusts the size of the cluster based on the demand for resources<strong>. </strong>It monitors this demand and adjusts the number of nodes accordingly based on metrics such as CPU and memory use, as well as network traffic. CA registers a decision status based on the satisfaction of these rules, and we can configure behaviour to scale up or down based on any of these metrics, or a combination within the limit of available nodes on the cluster.</p><p>CA creates a watch on the K8s API server and polls for unschedulable pods every 10 seconds. In the case of a satisfied scale up decision, a new node will be launched using a template and a check run to ensure the new machine has the available capacity for the pod. If all conditions are satisfied, it launches that workload.</p><p>In the case of scale down, a similar watch polls for unneeded nodes each 10 seconds. If the sum of CPU/Memory is less than 50% of allocatable resource on that node it triggers a scale down process, in which existing pods are moved to another node with availability and the original is marked for deletion. If no new workloads are assigned to this node after 10 minutes it is terminated.</p><p>An alternative to this approach would be to control our workload types using node pools and configuring pod affinity to always launch and scale within those pools. This is an approach we were already making use of to configure varying machine types based on performance needs, e.g. high memory nodes for memory intensive notifications processes.</p><p>We did consider scaling pools up and down based on schedule for cron workloads. However, this would involve greater management overhead to ensure pools scaled as expected and scripting actions manually, rather than them being automatically reactive. With CA being a well trodden path for exactly our aim of cost reduction, and given the flexibility of configuration and ease of integration with our existing stack, it was ultimately the obvious choice.</p><h3>Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A1euEsV2XlDNzkng" /><figcaption>Photo by <a href="https://unsplash.com/@tvick?utm_source=medium&amp;utm_medium=referral">Taylor Vick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Provisioning a cluster</strong></p><p>We began with Terraform, as our application cluster is created as code in the same way. This gave the perfect opportunity to refactor and introduce a lightweight and generic module that allows users to create Kubernetes clusters. We introduced the ability to configure cluster features on module calls such as toggling auto scaling and logging, configuring logging levels or IAM policies from a single point. This also meant testing was simplified, as a new cluster could be created and terminated with a small footprint.</p><p>We provisioned the new cluster for Songkick cron jobs with a single node group, and upper and lower count bounds which took some trial and error to balance. It’s important to note that, in our case, using GKE and a persistent Luigi deployment on our cluster, we always need a single node running making our lower count 1. For a max count, conservative configuration is not so important as with GKE you pay for the nodes you use, not the potential for CA to scale up.</p><p>We currently balance max node count at around 30% over average max scale, which can still be performant for any surges that might occur out of hours.</p><p>In order to maximise cost efficiency we tested several machine types for the cluster including high-CPU and high-memory types, using loads simulating the resource requests for both ETL cron jobs and our 3rd Party APIs ingestions. The most well balanced between cost and performance for our use case was actually the default e2-standard-4 machine, which comprises 4 vCPUs and 16GB Memory.</p><p><strong>Does it Scale?</strong></p><p>In initial tests we observed new nodes being added to the cluster and pods scheduled correctly onto them, however they would not initially scale down as expected. This is where Pod Disruption Budgets (PDBs) come to the rescue.</p><p>When a node needs to be terminated, Kubernetes attempts to reschedule the affected pods on other nodes. However, if there are not enough resources available on other nodes, or if rescheduling is taking longer than expected, it’s possible for pods to become unavailable. In the context of autoscaling, when there is a sudden increase in traffic or workload as nodes are added or removed, there is a risk of pods becoming unavailable due to rescheduling delays or resource constraints. PDBs help mitigate this risk by ensuring pods are not disrupted beyond a certain threshold. By setting these for critical workloads, we can ensure they remain available even as the cluster size changes.</p><p>As CA polls for a decision to scale down, we were noticing the log message “NotEnoughPDB”. In the same way PDBs protect applications from disruption, they also signal to the autoscaler whether disruptions are allowed. As shown in the image below, we needed to add theses not only to our workloads but also to the kube-system pods responsible for the system processes such as DNS and pod scheduling metrics collection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvpF2FOG_d1ftc146Z2qew.png" /><figcaption>kube-system Pod Disruption Budgets</figcaption></figure><p>After this addition we were greeted with the behaviour we originally expected as our cluster successfully scaled down to a single node after 10 minutes of inactivity. Success!</p><p><strong>Performance</strong></p><p>Below you can see an example of usage after migration, with our cluster scaling frequently during the running of jobs with a maximum of eight cores here (four nodes). Due to the daily schedule of varying processes, there’s a distinctive pattern of usage: a peak during late afternoon, a constant six nodes overnight, and a fluctuating load throughout the rest of the day.</p><p>For this iteration we didn’t modify any of the cron job schedules from their original as the business expects data to be loaded at certain times. One potential improvement for the future would be to group schedules to maximise node usage in one scale up and remain at a minimal scale once complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Veuy4xPprsGXuGZcIzzWhg.png" /><figcaption>Cluster monitoring of node autoscaling</figcaption></figure><h3>Conclusion</h3><p>We were now left with the ability to scale down our original cluster’s total nodes. Bearing in mind at maximum scale in our crons cluster we currently use five nodes, we were able to safely remove the total number of nodes running on the original cluster by 15, saving hundreds of dollars a month. Furthermore, by isolating this type of workload in its own cluster, we are able to test and make changes without the fear of affecting our critical services, giving us greater confidence in progression and incident management.</p><p>This work has also come with cross functional benefits in that the overall availability and reliability of our platform has improved. By automatically scaling the number of nodes based on demand, we can ensure our workloads are always running on nodes with available resources, reducing the risk of downtime and any need for us to manually intervene.</p><p>Hopefully you have gained some insight into the process of applying GKEs autoscaler and how it might help you. Overall I would recommend this solution if, like us, you are a small platform team with limited resources. While this post has focused on our cron job architecture, this solution is potentially an improvement for all workloads running on our Kubernetes clusters and something we’ll be investigating next!</p><p>Keep an eye on our<a href="https://devblog.songkick.com/"> tech blog</a> for more reads like this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ecd4b03252" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/optimising-platform-spend-with-cluster-autoscaling-7ecd4b03252">Optimising platform spend with cluster autoscaling</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Smooth [Prometheus] Operator]]></title>
            <link>https://medium.com/songkick-dev-blog/setting-up-a-smooth-prometheus-operator-c7fdc30176d1?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/c7fdc30176d1</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[alerts]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Eric Escalante]]></dc:creator>
            <pubDate>Fri, 23 Apr 2021 09:02:08 GMT</pubDate>
            <atom:updated>2021-04-23T09:02:08.690Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bE-B00CxdlflnXpstPhxWw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@salakoay_ola?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ayoola Salako</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Introduction</strong></p><p>In a previous <a href="https://devblog.songkick.com/getting-kubernetes-logs-into-bigquery-cost-effectively-bc6b67ddb923">post</a> we discussed how, as part of moving our apps to Kubernetes, we needed to get some basic foundations in place first. In that case, we looked at how we imported our Kubernetes logs into BigQuery. Now, it’s time to look at the work we embarked upon to get our metrics and alerts running from the new Kubernetes clusters!</p><p>For this piece of work, we wanted to deploy a full Prometheus stack to Kubernetes that would monitor applications that have been migrated to Kubernetes as well as the applications currently running in our (old!) Google Compute Engine instance groups. The aim was to accomplish a couple of goals: first, to have visibility into the apps migrated to Kubernetes while also keeping an eye on those waiting to be migrated. Second, to simplify things (no more Cortex and Cassandra!), plus save money as well, since we could deprecate the old, more resource-heavy Prometheus cluster.</p><p>Metrics are an integral part of keeping the service level smooth and top-notch, as music fans all over the world demand. With technical metrics like Requests per Second (RPS), CPU and memory usage, and so forth, we make sure our infrastructure is able to sustain the current user activity without disruption. With higher-level metrics, like emails and push notifications sent, we keep a finger on the pulse of our products’ health.</p><p>Alerting rules are the way we define what conditions have to be met for us to consider something is not quite right. Alerting routes ensure the right people are made aware in time — and via the right tool — to implement corrections and avert any user-facing service disruptions.</p><p>One of the many cool things you can do with Kubernetes is use a tool like <a href="https://helm.sh/">Helm</a> to install third party applications with extreme ease. Packages are called Charts in Helm’s world, and each instance of a given chart running in your cluster is called a release. Additionally, Helm allows us to very easily create our own charts by leveraging its powerful templating feature (based on Go templates), which we’ll discuss later on.</p><p><strong>Operator</strong></p><p>To get our metrics and alerts kick-started, we installed the<strong> Prometheus Operator</strong> by ways of the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack</a> chart, which sets up Prometheus, Grafana, Alertmanager and several other components in a matter of seconds. You can learn more about the Operator Pattern <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator">here</a>, but in essence it’s a way to to automate the deployment, configuration and maintenance of software packages in a Kubernetes cluster.</p><p>As previously mentioned in other posts, at Songkick we use Terraform to handle all our infrastructure-as-code needs, so installing the Prometheus Operator was as simple as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCMIVBFJmcG76DlxCAsvNA.png" /><figcaption>Installing the Prometheus Stack using Helm &amp; Terraform</figcaption></figure><p>And mic drop! 🤜 🎤</p><p>Ok, not just that, as always, the plot needs a little thickening to keep life somewhat interesting! You do need to follow that up with at least a few <em>set</em> statement blocks to customise the operator to your needs.</p><p>One of the main benefits of the Operator Pattern is that it leverages the experiences of many teams and comes with loads of configuration decisions already made for you in the form of default values. Still, you need to customize a few things here and there to settings that make sense to your particular use-case. Another strength of operators in general is that they extend the Kubernetes API by adding a couple of Prometheus-specific Custom Resource Definitions (CRDs to friends and family). CRDs allow you to create Kuberenetes resources that are not part of the standard API.</p><p>A good example of one of these is the <em>PrometheusRule</em> CRD. It lets us create Prometheus rule groups as a Kuberentes resource. These rules can be very low-level, like <em>sk-prometheus-error-rates</em> which keeps an eye on the number of errors in a given time period and raises an alert depending on the number and duration of them. An example of a higher-level one can be <em>sk-prometheus-daily-digest</em> which checks the health of daily email sending so we can be made aware if users are not getting notified about upcoming events.</p><p>The fact that these rules are created as cluster resources means they can very easily be created, updated, or deleted by standard Kubernetes actions and the Operator picks up any changes and keeps the Prometheus pod up to date with the new configuration.</p><p>Here’s how one of them looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmM-h-soDw9Q_76W-9kM7A.png" /><figcaption>Prometheus Alert Group as a Kubernetes Custom Resoure</figcaption></figure><p>Another custom resource is <em>AlertmanagerConfig</em>. This one gave us a bit of an achy-breaky heart (I <strong>did</strong> <strong>dance</strong> to the <a href="https://www.youtube.com/watch?v=byQIPdHMpjc">music video</a> during the writing of this post 🤠). An AlertmanagerConfig allows you to define two very crucial bits of information for Alertmanager: what are the available recipients for incoming alerts (for example Slack, email, or PagerDuty) and which routes should these alerts follow to end up in the hands of one, several, or none of these recipients. An added benefit of deploying your configuration via this CRD is that the operator checks your config against the CRD schema, which is always nice.</p><p>Sadly, it gave us two weird scenarios in which either most messages would end up in a predefined <em>null</em> recipient or only alerts coming from the <em>prometheus</em> namespace would be delivered. The good thing is that the operator configuration allowed us to override the use of this CRD and supply our own alerting configuration via a standard Kubernetes Secret. Have you been able to get this particular component working as expected? Please let us know in the comments! (Also click that bell icon - oh hold on…).</p><p>Instead of using the AlertmanagerConfig custom resource, we went with the other option that the Prometheus Operator offered: using a standard Kubernetes Secret to submit our routes and recipients. First, we defined a values file with all our route settings and a Helm template that interpolates them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ghoCYKxhhQBGSdT0RxAK8w.png" /><figcaption>A snippet of our alertmanager config template</figcaption></figure><p>The result of this process generates a Kubernetes Secret called <em>alertmanager-config</em>, and all that’s left to do is tell the operator to use it. This is very easy to configure using Terraform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AALPo_PTvJrdD_pC7CDttQ.png" /><figcaption>Telling Alertmanager to get its config from a Kubernetes Secret</figcaption></figure><p><strong>Enter Helm</strong></p><p>As mentioned before, Helm makes it deliciously easy to install applications in Kubernetes, but it also makes a for very powerful way — once you wrap your head around Go templating — to create new Charts that add configuration and customisation to existing releases (deployed Charts in Helmspeak).</p><p>With this in mind, we restricted the use of Terraform to just installing the Operator and configuring its main settings and secrets, but decided to create a new Helm Chart in a separate repository that would take care of all the Prometheus rules, Alertmanager configuration and such. This new Repo gives developers the power to add new alerting rules or custom metric scrape configurations just by editing easy to read YAML files, which the chart then deploys to the cluster. The Prometheus Operator keeps track of any changes made to these resources and synchronises the final Prometheus/Alertmanager configuration files accordingly.</p><p><strong>Prometheus service discovery part 1 — current applications</strong></p><p>Once Helm delivered our shiny brand-new Prometheus stack, we began working on telling it where the things that it needs to scrape are.</p><p>For our applications that are not yet running in Kubernetes we just pointed Prometheus to our <a href="https://www.consul.io/">Consul</a> service via a <em>consul_sd_configs</em> configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E1c2ckl0bfEBql0yWvH73Q.png" /><figcaption>Helm template to configure Prometheus target discovery using Consul</figcaption></figure><p>This will let Prometheus discover all the services that Consul knows about and Prometheus will just start looking for a <em>/metrics</em> endpoint and start scraping those metrics.</p><p>An additional step is needed for custom metrics that we have added which have a different endpoint, as well as Rabbitmq services since they serve metrics from <em>/api/metrics </em>instead of <em>/metrics</em>. For these cases, we list all the special Consul jobs in a values file as usual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zG8ZV3qZwjngGPa2gNms6w.png" /><figcaption>YAML description of custom Prometheus targets to scrape</figcaption></figure><p>These values are then interpolated into a template like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-c05OpBNW0WxcwpMFFb8A.png" /><figcaption>Template to generate custom target discovery from YAML</figcaption></figure><p>All the above then comes together on a final template that delivers the completed Prometheus scrape configuration as a Kubernetes Secret:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-IMMg3hJ1jIpqcd-ScYSQA.png" /><figcaption>Helm templates can have multiple layers, like Shrek</figcaption></figure><p><strong>Prometheus service discovery part 2 — applications migrated to Kubernetes</strong></p><p>Once migrated, our applications are deployed to Kubernetes as different releases of a single Helm chart we built called <em>sk-application</em>. This chart sets up all the Kubernetes resources needed to run the application, like a Deployment, Service, Jobs, and so on. The chart also creates a resource of special interest to us in this post: a <em>PodMonitor</em>.</p><p>A <a href="https://docs.openshift.com/container-platform/4.5/rest_api/monitoring_apis/podmonitor-monitoring-coreos-com-v1.html">PodMonitor</a> is a custom resource that lets you specify a group of selectors to match Pods running an application, then specify which of their containers can be scraped and how often. You can also list a group of Kubernetes labels from those Pods that can be transferred directly as Prometheus labels when scraping.</p><p>This is how one of the generated PodMonitors looks like after an application gets deployed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ati_u4LIK2E7Qe8TTOOFRA.png" /><figcaption>Look for any pods running charts-service and scrape both the app an haproxy containers for metrics</figcaption></figure><p><strong>Adding PrometheusRules</strong></p><p>Let’s take a deeper dive into how our new Chart gets a new group of Prometheus rules into the Prometheus stack. Our values file for rules contains a single key called <em>additionalPrometheusRulesMap </em>which is made up of groups of rules under a common key.</p><p>For example, let’s say we want to add a group of rules that will keep an eye on Redis and let us know if it’s having a bit of a rainy day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hf-niXvtPL-3CScvpfdy3A.png" /><figcaption>Sample rule to keep an eye on Redis</figcaption></figure><p>In this example, we defined a Redis key with a single group of rules, of which the first one will alert us when the Redis service is not operational on one of the Redis nodes.</p><p>Next let’s have a look at the template that our chart will use to create a PrometheusRule for each one our rule groups:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZkr3ec-WZaPD15wZDHz9A.png" /><figcaption>The notoriously mysterious Kubernetes List resource type</figcaption></figure><p>Let’s see step by step what’s going on here:</p><ul><li>This template creates Kubernetes <em>List</em> definition</li><li>Each item on the list is obtained by parsing <em>additionalPrometheusRulesMap</em> from the supplied values file and looping through each subkey. From each subkey the chart will create a <em>PrometheusRule</em> custom resource.</li><li>In our example above, we will have a PrometheusRule with a group called <em>redis</em> that holds all the alerts in the group, like our <em>RedisDownOnInstance</em> alert.</li><li>As soon as a new PrometheusRule resource is created the Operator picks it up and validates its rules. If all looks good, the new rules are incorporated into the Prometheus configuration.</li></ul><p><strong>Prometheus Adapter</strong></p><p>The next step is making use of Prometheus metrics to have a smarter way of scaling our application pods up or down. Out of the (proverbial) box, Kubernetes allows you to scale the pods in a deployment based on resource metrics like CPU, but the real benefit comes from setting up your autoscaler — called Horizontal Pod Autoscaler or <em>HPA</em> — with custom metrics.</p><p>The Prometheus adapter lets you make specific Prometheus metrics available to the cluster via the Kubernetes custom api. In our case, we wanted all HPAs related to our applications to use a custom Prometheus metric (<em>worker_usage</em>) to determine the number of pods at any given time.</p><p>Worker usage is the number of current Haproxy sessions vs the limit of sessions defined in the backend, i.e. the total number of workers. We could set our HPA to watch this metric and aim to keep it under 70%, for example. This will prevent requests from being queued by Haproxy and keep response times low.</p><p>The way to achieve this is relatively straightforward. All that’s needed is to create a <em>ConfigMap</em> Kubernetes resource that holds the new metric:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uaXCGj_-nwtHE2WfwjeDVw.png" /><figcaption>Defining a custom k8s metrics from a Prometheus query</figcaption></figure><p>Then all we have to do is tell the Prometheus adapter to use the <em>prometheus-adapter</em> configMap as the source of its rules. This is done via Terraform with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*e7iE4f7ZLJLitjojbBNXQA.png" /></figure><p>Once this is applied, our new metric should be available via the custom metrics endpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGDTYMovUXEjxl1300qvNQ.png" /><figcaption>Getting a raw custom metrics from k8s</figcaption></figure><p>In this case, worker usage is for one of the pods is currently at 50%.</p><p>Now we can set the final piece of this auto-scaling puzzle, and instruct the HPA template in our custom applications chart, to use that metric to determine when to scale pods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ShFDT6zGw2IkQVM9PuAxw.png" /><figcaption>Each app can have its own utilizationTarget</figcaption></figure><p><strong>Wrapping things up</strong></p><p>The final bit of configuration to get our new Prometheus stack ready for prime-time usage was securing the Prometheus web UI, Grafana, and Alertmanager behind the <a href="https://cloud.google.com/iap">Google Identity Aware Proxy</a>. This product makes web applications available to authenticated Gmail users within Songkick without the need for them to be on a VPN. In practice, this means that one of our engineers can get an alert via Slack on their phone, and click through to dive deeper into the Prometheus metric, have a look in a Grafana dashboard and determine the next course of action without having to open their laptop.</p><p>For us, the next phase of work is now to start migrating services and frontends in anger with the confidence that any changes in quality of service will be brought to our attention in a timely fashion so our users don’t miss out on any livestreams or upcoming live music events.</p><p>Let the good times roll! <br>(and get an alert when they ain’t so good)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7fdc30176d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/setting-up-a-smooth-prometheus-operator-c7fdc30176d1">Setting up a Smooth [Prometheus] Operator</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Kubernetes logs into BigQuery cost-effectively]]></title>
            <link>https://medium.com/songkick-dev-blog/getting-kubernetes-logs-into-bigquery-cost-effectively-bc6b67ddb923?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/bc6b67ddb923</guid>
            <category><![CDATA[ruby]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Eric Escalante]]></dc:creator>
            <pubDate>Tue, 17 Nov 2020 17:17:08 GMT</pubDate>
            <atom:updated>2020-11-17T17:17:08.207Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we navigated those JSON logs safely to BigQuery harbour</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jYvZuXd2aOkucaDM87kxLQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@usefulcollective?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mildly Useful</a> on <a href="https://unsplash.com/s/photos/logging?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>At Songkick, we’re always looking for new ways to make the most of available technology in order to work smarter, faster, and ensure we can sustain growth as efficiently and effectively as possible.</p><p>This spirit has inspired us to move away from a conventional datacenter to Google Cloud Computing Engine instances <a href="https://devblog.songkick.com/how-we-migrated-to-google-cloud-8f0b352fdde4">a couple of years ago</a>, and now has sparked a project to take on a gradual move towards Kubernetes clusters. This initiative requires us to establish first a firm foundation of some key pieces of technology before we start shifting and lifting applications into pods. Things like monitoring, security, and a couple more, chief among which is logging.</p><p>In our current cloud setup, we import all logs from our applications into BigQuery because it makes searching for patterns and debugging things extremely easy. We wanted to keep that functionality on applications running in Kubernetes.</p><h4>Kubernetes in Google Cloud</h4><p>Google offers a managed Kubernetes product called Google Kubernetes Engine (GKE) that allows us to focus more on the application side of things, and take the lower-level cluster management out of our hands. All logs generated from a GKE cluster are in JSON format and get ingested by default by the Cloud Logging product (formerly Stackdriver Logging). <br>Cloud logging has many interesting features, like setting filters to avoid ingestion charges (you have to pay for log message past a threshold) and the ability to route messages to destinations called “sinks”, one of which offers a direct-to-Big-Query option. This would have been optimal (<em>and would have made this an awkwardly brief post</em>) but this feature uses a streaming model, which given the volume of log messages our applications generate, would have been expensive in the long run.</p><p>So, the next logical step was to use a Google Cloud Storage (GCS) sink, which lets us save all the selected log messages into a bucket of our choosing, where they could be easily imported into BigQuery using the NEWLINE_DELIMITED_JSON format option.</p><h4>Setting up our apps to log to k8s</h4><p>The very first step was happily quite straightforward, thanks to the brilliant work of previous generations of songkickers on the songkick-logging gem. This little gem handles the logging functionality of all our ruby apps, adjusting its behaviour depending on which environment the code is running. It was architected with a plug-n-play model in mind, which allows it to channel messages to syslog, a test file, standard output, or to Sentry (an external error discovery service) very easily.</p><p>In order to get our apps Kubernetes-ready we needed to add a new component that would send log messages to stdout as JSON strings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovYtGTXHT53S42kq-9DNOQ.png" /></figure><p>Next, we just needed to tell songkick-logging to use the new StdoutJSONLayout class when our applications find themselves running inside a Kubernetes pod. Happy days! 💃🕺🏽</p><h4>But let’s talk about that JSON, though</h4><p>Here’s how a single Kubernetes log entry looks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZDwb7YElrfs8Xb8asOC2A.png" /></figure><p>As can be seen, the message resulting from a logger invocation from one of our apps, ends up in the jsonPayload field of the Kubernetes log entry and includes the fields we declared in our new logging class: hostname, level, message, and timestamp. The name of the pod, the environment, and other bits of information are part of the labels.</p><p>Brief diversion first, though: let’s talk about those labels (cue suspenseful track).</p><p>Kubernetes uses labels as a way to tag and easily perform query-based operations on all the variety of objects that live in the cluster. These labels are part of the metadata definition of each resource. In the <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/">official documentation </a>we find:</p><blockquote>Labels are key/value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (/).</blockquote><p>This optional prefix differentiates between keys that are private to your cluster setup and keys that are provided by Kubernetes or some other third party.</p><p>This feature became a multiple-slash-shaped stone in our cool suede trainers, since while the JSON produced by Kubernetes is valid, BigQuery had a very distinct opinion about what constitutes a valid column name:</p><blockquote>Invalid field name “k8s-pod/environment”. Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long.</blockquote><p>This means we can’t use NEWLINE_DELIMITED_JSON to import JSON directly, indeed a bummer, but not a show-stopper at all. BigQuery gives you ways to handle all sorts of format issues, and the option we went for was this: Import all JSON messages as CSV into a temporary table with a single STRING column then proceed to use the JSON_EXTRACT SQL function to extract the fields we actually cared about and append them to the real logs table.</p><h4>Listing our ingredients</h4><p>Now that we have a (hopefully) working plan, let’s outline the things we need to get this little little boat merrily down the stream:</p><ul><li>A GCS bucket</li><li>A sink and a filter definition in Cloud Logging to send selected messages to that bucket and discard the unwanted ones to avoid ingestion charges</li><li>A dataset and a table to host the logs in BigQuery</li><li>A service account with the right permissions to read from the GCS bucket and perform data operations in BigQuery</li><li>A shell script that handles the import operation</li><li>Add a cron job that invokes the script at a time of our choosing</li></ul><h4>Setting up the bucket, sink, and table (without mopping)</h4><p>Whenever possible, at Songkick we aim to treat infrastructure as code. For this purpose, we rely on Terraform, which makes it very easy to work with Google Cloud resources.</p><p>We start by setting up a bucket with the low cost storage class called NEARLINE and adding an expiration rule that deletes all files after 30 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*fCqTFkIXKOmC2mj4mdf0cQ.png" /></figure><p>Next we need to configure Cloud logging to achieve two objectives: first, to save in the bucket we just created all entries that both have a type called k8s_container and also belong to any container named app. Second, to exclude from ingestion any other log entries with the type k8s_container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*smgAGmaXPGwt-mH-oEqY7g.png" /></figure><p>This sets up the “source” side of things, so next step is prepare the “target”. For that purpose, we need a dataset and a table in BigQuery that will host all the logs fetched from the GCS bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyxoJqNiV_-chE0KMJIp9Q.png" /></figure><p>The above creates a dataset for our logs table, this next bit creates it and defines the schema we want to work with (timestamp, level, message, application, etc)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPCQqYt0fsCUpE_9GWX7HA.png" /></figure><p>Since log messages can reach high volumes, we decided to partition the table on periods of one day using the timestamp column. We don’t care for log messages older than 30 days, so they can be purged after that.</p><h4>It’s time to kube!</h4><p>Now that the source and the target have been created, it’s time to define the Kubernetes resources that will do the actual deed! Let’s work our way from the inside out (like that eager little alien in the 1979 classic family film).</p><ul><li>First, we need access to the the gcloud and bq commands.</li><li>Next we need a shell script that will do a bit of google cloud setup, execute all the bq queries for the desired date, and perform some cleanup when done.</li><li>Next, we need to add that script to the Kubernetes cluster.</li><li>Finally set up a cron that invokes the script regularly.</li></ul><p>So, to start with, here’s the heart of what the shell script does (brace for a slashes galore)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujO2oCmWM2x2KPKVQFFFiQ.png" /></figure><p>We resorted replacing the characters BigQuery didn’t like in the keys with underscores, which get cleaned up and turned into something sensible at a later stage. Since the jsonPayload contents are escaped JSON strings, we also need to remove double quotes and slashes. It’s a bit of an eyeful, but it gets the work done.</p><p>After this, our next mission is to have this script available inside the cluster. For this, we can make use of the ConfigMap resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kopwq7VHhEO_EJRx73gc5A.png" /></figure><p>Now, our shell script can be mounted anywhere within the current cluster namespace via the import-logs-script ConfigMap.</p><p>Finally, we need to define a cron job that provides gcloud, bq, and runs our nifty shell script at specific times. We can accomplish this using the CronJob resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qi0PE07sSASEUut4yTcySA.png" /></figure><p>This defines the following key items:</p><ul><li>The schedule: Our job will run every hour, on minute 1 . This is so, because GKE will dump all logs into our GCS bucket sink hourly.</li><li>The container image: We want our job to run in a Google Cloud SDK container, since it will have both the gcloud and the bq commands ready for us to use.</li><li>Mounts the ConfigMap that contains our shell script into the /root/script/import-logs.sh path (and makes it executable).</li><li>How many successful and failed job executions to keep.</li></ul><p>And that is it! We left out the service-accounts setup because there are tons of posts about that already and we wanted to focus a bit here. We now have a table in BigQuery that gets updated one minute past each our with all application log messages from our clusters. Here’s a sample of how the imported data looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N4iIo--5SqPIv41MAHo_TA.png" /></figure><p>Thanks for reading this far, we hope this short coming-of-age story of log message transportation and transformation managed to add a few insights (or at least a little smile).</p><p>Keep an eye out for this space, as more stories of our Kubernetes migration are sure to appear every now and then!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc6b67ddb923" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/getting-kubernetes-logs-into-bigquery-cost-effectively-bc6b67ddb923">Getting Kubernetes logs into BigQuery cost-effectively</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iOS App Architecture for Web Developers]]></title>
            <link>https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-c289c191052c?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/c289c191052c</guid>
            <category><![CDATA[view-controller]]></category>
            <category><![CDATA[mvc]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Chloe Verity]]></dc:creator>
            <pubDate>Fri, 16 Oct 2020 13:31:20 GMT</pubDate>
            <atom:updated>2020-10-16T13:30:39.138Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 3: View Controllers</h3><p><em>This article is part of a series of posts dedicated to helping web developers new to mobile development bridge the gap between building web apps and iOS using the common MVC pattern, as told by two developers at Songkick who have recently made the jump themselves.</em></p><p><em>If you haven’t read the previous post, please </em><a href="https://devblog.songkick.com/ios-app-architecture-for-web-developers-f61ded0ef427"><em>click here.</em></a></p><p>If you’ve made it this far, well done!</p><p>There’s only so many times I can introduce what the MVC pattern is using the same diagram — so this time, let’s use an analogy. I want you to think of your favourite meal. Mine is a good ol’ pepperoni pizza. Now, let’s say, for some reason I wanted to make my own pizzas for a pizza party I’m having with my friends; imagine I’ve filled the fridge with all the raw ingredients. We have all the makings and mechanics of a pizza stored away in there: the necessities like dough, tomato sauce, mozzarella, and some other bits and pieces for toppings, that will be used depending on the wants of the person eating it. That’s basically the Model in a nutshell; a data blueprint (of a pizza).</p><p>And if, as we saw from the previous instalment in our series, a View encapsulates all that the user sees and is able to interact with, then at our pizza party, the View is each encapsulation of a pizza that sits before one of my friends; a combination of their choices (toppings) while still abiding by the fundamental architectural rules of the pizza (for example, without dough, it’s not a pizza, and no, we’re not counting cauliflower pizzas). So, then what the hell is a View Controller?</p><p>In this case, the View Controller would be the recipe. The recipe dictates the logic behind how we use the raw materials/data from the Model, and how we turn it into something tangible that the user can see and interact with (the View). It’s flexible (there are many different recipes that give you a pizza), yet if the View Controller isn’t doing it’s job correctly, then the final product isn’t displayed on the screen properly, much like how you could assemble the ingredients in any way, but you might not end up with something that constitutes a pizza.</p><p><strong>So what actually goes in the View Controller?</strong></p><p>View Controllers, considering they are the bridge between the Model and the View, are largely considered to be skewed towards the View; each View Controller (usually) represents one screen of the app, and has one or more corresponding Views to make up that screen.</p><p>As discussed in the View portion of this series, the View Controller contains references to most, if not all, of the objects that appear on the View. This is to handle user input, or so that they can be interacted with; for example, in the View Controller, we can give a button specific instructions on what it needs to do when pressed. We do that like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qRnLjrtblO-4HyC0" /><figcaption>Anatomy of a button created programmatically</figcaption></figure><p>In this example, we can see that on loading the View, we first give the viewAllButton several properties, such as colour, title colour and title text. The reasons you might want to set these attributes programmatically are manyfold; you might want to localise the text so it can appear in 15 different languages, you might want to, as shown above, manipulate an attribute depending on the state of the object (as we can see where we set a different title colour for the button depending on whether the button is selected or not), perhaps you want to reuse the object later, perhaps you want to populate text or manipulate the View depending on data from the Model (or maybe you just have a general aversion to interface builder), all are perfectly valid.</p><p>The next thing to do is add a target to the button, which has three parameters: the target, an action, and the control events. The target is the object whose action method is called. The action is the method you want to invoke when the object is interacted with, and the control event is the type of gesture event that launches it (touchUpInside is the most used gesture, for it reflects a simple tap, but there are several that you can invoke this way). Think of this as the Swift version of an event listener.</p><p>And finally, you need to create the method that you want to call. Throwing it back to Models in Section 1, we might need to create, update or delete database information via our Model. Imagine serving your friend a pizza, but lo and behold you’ve forgotten something: you go back to the recipe (View Controller) and realise you’ve forgotten the anchovies, so you go to the fridge (Model) to grab them, use the recipe again to understand that, yes, they’re placed on top of the cheese, and once you’ve executed those instructions, voilà, your friend is served again with a delicious, updated pizza (View).</p><p>It might be apt just to call a method in our View Controller, or, if the View Controller in which this method lives is the child of another, and we need access to the parent controller’s methods and variables, we might need to pass this method call up to the parent via a delegate. Above, I’ve simply printed something, but as I said, this could be anything your heart desires.</p><p><strong>Why MVC?</strong></p><p>MVC is not unique to Swift; if you’ve ever built a Rails project you’ll know that you should have separate folders for each Models, Views, and Controllers, making it easier for you to identify and understand what each individual piece of code does, and where it fits based on that functionality.</p><p>Now, the ideal scenario is where responsibility is pretty equally spread between the Model, View and Controller. The reality is, however, that the View Controller ends up with disproportionately more responsibility than the other sections, since they seem to lend themselves more naturally as the “place to put things”. What I mean by this is that, if something doesn’t immediately appear to fit the bill of “logic” or “design”, then it gets lumped in the View Controller without a second thought, and before we know it we end up with these massive View Controllers.</p><p>This impacts navigability, sure, but you’ll find that actually, if used as intended, MVC structure makes it easier to navigate through and understand what’s going on in the code base, and makes it possible for people to not only make modifications and debug, but for multiple people to do so simultaneously due to the separation of responsibilities.</p><p>For this reason, if your app necessitates structure and relies on performance, MVC is a good contender, if you are working on a single page app, you probably shouldn’t bother. MVVM (Model, View, View Model) is an alternative, which, rather than having a Controller which ensures that the Model and View don’t talk to each other, has a View Model that actually binds the data between the View and the Model, allowing them to directly talk to each other. MVVM would be the choice for a single page app, since it’s simple and allows direct communication and therefore can move quickly and continuously save to databases. Yet, since it relies on binding the data, it consumes a lot more memory, meaning that it would not be the first choice for larger applications, who as we’ve said should probably be using MVC.</p><p>In pizza terms, MVVM would be like me making all my friends’ pizzas relying solely on my memory — that’s fine if they all want the same thing, but if they don’t, I’d be better off having the recipes in front of me. Ultimately, choose the way that’s going to get the pizza on the table, fastest, easiest and leave everyone satisfied.</p><p>Happy p̶i̶z̶z̶a̶ ̶m̶a̶k̶i̶n̶g̶ engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c289c191052c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-c289c191052c">iOS App Architecture for Web Developers</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iOS App Architecture for Web Developers]]></title>
            <link>https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-f61ded0ef427?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/f61ded0ef427</guid>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Chloe Verity]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 12:51:34 GMT</pubDate>
            <atom:updated>2020-07-09T12:53:04.582Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2: Views</h3><p><em>This article is part of a series of posts dedicated to helping web developers new to mobile development bridge the gap between building web apps and iOS using the common MVC pattern, as told by three developers at Songkick who have recently made the jump themselves.</em></p><p><em>If you haven’t read the previous post, please </em><a href="https://devblog.songkick.com/ios-app-architecture-for-web-developers-da9d7a132ce9"><em>click here.</em></a></p><p><strong>Views in iOS</strong></p><p>As you will recall from our previous post (by one of our Swift Senseis in residence, Alex) on the Model-View-Controller pattern, Views are the third of the MVC trio that — yep, you guessed it — represents <em>all that we see</em>. It’s the representation of the user interface , the “face” of our app. Views are not where the business logic of the program resides — we’ve seen that in the models — instead, the View illustrates the interface, detecting user interaction and enabling navigation through the app. The interface allows the user to create an action that might send new information or instructions back to the Controller, which will then relay those message(s) to the Model. A View should not speak directly to the model; instead it should communicate to the ViewController that manages it<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*aneeATA8SIv06Zy9-AOIRw.png" /><figcaption>The MVC approach; as we saw in the last post</figcaption></figure><p>When a controller has decided <strong><em>what</em></strong> should be presented on the page, it loads the View. The View is likely composed of a set of UIView objects, which then draw their specified contents, which may in turn be other views.</p><p>Views, similarly to HTML elements, can be nested inside each other, where the parent views are a <em>superview</em>, and the children are <em>subviews</em>. Apple provides a base class (UIView) that all other view classes should inherit from. Since these view classes are now subclasses of UIView, they inherit all of its properties and methods; the parent class now becomes the superclass. Also similarly to HTML, this allows for the repurposing of views across the app; whether it be a more basic, built-in view like a UIButton or UILabel, or a more complex, custom view that is created by you for a specific purpose, unique to your app.</p><p><strong>How do I create a View?</strong></p><p>Great question. You have options! In general, your app (depending on its size) will likely use a combination of these options.</p><p><strong>XIB </strong>it!</p><p>One thing unique to Swift is Interface Builder. Where developing on the web can have a slow feedback loop, stopping and starting to check on the results of a small change, IB allows you to fully visualise your app (on any iOS device you want, no less). You can tie components to the edge of a screen, or to parent or sibling views using constraints in autolayout, leading to a great user experience. Changing things like size, colour, text, in the attributes and size inspector, and even managing its connections (i.e. interactions) to a specific event in the View Controller (we’ll talk more about this later) in the connections inspector means that everything is relatively easy to manage — especially for newcomers to both your codebase and Swift in general. It’s also a great way to prevent your codebase from becoming inflated with unnecessary lines of code, and reducing the amount of bugs you might produce in them. Saying this, it’s worth noting that version control in XIBs (and Storyboards too) can get a touch hairy if several developers on your team are working on the same files.</p><p>The<strong> Programmatic</strong> approach</p><p>You can of course, write the same code programmatically (and actually, more, since XIBs lack some customisations that are possible in code). If performance is key for your app, it’s important to know that the file loading time of a XIB file is slightly longer than building the same UI programmatically. Some developers also go programmatic over XIBs since you have more control, and it can be easier to debug, though more difficult to visualize.</p><p>A never-ending<strong> Storyboard</strong></p><p>A storyboard outlines the sequence of screens in an app, for example a signup flow, and the relationships between them. It can be a great way to visualize the UI flow of certain user journeys, and divide up what would otherwise be a complex navigation sequence into smaller, more manageable flows. Conversely, some dev teams avoid storyboards since it’s hard for multiple developers to work on them at once; and the larger your app the more complicated the web of flows may become.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7MENAyLkfal2YGjpDZgfw.png" /><figcaption>Example of how storyboard is used to visualise user flows</figcaption></figure><p>As I said earlier, the larger your app gets, the more likely it is that for certain situations and user flows, you will pick which of these suits your needs. Using any of these paths you can create a UIView, in each of which you will have access to the object library to use buttons, labels, switches, the whole shebang! This is particularly easy in IB or Storyboard since dragging and dropping one of these objects is essentially all it takes to add an instance of it into your View. Whichever option you choose, you should try and specify a View’s relative size and position to its superview; creating rules with Auto Layout will then govern how the size/position of the view will change in response to changes in the rest of the view hierarchy.</p><p><strong>Outlets and Actions</strong></p><p>As I said previously, we don’t want any business logic to reside inside the view, since its sole responsibility should be presenting the UI that the ViewController tells it to. Ideally, the View shouldn’t even be <em>aware</em> of the model.</p><p>So, how do we handle user interaction if there’s no logic allowed?</p><p>Outlets are used to make connections between objects in the views, and variables in the code. You can make these connections by ctrl-dragging from the XIB file into the ViewController. Now that we have the reference to the View stored in the ViewController, you have access to its properties and the ability to update them. You might need them, for example, to fetch dynamic text or images; so you simply place an outlet for a View in your ViewController. You don’t need to store references in the ViewController for each and every View in the hierarchy, really only the ones that need to be changed later, or with which you need to specify or report a user interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KB6R_fACQKKVgJ8oRfktw.png" /><figcaption>To add a new outlet, action, or outlet collection, ctrl-drag the component from the XIB to the ViewController</figcaption></figure><p>Actions are similar to Outlets, but go in the opposite direction. Outlets are variables that connect code to the layout, and are loaded when the XIB is loaded; using Actions is a way of making layouts trigger code — they are functions that the objects loaded by the XIB can call. An Action takes one argument: a sender. The sender is the specific UI element that will be calling the method. Actions are also created by ctrl-dragging from the View element to the ViewController.</p><p><strong>Context please!</strong></p><p>Let’s say we have a UIView that contains a UIButton and a UILabel. We want the UILabel to change its text when the UIButton is pressed.</p><p>When the UIView’s controller instructs the view to draw itself, the Outlet variables for both the UIButton and UILabel will be loaded, and hence creates their connection from the Interface Builder to the property in the ViewController. You can check that this connection exists by looking in the connections inspector in the inspector toolbar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*r3pMmxKAvrsndlfncvlQnw.jpeg" /><figcaption>The Connections inspector in XCode</figcaption></figure><p>When a user interacts with the View, for example pressing our UIButton, the relevant Action for that specific UIButton will be triggered, and the ViewController will interpret the interaction; usually a state and/or some information will be passed back to the Model. What will typically happen is the Model will use this information to perform subsequent logic, and will pass its findings to a ViewController (maybe the same one, maybe different). That ViewController in turn will interpret, and will load a View (or Views) which will then draw itself on the screen. In our case, once the Action is triggered, the ViewController will be made aware; and can subsequently perform a function that uses the Outlet tied to the label to change its text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-t6eJxLDBnNXQdXKi4T8Q.png" /><figcaption>In this image, the button is linked to an IBAction that will update the label’s text</figcaption></figure><p>Where the ViewController is what handles the events that come down the responder chain, it’s the View that usually handles their own touch events, and reports the results to the ViewController via an associated delegate i.e. our Action method.</p><p><strong>Conclusion</strong></p><p>In this part we’ve covered what a View is, how does it tie in with our MVC model, how we can set one up, and how to use one to allow a user to interact with your UI.</p><p>I’m aware I’ve slightly stepped over into ViewController territory in this post, though it’s difficult to separate the two since each View is managed by a ViewController. We’ll cover a lot more of the responsibilities of a ViewController in our third and final post by our next Swift Wizard, Felix!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f61ded0ef427" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-f61ded0ef427">iOS App Architecture for Web Developers</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iOS App Architecture for Web Developers]]></title>
            <link>https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-da9d7a132ce9?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/da9d7a132ce9</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Alex Dodd]]></dc:creator>
            <pubDate>Fri, 12 Jun 2020 09:43:06 GMT</pubDate>
            <atom:updated>2020-06-12T09:43:06.326Z</atom:updated>
            <content:encoded><![CDATA[<h4>Part 1: Models</h4><p><em>This article is part of a series of posts dedicated to helping web developers new to mobile development bridge the gap between building web apps and iOS using the common MVC pattern, as told by three developers at Songkick who have recently made the jump themselves.</em></p><p>Architecture Patterns in software development are helpful tools that help us compartmentalise the “how are we going to build this?” away from the “what are we going to build?”. Once we are familiar with these patterns we can start to translate early designs and ideas into organised projects and structured code bases which are easily extendable as we get closer to the final product!</p><p>One of the most common software architecture patterns is MVC, which stands for Model-View-Controller, and organises all of the objects in our apps by giving them one responsibility only (is this object a Model, a View or a Controller?), as well as dictates a simple basis of communication between these objects.</p><figure><img alt="Diagram showing the MVC architecture (model, view, controller)" src="https://cdn-images-1.medium.com/max/1024/1*kTAtul78NfgoXSah-6_nJA.png" /><figcaption><em>Models and Controllers can send and receive data to one another, so can Views and Controllers, but Models and Views cannot.</em></figcaption></figure><p>MVC is a common pattern for web apps, so much so that Microsoft named their web app framework after it! Ruby on Rails apps are also built on top of it, so chances are you’ve worked with an MVC web app before. What’s so great about the prolificness of this pattern is Apple also adopted it for its iOS app framework, making the jump from web development to iOS development that little bit easier, because we can structure apps the same way that we have done in the past for the web.</p><p>So let’s dive into the Model part of a typical iOS architecture and see how it feels familiar, take a look at some of the common patterns we can adopt to make our lives easier, as well as a few gotchas to look out for as a new app developer. Let’s begin!</p><h3>The Model</h3><p>Models are data, no change there. In iOS models are even simpler, usually taking the form of simple <strong>struct</strong>s that define the name of the model and its properties, as well as any methods it may need. This is probably the biggest departure from most web frameworks where this would usually be achieved with <strong>class</strong>es, but in Swift unless you need to use features like <a href="https://docs.swift.org/swift-book/LanguageGuide/Inheritance.html">inheritance</a>, <a href="https://docs.swift.org/swift-book/LanguageGuide/TypeCasting.html">type casting</a> or <a href="https://docs.swift.org/swift-book/LanguageGuide/AutomaticReferenceCounting.html">passing objects by reference</a> (psst, you probably don’t!), you can get along just fine with plain ol’ <strong>struct</strong>s.</p><h3>Repository Pattern</h3><p>In order for our mobile apps to show things to users we need to actually get the data from somewhere! Additionally we need to be able to create, update and delete our various models as the user completes tasks in our app, and all these things will probably need to be achieved from various parts of our app as well! Suddenly what was originally a simple task has turned into a complicated and critical part of our project. Luckily for us we have a solution: The Repository Pattern — which is simple, extendable, and <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>.</p><figure><img alt="Diagram of the repository pattern" src="https://cdn-images-1.medium.com/max/1024/1*d0L8HEph5mIowrmyuxs7qQ.jpeg" /><figcaption><em>The app’s controller layer only ever interacts with the data layer via repositories. The repositories then figure out where they actually get the data from.</em></figcaption></figure><p>In short, each of your business models have their own repository, which is used as a service layer between your app and your data. So if your app needs to show a user a list of their favourite musicians, it will ask the <strong>FavoritesRepository</strong> to fetch all the musicians liked by that user. How the repository does this doesn’t matter to the business logic. If the user then decides they want to add another musician to their favourites while in another part of the app, then the business logic does this via the same repository. This way the app only has one place that can talk to the various data sources we have (such as an external API or a local cache) to handle any new, updated or deleted models. Tidy!</p><h3>Observer Pattern</h3><p>Repositories are not only excellent data gate-keepers but they can also be used as the single point for a model to let other parts of your app know that the state of your app has changed. This can be achieved by adopting the Observer Pattern in your models.</p><figure><img alt="Diagram showing the observer pattern" src="https://cdn-images-1.medium.com/max/1024/1*qKk90CsCM6su06CK4fszXQ.jpeg" /><figcaption><em>Any number of objects can listen for a message, and any number of objects can post them. The NotificationCenter will figure out where to send them.</em></figcaption></figure><p>The Observer Pattern is simple: iOS’ Foundation framework has something called <a href="https://developer.apple.com/documentation/foundation/notificationcenter">NotificationCenter</a>, which is available out of the box. It has nothing to do with push notifications, and everything to do with Observers. An Observer is any object that subscribes to the NotificationCenter, by asking to be notified whenever a specific notification is sent there. Notifications always have a name, which is what the Observer uses to specify which notifications it wants to observe. Then on the other side of the fence is the Subject, which will post notifications to the NotificationCenter when something happens in its lifetime, complete with a name, and then the NotificationCenter will in turn notify all the different objects that are observing for notifications of that name.</p><p>Using this pattern it is possible for any number of Subjects (however it is usually only one) to send a named notification to any number of Observers. For a Repository, this is very handy as they handle all updates to the Model, which means that when they see that something has updated, they can broadcast the fact that something has updated to whichever objects want to listen.</p><p>For example, let’s say we have two lists in our app that contain all of the user’s favourite artists, called FavoriteArtistsList (which shows all their favourite artists) and RecentFavoriteArtistsList (which just shows their 5 most recently favourited artists). Both of these lists need to stay up to date with any changes to a user’s favourite artists (the user might unfavourite one, or add new ones, etc), so they subscribe themselves as observers for the <strong>didUpdateFavorites</strong> notifications with the NotificationCenter. Separately, the FavoriteArtists repository posts a notification to the NotificationCenter, with the name of <strong>didUpdateFavorites</strong> when the user adds a new favourite artist. Both of the subscribed lists receive a notification from the NotificationCenter, and in return update themselves to show the latest list of favourite artists.</p><h3>A patterned future</h3><p>As we have observed (sorry) with three simple patterns we can safely read, write and update our application data by containing it in models, use repositories to keep it in silos, and actually change the state of our app as our data changes using observers.</p><p>The M in an iOS app’s MVC can also be responsible for many other things like utility classes, parsing logic and so on, each coming with their own set of patterns that can help keep the code smells and lost hours to trying to understand an unorganised codebase at bay. However with the three patterns we’ve learned today the simple–sounding yet complicated task of getting data in and out of our application is tamed. How we actually show our users that data will be explained by our mobile pro Chloe with our next post in the series, Views in iOS. Until next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da9d7a132ce9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/ios-app-architecture-for-web-developers-da9d7a132ce9">iOS App Architecture for Web Developers</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Ran a Hack Day in the Age of Social Distancing]]></title>
            <link>https://medium.com/songkick-dev-blog/how-we-ran-a-hack-day-in-the-age-of-social-distancing-718b41b4c45?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/718b41b4c45</guid>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[work-from-home]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[songkick]]></category>
            <dc:creator><![CDATA[Laurent Baeriswyl]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 14:41:14 GMT</pubDate>
            <atom:updated>2020-04-28T14:41:14.785Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>As social distancing measures continue to disrupt our day-to-day workflows, we want to share some valuable lessons we learned from taking an intense and collaborative process into the online world.</strong></h4><p>At Songkick, we try to have a company-internal hack day once per quarter. Hack days are an opportunity for us to hit pause on our day-to-day work and explore new ideas for the company. It’s a way to freshen minds, think outside the box, and fuel our idea back-log for the coming months.</p><p>It’s important to note that hack days at Songkick are not competitive: the goal is for people to be working on things they usually wouldn’t, and to (hopefully) enjoy themselves in doing so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*enyxryP7oLVLl6sbX5QVyg.png" /><figcaption>Slide taken from an internal presentation</figcaption></figure><p>For this year’s first hack day, we gave ourselves the space to try a new format. We wanted to find better ways of including non-developers, new ways to spark ideas, and better ways of documenting and sharing our hacks. Unexpectedly, social distancing also prompted us to explore new ways for productive remote collaboration.</p><p>Hack days rely on people being in close proximity. Because you can get feedback very quickly, they are often deliberately unstructured, leaving space for people to organise themselves. When working from home this presents new challenges, and we had to make some adjustments to our usual format in order to tackle them.</p><p>Most of the changes were small and easy to implement, but made a big difference. As a result, the day ran smoothly and, surprisingly, more on-schedule than many in-person hack days before it.</p><p>Here are some of the things that helped us:</p><h4><strong>Structure is key</strong></h4><p><em>Adding more structure to processes that would usually be played by ear allowed us to reduce uncertainty for the team and ensured everyone has clarity.</em></p><p>We usually allow teams to choose how they will work together and communicate. Often, the priorities and work of a team change fast, and the lack of a defined structure can be conducive to creativity. When everybody is remote, the risk is that if communication breaks down, people may spend hours in front of their screen not exactly knowing what they should do or how they can contribute. Adding more structure presents a tradeoff: Teams lose some of their freedom, but in exchange gain confidence that everyone will be on the same page.</p><p>For our hack day, this meant that things that would usually be worked out on the day (teams and the division of work) needed to be planned in advance, and we made sure to set and communicate a clear schedule to everybody:</p><ul><li><strong>One week before:</strong> Teams are formed</li><li><strong>During the week before:</strong> Teams have a planning meeting, clearly defining what they hope to achieve and how they would split their work up</li><li><strong>10:00 AM on the day:</strong> Kick-off meeting, where every team introduces their hack</li><li><strong>5 PM on the day:</strong> Teams get to share their demos</li></ul><p>There was a strictly defined format and time limit to all demos. Teams had to nominate one person to speak on their behalf during the kick-off, as well as decide who would present the demo.</p><p>All of these things are small considerations individually, but they helped keep things snappy and moving along, allowing us to avoid stress and focus more on the actual hacking and building parts of the day.</p><h4><strong>Default to transparency</strong></h4><p><em>Sharing process and in-progress work with the whole team regularly allowed people to get a sense of what everyone is working on, alongside them.</em></p><p>Hack days give us the opportunity to look at what other people are building. We can see things come together in real time, and learn from each other. When everyone is working from home, it’s easy to miss out on that shared experience.</p><p>We wanted to replicate the feeling of being able to walk up to another team and ask: “Oh cool, what are you working on?” This meant that we needed a way of being able to look at another team’s progress while working. To try and get as close to possible to the in-person ideal, we went with the philosophy of making everything as accessible as possible.</p><p>For the day itself, every team had a public slack channel to communicate with each other on progress, post links, or discuss potential solutions and implementations.</p><p>Everybody was encouraged to join other groups’ channels if they were interested, just to spectate the progress.</p><p>For later reference, each team submitted a short doc; sharing what they did, why it’s important, and linking to any images, graphs, or code that came as a result of their hack. We could then easily compile and share them with the whole company, so that people could go back and see the things they may have missed on the day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhDqgqrLRmCRbDP6x4u9Ew.png" /><figcaption>Our hack day meeting minutes</figcaption></figure><p>As organisers, we also made the minutes of the meetings held in preparation of the hack day, including decisions made and follow-up actions, publicly available to anyone interested and who wanted to share their thoughts on what is important to them in a remote hack day.</p><p>These decisions meant that we maximised visibility across different projects on the day itself and could rest easy knowing that the things we built wouldn’t just disappear into the aether, but could serve as references and learning resources in the future.</p><h4><strong>Keep it human</strong></h4><p><em>Making sure people feel connected and have a social outlet while working keeps us grounded and happy in this difficult time.</em></p><p>Lockdown loneliness is a real phenomenon, and it can easily creep up on us and make the days difficult to get through. Hack days are meant to be fun, and they often are, in no small part due to the collaborative aspect of the work and the human interaction we get to indulge in.</p><p>Despite everybody working from home, we didn’t want to allow individual projects. The advantages are similar to pair programming: people are less likely to get stuck, and they are naturally sharing knowledge as they go. In this case there was another benefit: it meant people were talking to and engaging with each other.</p><p>Teams were encouraged to be on calls while working, even when not directly collaborating. While waiting for your code to finish running, why not share a few thoughts you had about <em>that new album</em>? Did you love it? Did you hate it?</p><p>Maintaining and investing in the little things that make days more enjoyable is more important now than ever, and will lead to people feeling happier and more included.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4k3-H7tWJz-cFdPGaB9dcw.jpeg" /><figcaption>Some happy Songkicker faces</figcaption></figure><p>The feedback we have received from the team so far has been largely positive, and it was great to see things running as smoothly as they did. As always, people worked on amazing projects, and we were happy to see that people seemed to enjoy themselves despite everything.</p><p>There are still some things that could be improved: Are there better ways to share demos in an online meeting? How much time should we allocate for them? We are still getting feedback and learning what people want us to work on in the future.</p><p>Not only will we probably continue working this way for a while, but even without a crisis, working from home and practicing flexibility is becoming increasingly common. As such, it’s important for us to keep figuring out how we can keep people included and productive.</p><p>We hope that the next hack day we run at Songkick will be from our offices again, and that we can then play things a bit looser — but until then, we will do our best to keep things running.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=718b41b4c45" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/how-we-ran-a-hack-day-in-the-age-of-social-distancing-718b41b4c45">How We Ran a Hack Day in the Age of Social Distancing</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Growth Framework Pt II: No Tester Left Behind]]></title>
            <link>https://medium.com/songkick-dev-blog/building-a-growth-framework-pt-ii-no-tester-left-behind-edf39277f7eb?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/edf39277f7eb</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[growth-mindset]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Lauren Braid]]></dc:creator>
            <pubDate>Tue, 31 Mar 2020 15:49:54 GMT</pubDate>
            <atom:updated>2020-03-31T16:16:51.618Z</atom:updated>
            <content:encoded><![CDATA[<p>In April 2018 the Engineering team at <a href="https://blog.songkick.com/building-a-growth-framework-2464435e9e46">Songkick started using a Growth framework;</a> a way for developers to simply and clearly understand where they currently are in their roles and what they need to do to grow.</p><p>This framework was an incredible success. Engineers and their managers were having regular and meaningful growth conversations, and were producing progression plans off the back of those conversations to ensure real growth occurred. As a manager of 4 developers this was awesome for me, however I felt like the tester I managed was getting short-changed. The growth framework was awesome but was made for developers and more senior levels of engineering. Career conversations with the tester felt ad-hoc and less meaningful for her. So I decided to create a Growth Framework for Test and <a href="https://www.songkick.com/downloads/growth-framework/sk-growth-framework-for-test.pdf">here it is</a> :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dkMEKP8se_Mfe1cBJ5drlQ.png" /></figure><p><strong>What does it look like?</strong></p><p>The amazing thing about <a href="https://www.songkick.com/downloads/growth-framework/sk-growth-framework.pdf">Songkick’s Growth Framework for Engineering </a>is how it could so easily be mapped to other disciplines across the company, which meant I could use the same 7 skills to measure growth in testing as we were using for the rest of Engineering:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uzbGOY3VeDFMHYxK" /><figcaption>Songkick’s 7 skill areas for growth</figcaption></figure><p>The Growth Framework for Engineering has 9 roles in it, across individual contributor and people manager tracks, however making a framework with that many roles for test, when we only have a very small test team, felt unnecessary. Instead I made it as complete as it needed to be for now, so the Growth Framework for Test has 5 roles within it; Software Tester 1, Software Tester 2, Senior Software Tester, Test Lead and Engineering Manager.</p><p>You can see here how I have created one of the roles using the 7 skills and example behaviours:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hBfQXWqAQ3KjoJ_M" /><figcaption>Software Tester (1) role</figcaption></figure><p>As you explore the Growth Framework you will also notice I do not dictate proficiency in specific tools in order to progress; such as automation.</p><p><strong>You do not need to know how to automate to become a senior level tester at Songkick. </strong>What you do need is to know is when to use the right tools for the job. Automation is a tool to help us with checking so we have more time for exploratory testing. The example behaviours, are just that, examples and should be taken as such.</p><p><strong>Using the Growth Framework</strong></p><p>In line with the rest of the Engineering team, every 3 months, the tester and I run through the Growth Framework for Test and work together to create a progression plan and OKRs. The Growth Framework has meant that we now discuss growth together at regular, and expected, intervals instead of the ad-hoc nature of the past. We have done this 3 times over the last 9 months and every time we have had productive and meaningful conversations. She is in fact now a Senior Software Tester :) Using this tool made that clear and easy to achieve! We used the framework to identify areas in Leadership and Communication that she needed to improve and as a result she took ownership of the release process for our iOS app, and also ownership of our bug process, and reporting, across the whole company.</p><p><strong>Introducing a Growth Framework to your test team</strong></p><p>Understanding what needs to be achieved for growth is incredibly important for both manager and report. The Songkick Growth Framework for Test means we now have a shared understanding, as well as a process to facilitate that growth. You may look at my Growth Framework and think it won’t fit your team or company. That’s ok. You can start how I started, by picking out the key elements that work for you, and then expanding and evolving them to suit your purpose. Create something that fits your needs, for now, and build on it. Once you have something, you can share it with your team and get feedback so you can understand what is missing and what isn’t right. Through collaboration and iterative improvement you can create something that you and your team buy into and value.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf39277f7eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/building-a-growth-framework-pt-ii-no-tester-left-behind-edf39277f7eb">Building a Growth Framework Pt II: No Tester Left Behind</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2019 in technology]]></title>
            <link>https://medium.com/songkick-dev-blog/2019-in-technology-b9fe9182b7e2?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/b9fe9182b7e2</guid>
            <category><![CDATA[react]]></category>
            <category><![CDATA[platform-migration]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Joe Rivett]]></dc:creator>
            <pubDate>Mon, 02 Mar 2020 12:03:08 GMT</pubDate>
            <atom:updated>2020-03-02T15:53:54.985Z</atom:updated>
            <content:encoded><![CDATA[<p>At its height songkick.com reaches over a hundred requests per second with upwards of 7000 users active on the site at a time, whilst our apps serve millions of users per month worldwide. To help us build products at this scale, our platform is built on the service oriented architecture approach where we have many applications each dedicated to a very specific aspect of the Songkick product offering. With this architecture comes some rather mundane maintenance tasks — like ensuring our frontend applications are running the latest version of <a href="https://rubyonrails.org/">Ruby on Rails</a> or ensuring our services are running the latest version of <a href="http://sinatrarb.com/">Sinatra</a> — but it also comes with some more exciting technical challenges, such as asking what technology choices would allow us to best serve our users or how we should build applications that scale. Our technology choices and achievements last year meant 2019 was one of the biggest years of change ever in Songkick’s technology history. Here are a few highlights:</p><p><strong>Cloud migration</strong></p><p>We can’t review the tech achievements of 2019 without opening with the cloud platform migration. The year saw the completion of an 18 month project to migrate all our applications and databases out of dedicated servers in a data centre and onto Google Cloud Platform. Our aim was to get to a position where we could <a href="http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/">treat our servers like cattle, not pets</a>, which meant we must be able to cope when servers are removed from instance groups, or ensure that servers are able to bring themselves online without manual intervention when we spin up new ones. This approach brings many advantages — for instance, servers no longer require complex manual set up tasks and subsequent long-term nurturing; instead we can spin them up on an ad hoc basis when we need them (such as scaling up to handle load). Why is this particularly useful for us? Well, not everyone in Songkick’s tech team would feel comfortable bringing new Linux servers online, but now anyone can scale up any of our applications with the click of a button in a web interface. The result of 18 months of hard migrating is an infrastructure that sits entirely within Google’s Cloud offering, with Cloud Instances for servers, Cloud SQL for databases and Cloud Storage for persistent data stores.</p><p>Interested in how we managed the migration process? Have a read of the blog post <a href="https://devblog.songkick.com/how-we-migrated-to-google-cloud-8f0b352fdde4">here</a>.</p><p><strong>Three’s a cloud</strong></p><p>In some cases, migrating applications to the cloud platform required an entire reworking to ensure they fitted well within the cloud model. Our ETL (Extract, Transform, Load) data processing suite is a good example. When you have a complex web of data processing tasks that must be run in sequence, it is not enough to rely on a single cron job kicking off the pipeline on a machine that in theory should cope with being killed at any time. To address this we migrated our ETL pipeline to Spotify’s <a href="https://github.com/spotify/luigi">Luigi</a> platform, which is designed to handle complex task dependencies and play very nicely in cloud environments. Check out <a href="https://devblog.songkick.com/migrating-our-etl-pipeline-to-luigi-on-a-cloud-8fde921cec51">this blog post</a> if you want to learn more about how we went about this.</p><p>Another example is our search system, used to search for events, artists, venues and cities on Songkick products. We previously used <a href="https://lucene.apache.org/solr/">Apache’s Solr</a> product for our search needs, but our team’s understanding of Solr was limited and moving to Solr’s cloud offering would have been an extra steep learning curve given other dependencies like Zookeeper. As a result we opted to migrate to Solr’s biggest open-source competitor, <a href="https://www.elastic.co/">Elasticsearch</a>. Elasticsearch makes it easy to set up clusters in cloud environments, and we already used it for our log processing, so it made sense to bring our generic searching into the same technology and reduce the learning requirements of our tech team.</p><p><strong>New products, new tech</strong></p><p>2019 saw the advent of Songkick’s dedicated Presales tool, which gives artists the ability to offer early access to tickets to their fans on Songkick. Since its launch in August, the product has been used to get presale codes into the hands of over six million live music fans. This product offered exciting technological challenges — how do we build an intuitive user interface where artists and their managers know exactly how the presale will be marketed? How do we build a reliable, scalable notification platform that guarantees users receive a presale code at the right time before on-sale? How do we ensure users across all our products get the best out of presales? The resulting product includes a tool to build presales in Songkick’s <a href="https://tourbox.songkick.com/">Tourbox platform</a>, and with a vision to build a reactive, user-friendly interface with real-time preview of the presale email, we opted for — you guessed it — React. React allowed us to quickly build out a presale compose tool that was easy to integrate into Tourbox, easy to iterate on and (relatively!) easy for our JavaScript-friendly developers to sink their teeth into. Behind the scenes, we use background jobs and RabbitMQ to ensure delivery of presale notifications — with queue consumers that are designed to be reliable, delivery-guaranteeing and easily scalable to cope with larger presales and to improve delivery speed of the notifications.</p><p>We also built presale information components on our iOS app, Android app, and songkick.com website so all our users can benefit from presale tickets.</p><p><strong>Playing catch-up with frontend technologies</strong></p><p>2019 also saw us brush off the cobwebs from some of our longest-serving frontend applications and modernise some of the technologies we use behind the scenes. The songkick.com website has finally got full SASS capability and the CSS to SASS migration is well underway. With this we can build more consistent features on songkick.com quicker and easier than we have previously been able to. We migrated our JavaScript tests out of a deprecated technology called JSTest into the much-loved <a href="https://jasmine.github.io/">Jasmine</a> framework, which we now use for all JavaScript tests across our products.</p><p>Frontend applications include our apps as well as our websites — so they got a bit of love too. We pushed to migrate all the Objective-C code in our iOS app over to Swift, because we feel that Swift is a more approachable, familiar language to our developers who mostly work with Ruby and JavaScript in their day to day. We didn’t quite migrate the entire app, but we’re at around 95% of Swift code coverage. In a similar fashion we shifted some of the Java in our Android app to Kotlin — Kotlin and Swift are similar enough to follow the same learning approach and reduce the learning deficit in our team.</p><p><strong>Education, education, education</strong></p><p>Inevitably with change on this scale comes the question of how we ensure our tech team are up-skilled to maintain our products and feel comfortable fixing bugs when they come up. To tackle this, we’ve run weekly talks for the whole engineering team on topics ranging from <a href="https://www.docker.com/">Docker</a> to SASS to the React component lifecycle. Further, we employ regular pairing in our day-to-day, we’ve run mob programming sessions and whiteboarding sessions, and will soon be trialling small, targeted sessions for developers where they work in small groups with a technology <em>knowner</em> to learn areas of specific interest.</p><p><strong>2020 and beyond</strong></p><p>So 2019 was an exciting time for technology at Songkick, and there’s a lot more coming in 2020. From moving to the amazing <a href="https://sentry.io/">Sentry</a> for all our error reporting needs, to improving the scalability of songkick.com by allowing us to cache pages in <a href="https://fastly.com/">Fastly</a>, watch this space for more exciting things coming up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9fe9182b7e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/2019-in-technology-b9fe9182b7e2">2019 in technology</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we migrated to Google Cloud]]></title>
            <link>https://medium.com/songkick-dev-blog/how-we-migrated-to-google-cloud-8f0b352fdde4?source=rss----e0e3c74f91b8---4</link>
            <guid isPermaLink="false">https://medium.com/p/8f0b352fdde4</guid>
            <category><![CDATA[songkick]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Paul Lawson]]></dc:creator>
            <pubDate>Thu, 23 Jan 2020 13:50:18 GMT</pubDate>
            <atom:updated>2020-01-23T13:50:18.010Z</atom:updated>
            <content:encoded><![CDATA[<p>At Songkick, we recently completed a successful migration from our data centre to Google Cloud Platform. I’d like to give an overall account of how we successfully delivered this project with a small team in around 18 months, while allowing the rest of the company to continue delivering exciting products and features to our users.</p><h3>Why migrate?</h3><p>For roughly 9 years before the migration, our infrastructure lived in a data centre just outside of London. This had served us really well over the years, but had a number of important issues:</p><ul><li><strong>Only a very small number developers on our team had the experience to debug and fix some low-level infrastructure problems</strong> that could occur, for example tricky networking issues. This led to us being over-reliant on those developers to fix occasional out of hours fires, and consequently them not getting enough sleep!</li><li>During recent years, <strong>we’d done enough to put out fires and fix bugs in our infrastructure, but not enough to actually improve it</strong>, keep it up to date, and bring it in line with modern practices. We were at a point where doing that would be a huge task (likely larger than migrating to the cloud).</li><li><strong>Adding capacity or making major changes to our infrastructure took too long</strong>. Buying new hardware, installing it and manually moving VMs around was time-consuming and made us slower to deliver as a team.</li><li><strong>Moving to the cloud would mean that we have access to new services and tools</strong> that would allow us to build even better things. For example, we are now using Google Cloud Machine Learning, Cloud Dataflow and BigQuery, and these have had a huge impact.</li></ul><h3>How did we start?</h3><p>With a minimum of fuss! Once we’d decided that something needed to be done, we assembled a small team of 4 engineers, had a meeting to discuss first steps, and sent an email to the company:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jlOW7IJLsR5DELAxleAhuw.png" /></figure><p>We then worked through the migration in <strong>5 phases</strong>:</p><h3>Phase 1: MVP</h3><p><strong>Goal: work in a small team to deliver an </strong><a href="https://en.wikipedia.org/wiki/Minimum_viable_product"><strong>MVP</strong></a><strong> of a single service to production in 3 months</strong></p><p>This included deciding on a hosting solution (self-hosted vs managed hosting vs various cloud options), deciding high level platform architecture and major technology choices (for example using terraform/puppet for provisioning), and migrating a single back-end service fully serving production traffic. And all of this with only four engineers in three months!</p><p>Some important aspects of this were:</p><ol><li>Getting a production service migrated as quickly as possible allowed us to find out very quickly if our overall approach was likely to succeed. Instead of building a non-production prototype in a fully-fledged infrastructure that looked great, and only further down the line finding that we couldn’t handle the levels of traffic we needed, or that the new service was much less stable than the old one, this approach allowed us to fail fast if we were to fail at all.</li><li>The small team effect: we’ve consistently found at Songkick that small autonomous teams tend to achieve more (per team-member) than larger teams, have more ownership of what they are building and are happier and more effective as a result. This recipe is even more successful when the team has a short deadline and a clear goal. Having a small team also allowed us to get started with minimal disruption to the rest of the company.</li><li>To achieve the above with only four engineers, we had to be very ruthless with our prioritisation, and defer building a bunch of infrastructure that we would normally require for a production service. The most extreme example of this was that for the MVP service we would not have any monitoring or logging! This is a decision we’d normally be extremely uncomfortable with, but decided it really was that important to gain confidence in the overall approach as quickly as possible.</li></ol><p>The outcome: we did it! After three months of work, we had a single small back-end service deployed to Google Cloud and serving production traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*PWUI0ob0joSisuwfFSWDBA.png" /></figure><h3>Phase 2: Build confidence</h3><p><strong>Goal: Prove that our high level technology and architectural choices will work (3 months)</strong></p><p>At this stage, the highest priority was to understand the biggest risks facing the success of the project and make sure that we could overcome them. As with phase 1, the idea here is that it’s much better to find that the approach needed a major re-think as early as possible, rather than when we’d already spent eight months migrating.</p><p>To do this, we sat down as a team and listed all the risks and uncertainties with our approach, and prioritised the most serious ones. A couple of examples of our biggest concerns were:</p><ol><li>Will it be too expensive?</li><li>Will the network performance be good enough (we had very stable network performance in our data centre and some of our applications may rely on this)?</li></ol><p>Next, for each prioritised issue, we came up with a plan to test how serious each concern was. For example, for the above questions:</p><ol><li>We spent time modelling everything we would need for our new platform, and then calculating likely best case and worst case costs.</li><li>We migrated the application that most relied on network stability and monitored it over several months for any issues.</li></ol><p>At the end of this phase, we found that we had no major issues, and we were ready to move onto phase 3:</p><h3>Phase 3: Building the foundations</h3><p><strong>Goal: Build our monitoring, alerting, logging, and deployment infrastructure (3 months)</strong></p><p>For each of the above, we got together to discuss requirements, then investigated the options available, then as a team made a decision (or decided to investigate more) before building the solution. Where possible, we tried to stick with solutions that were similar to our existing infrastructure, in order to reduce unknowns, and make it easier for engineers to transition between platforms. On the other hand, where we saw large benefits to moving to something new, we did so. For example, we completely overhauled our monitoring system to use prometheus, as this was much more cloud-friendly than our existing monitoring system.</p><h3>Phase 4: Execution</h3><p><strong>Goal: Move everything to the new platform! (9 months)</strong></p><p>The most exciting part of the project! This included moving all of our applications to the platform, including our frontend applications, services, databases, ETL system and more.</p><p>Some key aspects of this phase:</p><ol><li>We wanted to fully migrate one component at a time (and deprecate the data centre version of that component), rather than duplicate all of our infrastructure then do a big-bang release at the end. This meant we could deliver value to the company (taking advantage of our cloud infrastructure) as we migrated, pressure test our solution gradually and overcome issues early.</li><li>At this stage of the project, education for the rest of the engineering team was key. As well as the 4 core engineers on the team, the rest of the engineering team had a weekly rota where one engineer would join us each week and help to migrate our applications. This meant that everyone on the team could gain confidence working with, and maintaining, the new platform. As well as this, we ran regular talks and workshops on various aspects of working with our new infrastructure.</li><li>We made completing the project, and disbanding the migration team, one of our <a href="https://devblog.songkick.com/creating-a-technology-plan-for-2019-825cf88711d">Tech Team Goals 2019</a></li><li>About two months from completion, we decided on a deadline for turning off all the servers in the data centre. For any long-running project, you’ll find that there’s always more you can do as you get towards the end, and the final two months has a danger of turning into four months, six months or more. To avoid this, we regularly reprioritised and decided what corners we needed to cut to meet the deadline. For example, certain parts of our ETL infrastructure only run once per month, so we decided to not migrate them at all until after we shut off the data centre. Similarly, there were certain tasks that we had automated in the data centre that we decided to run manually and have the engineering team migrate them later down the line if we decided they needed automating.</li></ol><h3>Phase 5: Celebration!</h3><p>After 18 months of hard work we had finally done it, the data centre was no more!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*Nf27zu8PoT9Vjkn1Z2jLAQ.png" /></figure><p>After such a huge project it’s important to celebrate your success!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_RlH2UeYmRZpt0So9ny4g.jpeg" /><figcaption>celebration cakes (members of the migration team were code-named the badgers)!</figcaption></figure><h3>What’s next?</h3><p>After completing the migration, we spent a couple of months ironing out some of the tech debt and uncompleted work that had been de-prioritised to get the migration over the line.</p><p>Now, we’re working on the future of our platform, including:</p><ul><li>keeping on top of tech debt and upgrading things.</li><li>being pro-active about improving our platform, so that we don’t have to migrate ever again.</li><li>building amazing features for our users that we couldn’t build before.</li></ul><p>We’ll also be sharing more around our architecture, technology choices and implementation in future posts.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f0b352fdde4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/songkick-dev-blog/how-we-migrated-to-google-cloud-8f0b352fdde4">How we migrated to Google Cloud</a> was originally published in <a href="https://medium.com/songkick-dev-blog">Songkick Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>