<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Morten Hanshaugen on Medium]]></title>
        <description><![CDATA[Stories by Morten Hanshaugen on Medium]]></description>
        <link>https://medium.com/@morten.hanshaugen?source=rss-9d484a8586f0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*DwrpyMS3RhDkJI8_.jpg</url>
            <title>Stories by Morten Hanshaugen on Medium</title>
            <link>https://medium.com/@morten.hanshaugen?source=rss-9d484a8586f0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:01:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@morten.hanshaugen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From data center to Polycloud]]></title>
            <link>https://medium.com/finn-no/from-data-center-to-polycloud-8671eceb04dc?source=rss-9d484a8586f0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8671eceb04dc</guid>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Morten Hanshaugen]]></dc:creator>
            <pubDate>Mon, 07 Oct 2019 13:51:55 GMT</pubDate>
            <atom:updated>2019-10-07T14:52:43.753Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A cloud containing Google cloud and AWS, Fastly and IBM cloud in the background" src="https://cdn-images-1.medium.com/max/652/0*NEeUDJLnBYtVrdoH" /></figure><p><strong>We had to move</strong> out of the data center. Where do you move a technology company these days?</p><p><strong>As we improved</strong> <a href="https://medium.com/finn-no/infrastructure-at-finn-1c25c0fe3790">the way we develop</a> FINN.no we overloaded the underlying server network. We needed an alternative to on-premise hosting or risk not being able to grow our hundred million USD business any further.</p><p><strong>A network upgrade</strong> would require work that likely would lead to service disruptions. And seriously, a better network was not what we wanted; our real mission is to increase the company’s <em>opportunities to innovate</em>. The “good old data center” had become a liability. We decided to move to the cloud.</p><figure><img alt="Arrow pointing from racks and network to cloud concepts representing agility, savings, choices, flexibility, reliability, etc" src="https://cdn-images-1.medium.com/max/890/1*RHiJgLPk4HtUxTTEoUSNzg.png" /><figcaption>The journey from VMs to trusted Cloud Concepts required a containerisation process</figcaption></figure><h3><strong>Infrastructure as a competitive edge</strong></h3><p><strong>FINN.no is</strong> the dominating e-commerce site in the Norwegian classified advertising market. Facebook marketplace compete with us using their global platform and startups compete using cloud technologies. We did consider moving to a different on-premise infrastructure supplier, but local hosting companies focus on enterprise. Enterprise IT can not provide the required speed and flexibility needed to improve our products and capture new markets. The cloud offers combination of modern infrastructure concepts and easier integrations with business partners and services. Some clouds offer technologies that will propel us forward, others do not. How to choose?</p><p><strong>We are known</strong> for adopting technology to achieve competitive advantages. Early heavy adoption of Kafka, Kubernetes, Prometheus and Terraform has accelerated how we work and our ability to deliver features faster with less friction. Our focus on technology help attract outstanding talent — talent that expect to be working with great technology. We decided to focus on technology when selecting a cloud partner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*zmXYNrfbdjcpFoGSJ5ruuA.png" /><figcaption>High throughput Web businesses require infrastructure services layers that can be trusted</figcaption></figure><h3><strong>A solution for a technology company</strong></h3><p><strong>FINN implemented DevOps</strong> years ago. We have a true DevOps culture — making 200 production changes per day. We developed our own deployment abstraction tool for Kubernetes (known as <a href="https://fiaas.github.io/">FIAAS — FINN Infrastructure as a Service</a>) to help developers quickly and safely manage their deployments. FIAAS merges simple application configuration with infrastructure defaults. Once tests are green and a service is deployed — statistics, errors, and alarms are built-in to help make ongoing work efficient and comfortable. This way both application developers and infrastructure developers can concentrate on enhancing their products.</p><p><strong>All this is well</strong> and good! -But the modern and efficient continuous deployment platform balances on top of an outdated server solution 😬.</p><p>Servers and storage are expensive to administrate (and seriously, a virtual server is just a server). What we need is <em>solid abstraction concepts</em> to support elevated software ecosystems. “A platform for automating deployment, scaling, and operations of application containers across clusters of hosts” — <a href="https://en.wikipedia.org/wiki/Kubernetes">Kubernetes</a>. The early adoption of Kubernetes in 2016 to have 97% of apps running in one large Kubernetes cluster took 3 years. There is no reasons to turn back.</p><figure><img alt="Schematic picture of how apps are packed in a physical server, a virtual server and in a containerised stack" src="https://cdn-images-1.medium.com/max/797/1*exZn5ssuo_10uBaopS8dfA.png" /><figcaption>From hardware via VM to lightweight app deployment</figcaption></figure><h3><strong>Kubernetes</strong></h3><p><strong>Kubernetes</strong>, the key element of all that we do these days, provides pretty good abstraction. -Containers and Kubernetes rely on simple principles for how a process is run in the operating system. As a result you have fine-tuned control over important application behaviours. Introducing Kubernetes in combination with breaking the monolith into microservices has proven a huge success. The number of microservices in production grows by the week (739 being the current number).</p><p><strong>We used to manage</strong> hundreds of servers. Today we have just two dozen identical Kubernetes nodes. Managing a few identical servers with only stateless Kubernetes microservices running on them require just a fraction of the manpower required to maintain hundreds of pet VMs. Kubernetes herd itself to large degree, freeing up people to working on evolving the infrastructure services even further.</p><p><strong>We plan</strong> to migrate our valuable Kubernetes workload to an quality managed cloud based Kubernetes service. We plan to do this using our normal deployment capabilities, as we just “start deploying to an <em>additional</em> Kubernetes target cluster”.</p><figure><img alt="21 Kubernetes nodes is running around 1500 Kubernetes pods, a few pods are red indicating error state" src="https://cdn-images-1.medium.com/max/1024/1*-TDWiWcs5Z2zhysDr8B03Q.png" /><figcaption>A snapshot of FINNs Kubernetes microservice workload. We will move it to a managed Kubernetes in the cloud</figcaption></figure><h3><strong>The cloud</strong></h3><p><strong>Securing</strong> a good Kubernetes runtime for FINN.no services is the number one priority. The <em>Long term</em> goal is the potential for our business. Professional cloud services will support us in providing our customers with stable services as we continue to grow. What we really want to get out of this journey is access to technologies, abstractions and services, and in particular <em>room for more experimentation</em>. We are adopting a Polycloud strategy because we prefer to use the best managed services from the best cloud providers at any given time. The current plan include using managed Kubernetes (GKE) from Google, Amazon Redshift for data warehouse and Fastly as CDN. Business needs and cloud service maturity will determine what additional services will be adopted.</p><h3><strong>Build vs buy</strong></h3><p><strong>We want access</strong> to partners, services, and APIs for new cool technologies like image recognition, machine learning, AR, etc. We want access to capacity, to burst resource consumption in short periods of time, and the ability to duplicate test systems. We want access to new and potent technologies. In short, we look for possibilities to work differently, and more efficiently. We want more freedom. As Albert Camus said, “Freedom is nothing else but a chance to be better.’’ We think the cloud will increase our freedom to become faster and more innovative. To succeed, we need to identify cloud services and partners providing sensible cost, quality and velocity.</p><figure><img alt="Graphs showing how memory, cpu, number of instances and cost information changes between versions of the application" src="https://cdn-images-1.medium.com/max/433/1*kf19XK4Rl6glNaaAygUrrA.png" /><figcaption>Near realtime cost information per team / application to catch cost effects early</figcaption></figure><h3><strong>Metrics</strong></h3><p><strong>Freedom combined</strong> with accurate feedback provides a good foundation for growing a business. FINN is data-driven. We have millions of “time series” that track how infrastructure, applications, and customers behave. Developers combine metrics from different parts of the business to figure out how well a service behaves. As we relocate to the cloud, we want to expand our metrics to include cost details for all applications. Developers should be able to act on increased cost of operations as they normally do on metrics reporting increase in user facing error messages or how much memory an application use.</p><h3><strong>Timeline</strong></h3><p><strong>We expect</strong> to start migrating FINN.no from on-premise to the cloud in the beginning of 2020. There are hundreds of applications and terabytes of data to move, dozens of database clusters, message buses, monitoring and metrics stacks to redeploy, and new environments to implement. All using cloud native paradigms. We expect the migration process to last one year. This estimate is based on experience from moving our dev environment and on qualified guessing.</p><p><strong>We will use</strong> the next months for preparations: What is the best way to provision our workload in the cloud, how do we improve security, how should the organization structure be, how do we link the cloud account to our mother company, last but not least: how do we collect and make cost information available to individual developer teams.</p><p><strong>We will be writing more</strong> about how we prepare and perform FINNs journey to the cloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8671eceb04dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/finn-no/from-data-center-to-polycloud-8671eceb04dc">From data center to Polycloud</a> was originally published in <a href="https://medium.com/finn-no">FINN.no Blog – Product, Design, and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to manage the FINN.no infrastructure where 170 developers deploy 1000 times a week]]></title>
            <link>https://medium.com/finn-no/infrastructure-at-finn-1c25c0fe3790?source=rss-9d484a8586f0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c25c0fe3790</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[pipeline-as-code]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Morten Hanshaugen]]></dc:creator>
            <pubDate>Thu, 07 Mar 2019 13:31:24 GMT</pubDate>
            <atom:updated>2019-12-16T14:29:48.036Z</atom:updated>
            <content:encoded><![CDATA[<p>FINN.no is the largest marketplace in Norway, and one of Norway’s busiest Web destinations. We need stable and real-time consumable infrastructure services to run and innovate our service, and to evolve finn.no.</p><p>This post is about how we do infrastructure at FINN, and a bit about how we got where we are.</p><h3>Introduction</h3><h4>Continuous Deployment</h4><figure><img alt="Two pipelines, the more automated Continuous deployment pipeline has automated progress between all steps." src="https://cdn-images-1.medium.com/max/1000/0*QlR5Lv4l_wz23AZS" /><figcaption>Attributions to: <a href="https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff">https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff</a></figcaption></figure><p>Our developers commit code to Github thousands of times a day. A code commit may trigger a build. The Pipeline build create and test a new deployment artifact in a Dev environment. This may result in an automated deploy to production. We release both small and large changes to production a 1000 times per week this way. The actual deployments are performed by FIAAS. -<a href="https://fiaas.github.io/">FIAAS</a> is an <a href="https://finn-no.github.io/">open-sourced</a> FINN infrastructure as a service deployment mechanism. It has a <a href="https://medium.com/@pavanbelagatti/declarative-continuous-deployment-pipelines-9d30807f7d0d">declarative</a> nature, scheduling verified artifact deployments as one or more Kubernetes pods. Application updates are typically deployed and verified on the side of running production versions. The systematic verification version switching and auto-rollback make releasing to production quite safe.</p><p>To get a more complete picture of how this works, let’s summarise the technologies in use:</p><h4>Application development and storage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*GQe1Y6Lpwk7CoXi5Zi9wIw.png" /><figcaption>Application development and storage technologies in use at FINN</figcaption></figure><p>Developer teams are free to use feasible programming languages. The majority of existing code is written in Java. New stuff is a mix of Java, Node, Kotlin, Scala, Haskell, React, Ruby, Python, and Go.<br>Most persistent structured data is stored in PostgreSQL database clusters. Ads are stored in a legacy Sybase database. Kafka messages are used to synchronize the 700 microservices. Solr is our main search technology. Redis is a popular in-memory storage for fast handling of i.e session data. Kibana is the favoured analytics and visualization platform. Our teams use it to get feedback on how applications are performing. Redshift is our data warehouse platform.<br>We use Github and Github Enterprise for all code revision control. Maven, Bamboo, Gradle and Travis are heavily used to achieve Continuous Integration and Continuous Deployment throughout the company. Artifactory and Nexus help us store the various CI/CD artifacts.</p><h4>Orchestration and Management</h4><figure><img alt="Orchestration consist of categories scheduling, services support, logs &amp; metrics, monitoring and service management services." src="https://cdn-images-1.medium.com/max/779/1*VJCHbWVxSJ59odrbw92HkA.png" /><figcaption>Orchestration and management technologies in use at FINN</figcaption></figure><p>The FIAAS deployment daemon schedule and deploy our container workload in Kubernetes dev and prod environments.<br>Distributed configuration is stored in etcd. Secrets are stored in Consul. A/B testing and feature toggle roll-out is performed using Unleash.<br>We collect logs using Fluentd and Logstash, and aggregate using Elasticsearch. We store millions of time series in Prometheus.<br>We have some limited use of RabbitMQ in connection to Sensu monitoring of all our services.<br>All FINN load balancers are implemented using HAProxy. Some parts of finn.no traffic, like images, are served by Fastly (Varnish technology). The storage of the images is handled by a YAMS image storage service.</p><h4>Runtime</h4><figure><img alt="The Runtime is divided into distributed storage, container runtime and cloud native categories." src="https://cdn-images-1.medium.com/max/800/1*x9nr_k8w4FC_5z_HokOaWQ.png" /><figcaption>Runtime technologies in use at FINN</figcaption></figure><p>We solve storage requirements by using Ceph block storage and S3 buckets/cloud storage. We use Docker network and runtimes, and we use Flannel for virtual networks and to attach IP addresses to containers.</p><h4>Provisioning</h4><figure><img alt="Provisioning consist of infrastructure automation, host management tooling and image security." src="https://cdn-images-1.medium.com/max/800/1*JXvx6jVkFvR4-xiESQmIGg.png" /><figcaption>Provisioning technologies in use at FINN</figcaption></figure><p>We use Terraform to provision infrastructure resources like compute, storage, network, identity. Orchestration and configuration of the provisioned resources is done by Puppet and helm. All the secrets we have are stored in Vault.</p><h4>Infrastructure</h4><figure><img alt="Infrastructure consist of AWS services, on-premise hardware resources with OpenStack, Google cloud and IBM cloud." src="https://cdn-images-1.medium.com/max/800/1*nU3KNv0J-NfL0tmWxvdaEw.png" /><figcaption>Infrastructure options in use at FINN</figcaption></figure><p>Most FINN services operate from on-premise servers located close to Oslo. VMs are created in a Mirantis OpenStack cluster, pods are scheduled in Kubernetes clusters. We run dev environments in IBM Cloud (Softlayer). Data intelligence workloads run in Google Cloud and data warehouse run in Amazon Redshift. We use additional Schibsted provided services running in AWS.</p><h4>The whole picture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*I2wvVWDXqNqXfYO44ai5ZA.png" /><figcaption>Technologies involved in development, orchestration, provisioning and releasing at FINN</figcaption></figure><h3>Infrastructure as code</h3><p>FINN has a need for efficient infrastructure management to let 175 developers develop at good speed.</p><p>We are walking the <a href="https://github.com/cncf/landscape/#trail-map">”Cloud Native” Trail</a> to provide efficient self-serve infrastructure services. FINN has two <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">infrastructure as code implementations</a>:</p><ol><li>Terraform &amp; Puppet — legacy</li><li>Kubernetes &amp; FIAAS — the cool stuff</li></ol><p>1. Our use of Terraform and Puppet to provision infrastructure is similar to what you will find in many places. You edit config files and infrastructure changes happen after one or more Puppet/Chef runs. Or after invoking Ansible or Salt. New VMs are created, configured and secured. Applications are installed programmatically. We actively maintain around 350 VMs with Terraform and Puppet this way. One third of VMs are 20–30 PostgreSQL clusters. The ambition is to reduce the number of these legacy servers by 50% year over year. Manual or interactive server configuration is a no-go.</p><figure><img alt="The configuration snipplet shows a Consul production instance specification." src="https://cdn-images-1.medium.com/max/948/0*FKd4UOO7ttbcTaH0" /><figcaption>Terraform example specification</figcaption></figure><p>2. Using FIAAS to run 700 applications in Kubernetes as 1.500 pods is quite cool! Developers use the FIAAS <em>declarative</em> interface to get new stuff into production in minutes. All they have to do is add a simple yaml file to their code repository, specifying: ports, health and readiness checks, required memory and CPU, replicas, etc.</p><figure><img alt="The yaml snipplet shows a resource, port and health check specification." src="https://cdn-images-1.medium.com/max/343/0*-prnC0kQirj9y3Xk" /><figcaption>fiaas.yml example specification</figcaption></figure><p>FIAAS create Kubernetes pods using specified requirements and cluster defaults. Autoscaling, logs, metrics and alerts are automatically available, at well defined predictable addresses.</p><blockquote>Read more about this in Øyvind Ingebrigtsen Øvergaards excellent <a href="https://medium.com/finn-no/open-sourcing-fiaas-1d495abf96b0">open sourcing of FIAAS</a> article.</blockquote><p>Around 2015 as FINN adopted Domain Driven Design and Event Driven Architecture, FINN evaluated several cluster orchestration tools. The candidates where:</p><ol><li>Mesos + Marathon/Cisco cloud/Mantl</li><li>Openshift</li><li>Kubernetes</li></ol><p>Mesos was judged to have lacking component integrations.<br>Openshift was viewed to be expensive, not so much improvement over plain Kubernetes.<br>Kubernetes provided the desired level of flexibility. It was Open Source and the project had excellent traction.</p><figure><img alt="Showing how the yaml file specifies autoscaler, ingress, and deployment parameters." src="https://cdn-images-1.medium.com/max/1024/0*1irumEr23idfdClG" /><figcaption>The FIAAS YAML file is the developer interface to the Kubernetes production cluster</figcaption></figure><p>The use of the FINN-opinionated FIAAS deployment daemon make everyone’s world easier. Developers do not have to relate to the Kubernetes setup.</p><p>The declarative nature of Kubernetes has proven a great benefit. Kubernetes create resources like Deployment, Service, Ingress, and HorizontalPodAutoscaler approximating the desired state of your application. The Kubernetes control plane start containers, configure network resources, and instruct load balancers to achieve a deployment that correspond to each app’s resource configuration. All the resources are configured with YAML via the Kubernetes API.</p><h4>Configuration as Code</h4><p>All configuration for development, testing and production is part of the application code. Configuration as Code requires proper handling of secrets. We use <a href="https://www.vaultproject.io/">Vault</a> with persistent storage in <a href="https://www.consul.io/">Consul</a> to provide on demand granular secret management. <a href="https://coreos.com/etcd/">etcd</a> is an extremely important component, as it functions as a shared configuration and service discovery for all containers.</p><h4>Pipeline as Code</h4><p>FINN uses an in-house developed Pipeline that builds, tests and triggers deployments. On legacy servers and as containers in Kubernetes.</p><h4>Everything as Code</h4><p>FIAAS sits in front of the Kubernetes APIs. It reads a fiaas.yml input and creates Kubernetes resources. It triggers the Kubernetes control plane to deploy all our applications. Fiaas.yml follows the “FIAAS config format”, an abstraction of the Kubernetes API.</p><h4>Less Code</h4><p>Configuring Kubernetes resources usually require a lot of YAML. Deploying an example application requires 303 lines of resource configuration. The corresponding FIAAS configuration requires only 25 lines, saving developers 90% of the time and hassle.</p><p>Configuration not specified in FIAAS yaml file is generated by a common mechanism for every application. This provides flexibility to change underlying configuration. We can change infrastructure across all applications without altering any of the applications themselves.</p><h3>Contracts</h3><p>Contracts are an important part of why we move so quickly at FINN. Contracts enforce standards. Standards make it easier to deploy applications. Developers are empowered and can focus on the important decisions. They know that logs, metrics, and applications will appear in a standard format, under standard URLs. It helps make the infrastructure easier to manage as communication around features, changes and expectations are clear.</p><h4>Developer contract</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*hofRzG8PDbSVQcSS_6954g.png" /><figcaption>Infrastructure services, ingress, egress, autoscaling, log endpoints, etc are explicitly defined</figcaption></figure><p>We enforce a contract between Developers and the Infrastructure:</p><ul><li>Deployment</li><li>Traffic ingress</li><li>Service discovery</li><li>Observability</li></ul><h4>Platform Contract</h4><ul><li>Kubernetes</li><li>Metrics -&gt; Prometheus</li><li>Aggregated logs -&gt; Fluentd</li><li>Feature toggled migration with Unleash + custom ingress controller</li></ul><h3>We organise for de-coupled communication</h3><p>The organising into functional domains in FINN is inspired and derived from:</p><ul><li><a href="https://scs-architecture.org/">Self Contained Systems</a> — separation of functionality into many independent systems</li><li><a href="https://blog.scottlogic.com/2017/07/17/successful-microservices-with-12factor-app.html">12 factor model</a> — declarative setup automation, lower cost of adding employees</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-B8ZeAE97li0w0JO" /><figcaption>Domains communicate events on a common bus, results from the different domains are glued together in the frontend</figcaption></figure><p>The goal is De-coupled Event Driven Communication between Domains Services. Business logic should reside entirely inside the domain service. Domain functionalities are called in parallel to construct customer facing views.</p><h4>Technology tiers</h4><p>FINN technology is conceptually divided into 3 tiers: Business area services, Platform services, and Infrastructure.</p><figure><img alt="We have divided our technology into 3 tiers: Business areas, Platform and Infrastructure." src="https://cdn-images-1.medium.com/max/1024/0*14UglRPooLMrtvJL" /><figcaption>The 3 technology tiers</figcaption></figure><p>Infrastructure is further divided into: Development support, Observability, Common services, Application platforms, Custom services and Legacy services.</p><figure><img alt="Infrastructure services at FINN: Development support, Observability, Basic services, Application plattform and Custom service" src="https://cdn-images-1.medium.com/max/1024/0*XAJOGYG1RVNIs57Z" /><figcaption>Infrastructure services at FINN</figcaption></figure><p>It took years and a lot of effort for FINN to become able to release to production at a high frequency. See data from FINNs Pipeline below. In the same period we managed to reduce the build time for an artifact from several hours to minutes. This changed our way of working. It became easy to release a fix to production, instead of rolling back to a previous version.</p><figure><img alt="The number of production deployments has increased from 1 per week in 2012 to more than 1000 in 2019, peaking at 2500." src="https://cdn-images-1.medium.com/max/986/1*KgXg8HR0kExTnevHlvJJhA.png" /><figcaption><em>FINN Growth of number of production deployments per week from 2012 to 2018.</em></figcaption></figure><p>In the period from 2010 to 2015, the number of developers increased to ~120. The number of services increased from a handful to several 100s. One deploy every 3 months changed to 1000 deploys every week. The some 10s of managed physical and virtual servers increased to 100s of self-served virtual servers. We could provide required compute resources. But developers did not have efficient enough and safe enough tooling. We did not achieve the required developer efficiency. The move to containers, Kubernetes deployments combined with the FIAAS abstraction and metrics, logging, alerts and notifications out of the box fixed this.</p><p>We used to run a single Puppet run for all clusters. Failures in one team could easily block deployments for other teams. Developer teams would ask for machines. Infrastructure people struggled to create enough server resources. Our current container-driven infrastructure has removed these constraints. The velocity of development and deployments has improved a lot. We no longer have interdependencies between deployments. Developers are mostly self-served, declaring simple infrastructure concepts in yaml. Developer teams are free to run whatever runtime gets the job done.</p><p>As a bonus we are free to move our considerable application payload anywhere that can run containers.</p><h4>Designed for failure!</h4><p>We allow and celebrate failure. We can not be fast if we can not risk failures! Our infrastructure, the release pipeline, the microservice architecture, it is all about being able to move more quickly. We need to be able to release new features easily and safely. We need to release often to reduce the impact when a change fail.</p><h4>Measure everything</h4><figure><img alt="Showing the number of time series, queries and scrape durations varying over time." src="https://cdn-images-1.medium.com/max/1024/0*L_eRJHTJ4wv-Wehw" /><figcaption>Developers have millions of time series available to track performance and spot problems.</figcaption></figure><p>We strive to measure everything. We collect millions of time series related to production services. The data provide real-time information on infrastructure, services, applications, customer behaviour, and economy. We also collect, analyse and aggregate tons of logs in near real-time. All this information is used to generate alarms early in such a way that we can handle most problems during normal work hours.</p><h3>FINN in a nutshell</h3><figure><img alt="Showing the steps involved from developer commits code till it is visible for user, including internal services involved." src="https://cdn-images-1.medium.com/max/573/1*R7Th43kz6xY2bo3UVx3Yjw.png" /><figcaption>Commit -&gt; to user facing product</figcaption></figure><p>FINN is one of Norway’s largest in-house tech environments. And we make one of Norway’s largest Web destinations. We have a working culture focusing on removing dependencies. We organise as autonomous teams with all decision makers as part of the team. We strive to have processes and ways of working that minimise waste and waiting time. The loosely coupled architecture allows teams to move fast without fear of breaking other people’s stuff.</p><p>FINN has a strong tradition of creating bold strategies and delivering on them. Read about <a href="https://medium.com/finn-no/from-data-center-to-polycloud-8671eceb04dc">why Polycloud is a central part of our technology strategy</a>. Look out for more posts about how we are moving finn.no from on-premise to the cloud as the work proceeds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c25c0fe3790" width="1" height="1" alt=""><hr><p><a href="https://medium.com/finn-no/infrastructure-at-finn-1c25c0fe3790">How to manage the FINN.no infrastructure where 170 developers deploy 1000 times a week</a> was originally published in <a href="https://medium.com/finn-no">FINN.no Blog – Product, Design, and Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>