Monitorama is a conference that focuses on Observability, Monitoring, Metrics, Logs, Traces, Dashboards, Incident Management, and just about everything else in the area.

An annual event (apart from during the COVID Pandemic), the conference is a fantastic blend of industry experts, beginners, and vendors sharing their stories either on stage, in the corridors of the conference space, or at the various restaurants, bars, and parks in the surrounding areas.

This was my first time visiting Monitorama, so here (in no particular order!) are the topics of conversation, conference talks, and vendors that caught my attention over the past few days.

The main topic that grabbed my attention was “How do we help our teams/clients write good SLO’s and SLI’s?”

Inspired by Sophia Russell’s talk “The Little SLI That Could”, in which Sophia talked about how her team had started decorating their ruby classes with attributes that could then be turned into SLI’s via some custom scripts and other tooling as part of the CI pipeline whilst demonstrating her point using toy trains and the story of “the little engine that could”. This topic continued throughout the week, with various people talking about how similar approaches could be used to provide a wider framework for all kinds of languages and organisations.

Obviously, I wasn’t going to miss an opportunity to share more information about the OpenSLO and SLODLC projects that I’m involved in, however the relative simplicity with which Sophia and her team had managed to enable developers to create and deploy SLI’s was generally agreed by all to be really quite impressive and more accessible than having to write multiple lines of YAML.

A few of us even started wondering if we could write a Gherkin syntax compiler to describe SLI’s as user stories, so stay tuned to see if that actually happens!

Other conversations ranged from “Do we really need to do EVERYTHING in order to do SRE?” (you don’t), and “Where shall we socialise tonight?” (Momo’s Bar), to “That’s a really cute dog, I really miss my dog…” (it was cute, a disproportionate number of people appear to own dogs at Monitorama), however it was a chance encounter with Steve McGhee from Google that led me to finding out about the embryonic r9y.dev project.

R9y (which stands for “Reliability” just as “o11y” stands for Observability) is a project started by Steve to try and help organisations work out how they can get to the appropriate “number of nines”.

The concept is based on a “tech tree” such as you would find in Sid Meyer’s Civilisation or Kerbal Space Program and requires you to achieve competency in each of the proceeding areas before you can move on to the next and achieve your goal.

This is, of course, after you’ve helped the organisation you’re introducing this to understand that “five nines” (99.999% uptime) is probably not what they actually need, and they can usually get away with 99.95%.

I can’t wait to get more involved in the r9y project and help it progress and become even more useful to our customers!

There were many more conversations about how we can present data to customers, with Fred Moyer showing some really simple ways to break down SLI’s and SLO’s into their component parts, leaving many in the audience (including me!) scratching their heads and wondering why they hadn’t thought of doing it that way!

Corey Quinn also delivered a devastating talk on whether to build or buy, with the answer being “it depends”, however for me the main takeaway was that egress traffic is really expensive, so every time your applications in AWS send logs or metrics to your cloud hosting provider, you start running up charges on your bill.

Couple that with the fact that most SaaS offerings make requests to the CloudWatch API, and each of those requests is chargeable above a certain level, so it’s very easy to find that the highest part of your AWS bill is actually the calls from outside your applications and infrastructure from your Observability stack — something to keep in mind in future!

I also want to say thank you to all the vendors that I spoke to. All of them engaged with me even though it was clear as a consultant I wasn’t their immediate target market, and pretty much everyone I spoke to was keen to come and speak to our engineers and consultants at Contino to help us understand where their product might help our customers.

They also gave me so much swag I’m going to struggle to get it all home again!

Finally, I want to say a massive thank you to everyone who was involved in organising and running Monitorama this year, especially Jason Dixon and Pete Cheslock, both of whom did a fantastic job of keeping it running smoothly.

The catering team made sure that we had more than enough food and drink each day (I’m pretty sure I’m going back at least 2lb heavier than I arrived as a result!), and the team at Portland Center Stage provided us all with a safe, comfortable space in which we could all enjoy ourselves.

So what’s next? I’ll be exploring r9y.dev more, working on improving OpenSLO and the SLODLC based on all the things we discovered and spoke about at Monitorama, and who knows, I might even update The Observability River as a result! For now, I’m off to find my flight back home to the UK.

Thanks Portland, you’ve been amazing, I’m looking forward to coming back for another Monitorama in the very near future!

In the past, I’ve spoken about the Observability River and the journey that we all need to take in order to get from “is the platform up?” through to “how are my users interacting with the application”, and today I’m going to build on that to look at how we can integrate the various stages of the monitoring river into our application.

Often as engineers, we (rightly!) focus our development stories on the user interaction. Stories such as “As a user, I want to be able to add an item to my basket, so I can buy it” are not uncommon, but what about the internal requirements for our team and those around us within our own organisation that fall outside the “end-user” customer?

When we talk about monitoring and all the goodness that goes with it, we frequently focus (again, rightly!) on what is important to the development teams and those who are going to support it, however, it’s time that we started to move beyond that small circle of individuals and out to the wider organisation.

What would the change be within your organisation’s approach to monitoring if you started to add in the following stories?

As a developer
I want to see how my application is performing
So I can refactor the appropriate parts of the codebase to improve user experience

This is a story about a process we’re all familiar with — we check the data, then we refactor and see if the data improves — however when was the last time you wrote it down as a user story on the backlog?

As a product owner
I want to know how my application is performing
So I can prioritise the backlog more effectively

Similar to the above, application performance can help product owners, tech-leads, scrum facilitators, and agile coaches move stories around the backlog to prioritise performance enhancements alongside new features — If your agile “leadership” team doesn’t have access to your monitoring metrics, perhaps they should do…

As a marketing designer
I want to understand page load times
So I can ensure our content is optimised appropriately

OK, so perhaps not every marketing exec is going to be interested in page load times, however, they may want to understand how different sizes and formats of images affect how quickly prospective leads can read about your product.

Both image optimisation and compression of assets are known to be reasons for long page load times and, with access to the monitoring dashboards, your marketing teams can start to look at the way they generate media for the site and find the levels of compression/quality loss that are acceptable.

As a sales executive
I want to ensure the website is serving the correct content
So my targeted mailings don’t go unanswered

Let’s imagine that we’re about to send out a sales email to 2,000 existing customers inviting them to sign up for a new service based on their current preferences. That email has a link in it to an asset on the website, and we’re going to track how many customers click that link and who they are using a marketing automation tool.

As part of our pre-send checks we’ll want to confirm that the asset is in place, but what happens if it disappears for whatever reason halfway through sending them emails, or 24 hours after the email is sent out?

Providing the Sales team with observability and insight on their assets or similar suddenly allows them to halt sending out the mailshot, follow up directly with those who clicked the link but didn’t get the asset, or even look at whether people preferred a particular format of the document to another.

With the exception of the first ticket in the list above all of these tickets are aimed at improving observability for the wider organisation, not just the development teams, but when taken together they improve the platform, the application, and the user experience.

What other user stories would you add to improve observability across your organisation? Let us know in the comments below!

Writing user stories to drive observability usage was originally published in Contino Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

It’s frequently the case that neither the most expensive nor the cheapest monitoring solution is the answer when starting on your observability journey, so let’s take a look at the core decisions you need to make when selecting a monitoring solution for your project.

Question 1: Do you need a team that can support a monitoring solution?

Well before we start worrying about whether we’re going to run a monitoring solution in-house, in the cloud, or purchase a managed service, we need to know if we have the knowledge and capability to run that service.

It’s all well and good deciding that it’s cheaper from a licensing perspective to run something like Prometheus.io because it’s “free”, but that doesn’t take into account the cost of training up your staff or hiring in new team members to run it for you.

Let’s assume you need a team of 5 people to design, run, support, and improve a Prometheus at scale (at least 1000 instances/servers/containers across used by multiple teams) for you on an average salary of £40,000/year, and that you need to spend an additional £250/person in training, along with internal charge-backs of around £2500/month for the infrastructure:

Annual Salary: £40,000 * 5 = £200,000
Annual Hosting: £2,500 * 12 = £30,000
First Year Training Costs: £250 * 5 = £1,250
Total First Year Costs: £231,250
Ongoing Annual Costs: £230,000

That’s a lot of money to spend on a solution that you thought was “free”!

You may have people within your organisation who already know your monitoring solution of choice well enough to deploy it, and you may want to “hide” the cost of moving them to the new team because it’s a cost the company is already bearing, but it’s still not going to be “free” and you’ll still need a team to manage the solution.

Question 2: Are we happy to allow the monitoring data to exist outside our environment?

When running a monitoring solution, you generally have three options:

1. “Behind the firewall” — Generally considered to be the “most secure” by many enterprise organisations, the theory is that by ensuring the data is kept within the environmental boundaries there is less risk of monitoring data (and therefore details of your infrastructure layout) leaking into the public domain.
2. Software as a Service (SaaS) — This is rapidly becoming the primary way for organisations to purchase monitoring solutions, especially in cloud environments. It reduces the need for a dedicated team within your organisation and passes the difficulties of maintaining alerting platforms etc. on to a third party. The downside is that your data goes into a dedicated “bucket” on a platform that is shared with multiple other companies, so if your monitoring provider gets hacked then the attackers will gain information about the state of your infrastructure
3. Cloud Monitoring — Cloud monitoring can fall between the SaaS and “Behind the Firewall” offerings as quite frequently cloud providers such as Amazon and Google will provide both fully-managed monitoring platforms (such as CloudWatch), and the ability to build your own from a managed service (AWS Managed Grafana coupled with AWS Elasticsearch Service for instance). Whilst this might be seen as a solution with all the pitfalls of both of the others, if managed correctly this can provide a semi-hosted solution at a lower cost than a full SaaS offering in some cases

Whether you chose to pick 1, 2, or 3 from the above list, you’re going to need to think about your data storage needs before making the decision, especially as many SaaS providers will only host your data in certain geographical locations.

Question 3: What do we want to monitor?

Choosing what to monitor and how you are going to monitor it is a massive part of deciding which tool to use and will form a significant part of a future blog post to address it in more detail.

Some tools are better at particular jobs than others — for example, I wouldn’t recommend using InfluxDB for log storage and querying, but on the other hand, Elasticsearch is still relatively new to dealing with time-series metrics.

If we go back and look at the Monitoring River, we can see that there are five key areas that we can address when monitoring:

Whilst combining the 5 into a single platform is absolutely possible, we need to ask “is it necessary?”

It may be that the platform you’re running hands-off security and audit logging to another team, you might not do FinOps yet, or you could be providing virtual instances to your customers and therefore not be overly concerned about Application Performance.

The reason I bring this up is that I’ve seen many organisations go for the most expensive/feature-rich/darling-of-the-moment monitoring platform that provides everything anyone could ever need, only to find out they didn’t need half of those features and are now lumbered with a hefty monthly charge.

Question 4: Which languages are we writing our applications in?

You need to be aware that some of the tools are better suited to particular languages than others.

As an example, AppDynamics is very popular amongst enterprise organisations for Application Performance Monitoring, and if you’re working with a Java or C# monolith then it’s probably fine, but as soon as you move into other languages such as Python or Rust, you’re going to want to look for something else because the support for those languages is really poor.

It used to be that AppDynamics was the only option if you wanted to visualise your application request flow however, these days pretty much every monitoring system out there can do the same thing including many of the Open Source ones, so there really is very little reason to stick with AppDynamics unless you’re bound by a license agreement or your team know AppD better than anything else (see Question 1!).

Newer offerings such as Datadog, NewRelic, or even Grafana coupled with an OpenTelemetry backend such as Tempo are all excellent offerings for application performance monitoring if you’re using more modern languages.

Question 5: OK, so what do I choose then?

This is a question that only your organisation can really answer because as we’ve seen it depends heavily on your environment and setup.

Having said that, my personal preference at the moment is to use SaaS where I can (Prometheus, Grafana, Loki, and Tempo if I can’t!), and having worked with the majority of the offerings out there I’d probably recommend Datadog.

Datadog isn’t the cheapest, but it’s not the most expensive either, and if you go past 500 hosts/instances then you qualify for a “bulk purchase discount”.

It has a “pay as you go” model that easily translates to licensing for cloud, it can do instance/host-level monitoring, log analysis, application performance monitoring, network monitoring, and many, many more things that you may never need, and it’s as simple as installing the agent, deploying the configuration file, and watching your data roll into the pre-packaged dashboards.

Question 6: So where does that leave us?

Choosing a monitoring solution has never been easy, and the plethora of products hitting the market at the moment as everyone tries to jump on the SRE/Observability bandwagon makes it difficult to pick the outliers and anomalies from the providers that actually work for you however, the above five questions should set you well on your way.

If you need a bit more help with your observability, then get in touch with the folk at Contino who will be only too happy to support you on your journey down the monitoring river!

How to pick the best observability solution for your organisation was originally published in Contino Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWStats (now nearly 20 years old!) was all the rage for analysing your Apache web server logs to see what site visitors had been up to, and monitoring alerts generally fell into three categories:

• Everything is working
• It’s about to fall over
• It’s fallen over and your customers are angry

For a good few years this approach had worked on platforms hosted in physical data centres such as shared hosting clusters with thousands of websites, however as “the cloud” gained popularity it had become clear that a seismic shift was required in the way we observed our platforms and how our customers interacted with them.

In this post, I want to introduce something called the “observability river”. This is based heavily on my experience of layering monitoring tools and solutions on top of each other, but sets it out in an easy to follow way, demonstrating that you don’t need a “big bang” monitoring solution, and that you can easily add to it as your team or organisation grows.

As we go through this series of posts (it’s to large a subject to do justice in just one post!) we’ve tried where we can to recommend specific tools that can help, and whilst most of these will be Open Source and freely available, some are inevitably commercial.

If you’re struggling with your journey down the monitoring river then you can get in contact with us at Contino and see how we can help, but for now sit back, relax, keep your hands and feet inside the boat at all times, and join us as we sail towards monitoring bliss.

The River

Like all rivers, ours starts with a small spring of a source and gradually increases in size as tributaries flow into it, before finally arriving at a larger body of water — in our case, a “data lake” (Sorry! Ed.)

Our source and starting point is audit and security logging, enabling us to trust that our platform (and more importantly the data of our customers!) is secure and that we have traceability and accountability for all actions taken on the platform.

As we move downstream, we start to pull in data about the platform itself. These could be provided by the underlying cloud platform (CloudWatch, Azure Monitoring), or collected directly from the instances, load balancers, and containers that are deployed.

Gradually picking up pace, application monitoring starts to enter our stream, widening the available data once again and adding information about components such as application response times, database query performance, and cache hits/misses.

Every good river exploration needs a set of rapids, and our journey down the monitoring river is no exception. As data starts to roll in from our users’ end devices, we start to see how well our app or website responds on various mobile operating systems or across differing connectivity options — your app loads fine over the WiFi in the office, but how well does it perform when your user is connected via LTE or HSDPA? This layer of monitoring helps you answer these important questions.

As the river reaches the end of its journey and we drift out onto our data lake, we start to look at visualisation — how do you ensure that you present your data to your customers (both internal and external) in a way that is understandable and constructive? This is the point at which we start to look at existing methodologies of data analysis, including the “4 Golden Signals” from the Google SRE handbook, and the combination of USE and RED.

Expeditions can be expensive, and a digital journey is no exception, so in the middle of the lake, we find “FinOps Island” — rapidly becoming a vital part of the DevOps movement and cloud adoption, FinOps (Financial Operations) allows us to ensure that we are getting value from our environments by keeping track on what we’re spending, how we’re spending it, and (perhaps most importantly!) whether we can reduce spending or make our spending more efficient.

Finally, as we approach the far shores of the lake we start to add alerting. This could be manually configured, it could use the latest trends in Artificial Intelligence, it doesn’t really matter. What matters is that it is relevant and can lead quickly to a resolution. I’ll give examples of both good and bad alerts when we reach that point.

Now that we understand the flow of our monitoring, we’ll be looking at each part of the river in more depth in future posts, so join me as we set sail (Sorry again! Ed.) towards monitoring perfection!

The Observability River was originally published in Contino Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.