<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gianluca Insolvibile on Medium]]></title>
        <description><![CDATA[Stories by Gianluca Insolvibile on Medium]]></description>
        <link>https://medium.com/@ginso?source=rss-ed773c6dbf4b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*O9vCKVNvK9Zncd0Q.</url>
            <title>Stories by Gianluca Insolvibile on Medium</title>
            <link>https://medium.com/@ginso?source=rss-ed773c6dbf4b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:23:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ginso/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Taming the log storm: how to improve your logging]]></title>
            <link>https://medium.com/@ginso/taming-the-log-storm-how-to-improve-your-logging-c88f75752811?source=rss-ed773c6dbf4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c88f75752811</guid>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Gianluca Insolvibile]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 13:49:08 GMT</pubDate>
            <atom:updated>2024-06-23T13:49:08.991Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*89pfGMZtpRc0PhqY" /><figcaption>A log storm, generated by Dall-E</figcaption></figure><p>As we have seen in <a href="https://medium.com/@ginso/taming-the-log-storm-part-1-10da1f7c66b6">Part 1</a>, until the late 1990s, the log level was the main mechanism available to developers to regulate the amount of output generated by their programs. Then things got complicated.</p><p>Between 2000 and 2010 there has been a major paradigm shift, largely driven by increased complexity, distributed systems, microservices and cloud-based solutions.</p><h3>A moden approach to logging</h3><p>To fully understand the modern approach to logging, we need to revisit the questions posed in the previous article regarding the rationale, the methodology and the objectives of writing logs.</p><p>We mentioned that log “lines” should represent a state of the application, or a change in it. When we talked about levels, we also advised that we should print a message (usually at the <em>info</em> level) whenever something happens that causes a “significant change” in the state. In other words,</p><blockquote>We emit a log to report an <strong>event</strong>.</blockquote><p>This is a deep conceptual shift that will force us to think “<em>what is the event that I need to communicate?</em>” for a large part of our logs. Indirectly, recognizing that an event happened will also force us to be aware that besides logging it we should also handle it, leading to a more resilient software structure. In fact, if you think in terms of events, there is a strong parallel between the log structure and the error handling structure.</p><p>If we then look at some of the reasons for logging that we previously discussed (preventing issues, analysing anomalies, tracking program behaviour) we can ask ourselves whether printing lines of text is actually the best way to do it.</p><p>Going beyond the idea of printing messages and having someone or something to read them, we find more appropriate solutions to this task in specific tools for health and performance monitoring, incident analysis and distributed tracking. The common approach to all of these types of tools is to have a centralised processor that collects information (often called <em>metrics</em>) from a number of services, and provides a unified interface for their monitoring and analysis.</p><blockquote>So instead of “printing lines”, we now need to send (or expose) specific information blocks to these external processors.</blockquote><p>From this perspective, logging assumes its own identity as one of the tools within the broader field of “observability”, which in turn is part of the discipline of <em>System Reliability Engineering</em> (SRE).</p><p>The final step in this transition to “modern logging” (aptly called “<strong>structured logging</strong>”) is to move away from the simple “line of text” to use a real data structure, capable of representing the much richer set of information we now need. A structured log usually contains these information elements:</p><ul><li>a message</li><li>a timestamp</li><li>a “context” made of several fields providing addition information about the event. They typically include origin, level, tags, session IDs, user IDs, function names, request parameters, and so forth.</li></ul><blockquote>A structured log is a format that allows for the comprehensive and accurate description of an event.</blockquote><p>It can be converted back to a “text line” format for printing purposes via an appropriate formatter, if the need arises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7_C6c865ffUEzze_uHagg.png" /><figcaption>Dall-E, visualize the concept of structured logging!</figcaption></figure><p>Most frequently, the data is transmitted to an “aggregator” which, at the minimum, stores it in an indexed database for more sophisticated querying.</p><h4>Pros and Cons</h4><p>The main advantages of structured logging are:</p><ul><li>it helps us to think about the context when defining the contents of the structure, which is a great way to identify relevant information about the event</li><li>it allows for simplified analysis and querying in a vast amount of information (e.g. let me see all the events related to a specific API request)</li><li>when collecting events from several different services, it allows for efficient tracking across a distributed environment</li><li>having specific pieces of information in well-defined fields allows the construction of effective automation flows that can react to specific events.</li></ul><p>The main disadvantage of this approach is the greater need for computing resources (in terms of CPU, disk and network bandwidth), which makes it less suitable for more constrained contexts such as embedded devices. Even in unconstrained conditions, such as those found in a server farm or cloud environment, the additional resources required by structured logging result in greater energy consumption and an increase in operational costs. Therefore, it is important to always be judicious in throttling the amount of information we generate.</p><h4>Commonly used aggregators</h4><p>We will not go into the details of how log aggregators are used, but it may still be useful to give a quick overview of the most commonly used ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_IgF0BHYEFa66sUPEjAZHQ.png" /><figcaption>A donut-shaped metrics aggregator (courtesy of Dall-E)</figcaption></figure><p>Many solutions are Open Source projects and can be installed on-premise to keep everything local. Among them we can mention</p><ul><li>the <a href="https://www.elastic.co/elastic-stack/">ELK stack</a> (ElasticSearch, Logstash, Kibana), probably the first and most widespread solution for log aggregation, search and analysis</li><li><a href="https://graylog.org/">Graylog</a>, a powerful log management platform specialized in threat detection and incident response</li><li><a href="https://www.fluentd.org/">Fluentd</a>, an Open Source data collector which is also part of the CNCF (<em>Cloud Native Computing Foundation</em>)</li></ul><p>There is an even wider range of alternatives when considering proprietary log aggregators, many of which are also available as a service:</p><ul><li><a href="https://logz.io/">logz.io</a></li><li><a href="https://www.splunk.com/">Splunk</a></li><li><a href="https://www.datadoghq.com/">Datadog</a></li><li><a href="https://www.loggly.com/">Loggly</a></li><li><a href="https://www.papertrail.com/">Papertrail</a></li><li><a href="https://www.jaegertracing.io/">Jaeger</a></li></ul><p>Most of these solutions are not mere log aggregators but real observability platforms, many of them equipped with specific functions for cybersecurity applications.</p><p>Speaking of observability, we should also mention solutions which provide both log and metrics collection, like <a href="https://opentelemetry.io/">OpenTelemetry</a>, <a href="https://prometheus.io/">Prometheus</a>, and <a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a>.</p><p>As mentioned in the previous paragraph, metrics are an effective substitute for logs in all those cases in which we need to communicate values relating to the operation of our system to the outside world. These can be average processing times, latency in responses, RAM and CPU load, number of requests processed, and so on. All information that used to be printed in logs to be later extracted in post-processing, and that can be effectively monitored in real-time by using modern observability systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cryvb4O2dF3OYfYM" /><figcaption>Steampunk metrics dashboard (courtesy of Dall-E)</figcaption></figure><p>Observability solutions are extremely interesting, but have led us astray. We may return to this topic in a future article; for the present, let us turn our attention back to logs.</p><h3>Refining your logging strategy</h3><p>As we discussed in the first part of this series, the process of logging is not governed by strict rules. To enhance our logging habits, it’s recommended to incorporate insights derived from both practical experience and established best practices.</p><p>In the following, I’ll go through some of the rules I have put together over the years. I hope you will find them useful!</p><h4>A log message is a form a communication</h4><p>The first and most important thing to bear in mind is that a log message is a form of communication. As in all communication, there is a person on the other side who has to receive it, and it will be all the more effective the more we are able to put ourselves in the recipient’s shoes.</p><blockquote>A log message is a form of communication.</blockquote><p>Even if you feed the logs into a centralised indexing system, they will still eventually be consumed by another human. And if you’re thinking about LLMs processing the messages, which is already happening, remember that these AIs are modelled on human language and will therefore be much more effective if the messages are expressed clearly.</p><p>All messages emitted at levels between <em>info</em> and <em>critical</em> should always be understandable by a reasonably skilled operations engineer. When you write those logs, forget all the details you know as a developer and focus on the information your reader will need and be able to extract from the message. And remember, your reader might as well be you, in 6 months or maybe 2 years. Trust me: all the details in your head will be gone.</p><p>The real goal of these logs is for ops engineers to understand what is going on with the system, and whether they should involve someone from the development. The clearer the information you provide, the fewer useless calls you receive from the field.</p><p><em>Debug</em> and <em>trace</em> level messages are generally addressed to other developers and may contain a sensible amount of detailed low-level information (even more so if dynamic level filtering, as described below, is in place).</p><blockquote>As a rule of thumb, the frequency of logs at a given level should be inversely proportional to the level itself.</blockquote><p>That is to say, in a well-balanced log you would expect to find extremely infrequent (or even no) <em>critical</em> messages, a few <em>error</em>s, some <em>warning</em>s, and so on. An inversion of this rule, i.e. the presence of many <em>critical</em>s, is an almost certain indicator of a lack of fine-tuning of the levels, or worse, of very poorly behaving software.</p><h4>Release-mode log setting</h4><p>It may seem superfluous to mention this, but simple rules are easily forgotten: when a software is released to production, the default log level should be set to “info” (which means that <em>debug</em> and <em>trace</em> messages should not be visible).</p><blockquote>when a software is released to production, the default log level should be set to “info”</blockquote><p>If you can afford a small runtime overhead, and if your logging library supports it, consider using mechanisms to dynamically change the active log level. This is extremely useful for extracting information from running systems that have ended up in an inconsistent state, without having to restart them (which would cause the state to be lost).</p><p>Of course, if you are using an aggregator and have a lot of resources to waste, you can always keep all of your log levels active and use offline filtering to extract information as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*alwlkBlPtchLm5y_8tWFOA.png" /><figcaption>Image by author</figcaption></figure><p>In any case, always bear in mind that under normal working conditions (hence at <em>info</em> level) there should be no repeated messages, unless they are relatively sparse. Any periodic checks, heartbeats and the like should not be logged at info level, since they usually do not represent “significant changes in the state of the system” (see <a href="https://medium.com/@ginso/taming-the-log-storm-part-1-10da1f7c66b6">Part 1</a> of this series for more details). Exceptions are always possible, but remember: issuing a log every day is one thing, doing it every minute is quite another.</p><p>Finally, remember to clean up the logs you used during development and debugging before the final release. For those debug messages that you do decide to keep, consider the previous rule about clarity.</p><h4>About information</h4><p>In the previous paragraphs we have often talked about “communicating information”, but what information do we actually need exactly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qf8A-sU6WkWAm3MK" /><figcaption>Generated with Dall-E</figcaption></figure><p>Reducing the signal-to-noise ratio is one of the most important things we should do to ensure that relevant information is not lost in a haystack of meaningless messages. Many of the recommendations we have seen so far are indeed aimed at this. This is easier said than done, of course, since even the definition of “relevant” can change depending on the specific problem we are trying to debug.</p><p>Again, let’s try to review some simple rules that may seem trivial but are often overlooked.</p><ul><li>Don’t put random strings in the messages, like “AIEEEEEEE” o “TCPTCPTCPTCP” (real-world examples here). They don’t convey any meaning and usually are just markers to be found with grep, useful during development but rarely in production.</li><li>The same goes for swearing, it just indicates that the author is somewhat angry or frustrated. Nevertheless, there is some evidence that swearwords do actually improve the quality of code :), see the Resources for more information about this. I personally use them quite often in development as markers of messages that I don’t want to keep in released code, they are easier to grep.</li><li>When using structured logging, the whole team needs to be consistent with the names of fields and tags, both within each service and across services. For example, if the tag is user_id, don’t mix it up with uid or user. This allows you to effectively correlate messages and events across all logs. A good way to achieve this is to plan the tags you will be using in advance and share them with the team. This exercise will also make you think about what relevant information you expect to be useful in monitoring the operation of the system.</li><li>Keep the text part of the message short and to the point. If you’re going to print the messages to a text file, the message should be on a single line (aim for a maximum of 80–100 characters).</li></ul><h4>Stack dumps and exceptions</h4><p>Many languages dump the call stack when an exception is not caught (like Python, Java, C#). When tracebacks appear in a log, it’s usually a sign that something has gone very wrong, and the diligent devops engineer will call the developers immediately.</p><p>This also means that the diligent developer will want to prevent easily predictable errors from becoming run-time exceptions: catching exceptions allows you to provide meaningful error messages, log the error details for debugging, and possibly recover from the error in a way that keeps the software running. Unhandled exceptions can sometimes expose sensitive information in stack traces, so catching them can also help prevent security vulnerabilities. Finally, exceptions can interrupt processes and leave resources in an inconsistent state. Catching them allows you to clean up these resources properly.</p><p>On the other side, it is not always possible to anticipate every possible exception that might occur (the infamous “unknown unknowns”). Trying to catch everything could lead to overly complex code and obscure the root cause of issues, potentially masking underlying problems in the code and making it harder to diagnose and fix them.</p><p>So you need to strike a balance and focus on catching exceptions that you can reasonably anticipate and handle gracefully. This includes exceptions that are specific to your application’s logic or external dependencies.</p><h4>Logging plans and reviews</h4><p>As discussed earlier, logging is a form of communication involving humans, more specifically development and operations teams. There are two things you can do to make this communication more effective.</p><blockquote>In the early stages of development, create a <strong>logging plan</strong>, share it with the team and make sure it is adopted correctly.</blockquote><p>First, in the early stages of development, create a “logging plan” to define things like what kind of events you want to track, which ones you would consider critical to the application, which ones would be errors rather than warnings, how you will collect, monitor and respond to the logs, which pieces of information are relevant for the analysis, which events should be automatically handled, and so on.</p><p>Thinking about the log plan will also have the side effect of getting you thinking about the incident management strategy: how you are going to handle incidents at different levels, when the ops team should involve the development team, when and how the user should be informed about an issue.</p><p>The logging plan does not need to be overly complex: at the very least, it could simply define how the levels and tags will be used. The most important thing is to share it with the team and make sure it is adopted correctly.</p><p>To this end, the second and most important thing you should do is to hold a couple of “log review” meetings after the application has been running for a few weeks.</p><blockquote>The second and most important thing you should do is to hold a couple of <strong>log review</strong> meetings after the application has been running for a few weeks.</blockquote><p>The log review is an invaluable tool for bringing together the key people involved in the maintenance process (developers, system administrators, operators, etc.) and having them read and comment on the most significant log messages. The purpose of the meeting is to assess issues such as:</p><ul><li>Did the ops team manage to understand whether the system was behaving correctly or whether they needed to involve their colleagues in development?</li><li>Did the development team manage to gather enough information to effectively address anomalies and debug problems?</li><li>Is it possible (or even necessary) to set up automation rules to respond to specific events?</li><li>Is the signal-to-noise ratio acceptable? Are the logs cluttered with too many useless messages? Are there endlessly repeating lines? Are there any conditions that make the production environment different (noisier) than the development environment?</li></ul><h4>One final suggestion</h4><p>This has long been one of my favourite quotes when writing code, but it also applies well to logging:</p><blockquote>“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.” [John Woods, comp.lang.c++, 1991]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*NiJjyg96zGroGOh3" /><figcaption>What exactly did you mean when you wrote…</figcaption></figure><h3>Conclusions</h3><p>Logging is indeed a rather boring topic to talk about, even more so if compared with fancier topics like LLMs, a shiny new programming language or the latest front end development framework.</p><p>Nevertheless, I hope that these two articles have convinced you that a proper approach to logging will reward your efforts when the product will be in maintenance phase.</p><p>At the very beginning of part 1 I asked: “What do logging, unit testing and writing documentation have in common?” The real answer to the question is that these are three fundamental tools that help us developers create more maintainable software!</p><h3>Resources</h3><p>[<em>this is part 2 in a short series of articles about logging written along the lines of the talk I gave at GDG DevFest Pisa 2024, </em><a href="https://devfest.gdgpisa.it/">https://devfest.gdgpisa.it</a>.]</p><p>You can find Part 1 of this series at <a href="https://medium.com/@ginso/taming-the-log-storm-part-1-10da1f7c66b6">https://medium.com/@ginso/taming-the-log-storm-part-1-10da1f7c66b6</a></p><p>If you feel guilty about using too many swear words in your code, have a look at Jan Strehmel, “<em>Is there a Correlation between the Use of Swearwords and Code Quality in Open Source Code?</em>”, <a href="https://cme.h-its.org/exelixis/pubs/JanThesis.pdf">https://cme.h-its.org/exelixis/pubs/JanThesis.pdf</a></p><p>A good introduction to the wider subject of Site Reliability Engineering (SRE) is the book by Beyer, Jones, Murphy, Petoff, “<em>Site Reliability Engineering</em>”, published by O’Reilly 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c88f75752811" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taming the log storm, part 1]]></title>
            <link>https://medium.com/@ginso/taming-the-log-storm-part-1-10da1f7c66b6?source=rss-ed773c6dbf4b------2</link>
            <guid isPermaLink="false">https://medium.com/p/10da1f7c66b6</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Gianluca Insolvibile]]></dc:creator>
            <pubDate>Tue, 04 Jun 2024 23:02:31 GMT</pubDate>
            <atom:updated>2024-06-17T05:31:31.117Z</atom:updated>
            <content:encoded><![CDATA[<h3>Taming the Log Storm, part 1</h3><p>[<em>this is part 1 in a short series of articles about logging written along the lines of the talk I gave at GDG DevFest Pisa 2024, </em><a href="https://devfest.gdgpisa.it/">https://devfest.gdgpisa.it</a>. <em>Stay tuned for the next parts!</em>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y1LDbGUtL9U40zKT" /><figcaption>A logging infrastructure, as imagined by chatGPT</figcaption></figure><p>What do logging, unit testing and writing documentation have in common? They are probably the 3 most boring things in software development! :)</p><p>I am not an “expert in logging”. And it’s very likely that you won’t be either, since there are generally no logging experts (except those who write logging libraries, that is). There is no such thing as a “logging certification course” or a M.Sc. in logging technology. Logging is just a tool that we use when we need it, like a screwdriver; and there are no experts in screwdrivers.</p><p>When we are writing code, we just throw a console.log line here or a log.debug there, with no time to think too much about what we are logging. There are always more important things to do, we are writing code to implement a specific feature or solve a complex problem, and have no time to carefully craft our log messages. We’ve all been there.</p><p>But when our code goes in production and something behaves differently than expected, and sure enough it will do, our hastily written log messages will be the first and perhaps only foothold in understanding what’s happening. And this is where the pain begins.</p><p>The purpose of these articles is to find a reasoned approach to logging, to try and think with a cool head about how to do it properly. This way, the next time we are implementing a feature or solving a problem, we can write correct logs without having to think about it too much.</p><p>Let’s start!</p><h3>The Basics of Logging</h3><p>First things first… do you know why are they called “logs” and are recorded in a “journal” ? Well, for a start you must know that the very first line of log dates back to … 1689!</p><p>At that time, the “log” was actually a real piece of wood, tied to a long rope on which regularly distanced knots where tied. The log would be thrown overboard and acted as a weight to drag the rope, which unwound as the ship moved out to sea. By counting the number of knots and their rate with an hourglass, sailors were able to measure distance travelled and speed, which were dutifully annotated in the “log journal”.</p><p>So that’s the reason why we still call them “logs”, and incidentally it is also the reason why we continue to measure speed at sea using “knots”. If you want to know more about this, see the first link in the Resources section below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UdCYA8eMtT4lUA1F" /><figcaption><a href="https://www.flickr.com/photos/lightcliff/">Kate’s Photo Diary</a>, CC BY-NC-ND 2.0</figcaption></figure><p>Now let’s jump ahead a few centuries and talk about “our” logs.</p><h4>Let’s start with a few questions</h4><p>There are a few basic questions that can help us put logging in the right perspective.</p><p>The first one is: <strong>what is the purpose of logging?</strong></p><p>Logging is primarly used to <strong>communicate the status of our application</strong> to the outside world. In fact, when a software component is released in production it becomes a “black box” of code and data, whose actual internal status is unknown (excluding what can be deduced from the UI, which the developer generally cares little about, anyway). Log messages thus become the only clue we have to reconstruct the internal state.</p><p>A second objective we commonly have is to <strong>track the evolution of the software state</strong>. To this purpose, we look at the entire stream of log messages to understand how, and possibly why, the state evolved in some specific way.</p><p>Second question: <strong>how do we usually write log messages?</strong> Here the <em>traditional</em> answer is very easy: we print lines of text. They are usually dumped to a console or written to a file. Both will eventually overflow, files will be rotated or, if you messed up something in the system configuration, hard drives will be eventually filled. We’ve all been there.</p><p>Third question: ok, but <strong>why do we really need logging</strong>?</p><p>The answer depends on whether the software is <strong>under development</strong> or <strong>in production</strong>. In the former case, logging is used for debugging the code we are writing; in the latter case, we have three main needs:</p><ul><li>providing information on the application state</li><li>preventing potential issues</li><li>analysing what happened after a problem occurred</li></ul><p>The distinction between the two cases is very important, and gives us an indication of how the writing of a log can differ substantially.</p><p>When the software is <strong>under development</strong>,</p><ul><li>logs are often used to follow the program’s flow (did the code jump into a specific branch? what was the value of a variable before calling the function? etc.)</li><li>the recipient of the messages is the developer him/herself, or someone in the same team;</li><li>messages are continuously read as the program executes, since the developer is patiently sitting right there waiting for something to happen.</li></ul><p>When the software is <strong>in production</strong>,</p><ul><li>logs are used to provide status information, as discussed earlier;</li><li>the recipient of messages is an operations technician, seldomly a developer;</li><li>no one stands in front of the console watching endless streams of messages flowing. Instead, logs are only read when a problem arises.</li></ul><p>What clearly emerges from these first considerations is that we cannot simply use the same log messages under development and in production. Or, at least, we need something to throttle the type and amount of information that our software spews out.</p><h3>Log Levels</h3><p>The traditional tool used to regulate the amount of information emitted by software logs is the <strong>level</strong> (sometimes also called the <strong>priority</strong> or <strong>severity</strong>).</p><p>I found no formal definition of log levels, except for the severity as defined in RFC5424 (“The syslog protocol”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*i5DOAgF7eDaDCJzz2nhd8Q.png" /><figcaption>RFC5424 levels [image by author]</figcaption></figure><p>This document, published by the IETF, is mainly focused on networked systems and less so on generic software components. The document dates back to 2009, updates a previous version dated 2001, which in turn begins by stating “<em>This document describes the observed behavior of the syslog protocol. This protocol has been used for the transmission of event notification messages across networks for many years.</em>” So this is by no means a top-down systematic definition of levels, but rather the acknowledgement of a state of the art consolidated over the previous 20 years.</p><p>In the syslog protocol, severity has a value between 0 and 8 and is ‘weighted’ according to the “facility” (multiplied by 8). In short, it is a way to represent the “importance” of a message with a numeric value which can be used as a threshold by a filtering system.</p><p>In the software development world, things are a bit simpler and most libraries in different languages stick to the “standard” levels shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*VYN7u3NpAxIgGOIPGH6I_A.png" /><figcaption>Log levels in software development [image by author]</figcaption></figure><p>There are no hard and fast rules that tell you which level to use for each message, but in principle we can make some considerations as a guideline:</p><ul><li>The <strong>critical</strong> (or <strong>fatal</strong>) level represents a major fault that the application was unable to handle and due to which the functioning of the system can no longer be guaranteed. When a “critical” message appears on the log, you need to wake up the system administrator in charge (because a proper critical error would always occur during the night, possibly on a week-end). Examples of such situations include database inconsistencies, no space left on hard drives, hardware faults;</li><li>The <strong>error </strong>level typically represents an unforeseen event that the application could not handle, resulting in external effects. By “external effects” here we mean that an outside user (human or machine) of the application did not receive the expected outcome. Errors do not usually require immediate attention, but they do need to be analysed with “moderate urgency”. More often than not, an <em>error</em> message is caused by an internal issue in the software/system rather than by an invalid input. For example, in an application that takes a value as an input and produces the result of a calculation as an output, an out-of-range input should not generate an error log, but an internal overflow should.</li><li>The <strong>warning</strong> level can be used to log an unforeseen event which was correctly handled by the application, and did not cause any external effects. As an example, if our application fails to connect to a database but retries and succeeds after 5 seconds, completing the required operation, a warning can be logged to keep track of the event. A series of subsequent failures could lead to an error log, and repeated failures over a given threshold might end up in a critical log.</li><li>The <strong>info</strong> level is the level responsible for communicating changes in the state of the application, and specifically fulfils one of the main purposes of logging as seen above. Generally, an info-level log is issued each time a significant event occurs in the application’s life cycle.</li><li>The <strong>debug</strong> level is probably the most intuitive log level to use, but precisely for this reason, it is also perhaps the one to be used with the greatest caution. During development, we distribute debug logs around critical parts of our software to print out runtime values, see which branches of the code are being executed, dump pointers and structures, and so on. When the code moves into production, it is tempting to leave these messages as they are, thinking that they might be useful if a stubborn bug decided to surface after remaining silent in testing. However, for the reasons discussed in the previous paragraph, it may be useful to check these messages carefully to ensure that the information they provide is actually usable (we will see this suggestion again in the following). In the days when CPU and memory resources were limited, it was quite common to remove debug messages entirely from the release mode binary; nowadays this is rarely done, even in embedded systems.</li><li>The <strong>trace </strong>level is intended for printing messages that track the execution flow of the application in detail. When required, traces are typically placed at the beginning and end of important functions, along with their input and output parameters. Many logging libraries in different languages provide some form of code annotation to automate this task. Of course, traces can also be placed in the middle of a function to accurately track branches, looping and the like. Since they are often invasive and can have a considerable impact on system performance, trace-level prints are generally only activated during the resolution of a specific problem.</li></ul><h4>Important things to remember</h4><p>After this brief overview of the most commonly used log levels, there are a few considerations that should be kept in mind.</p><p>While taking into account the guidelines proposed in the previous paragraph, the choice of level for a log message is largely left to the sensitivity of the developer and the type of application he or she is making. A temporary loss of connectivity in a desktop application may be a warning; the same issue in a backend that fails to communicate with a database becomes an error, but if the database in question is part of a high-reliability cluster it could also be considered critical. Thinking about potential “external outcomes”, as described above, can be a useful guide to deciding whether a log should be an error or a warning.</p><p>More important than correctness is <strong>consistency</strong>. Whatever criteria you or your team choose for assigning log levels, always try to be consistent. The same type of situation, in terms of severity, impact and urgency, should always result in the same log level being used. Of course, the devil is in the detail when it comes to deciding what exactly “the same type of situation” means on a case-by-case basis.</p><p>The effort of carefully selecting the log levels is easily rewarded, however: if the levels are used correctly, a simple grep in a log file can be a good indicator of the maturity of a system (too many “<em>errors”</em> in a short time during a beta test immediately tell us that the application is not ready for prime time). The same grep used in production can help operators quickly skim over situations that require attention from developers.</p><h3>Beyond Log Levels</h3><p>Until the late 1990s, log levels were the main, if not the only, means of controlling the amount of output generated by software.</p><p>Since then, a number of concurrent factors have led to the increasing complexity of computing systems, including the greater availability of computing resources, the proliferation of cloud-based solutions, and the rise of microservice architectures.</p><p>This required a revised approach to logging, which we need to understand by going back to our original questions. But that will be the subject of the next article!</p><p>In the meantime, I hope this article has helped to take some of the boredom out of logging!</p><h3>Resources</h3><ul><li>The secret origin of “log in”, <a href="https://www.designcult.org/2011/08/why-do-we-call-in-logging-in.html">https://www.designcult.org/2011/08/why-do-we-call-in-logging-in.html</a></li><li>RFC5424: The Syslog protocol, <a href="https://datatracker.ietf.org/doc/html/rfc5424">https://datatracker.ietf.org/doc/html/rfc5424</a></li><li>Interesting thread on the use of different log levels, <a href="https://stackoverflow.com/questions/2031163/when-to-use-the-different-log-levels">https://stackoverflow.com/questions/2031163/when-to-use-the-different-log-levels</a></li><li>The Art of Logging, <a href="https://www.codeproject.com/Articles/42354/The-Art-of-Logging">https://www.codeproject.com/Articles/42354/The-Art-of-Logging</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10da1f7c66b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>