While My Python Script Gently Logs
A class template for dual logging using the structlog library
Logging is an essential aspect of any project that involves writing code. It helps tracking and debugging errors, it improves monitoring the performance of your code, and it provides valuable insights into the general behavior of your application. Structured logging is crucial for easily parsing and analyzing log data, making it an indispensable tool for software engineers, data engineers, data scientists and alike.
A logger class template
We’ll explore a simple class template that configures a useful structured logging setup using the structlog library in Python. Without further ado, below is the full class definition.
import logging
import sys
from logging.handlers import WatchedFileHandler
import structlog
class Logger:
def __init__(self, log_name, log_file, level):
# define basic configuration
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M:%S"),
structlog.stdlib.add_log_level,
structlog.stdlib.add_logger_name,
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.stdlib.ProcessorFormatter.wrap_for_formatter
],
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True
)
# define stdout handler
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(
structlog.stdlib.ProcessorFormatter(
processor=structlog.dev.ConsoleRenderer(colors=True)
)
)
# define log file handler
file_handler = WatchedFileHandler(filename=log_file, mode="wt")
file_handler.setFormatter(
structlog.stdlib.ProcessorFormatter(
processor=structlog.processors.JSONRenderer()
)
)
# add handlers to single root logger
root_logger = structlog.getLogger(log_name)
root_logger.addHandler(console_handler)
root_logger.addHandler(file_handler)
# set logging level
root_logger.setLevel(level)
# add logger to class instance
self.logger = root_logger
@classmethod
def get_logger(cls, log_name, log_file, level=logging.INFO):
return cls(log_name, log_file, level).loggerLet’s go over its components step by step.
When prefixed by
stdlib, the functionality is taken from Python’s standard logging library. In fact, structlog is built on top of that library!
Step 1: Set basic configuration
The first step is to define a set of processors that will be applied to each log entry. These processors sequentially add essential information to the log messages, making them more informative and (wait for it…) structured.
These are the six processors involved:
- TimeStamper() → Adds a timestamp to each log entry, using the specified format. A useful alternative format is
fmt="iso". - add_log_level → Ensures that the log level (e.g. INFO or WARNING) is included in the log entries.
- add_logger_name → Adds the logger’s name to the log entries.
- StackInfoRenderer() → Appends stack information to log entries, helping to identify where the log message originated.
- format_exc_info → If an exception occurs, this processor formats and includes exception information in the log message.
- wrap_for_formatter → Prepares the log entries for further downstream formatting.
You can always modify the processors. Be mindful that the order matters, because the output of one processor is served as the input of the next.
There are three other arguments in the configuration. These are used to customize the behavior of the logger. The values chosen are rather standard, so don’t worry if you can’t understand what they exactly do. Here’s an explanation of each:
The logger_factory argument specifies the factory function that is responsible for creating logger objects and configuring them with the desired behavior. With structlog.stdlib.LoggerFactory(), the logger instances are compatible with Python standard’s logging module. This means you can seamlessly use structlog in conjunction with existing logging infrastructure.
The wrapper_class argument stipulates the class that will be used to wrap the underlying logger instances created by the logger factory. This choice affects how you log messages and the methods available for logging. In other words, it defines the interface to interact with the logger. The class structlog.stdlib.BoundLogger provides a convenient way to log messages with structured data. It can be customized if you have specific requirements for your logger’s behavior.
The cache_logger_on_first_use argument is a boolean flag that controls whether the logger instance created by the logger factory should be cached. If enabled, the first time you request a logger for a specific name, the logger will be created and cached. Subsequent requests for the same logger name reuse the cached instance instead of creating a new instance. This can lead to performance gains, as it avoids creating multiple logger instances for the same name.
Step 2: Define handlers
Next, we define two log handlers, one for console output and another for writing log entries to a file.
Get Sam Borms’s stories in your inbox
Join Medium for free to get updates from this writer.
For the console handler, we create a StreamHandler() to direct log entries to the console (or more precisely, the standard output stream). The ConsoleRenderer() processor renders log entries with colors for improved readability, and because it looks cool.
For the persistent log storage, we use the WatchedFileHandler() to write log entries to a specified file in JSON format using the JSONRenderer() processor. This structured format makes log analysis more accessible. The mode is set to "wt" which means “open a text file for writing.” If the file already exists, its contents are destroyed and a new file is created.
Step 3: Add handlers to root logger
Now, we add both handlers to a single root logger. The root logger is the central point where log entries are collected before being processed and dispatched to the specified outputs.
In our case, logs are written simultaneously to the console and to the specified file — hence the concept of dual logging. It combines colorful and pretty printing handy during local development, and structured JSON log output that is a lot easier to parse and analyze by so-called log aggregators.
Step 4: Pass logging level
We set the logging level of the root logger to INFO by default. This means that log entries with a level of INFO or higher will be processed and recorded. This is an overview of available logging levels according to the official Python logging documentation. In order, the main levels are DEBUG, INFO, WARNING, ERROR, CRITICAL.
Step 5: Assign logger to object
Finally, we assign the root logger to the object itself and create a get_logger() class method. This allows to use the logger throughout your codebase by immediately calling this method with the desired parameters. Most loggers work with such an initialization method.
How to use the logger?
To use the logging functionality provided by the above class, first store the Logger class separately in a logger.py script.
This short Python script demonstrates how to import the Logger class and initialize two instances. The instances each write to a different log file.
from logger import Logger
# initialize two logger instances with different log files
logger1 = Logger.get_logger("Logger1", "log1.log")
logger2 = Logger.get_logger("Logger2", "log2.log")
# log some messages using each logger
logger1.info("This is a message from Logger1.")
logger2.info("This is a message from Logger2.")
logger1.warning("This is a warning from Logger1.")
logger2.error("This is an error from Logger2.")We then use these loggers to log some messages with different log levels. The log messages will be directed both to the output and to their respective log files. Based on the name of the logger, you can see which logger wrote what message.
This is how the logs appear in the console:
This is how the logs for the second logger appear in the log file:
Neat, isn’t it?
Conclusion
Structured logs are invaluable for understanding the behavior of your code and ensuring its reliability. We’ve seen how to cook up a Python class to carry out structured logging using the structlog library. Embedding such class in your code will ensure that your projects are equipped with comprehensive, structured and easy on the eyes logs.
That’s a wrap! I hope this will help you to smoothly integrate logging into your projects. Oh, and this. 🎸










