Logs — Never tells a LIE..

Logs is the earliest, most trusted and core observability signal (others are built on top of it). Logs tell your system’s story, make sure you’re listening — long before metrics, events, or traces gained popularity. From early UNIX systems to modern distributed microservices on Kubernetes, logs have remained the primary source of truth for understanding what really happened inside a system. They capture the story behind every request, failure, warning, or anomaly — not as aggregated numbers but as raw, granular, contextual evidence.

Basic Log Structure: What Makes a Useful Log?

A well-structured log entry is more than just a line of text. It should be machine-readable and consistently formatted. Here’s a basic structure every log should include:

• Timestamp: Backbone info of logs, when did it happen? (e.g., 2025-12-11T20:45:12Z)
• Log Level: INFO by default, use DEBUG temporarily for troubleshooting — continuous DEBUG logging can quickly fill disk space.

• Message: Human-readable description of the event
• Source/Service: Which component generated the log?
• Context/Fields: Additional structured data (request ID, user ID, status codes, latency)

{
 "timestamp": "2025–12–11T20:45:12Z",
 "level": "ERROR",
 "message": "Payment processing failed",
 "service": "payment-service",
 "status_code": 500,
 "request_id": "req-abc123"
}

Where Do Logs Come From? Common Sources

Logs are generated everywhere in modern systems:

• Servers: System logs (syslog), authentication logs, kernel logs, journal logs.
• Cloud Platforms: AWS CloudWatch log, Azure Activity logs, GCP Audit logs
• Containers & Orchestration: Docker container logs, Kubernetes pod/control-plane logs
• Applications: Logs for services like Java, Python, Golang, Php, NodeJs etc
• Middleware: Webservers (Nginx, Apache), ApiGateway (Kong), ServiceMesh (Istio) logs
• Network & Security: VPC Flow logs, Firewall logs, IDS/IPS alerts, VPN logs etc

Each source may have different formats, but the goal is the same: to capture what happened.

Logs to Logging -

Logs are ephemeral artifacts and can be lost during log rotations or server termination in auto-scaling group. Secondally, manual inspection is unreliable for 24x7 coverage. Automated logging pipeline essential to ensure data is captured, stored, and accessible —

1. ⚙️ Collection

Agents (Vector, Filebeat, Fluentbit, Promtail etc) are installed near the source to collect, initially process and ship the logs. This is the pipeline’s crucial first mile.

Get Ashish Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A log agent can operate in two distinct collection modes:

• Pull is agent-driven: the agent continuously monitors and reads from sources. This is the industry-standard approach for capturing the logs.
• Push is source-driven: external systems deliver logs directly into the agent.

sources:
  k8s_cluster_logs:
    type: kubernetes_logs 
    #fundamentally file-based
    exclude_paths_glob_patterns: 
    - "/var/log/pods/kube-system_*/**" 
    
  syslog_udp:
    type: socket
    mode: udp
    address: 0.0.0.0:514  

Look at the configuration —

• k8s_cluster_logs (type: kubernetes_logs) This source tails log files under /var/log/pods. Since the agent is actively reading new lines from files, it represents a pull-based mechanism.
• syslog_udp (type: socket) This source binds to UDP port 514 and waits for external syslog daemons or applications to send packets. Because the external system initiates the transfer by writing directly into the agent’s socket, it is a push-based mechanism.

2. 🧩 Processing

Manual log inspection doesn’t scale, Automation is key. Logs must be structured and cleaned before storage. The Processing layer (Logstash, Fluent/Loki) transforms the raw data -

• Parsing: Structuring raw, plain text log lines into machine-readable fields (e.g., separating timestamp, log level, message body). This typically involves using tools or custom regular expressions.
• Enrichment: Adding context to the log event (e.g., Geo-IP lookup, mapping internal IDs to usernames, adding environment tags, or linking to tracing IDs). This makes logs more useful for analysis.
• Filtering: Dropping irrelevant or high-volume noise (e.g., routine health checks, debug logs) early in the pipeline to save on storage and processing costs.

3. 📊 Visualization

This is how data transforms into insights. Visualization tools connect to the Short-term Storage (OpenSearch/ElasticSearch) to query and display log data.

• Tools: Kibana, Grafana, Redash.
• Purpose: Search, analyze trends, troubleshoot incidents, and monitor system health using dashboards.

Best Practices for Search Performance:

• Restrict to relevant indices or index patterns so only the required data is scanned.
• Use aliases or filtered views that inherently narrow search scope.
• Avoid overly broad wildcard patterns that match many old or unrelated indices.
• Tailor your queries (e.g., time range filters) to reduce the amount of data examined.

4. 🚨 Alerting

Proactive notification based on log events. The alerting mechanism continuously monitors the short-term index for patterns or thresholds that indicate a problem.

• Tool: ElastAlert2.
• Action: Triggers a notification when predefined log conditions (e.g., error rate spike, specific critical message) are met.

5. 🗄️ Storage & Retention Policy (Hot, Warm, Cold Storage)

The Storage and Retention Policy time windows are defined based on debugging needs and compliance (GDPR, PCI DSS) meet. Based on these, logs can be categorized in 3 phases (hot, warm & cold) as document above. Recycle — another one more important phase, its important to discard logs that become obsolete over time to avoid unnecessary storage costs

SLA Management: From MTTR to Uptime

Logs directly support Service Level Agreements through -

Mean Time to Resolution (MTTR):

Structured logs with unique identifiers (request_id, trace_id) allow rapid incident investigation. Instead of grepping through terabytes of text, engineers query specific transactions:

status: 500 AND service: "payment-service" AND request_id:"req_abc123"

Request-Based Uptime:

Uptime can be calculated based on number of 5xx vs total number of requests, its 99.26% (except — rate of server errors 5xx, 0.74% red slice).

Beyond this, a comprehensive system state can be defined using other metrics derived from logs metadata, such as client errors (4xx), latency, and request patterns. My other blog on SLA, you may refer in case want to deep dive into it.