The Linux Server Nobody Was Watching
Break the Mold —Project 01 — SirhurryUp Cloud Engineering Accelerator
There is something almost comical about a server going down while everyone responsible for it goes about their day believing the world is still turning.
The server, of course, knows better.
Customers are waiting. Requests are going nowhere. Somewhere, an employee is staring at a screen, refreshing a page as though persistence might persuade the machine to reconsider its position.
And nobody knows a damn thing.
At Northstar Business Services, a fictional company I created for this engineering scenario, the application had been unavailable for approximately forty minutes. Someone eventually noticed. An engineer restarted the service, and just like that, everybody could get back to business.
Awesome!
Except nobody knew why it failed.
A restart can be a remarkable thing. One command, a few seconds, and suddenly everyone is ready to declare victory. But a server returning to life does not mean the mystery of its disappearance has been solved.
That distinction became the starting point for my first Break the Mold eight-week engineering projects.
This assignment was straightforward: build a lightweight monitoring system for an Apache web server running on Linux. Detect when it stops responding, distinguish an occasional hiccup from a persistent failure, and produce enough evidence for someone to investigate.
I chose Python even though I prefer R for the health check and systemd to handle the scheduling. No elaborate monitoring platform. No expensive collection of services assembled merely because AWS makes them available. Old school forgotten practices.
Just a small system with a serious responsibility.
And before long, I discovered that getting a script to run was the easy part.
Getting it to tell the truth required considerably more thought.
The trouble with a green light
My first instinct was simple enough. Send an HTTP request to Apache. If the server responds, report healthy. If it doesn’t report a failure.
Python’s urlib.request could handle that without installing another library. I pointed the request to http://localhost and sent a five-second timeout.
There was a practical reason for choosing localhost. The monitoring script was running on the same EC2 instance as Apache. It didn’t need to travel across the internet to ask a service sitting a few feet away, figuratively speaking, whether it was still alive.
But I had to be careful about what a successful response meant.
An HTTP 200 response tells me that the endpoint answered. It does not tell me whether a customer can submit a service request, whether a database is available, or whether the application is quietly making a mess of somebody’s afternoon.
A green light can be useful. It can also be terribly persuasive.
For this project, I limited the claim: Apache responded to the local HTTP check. Nothing more, nothing less.
Three strikes, but nobody’s out
A single failed request isn’t always an outage. Networks hesitate. Processes get busy. Machines, despite their impressive reputations, occasionally behave like the rest of us on Monday morning.
I didn’t want the monitoring system declaring catastrophe every time it encountered one unsuccessful check.
So I introduced a threshold.
The first failure would produce a warning. The second would produce another warning. The third consecutive failure would become critical.
That sounds straightforward until you remember that each execution of the Python script starts a new process. Why did I divorce R?
The script can’t simply remember what happened a minute ago by wishing hard enough.
I needed or should I say wanted a persistent state.
I stored the failure count in a file under /var/lib/northstar-monitoring/. Each execution could read the previous count, update it, and write it back. A successful health check reset the counter to zero.
This gave the monitoring logic a modest but important memory.
It also introduced a tradeoff. Waiting for three failures reduces noise, but it delays escalation. With checks scheduled approximately once per minute, that delay matters when the fictional customer’s requirement is notification within five minutes.
A threshold isn’t automatically a good decision because it sounds sensible. It has to be tested against what the customer actually needs.
The scheduler that didn’t need an audience
Writing a health check is one thing. Remembering to run it is another.
I could execute the Python script manually all day, but that would make me the monitoring system. And, frankly, I have other things to do.
Linux already had a solution: systemd.
I created a oneshot service to execute the Python script and a timer to trigger that service approximately every sixty seconds.
The service handled execution. The timer handled scheduling. The journal recorded what happened.
There was no need to build a scheduling framework to perform a task the operating system already knew how to do. In other words, handle it systemd!
More importantly, I could inspect the service and timer independently. If the script worked manually but failed when invoked by systemd, I had a specific boundary to investigate.
Get Dottington Fullwood’s stories in your inbox
Join Medium for free to get updates from this writer.
That separation would become useful during testing.
I broke it on purpose
Eventually, there comes a moment when looking at a successful command is no longer enough.
I had a script that could report healthy. Lovely. But a monitoring system that only works when everything is working isn’t much of a monitoring system.
So I stopped Apache.
The first check reported a connection refusal and warning one of three.
The second reported warning two of three.
The third reported a critical condition after three consecutive failures.
Then I restarted Apache and ran the check again. The script reported healthy and reset the stored failure count to zero.
This was the first point where I had evidence that the failure-counting logic behaved as designed under a controlled outage.
I also confirmed that the systemd timer was invoking the service automatically and that the journal contained successive healthy executions.
Of course, I made mistakes along the way.
One revision of the script incremented the failure counter twice. Another referenced the wrong variable inside an exception handler. Neither mistake was particularly glamorous, but both mattered.
A counter that advances incorrectly can escalate too early. An exception handler with the wrong variable can fail precisely when it is supposed to explain a failure.
I corrected the code, checked its syntax, and repeated the controlled tests.
There is a difference between a script that looks reasonable and one whose behavior you’ve actually observed.
The terminal is remarkably indifferent to how confident you feel.
The email that exposed the missing bridge
The original requirement wasn’t merely to detect an outage. Northstar wanted its IT team notified within five minutes.
I configured an Amazon SNS topic, confirmed an email subscription, and assigned the EC2 instance an IAM role with permission to publish to that specific topic.
Then came the awkward party. Take my word, you did not want to attend.
The AWS CLI could identify the instance’s AWS identity, but attempts to publish an SNS message repeatedly failed with a badly formed help string error. Reinstalling the CLI didn’t resolve it, and I didn’t establish the root cause.
I could publish a test notification from the SNS console, and the email arrived.
But that proved only that the SNS topic and email subscription could deliver a manually published message.
It did not prove that my Python monitoring script could publish an alert.
The health check worked. The timer worked. The SNS email test worked.
The connection between the monitoring script and SNS was still missing.
I could have written that the monitoring solution was complete, pointed to the email in my inbox, and hoped nobody asked how it got there.
But that would have confused a successful component test with a successful end-to-end test.
And it would have defeated the purpose of the exercise.
The five-minute notification requirement remains verified. That is not a footnote to hide. It is an engineering finding to document.
What the server taught me
By the end of the project, I had a Python health check, persistent failure tracking, systemd scheduling, journal evidence, and a controlled test showing warning, critical, and recovery behavior.
I also had an unfinished alerting integration and an AWS CLI problem I had not resolved.
Both belonged in the record.
What interested me the most was how many decisions lived inside what initially looked like a small assignment.
Should the health check run locally or remotely? What does an HTTP 200 actually prove? How many failures justify escalation? Where should state survive between executions? Who is allowed to publish notifications? How do I know the scheduler is doing its job?
And perhaps the most important question: what evidence would persuade someone other than me that the system works?
Those questions turned a short Python script into an engineering scenario.
I documented the architecture, decisions, tests, and limitations in the project repository. Not because every command deserves a monument, but because someone reviewing the work should be able to distinguish what I intended from what I actually verified.
Northstar’s fictional server now has a mechanism for detecting when Apache stops responding.
Its notification problem is not fully solved.
I know the difference.
And that, for Project 01, is progress worth keeping.
Break the Mold — Project 01 of 08
GitHub: The Linux Server Nobody Was Watching — Project Repository
Next: Project 02. Another business problem. Another set of engineering decisions. And, I suspect, another opportunity for a perfectly innocent command to ruin my afternoon.
Pace and purpose.
