From Up and Running to No Longer Reachable
Overview
CumulAI is an early-stage SaaS company that decided to run it’s internal tools on a small fleet of Linux VMs. As the on-call Linux administrator I’ve been tasked with ensuring the site hosting these tools remains accessible.
The issue…
A ticket was escalated to me reading: “Our internal status site stopped responding. Browsers and curl to http://localhost:8080 just hang or time out. We already checked — the web service process is running, so we’re stumped. Please get the site reachable again.”
The service is a small static site served by systemd as sitehttp.service on port 8080. The reporter is right that the process is up — so the problem isn’t the app. Something on this host is stopping traffic from reaching the port. My task is to find it, remove it, and confirm the site answers.
Get Stephan Barlatier’s stories in your inbox
Join Medium for free to get updates from this writer.
After getting logged into the server, the first step is to confirm the service is up and running.
systemctl status sitehttp.service --no-pager
It’s clear the service is still running. Now I need to verify what the site is supposed to return.
cat /srv/site/index.html
Now to test the same way the reporter did.
curl -v --max-time 5 http://localhost:8080; echo "Exit code: $?"
Next, I confirmed that the service is listening on the appropriate port.
ss -tlnp | grep 8080
The service is listening on port 8080. It’s now to time to check the firewall and see if there is an issue with the configuration for incoming traffic.
sudo iptables -L INPUT -n --line-numbers
The output states that all incoming traffic on port 8080 is being dropped, leaving the site unreachable. To resolve this issue, the drop rule has to be deleted from the firewall.
sudo iptables -D INPUT -p tcp --dport 8080 -j DROP
## to verify the rule has been removed, run
sudo iptables -S INPUT
## verify the site is reachable
curl -s --max-time 5 http://localhost:8080
## verify with a live status status returned
curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 http://localhost:8080200 code is returned. Now to close out the ticket.
What: The internal site on port 8080 was not reachable. Both curl and browser access would hang/timeout.
Why: the sitehttp.service was running and listening fine, but a firewall rule on the INPUT chain (-p tcp — dport 8080 -j DROP) was silently dropping all traffic to the port 8080.
Fix: Deleting the DROP rule with iptables -D, then verifying curl http://localhost:8080 returns 200 code with the expected CUMULAI-OK page.
How does this affect the business?
A site hosting internal tools being unreachable creates a reliability issue. Down time is an achilles heel to business continuity, and beyond being undesirable, it can be unaffordable. Every second a system or service is down for a business like CumulAI, the costs can become unimaginable.
It is important to note that with proper change management processes and formal auditing, this issue could have been avoided all together before ever hitting production.
Thanks for making it this far. I hope you found this article helpful and if you have any questions or feedback leave a comment.
Let’s connect on LinkedIn!
