Azure Iot Edge
Cloud
Reliability
Security
Edge Computing

#11. Designing Edge Devices for Stable 24/7 Unattended Operation

Secure Edge Ops

13 min readSep 26, 2026

Introduction

I work on improving the security and reliability of edge/cloud environments used in medical systems, especially in environments where devices are expected to operate unattended for long periods.

Some of the main topics I focus on are:

recovery after power failures
retaining data during offline periods
detecting communication failures
applying security patches safely
maintaining long-term system stability

In the previous article, I discussed an important question:

How much of Azure IoT Edge and the container runtime should be fixed to a specific version, and how much should be allowed to update?

The policy I adopted was roughly:

Security updates
↓
Apply them

Patches within the same release generation
↓
Apply them under controlled conditionsUnvalidated major or generation changes
↓
Do not apply automatically

The goal was to balance two requirements that often conflict:

Security and operational stability

But version management alone is not enough to operate an unattended edge device 24 hours a day, 365 days a year.

In a real edge PC, many other things can go wrong:

System clock drifts

DNS configuration changes unexpectedlyNetwork routes are not restored correctlyFirewall or NAT state becomes inconsistentA NIC does not recover properlyAn update requires a rebootContainer runtime state changesOnly part of the system recovers after a power failure

So the design principle for this article is different.

Instead of trying to guarantee that failures never happen:

Design the system so that it can return to a known-good state when something goes wrong.

One of the main tools I used for this was:

systemd timer

1. Booting Successfully Is Not Enough

During development, we often verify something like this:

Power ON
↓
Ubuntu boots
↓
Docker starts
↓
Azure IoT Edge starts
↓
Application starts
↓
Cloud connection established

At that point, the system may appear complete.

For a device that runs unattended 24/7, however, that initial success is not the most important thing.

The real question is whether the system can return to the same working state:

One day later

One week laterOne month laterAfter an updateAfter a network outageAfter a power failure

That changed how I thought about the system.

Instead of focusing only on:

Creating the correct initial state

I also focused on:

Recreating the correct state periodically.

In long-term unattended operation, it is almost impossible to prevent every possible abnormal condition.

For example:

Temporary network failures

Changes on the router or DHCP sideClock driftPackage updatesStopped processesUnexpected rebootsPower failures

Some of these events are outside the control of the edge PC itself.

So the design changed from:

Prevent every abnormal condition

to:

If an abnormal condition occurs,
return to the expected state during the next recovery process

Conceptually:

Normal State
             ↑
             │
          Verify
             │
        Repair if needed
             │
             │
     Temporary Failure

In other words:

Define the expected state, detect deviation from it, and restore it when necessary.

I applied this idea to each maintenance process.

2. Do Not Put Everything Into One Huge Maintenance Script

One approach I wanted to avoid was something like:

Every night at 01:00

Check everything
Update everything
Reconfigure the network
Restart everything
Reboot

Combining everything into one script may reduce the number of files that need to be managed.

But when something fails, troubleshooting becomes much more difficult.

For example:

Where exactly did the process fail?

Does one failed step stop everything after it?Did the network process conflict with an update?Did a reboot make it difficult to inspect the logs?

Instead, I separated maintenance processes according to their responsibilities.

For example:

NTP

NATDNSIoT Edge updatesOS reboot checksIoT Edge reboot checksNIC recoveryRoute repairFirewall verificationDiagnostics

Each process has its own service and timer.

This makes the system easier to understand and easier to troubleshoot.

3. Separating Responsibilities with systemd Timers

For this system, I use standard Linux mechanisms:

systemd service
+
systemd timer

The basic relationship is:

.timer
↓
Starts at a defined time or condition

.service
↓
Performs the actual operation

For example:

mdlinkage-ntp-sync.timer
↓
mdlinkage-ntp-sync.service
↓
NTP synchronization

With this structure, I can check whether the timer itself is running:

systemctl status xxx.timer

And I can inspect logs for only the corresponding operation:

journalctl -u xxx.service

The important benefit is that it separates:

When something should run

from:

What the process actually does

That separation becomes extremely useful when debugging an unattended device.

4. The Maintenance Schedule

For this edge PC, the main maintenance processes are distributed across several hours.

00:15
Scheduled reboot

00:45
NTP synchronization
NAT verification01:00
DNS verification and recovery01:30
IoT Edge / Moby patch check02:30
Check whether OS updates require a reboot03:30
Check the reboot flag created by
IoT Edge / Moby updates04:30
Verify and recover NIC state

In addition, immediately after boot, several network-related states are checked again:

Route

NATFirewallNICNetwork configuration

The important point is:

These processes do not all run at the same time.

5. Why Maintenance Tasks Are Staggered

Imagine that all of the following occur simultaneously:

NTP synchronization

Package updatesDNS configurationContainer restartNetwork reconfigurationSystem reboot

Individually, every operation may be valid.

But when they happen at the same time, they can interfere with one another.

For example:

The network temporarily disconnects

The machine reboots during an updateThe NIC restarts while DNS is being checkedThe container runtime changes while container health is being verified

If something fails, troubleshooting also becomes difficult.

You are left asking:

Was it NTP?

DNS?An update?The network?The reboot?

So the schedule looks more like this:

Process A
↓
Wait
↓
Process B
↓
Wait
↓
Process C

The key assumption is:

Maintenance processes themselves can become a source of failure.

This is why I deliberately separate them in time.

6. 00:15 — Scheduled Reboot

The system also performs a scheduled reboot every day at:

00:15

using a timer such as:

scheduled-reboot.timer

On an edge PC running continuously, temporary states can gradually accumulate.

For example:

Long-running application state

Runtime state changesTemporary network stack issuesUnexpected process states

A scheduled maintenance window provides an opportunity to periodically return the entire system to a clean boot state.

But simply rebooting the device is not enough.

7. The System Must Recover Automatically After Reboot

A scheduled reboot becomes dangerous if the device does not automatically recover afterward.

For example:

Network does not return

Route is missingFirewall configuration is not restoredNAT rules are missingNIC remains downIoT Edge does not start

In that case, the scheduled reboot itself becomes the cause of downtime.

That is why several recovery processes run after boot.

For example, a timer such as:

fix-route.timer

checks the routing table after boot.

If the expected route already exists:

OK

and the process exits.

If the route is missing, the script restores the expected state.

The logic is:

Check current state
↓
If healthy, do nothing
↓
If unhealthy, repair it

rather than:

Rewrite the configuration every time

This distinction is important.

8. Firewall and NAT Are Runtime States Too

The same idea applies to firewall and NAT configuration.

Configuring:

nftables

during the initial setup is not enough.

After power failures, network restarts, or other system events, I want to verify that the runtime state is still what I expect.

For this reason, I use recovery processes conceptually similar to:

ensure-nft-filter

ensure-nft-nat

The logic is:

Check the current rules
↓
Healthy
↓
Do nothing

or:

Check the current rules
↓
Missing or abnormal
↓
Restore the expected state

The principle here is:

Do not only verify configuration files. Verify what is actually active at runtime.

9. 00:45 — Synchronizing Time

Time synchronization is also important for an edge gateway.

Significant clock drift can affect:

TLS certificates

Log timestampsCloud communicationTroubleshootingComparing logs across multiple systems

During troubleshooting, I may need to compare:

Edge PC logs

Network logsCloud-side logs

If their clocks are inconsistent, reconstructing the sequence of events becomes much more difficult.

In this configuration, NTP synchronization is performed as a controlled:

one-shot

operation at defined times rather than relying only on a continuously running daemon.

The synchronization is also triggered after boot:

Boot
↓
Network Ready
↓
NTP Sync

This means that after a power failure, the device has an opportunity to correct its clock as part of the recovery sequence.

10. 01:00 — Periodically Checking DNS

DNS is another critical dependency for Azure IoT Edge.

The network itself may appear healthy:

IP connectivity
↓
OK

DNS resolution
↓
NG

But if DNS resolution fails, the device may still be unable to communicate with Azure IoT Hub.

From the user’s perspective, the symptom may simply look like:

The network is connected,
but data is not reaching the cloud.

For this reason, DNS is not treated as something that is configured once and forgotten.

Its expected state is checked periodically.

It also helps to think about communication in layers:

Network
↓
Route
↓
DNS
↓
Cloud

Separating the layers makes troubleshooting easier.

11. 01:30 — A Dedicated Update Path for IoT Edge

At:

01:30

the device checks for IoT Edge and container runtime patches.

This is the update process discussed in the previous article.

Conceptually:

Check installed version
↓
Check candidate version
↓
Confirm that it remains within
the allowed release generation
↓
Dry-run
↓
Check dependencies
↓
Apply patch if conditions are satisfied
↓
Health check

This process does not run at the same time as OS updates or network maintenance.

I also use:

RandomizedDelaySec

so that if many edge PCs are deployed, they do not all begin maintenance at exactly the same moment.

For example:

Edge PC A
Edge PC B
Edge PC C
Edge PC D

As deployments grow, this becomes increasingly important.

A maintenance architecture designed for one device should not create a synchronization problem when the fleet grows.

12. 02:30 — Reboot Only When the OS Requires It

After certain Ubuntu updates, such as kernel or critical package updates, the system may create:

/var/run/reboot-required

However, I do not reboot immediately after the update.

The update process finishes first.

Get Secure Edge Ops’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Then, at a separate time:

02:30

another process checks whether a reboot is actually required.

No reboot-required file
↓
Do nothing

reboot-required exists
↓
Reboot

In other words:

The update process and the reboot decision are separated.

This reduces unnecessary coupling between maintenance tasks.

13. 03:30 — Reboots Required by IoT Edge Updates Are Also Separated

The same principle is applied to IoT Edge and Moby updates.

If a patch requires a reboot, the update process creates a flag such as:

IoT Edge / Moby Update
↓
pending-iotedge-reboot

Then, at:

03:30

a separate timer checks the flag.

No flag
↓
Do nothing

Flag exists
↓
Check conditions
↓
Reboot

The logic also considers the OS-level:

reboot-required

state so that the system does not reboot multiple times within a short period unnecessarily.

The architecture therefore becomes:

Update
↓
Flag
↓
Maintenance Window
↓
Reboot

instead of:

Update
↓
Immediate Reboot

14. 04:30 — Verify the NIC Last

Near the end of the maintenance window, the system also checks the:

NIC

A timer such as:

nic-wakeup.timer

runs both:

After boot

and:

Daily at 04:30

For an edge gateway, the application may be healthy, but if the network interface is not functioning correctly, the system cannot perform its role.

I therefore treat these as separate layers:

Application

ContainerOperating SystemNetwork Interface

For example:

Application healthy

butNIC unhealthy

still means the gateway as a whole is unhealthy.

15. Use Both Boot-Time Recovery and Scheduled Maintenance

Some timers run at a specific daily time.

Others also use:

OnBootSec

so that they can run shortly after the device starts.

Examples include:

NTP

NATRouteNICFirewall

This was important for handling unexpected power failures.

Imagine a power failure occurs at:

00:30

The device should not have to wait until the next day’s scheduled maintenance to recover.

Instead:

Power Failure
↓
Power Restored
↓
Ubuntu Boot
↓
Network Ready
↓
Verify Route
↓
Verify Firewall / NAT
↓
Verify NIC
↓
Synchronize Time
↓
Application Recovery

In other words:

Boot itself becomes a recovery trigger.

16. Use `Persistent=true` Selectively

systemd timers also support:

Persistent=true

For example, if a task was scheduled for 01:00 but the PC was powered off at that time, systemd can run the missed task after the next boot.

That is useful for some processes.

But it does not make sense for every timer.

Consider a scheduled reboot.

If the device was powered off during yesterday’s planned reboot window, there is little value in doing this immediately after the next boot:

Yesterday's reboot was missed
↓
Reboot immediately after startup

So I separate timers into:

Tasks whose missed execution
should be recovered after boot

andTasks whose missed execution
should simply be skipped

Again, the important question is not:

What systemd option should I enable?

The real question is:

Why does this timer exist, and what behavior is appropriate for that purpose?

17. Record Results, Not Just Execution

On an unattended device, nobody is watching the screen to confirm:

Success

So the result of every maintenance operation needs to be observable later.

With systemd, I can inspect logs per service:

journalctl -u mdlinkage-ntp-sync.service

And I can review timers with:

systemctl list-timers --all

This shows information such as:

When the timer last ran

When it will run nextWhether it is active

For example:

systemctl list-timers --all | grep -E \
'mdlinkage|mdl-|reboot-if'

can be used to review maintenance-related timers together.

18. Verify the Timers Themselves

Creating a timer file does not prove that the maintenance mechanism works.

I want to verify more than:

The timer file exists

I also check:

Is it enabled?

Is it active?Is the scheduled time correct?Did the corresponding service actually run?Was the exit code successful?

For example, during acceptance checks I verify that entries such as:

00:15 Scheduled Reboot

00:45 NTP00:45 NAT01:00 DNS01:30 IoT Edge Patch02:30 OS Reboot Check03:30 IoT Edge Reboot Check04:30 NIC Wakeup

are actually registered at the expected times.

After boot, I also verify evidence that processes such as:

Route

NTPNATFirewallNIC

actually executed during the current boot session.

The principle is:

Do not only check whether the configuration exists. Check evidence that it actually ran.

19. Maintenance Operations Should Be Safe to Run Repeatedly

Another principle I applied to recovery processes is:

Running the process against
an already healthy system
should not break it

For example, if a route is already correct:

OK: route already correct

and the process exits.

Similarly, if the expected firewall table and rules are already healthy:

already healthy

and nothing is changed.

The timer therefore behaves like:

Timer runs
↓
Check current state
↓
Healthy
↓
Exit

or:

Timer runs
↓
Check current state
↓
Unhealthy
↓
Repair

rather than:

Timer runs
↓
Forcefully rewrite everything

This reduces the chance that the maintenance mechanism itself becomes a new source of failure.

But Scheduled Maintenance Cannot Detect Everything

With these mechanisms, the edge PC can verify many aspects of its own health.

For example:

Time

DNSRouteFirewallNATNICIoT EdgeContainer RuntimeReboot Requirements

But there is one important limitation.

What happens if the edge PC completely freezes?

Edge PC
↓
Freeze
↓
CPU / Kernel / Network stop

At that point, the device’s own:

Timers

Health checksLog collectionRecovery scripts

also stop.

This is why internal recovery alone is not sufficient.

20. Self-Recovery and External Monitoring Have Different Roles

I eventually divided the reliability architecture into two responsibilities.

The edge PC itself is responsible for:

Edge PC
↓
Maintain the expected state
↓
Recover automatically when possible

An external monitoring device is responsible for:

External Monitor
↓
Check from outside
whether the Edge PC is actually alive

The external monitor can observe the device through multiple paths, such as:

Ping

HeartbeatTelemetryrsyslognetconsole

It is especially useful to separate an application-level heartbeat from lower-level signals such as kernel logging.

This makes it easier to distinguish between situations such as:

The application stopped,
but the kernel is still alive

Only the network stoppedThe entire operating system frozeA kernel crash occurred

I will cover this external monitoring architecture in more detail in the next article.

21. Monitoring Should Not Destabilize the System Being Monitored

More monitoring is not always better.

Imagine repeatedly running commands such as:

docker inspect

smartctlsystemctlpowertopmany short-lived processes

every few seconds.

Monitoring itself may begin to affect:

CPU load

Power statesDisk accessRuntime behavior

This is particularly important when investigating freezes.

The monitoring process itself should not significantly change the conditions you are trying to observe.

For that reason, I divide telemetry collection according to the cost of collecting each type of information.

For example:

Every 5 seconds
↓
Lightweight heartbeat / telemetry

Every 60 seconds
↓
C-state / PSI / vmstat /
IRQ / Docker stateEvery 10 minutes
↓
Network / filesystem /
NTP / SMARTEvery hour
↓
More detailed SMART information

The principle is:

Collect the information you need, but avoid changing the system too much just by monitoring it.

22. Putting the Architecture Together

For this 24/7 unattended edge device, I think about reliability in three layers.

1. Prevention

Version management
Security updates
Scheduled maintenance
Staggered maintenance windows
2. Self-RecoveryScheduled reboot
Route verification after boot
Firewall / NAT verification
DNS verification
NTP synchronization
NIC recovery
Conditional reboot
3. External DetectionPing
Heartbeat
Telemetry
rsyslog
netconsole

The overall strategy is:

Make failures less likely
↓
If something fails, recover
↓
If recovery fails, detect it externally

Conclusion: 24/7 Reliability Does Not Come From One Powerful Mechanism

There is no single special technology making this system reliable.

Most of the mechanisms are standard Linux tools:

systemd

systemd timerjournalnftablesShell scriptsHealth checks

The important part is how they are combined.

For every possible failure, I need to think about:

What type of failure is this?

At which layer should it be detected?When should the system check for it?How far should automatic recovery go?At what point should responsibility move
to external monitoring?

That system-level design is much more important than any individual command or timer.

For unattended 24/7 operation:

Reliability is not only about keeping the system from stopping. It is also about being able to return to a known-good state — and detecting when it cannot.

Next: Detecting an Edge PC Freeze From the Outside

At this point, the edge PC itself has:

Scheduled maintenance

State verificationSelf-recoveryControlled reboot mechanisms

But if the entire edge PC freezes, all monitoring running inside the device stops as well.

So in the next article, I will introduce a different approach:

Using a separate Linux device to monitor the edge PC from the outside.

In my actual setup, I collect signals through multiple paths and intervals:

1-second Ping

5-second Heartbeat5-second Telemetry60-second detailed Telemetry10-minute Health information1-hour detailed SMART informationrsyslognetconsole

The goal is not simply to collect as much information as possible.

Another important requirement is:

Keep the monitoring overhead low enough that the monitoring process itself does not interfere with freeze investigation.

The next article will focus on one question:

When the monitored device completely stops, what information needs to remain outside the device to help us understand what happened?

#11. Designing Edge Devices for Stable 24/7 Unattended Operation

Introduction

1. Booting Successfully Is Not Enough

2. Do Not Put Everything Into One Huge Maintenance Script

3. Separating Responsibilities with systemd Timers

4. The Maintenance Schedule

5. Why Maintenance Tasks Are Staggered

6. 00:15 — Scheduled Reboot

7. The System Must Recover Automatically After Reboot

8. Firewall and NAT Are Runtime States Too

9. 00:45 — Synchronizing Time

10. 01:00 — Periodically Checking DNS

11. 01:30 — A Dedicated Update Path for IoT Edge

12. 02:30 — Reboot Only When the OS Requires It

Get Secure Edge Ops’s stories in your inbox

13. 03:30 — Reboots Required by IoT Edge Updates Are Also Separated

14. 04:30 — Verify the NIC Last

15. Use Both Boot-Time Recovery and Scheduled Maintenance

16. Use Persistent=true Selectively

17. Record Results, Not Just Execution

18. Verify the Timers Themselves

19. Maintenance Operations Should Be Safe to Run Repeatedly

But Scheduled Maintenance Cannot Detect Everything

20. Self-Recovery and External Monitoring Have Different Roles

21. Monitoring Should Not Destabilize the System Being Monitored

22. Putting the Architecture Together

Conclusion: 24/7 Reliability Does Not Come From One Powerful Mechanism

Next: Detecting an Edge PC Freeze From the Outside

Written by Secure Edge Ops

No responses yet

More from Secure Edge Ops

#3. Keep Windows Update, Block Only Intel GPU Drivers (DenyDeviceIDs Design)

“I fixed it… and it came back.” Switching to MBDA (Microsoft Basic Display Adapter) made the system stable again. But Windows Update…

#2. Windows Stuck on a Black Screen: Freezes Before the Login Screen

(How I used boot logs + Event Viewer to zero in on the graphics driver)

#1. Windows 11 Pro Hard-Froze on an Edge PC (Ctrl+Alt+Del Didn’t Work) — Minimal Drivers Fixed It

Why edge reliability often comes from removing, not adding.

18. What Remote Monitoring Really Changes in Medical Device After-Sales Support

The real value isn’t just remote access. It’s helping support teams make better decisions, faster, with the right information.

Recommended from Medium

Open the App. Tap Nothing. You’re Already Being Tracked.

What an Oxford study of 1,297 Android apps taught me about consent, and why it matters for Nepal

Ants Solved Traffic Jams Before We Built Our First Highway

Millions of drivers still sit in gridlock every day. Tiny insects with no traffic lights, no maps and no leader figured it out long ago.

What Had to Come Has Come: The Lightweight Open-Source IDEA Is Here

How long has it been since you've done old-school programming, writing code with your own hands? As a new-age AI engineer, heavyweight…

Consciousness Unleashed

Consciousness existing outside the constraints of our 3D spatial reality is a profound concept explored by theoretical physicists…

Coding an Agent: A Harness That Rewrites Itself

A small model, a strong editor, and noise

5 AI utilities your project can’t live without

Stop burning tokens with just prompts

No responses yet

More from Secure Edge Ops

#3. Keep Windows Update, Block Only Intel GPU Drivers (DenyDeviceIDs Design)

“I fixed it… and it came back.” Switching to MBDA (Microsoft Basic Display Adapter) made the system stable again. But Windows Update…

#2. Windows Stuck on a Black Screen: Freezes Before the Login Screen

(How I used boot logs + Event Viewer to zero in on the graphics driver)

#1. Windows 11 Pro Hard-Froze on an Edge PC (Ctrl+Alt+Del Didn’t Work) — Minimal Drivers Fixed It

Why edge reliability often comes from removing, not adding.

18. What Remote Monitoring Really Changes in Medical Device After-Sales Support

The real value isn’t just remote access. It’s helping support teams make better decisions, faster, with the right information.

Recommended from Medium

Open the App. Tap Nothing. You’re Already Being Tracked.

What an Oxford study of 1,297 Android apps taught me about consent, and why it matters for Nepal

Ants Solved Traffic Jams Before We Built Our First Highway

Millions of drivers still sit in gridlock every day. Tiny insects with no traffic lights, no maps and no leader figured it out long ago.

What Had to Come Has Come: The Lightweight Open-Source IDEA Is Here

How long has it been since you've done old-school programming, writing code with your own hands? As a new-age AI engineer, heavyweight…

Consciousness Unleashed

Consciousness existing outside the constraints of our 3D spatial reality is a profound concept explored by theoretical physicists…

Coding an Agent: A Harness That Rewrites Itself

A small model, a strong editor, and noise

5 AI utilities your project can’t live without

Stop burning tokens with just prompts

16. Use `Persistent=true` Selectively