Lesson 6: Read-Only vs Read-Write Root Filesystems in Embedded Linux

Imagine this scenario — You compile a new version of IDS, connect to your board, and try to replace the existing executable:

cp ids-daemon /usr/bin/

But Linux responds:

cp: cannot create regular file
'/usr/bin/ids-daemon':
Read-only file system

Why?

You are logged in as root. You have administrative privileges. Yet Linux refuses to modify the file.

Meanwhile, IDS is still writing logs successfully.

How can Linux prevent you from modifying /usr/bin but allow your application to create log files?

That’s exactly what we’ll understand today.

What we’ll learn

1. What ro and rw actually mean.
2. How different directories can have different write permissions.
3. How read-only systems store logs and persistent data.
4. What OverlayFS is and how it works.
5. How development builds differ from production builds.
6. How Yocto creates read-only root filesystems.
7. How to safely investigate your physical Qualcomm board.

Part 1: What is a read-only root filesystem?

6.1 First, remember what a root filesystem is

In Lesson 1, we discussed the Linux filesystem hierarchy:

/
├── bin/
├── etc/
├── lib/
├── usr/
│   ├── bin/
│   └── lib/
├── var/
├── tmp/
├── dev/
├── proc/
└── sys/

The / directory is the starting point of the Linux filesystem hierarchy.

Your IDS application might be installed at:

/usr/bin/ids-daemon

Its configuration might be:

/etc/ids/config.json

And its logs might be:

/var/log/ids.log

Although these paths appear to belong to one directory tree, they do not necessarily live on the same physical partition or filesystem.

That distinction is very important.

6.2 Read-only (RO) vs read-write (RW)

Think of two different types of notebooks.

In Linux, filesystem mount options control whether writes are allowed.

For example, this is an illustrative entry from /proc/mounts:

/dev/mmcblk0p3 / ext4 ro,relatime 0 0

Now compare:

/dev/mmcblk0p3 / ext4 rw,relatime 0 0

Here rw means that the filesystem is mounted read-write.

Important: RW does not mean everyone can modify everything

Even if the filesystem is mounted rw, a normal user may not have permission to modify /usr/bin.

So two different mechanisms can prevent a write:

• Filesystem mounted read-only: even root generally cannot write through that mount.
• File permissions: the filesystem is writable, but a particular user lacks permission.

Also, read-only mounting and cryptographically verified images are different concepts. A read-only mount alone does not guarantee that nobody can change the underlying flash using privileged or offline tools.

6.3 Why do production devices use RO?

Consider a vehicle telematics unit.

It needs to run reliably for years, and users shouldn’t be able to accidentally damage core system files.

A read-only system partition helps because it reduces unintended filesystem changes, limits some forms of persistent modification, and supports predictable system-image updates.

It can also reduce routine writes to flash.

However, a read-only root filesystem is not sufficient by itself to guarantee security, integrity, or recovery from power failures. Those goals may require additional mechanisms such as verified boot, flash integrity checking, and robust update design.

Part 2: How can a read-only Linux system write logs?

Here’s the key idea:

A read-only root filesystem does not imply that every mounted filesystem is read-only.

Linux can combine several mounted filesystems into one directory hierarchy.

6.4 One Linux filesystem tree, multiple storage areas

Imagine your board has persistent flash divided into two main regions:

• rootfs — Contains the installed Linux system.
• userdata — Stores information that changes while the device runs.

The device could mount them like this:

For example, a hypothetical board could report:

/dev/mmcblk0p3  /         ext4   ro
/dev/mmcblk0p4  /data     ext4   rw
tmpfs           /tmp      tmpfs  rw
tmpfs           /run      tmpfs  rw

Although / is read-only, /data and /tmp can still accept writes.

Why? Because they are different mounted filesystems.

The kernel decides where to route a file operation based on the path and mount hierarchy.

6.5 What exactly is tmpfs?

tmpfs is a temporary filesystem whose contents are held in memory, with possible swap backing on systems that support it.

In an embedded system, /tmp and /run are often backed by tmpfs.

For example:

echo "IDS started" > /tmp/ids-test.log

On a system where /tmp is writable, this succeeds even when / is read-only.

But what happens after reboot?

The contents of tmpfs are lost.

So we should distinguish:

The distinction matters especially for logs.

Suppose IDS detects suspicious activity and writes important diagnostic information.

If the logs are stored only in tmpfs, a power loss could erase them.

For critical diagnostic data, the product may need persistent logging, remote log collection, or another suitable design.

6.6 Why /var/log can be confusing

You might see:

/var/log/ids.log

and assume it must be stored on the same flash partition as /.

But /var/log could be:

• A separate writable mount.
• A symbolic link to /data/logs.
• Backed by tmpfs.
• Part of a writable overlay.

Therefore, the directory name alone doesn’t tell you where the data is physically stored.

On your board, inspect the mount configuration rather than guessing from the path.

Part 3: OverlayFS — Making a read-only system appear writable

This is an interesting mechanism used in many embedded Linux systems.

6.7 The problem OverlayFS solves

Imagine your product uses a read-only root filesystem.

You want to keep the original system files protected, but developers still need to change configuration files or test some modifications.

How can you allow changes without rewriting the underlying rootfs?

One approach is OverlayFS.

Think of a textbook covered by a transparent sheet.

The textbook is the original, unmodified content.

You can write notes on the transparent sheet without changing the book itself.

When you look through the sheet, you see the original content plus your changes.

Get Ankush Dhanda’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That’s the basic idea behind OverlayFS.

6.8 How OverlayFS works

OverlayFS combines two directory trees:

• Lower layer: Original files, often read-only.
• Upper layer: Writable location for changes.

Linux presents a merged view to applications.

The application sees mode = B, because the upper layer takes priority over the lower layer.

But the original config.json in the lower layer remains unchanged.

When an existing lower-layer file needs modification, OverlayFS typically copies it to the upper layer and modifies that copy. This is called copy-up.

6.9 What happens when you reboot?

This depends on where the upper layer is stored.

Upper layer in RAM:Changes are temporary. They disappear after reboot, and the original files are visible again.

Upper layer in persistent flash:Changes can survive reboot, provided the filesystem is properly configured and mounted again.

Now imagine you’ve copied an updated IDS binary to an OverlayFS-backed /usr/bin.

It appears to work.

But if that overlay uses a temporary upper layer, your change disappears after reboot.

This is one reason you must understand the mount layout before trusting that a manual update is permanent.

Does OverlayFS always mean the complete rootfs is writable?

No.

OverlayFS can be mounted over a specific directory such as /etc or an application directory.

The rest of the root filesystem may remain read-only.

Also, the merged OverlayFS view may report rw while its underlying lower filesystem is still read-only.

Part 4: How Yocto creates read-only images

Now let’s connect everything back to Yocto.

6.10 Read-only is a system design choice

Remember that Yocto builds the root filesystem.

During image creation, it installs the selected packages, binaries, configurations, and services.

You can configure the Yocto image to support a read-only root filesystem.

For example, an image recipe can contain:

IMAGE_FEATURES += "read-only-rootfs"

This enables Yocto’s read-only-rootfs image feature.

Yocto also checks that required package post-installation steps can be completed during image creation rather than depending on writes to the read-only rootfs during the board’s first boot.

This is a build configuration example, not a command to run on your Qualcomm board.

Enabling the feature doesn’t automatically make every application compatible with a read-only system.

6.11 What changes when an image becomes RO?

Suppose your IDS application contains this code:

#include <fstream>
int main() {
    std::ofstream log("/var/log/ids.log");
    log << "IDS started\n";
}

In a development image, /var/log might be writable.

So the program works.

But in a new read-only production image, perhaps /var/log is not backed by writable storage.

Then the file open fails.

The application needs to detect and handle that error rather than silently assuming the log was written.

The correct solution isn’t simply to disable read-only mode.

Instead, the platform and application teams should agree on writable locations.

For example:

/usr/bin/ids-daemon       Program (RO)
/usr/lib/libids.so        Library (RO)
/etc/ids/config.json      Default config (RO)

/data/ids/                Persistent IDS data (RW)
/run/ids/                 Temporary state (RW)
/var/log/ids/             Logs, if writable storage is configured

If you want to modify IDS runtime behavior, one approach is to keep the default configuration under /etc but allow user-specific or persistent overrides under /data.

Yocto’s documentation discusses these runtime write requirements because applications designed for ordinary writable Linux distributions may not work unchanged on an RO rootfs.

6.12 Where do filesystem formats fit in?

Remember Lesson 1, where we learned about ext4, UBI and UBIFS.

Here’s how they relate to read-only and read-write behavior.

SquashFS and EROFS are specifically designed as read-only filesystem formats.

A crucial distinction:A read-only image is not the same thing as read-only flash hardware.

The physical flash can still support writes even though the Linux filesystem mounted from it is read-only.

This matters when updating a device. An approved updater or flashing tool may replace an entire RO filesystem image without modifying individual files through the mounted filesystem.

6.13 A real-world deployment example

Suppose your Qualcomm board uses this conceptual flash layout:

eMMC Flash
│
├── Boot Firmware
├── Kernel / Boot Partition
├── Rootfs A     (Read-only)
├── Rootfs B     (Read-only)
└── Userdata     (Read-write)

The board currently boots from Rootfs A.

You want to release a new version of IDS.

A possible A/B update flow is:

If the new system fails health checks, a properly designed A/B updater may roll back to the previous slot.

Not every Qualcomm board uses this design. It is a common embedded update pattern, not a guarantee about your specific device.

Notice that we didn’t need to make Rootfs A writable.

The updater installs a complete replacement into the inactive location.

This gives us an important distinction between updating the contents of a mounted filesystem and writing a replacement filesystem image into a partition.

Part 5: Hands-on investigation of your Qualcomm board

Now we’ll examine the actual Linux storage configuration.

For this exercise, we will only read information. We won’t remount partitions, erase flash, install packages, or restart services.

Step 1: Identify your root filesystem

On the Qualcomm board:

cat /proc/mounts | grep ' / '

You might see something like:

/dev/mmcblk0p3 / ext4 ro,relatime 0 0

This tells you that / is an ext4 filesystem mounted read-only.

Alternatively:

ubi0:rootfs / ubifs ro,relatime 0 0

Here Linux is using UBIFS, which would suggest a UBI-managed flash volume.

Or:

overlay / overlay rw,relatime,... 0 0

This indicates an OverlayFS root view. You would need to investigate the upper and lower layers to understand which parts are genuinely writable and persistent.

Step 2: Inspect all mounted filesystems

cat /proc/mounts

Look especially for mounts at:

/
/data
/tmp
/var
/var/log
/run

If your system provides findmnt, these commands are also useful:

findmnt -T /
findmnt -T /tmp
findmnt -T /var/log

findmnt -T helps identify the filesystem responsible for a given path, even when it is not itself a mount point.

Step 3: Identify where storage is available

df -h

And, if supported:

lsblk -f

df -h reports filesystem space, while lsblk -f can identify filesystem types on block devices.

For raw NAND systems, also inspect:

cat /proc/mtd

The command may fail if no MTD device is exposed.

Step 4: Inspect important directories

ls -ld /usr/bin
ls -ld /etc
ls -ld /tmp
ls -ld /var/log
ls -ld /data

These commands show directory permissions and existence, but they don’t prove the filesystem is writable. Combine them with the mount information.

Check whether /var/log is a symbolic link:

ls -ld /var/log

If it points somewhere else, inspect that destination’s mount as well.

Lesson 6: Final summary

The complete picture so far:

The most important takeaway is:

A Linux board can have a protected read-only operating system and still run applications that create writable logs, state files, and persistent data. What matters is the filesystem mounted at each path.