How To: Build A Read-Only Linux System

There seem to be a lot of people out there looking to run a custom application
on a Linux-based platform running on a solid-state storage device. From time to
time, we receive questions from customers looking to make their Linux platforms
read-only in order to maximize the longevity of their flash devices. I thought
I’d take the opportunity to create a blog post describing one way to do this.

There are a couple of different approaches to making a Linux system read-only.
Unfortunately, it is usually not as simple as using a conventional filesystem
mounted with the read-only option. Many programs assume that at least some
parts of the system are writable. In some cases, these programs will fail to
run correctly if this turns out not to be the case.

I’ll outline here what I think is the best approach for most applications. It is
similar to that taken by the current generation of live CD distributions.

Live CDs typically have read-only access to a root filesystem, which is often
compressed into a single file to be mounted later using a loopback device.
Knoppix broke new ground with its use of the cloop filesystem for this purpose.
More recent live distributions take this a step further by using a union
filesystem
to make the root filesystem writable. This approach is quite useful
for our purposes, as well.

Union Filesystems

Generally speaking, a union filesystem combines multiple filesystems into a
single virtual filesystem. There are two popular union fileystems that I’m
aware of: unionfs and aufs. Both have the same basic model. The following is a
dramatic simplification:

  • Filesystems are stacked vertically.
  • Read accesses are attempted on each filesystem in turn from top to bottom.
    The first filesystem that contains the file being read is used for the read
    operation.
  • Write accesses are performed similarly, but files that are written to are stored
    in the top-most writable filesystem. This usually means there is a single
    writable layer in the union. If files that exist in a read-only layer are written to,
    they are first copied to the next highest writable layer.

Obviously, there are a lot of subtleties and corner cases that I am not
presenting here. What’s important is that we can use a read-only filesystem
(which may be a compressed filesystem image or a flash device containing a more
conventional filesystem like ext3) and build a writable system on top of it.
All we need is a writable filesystem to union with the read-only layer.

The Writable Layer

What kind of writable filesystem you use depends on what you are trying to
achieve. If you don’t need any persistence between boots, it is pretty easy to
use tmpfs. Writes to the system will be preserved in RAM while the system is
up, but will disappear when the system is shutdown or rebooted.

If you want persistence over the whole system directory structure, you’ll need
to use a persistent writable layer. This is likely a conventional filesystem on
some other media (a second disk, perhaps). This is probably most useful for
live systems or thin clients where using a read-only base is not done so much
for longevity as it is to minimize local storage requirements.

In many cases, when you do need persistence, you only need it for specific
files. For instance, if you have a kiosk that stores user input in a local
database, the database must persist on disk, but you probably don’t want to
persist temporary files or other transient runtime data. The best approach for
dealing with this common use case is to have a tmpfs read/write layer and then
mount some writable media on an arbitrary mount point like /var/local/data (for
example).

Implementation

Implementing a read-only system requires hooking into the boot process. How
this is done varies from distribution to distribution, and can probably be done
in a variety of ways on a single distribution. In this article, I’ll
demonstrate an approach that works with Ubuntu 8.04.

By default, Ubuntu 8.04 uses an initramfs. This is the best place to make our
modification, as we can make sure that the union filesystem is mounted early on
in the boot process.

initramfs-tools

Ubuntu has an extensible system for building the initramfs called
“initramfs-tools”. We can use this to plug some scripts into the initramfs.
There are a few different ways that initramfs-tools can be extended: “hooks” and
“scripts.”

Hooks are run when the initramfs is being built, and are useful for adding
kernel modules or executables to the initramfs image. Hooks that are
distributed with packages are usually installed in
/usr/share/initramfs-tools/hooks, and they make use of the functions defined in
/usr/share/initramfs-tools/hook-functions. Local hooks should be placed in
/etc/initramfs-tools/hooks.

Scripts are run within the initramfs environment at boot time. These can be
used to modify the early boot process. As with hooks, scripts that are
distributed with packages are usually installed into
/usr/share/initramfs-tools/scripts. Local scripts should go into
/etc/initramfs-tools/scripts.

initramfs generation is controlled by the configuration files found in
/etc/initramfs-tools. /etc/initramfs-tools/initramfs.conf is the primary
configuration file, but files can also be placed in
/etc/initramfs-tools/conf.d. The primary boot method can be configured in
initramfs.conf by changing the value of variable “BOOT.” By default, it is
“local,” a boot method that mounts the root filesystem on local media like a
hard disk.

For each boot method, there is a script with that name that controls how that
boot method works. For instance, there is a script called “local” that defines
how a local boot is performed. Many such scripts also provide hooks for other
shell scripts to be executed at certain points during the boot process. For
instance, any scripts placed in /usr/share/initramfs-tools/local-premount will
be executed by the “local” script just prior to mounting the root filesystem.
The init script itself (which acts as process #1 up until the point where the
real init daemon is launched, after mounting the root filesystem) provides
similar hooks. See the contents of /usr/share/initramfs-tools/scripts to get an
idea of what other hooks are available.

Finally, both hooks and scripts must be written such that, if they are run with
a single argument “prereqs,” they print a space-separated list of the names of
other scripts or hooks that should be run before running this particular script
or hook. This provides a simple system of dependencies between hooks and
scripts. I find that I very rarely make use of this feature, but it is
available should your application require it.

Hooks and Scripts

We’ll implement our read-only system by introducing one hook and one script.
Our script will actually be an init-bottom script, run after the real root
device is already mounted. Our goal will be to take the already-mounted root
filesystem and shuffle it around as the base for an aufs union with a tmpfs
writable layer. This allows us to continue to use the standard Ubuntu
configuration mechanisms for specifying the device that contains the real root
filesystem.

We need a hook to tell initramfs-tools that we need a few kernel modules (aufs
and tmpfs, both of which are included with Ubuntu 8.04) and an executable (chmod). We’ll see why we need chmod shortly. Our
hook is quite simple (as most of them are). We’ll call this hooks/ro_root:

#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions
manual_add_modules aufs
manual_add_modules tmpfs
copy_exec /bin/chmod /bin

The script does the real work of making sure the filesystems are all mounted in
the right places. At this point in the boot process, the real root device has
been mounted on $rootmnt, and /sbin/init on that mount point is about to be
executed. At this point, we’ll be looking to move the root device mount to a
different mount point, and build our union mount in its place.

Here’s how we’ll do this:

  1. Move $rootmnt to /${rootmnt}.ro (this is the read-only layer).
  2. Mount our writable layer as tmpfs on /${rootmnt}.rw.
  3. Mount the union on ${rootmnt}.

Additionally, we may want to have access to the read-only and read/write layers
independently from the union. In order to maintain access to these mounts,
we’ll have to bind them to a new mount point under ${rootmnt}. We’ll do this
with “mount –bind”.

The union is still able to access the original read-only and read/write mounts
even after the root is rotated and init is launched, causing those mount points
to fall outside of the new root filesystem. I assume that aufs opens these
directories at mount time and the filesystems continue to be accessible as long
as processes have open file handles. The kernel seems to be pretty smart about
dealing with these kinds of interesting situations.

Getting back to things, here is the init-bottom script we’ll be using
(scripts/init-bottom/ro_root):

#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

ro_mount_point="${rootmnt%/}.ro"
rw_mount_point="${rootmnt%/}.rw"

# Create mount points for the read-only and read/write layers:
mkdir "${ro_mount_point}" "${rw_mount_point}"

# Move the already-mounted root filesystem to the ro mount point:
mount --move "${rootmnt}" "${ro_mount_point}"

# Mount the read/write filesystem:
mount -t tmpfs root.rw "${rw_mount_point}"

# Mount the union:
mount -t aufs -o "dirs=${rw_mount_point}=rw:${ro_mount_point}=ro" root.union "${rootmnt}"

# Correct the permissions of /:
chmod 755 "${rootmnt}"

# Make sure the individual ro and rw mounts are accessible from within the root
# once the union is assumed as /.  This makes it possible to access the
# component filesystems individually.
mkdir "${rootmnt}/ro" "${rootmnt}/rw"
mount --bind "${ro_mount_point}" "${rootmnt}/ro"
mount --bind "${rw_mount_point}" "${rootmnt}/rw"

Rebuilding The initramfs

The hook and init-bottom script that we wrote above can be installed in the
following locations, respectively:

  • /etc/initramfs-tools/hooks/ro_root.
  • /etc/initramfs-tools/scripts/init-bottom/ro_root.

They should both have the execute permission bit set.

After copying the files into place, regenerate your initramfs with
update-initramfs:

update-initramfs -u

The -u switch tells update-initramfs to update the initramfs for the most recent
kernel on the system. I assume that that is the kernel that you are running.
For most embedded or other single-purpose machines, there is typically only one
kernel installed.

Booting

The system should appear to boot as it would without the changes we made.
However, once it is finished booting, you can confirm that:

  • /ro contains the read-only base filesystem.
  • /rw contains the read/write layer, and will usually have some new files
    there immediately following boot (/var/run, etc.).
  • If you create a file and then reboot, the file will be gone.

Of course, a system like this has a few caveats:

  • The contents of /etc/mtab are likely not correct, so the output of the mount
    command is probably missing some information. There are steps we can take to
    correct /etc/mtab, but I won’t cover those in detail here.
  • No runtime state is preserved. Don’t forget that and save a file, expecting
    it to be around after a reboot!
  • Subtle semantic differences between aufs, tmpfs, and traditional filesystems
    may cause problems with some applications. Most applications won’t notice,
    but those that leverage more advanced filesystem features or rely on
    filesystem implementation details might run into errors or, worse, fail
    subtley. I believe most of these kinds of issues are now a thing of the
    past, but if you find yourself troubleshooting mysterious failures, keep it
    in mind.

This kind of system customization really demonstrates the power and flexibility
of the initramfs-tools configuration infrastructure. This architectural style
is common in Debian and Ubuntu, making these distributions ideal choices for
embedded and applied computing projects.

I hope this has been helpful. As always, I look forward to comments and
questions.

Improvements

[Section added 2009-02-23, updated 2009-04-28.]

The following updated script incorporates some improvements that helped with some problems that commenters ran into:

  • Boot with normally-mounted read/write root filesystem when the user requests single user mode (a.k.a. recovery mode).
  • Prevent /etc/init.d/checkroot.sh from running when booting into the read-only system.
  • Use mount –move instead of mount –bind when moving the ro and rw mount points into the new root.
#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

# Boot normally when the user selects single user mode.
if grep single /proc/cmdline >/dev/null; then
  exit 0
fi

ro_mount_point="${rootmnt%/}.ro"
rw_mount_point="${rootmnt%/}.rw"

# Create mount points for the read-only and read/write layers:
mkdir "${ro_mount_point}" "${rw_mount_point}"

# Move the already-mounted root filesystem to the ro mount point:
mount --move "${rootmnt}" "${ro_mount_point}"

# Mount the read/write filesystem:
mount -t tmpfs root.rw "${rw_mount_point}"

# Mount the union:
mount -t aufs -o "dirs=${rw_mount_point}=rw:${ro_mount_point}=ro" root.union "${rootmnt}"

# Correct the permissions of /:
chmod 755 "${rootmnt}"

# Make sure the individual ro and rw mounts are accessible from within the root
# once the union is assumed as /.  This makes it possible to access the
# component filesystems individually.
mkdir "${rootmnt}/ro" "${rootmnt}/rw"
mount --move "${ro_mount_point}" "${rootmnt}/ro"
mount --move "${rw_mount_point}" "${rootmnt}/rw"

# Make sure checkroot.sh doesn't run.  It might fail or erroneously remount /.
rm -f "${rootmnt}/etc/rcS.d"/S[0-9][0-9]checkroot.sh
About Forest Bond
Forest had been the lead software developer at Logic Supply since April 2005 before starting his own software company, RapidRollout in 2009. He occasionally peeks back in with us from time to time, but dedicates his energy to developing open source software platforms. We maintain his posts here, but he no longer writes for our blog. To contact him, you can visit RapidRollout.
This entry was posted in Linux, Technology. Bookmark the permalink.

59 Responses to How To: Build A Read-Only Linux System

  1. Matt Anderson says:

    This worked great for me on Ubuntu 10.04.01 LTS. Thanks for the great write-up.

    The only gotcha I ran into was the single user mode trick did not work for me. My system hung at a blank screen and was non-responsive on the network. I didn’t test this before making the RO root changes, so I’m not sure if this was an existing problem with the computer or not.

    Instead what I’ve been doing is once booted:
    # mount -o remount,ro /ro
    # cd /ro
    # chroot .

    That lets me edit my system in a persistent way. Once I’m done I simply:

    # exit (this gets me out of the chroot)
    # cd /
    # mount -o remount,ro /ro

    Usually I’ll throw a `sync` in there for good measure :)

  2. Francesco says:

    everything seems work, I’ve also added an item to grub, to start linux in normal mode, but I’ve encountered a bug: “Bug#607879: System hangs up with mmap.c:873″. Do you have any suggestion?
    Thanks again for your work

  3. Scott says:

    As an FYI, this guide is really superb. I’m using it with a Ubuntu 10.04 LTS server, booted off of a compact flash, in an embedded server application.

    One gajillion thanks!

  4. sparsile says:

    TK (or anyone else!), if you’re still listening, I would very much appreciate a tutorial on building a RO linux system for CentOS 5.

  5. Johnny says:

    This works except for one found one bug in Ubuntu 10.04:

    dhclient breaks.
    this is kind of a weird error. It reports that the libraries aren’t there, but they are.

    #dhclient: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
    # which dhclient
    /sbin/dhclient
    # ldd /sbin/dhclient
    linux-vdso.so.1 => (0x00007fffa8fff000)
    libc.so.6 => /lib/libc.so.6 (0x00007f4a3b820000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4a3be1f000)
    # ls -al /lib | grep libc.so.6
    lrwxrwxrwx 1 root root 14 2011-10-10 20:05 libc.so.6 -> libc-2.11.1.so
    root@Archives:/lib# ls -al /lib | grep libc-2.11.1.so
    -rwxr-xr-x 1 root root 1572232 2011-01-21 14:23 libc-2.11.1.so
    lrwxrwxrwx 1 root root 14 2011-10-10 20:05 libc.so.6 -> libc-2.11.1.so

  6. FBP says:

    I have a Seagate Dockstar (arm5 processor, 128 MB RAM, boots Debian squeeze from a usb flash drive). I have done this “Read-Only Linux System” procedure on a more “normal” computer (Intel Atom, Debian squeeze), and it works fine. But it fails on the Dockstar. It boots, it just isn’t read-only. I get the following in dmesg:

    [ 12.960395] aufs: module is from the staging directory, the quality is unknown, you have been warned.
    [ 12.993986] aufs 2-standalone.tree-32-20100125
    [ 13.003956] aufs test_add:218:mount[124]: unsupported filesystem, /root.ro (rootfs)

    I end up with no /rw or /ro mount points. Ideas?

  7. Pingback: Setting up a read-only Debian system?

  8. Forest Bond Forest Bond says:

    Hi everyone,

    I’m no longer at Logic Supply, but if people still have questions I can do my best to get them answered. I’ve created a new thread for this purpose on the RapidRollout forums:

    Configuring Ubuntu for read-only storage

    Go ahead and post your questions there and I’ll see what I can do to help!

    -Forest

  9. W3 Network Solutions says:

    Hey There. I found your blog the usage of msn. This is a really neatly written article. I will be sure to bookmark it and come back to read more of your useful information. Thanks for the post. I’ll definitely return.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>