How To: Build A Read-Only Linux System - I/O Hub

How To: Build A Read-Only Linux System

How To: Build A Read-Only Linux System

How To: Build A Read-Only Linux System

There seem to be a lot of people out there looking to run a custom application
on a Linux-based platform running on a solid-state storage device. From time to
time, we receive questions from customers looking to make their Linux platforms
read-only in order to maximize the longevity of their flash devices. I thought
I’d take the opportunity to create a blog post describing one way to do this.

There are a couple of different approaches to making a Linux system read-only.
Unfortunately, it is usually not as simple as using a conventional filesystem
mounted with the read-only option. Many programs assume that at least some
parts of the system are writable. In some cases, these programs will fail to
run correctly if this turns out not to be the case.

I’ll outline here what I think is the best approach for most applications. It is
similar to that taken by the current generation of live CD distributions.

Live CDs typically have read-only access to a root filesystem, which is often
compressed into a single file to be mounted later using a loopback device.
Knoppix broke new ground with its use of the cloop filesystem for this purpose.
More recent live distributions take this a step further by using a union
filesystem
to make the root filesystem writable. This approach is quite useful
for our purposes, as well.

Union Filesystems

Generally speaking, a union filesystem combines multiple filesystems into a
single virtual filesystem. There are two popular union fileystems that I’m
aware of: unionfs and aufs. Both have the same basic model. The following is a
dramatic simplification:

  • Filesystems are stacked vertically.
  • Read accesses are attempted on each filesystem in turn from top to bottom.
    The first filesystem that contains the file being read is used for the read
    operation.
  • Write accesses are performed similarly, but files that are written to are stored
    in the top-most writable filesystem. This usually means there is a single
    writable layer in the union. If files that exist in a read-only layer are written to,
    they are first copied to the next highest writable layer.

Obviously, there are a lot of subtleties and corner cases that I am not
presenting here. What’s important is that we can use a read-only filesystem
(which may be a compressed filesystem image or a flash device containing a more
conventional filesystem like ext3) and build a writable system on top of it.
All we need is a writable filesystem to union with the read-only layer.

The Writable Layer

What kind of writable filesystem you use depends on what you are trying to
achieve. If you don’t need any persistence between boots, it is pretty easy to
use tmpfs. Writes to the system will be preserved in RAM while the system is
up, but will disappear when the system is shutdown or rebooted.

If you want persistence over the whole system directory structure, you’ll need
to use a persistent writable layer. This is likely a conventional filesystem on
some other media (a second disk, perhaps). This is probably most useful for
live systems or thin clients where using a read-only base is not done so much
for longevity as it is to minimize local storage requirements.

In many cases, when you do need persistence, you only need it for specific
files. For instance, if you have a kiosk that stores user input in a local
database, the database must persist on disk, but you probably don’t want to
persist temporary files or other transient runtime data. The best approach for
dealing with this common use case is to have a tmpfs read/write layer and then
mount some writable media on an arbitrary mount point like /var/local/data (for
example).

Implementation

Implementing a read-only system requires hooking into the boot process. How
this is done varies from distribution to distribution, and can probably be done
in a variety of ways on a single distribution. In this article, I’ll
demonstrate an approach that works with Ubuntu 8.04.

By default, Ubuntu 8.04 uses an initramfs. This is the best place to make our
modification, as we can make sure that the union filesystem is mounted early on
in the boot process.

initramfs-tools

Ubuntu has an extensible system for building the initramfs called
“initramfs-tools”. We can use this to plug some scripts into the initramfs.
There are a few different ways that initramfs-tools can be extended: “hooks” and
“scripts.”

Hooks are run when the initramfs is being built, and are useful for adding
kernel modules or executables to the initramfs image. Hooks that are
distributed with packages are usually installed in
/usr/share/initramfs-tools/hooks, and they make use of the functions defined in
/usr/share/initramfs-tools/hook-functions. Local hooks should be placed in
/etc/initramfs-tools/hooks.

Scripts are run within the initramfs environment at boot time. These can be
used to modify the early boot process. As with hooks, scripts that are
distributed with packages are usually installed into
/usr/share/initramfs-tools/scripts. Local scripts should go into
/etc/initramfs-tools/scripts.

initramfs generation is controlled by the configuration files found in
/etc/initramfs-tools. /etc/initramfs-tools/initramfs.conf is the primary
configuration file, but files can also be placed in
/etc/initramfs-tools/conf.d. The primary boot method can be configured in
initramfs.conf by changing the value of variable “BOOT.” By default, it is
“local,” a boot method that mounts the root filesystem on local media like a
hard disk.

For each boot method, there is a script with that name that controls how that
boot method works. For instance, there is a script called “local” that defines
how a local boot is performed. Many such scripts also provide hooks for other
shell scripts to be executed at certain points during the boot process. For
instance, any scripts placed in /usr/share/initramfs-tools/local-premount will
be executed by the “local” script just prior to mounting the root filesystem.
The init script itself (which acts as process #1 up until the point where the
real init daemon is launched, after mounting the root filesystem) provides
similar hooks. See the contents of /usr/share/initramfs-tools/scripts to get an
idea of what other hooks are available.

Finally, both hooks and scripts must be written such that, if they are run with
a single argument “prereqs,” they print a space-separated list of the names of
other scripts or hooks that should be run before running this particular script
or hook. This provides a simple system of dependencies between hooks and
scripts. I find that I very rarely make use of this feature, but it is
available should your application require it.

Hooks and Scripts

We’ll implement our read-only system by introducing one hook and one script.
Our script will actually be an init-bottom script, run after the real root
device is already mounted. Our goal will be to take the already-mounted root
filesystem and shuffle it around as the base for an aufs union with a tmpfs
writable layer. This allows us to continue to use the standard Ubuntu
configuration mechanisms for specifying the device that contains the real root
filesystem.

We need a hook to tell initramfs-tools that we need a few kernel modules (aufs
and tmpfs, both of which are included with Ubuntu 8.04) and an executable (chmod). We’ll see why we need chmod shortly. Our
hook is quite simple (as most of them are). We’ll call this hooks/ro_root:

#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions
manual_add_modules aufs
manual_add_modules tmpfs
copy_exec /bin/chmod /bin

The script does the real work of making sure the filesystems are all mounted in
the right places. At this point in the boot process, the real root device has
been mounted on $rootmnt, and /sbin/init on that mount point is about to be
executed. At this point, we’ll be looking to move the root device mount to a
different mount point, and build our union mount in its place.

Here’s how we’ll do this:

  1. Move $rootmnt to /${rootmnt}.ro (this is the read-only layer).
  2. Mount our writable layer as tmpfs on /${rootmnt}.rw.
  3. Mount the union on ${rootmnt}.

Additionally, we may want to have access to the read-only and read/write layers
independently from the union. In order to maintain access to these mounts,
we’ll have to bind them to a new mount point under ${rootmnt}. We’ll do this
with “mount –bind”.

The union is still able to access the original read-only and read/write mounts
even after the root is rotated and init is launched, causing those mount points
to fall outside of the new root filesystem. I assume that aufs opens these
directories at mount time and the filesystems continue to be accessible as long
as processes have open file handles. The kernel seems to be pretty smart about
dealing with these kinds of interesting situations.

Getting back to things, here is the init-bottom script we’ll be using
(scripts/init-bottom/ro_root):

#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

ro_mount_point="${rootmnt%/}.ro"
rw_mount_point="${rootmnt%/}.rw"

# Create mount points for the read-only and read/write layers:
mkdir "${ro_mount_point}" "${rw_mount_point}"

# Move the already-mounted root filesystem to the ro mount point:
mount --move "${rootmnt}" "${ro_mount_point}"

# Mount the read/write filesystem:
mount -t tmpfs root.rw "${rw_mount_point}"

# Mount the union:
mount -t aufs -o "dirs=${rw_mount_point}=rw:${ro_mount_point}=ro" root.union "${rootmnt}"

# Correct the permissions of /:
chmod 755 "${rootmnt}"

# Make sure the individual ro and rw mounts are accessible from within the root
# once the union is assumed as /.  This makes it possible to access the
# component filesystems individually.
mkdir "${rootmnt}/ro" "${rootmnt}/rw"
mount --bind "${ro_mount_point}" "${rootmnt}/ro"
mount --bind "${rw_mount_point}" "${rootmnt}/rw"

Rebuilding The initramfs

The hook and init-bottom script that we wrote above can be installed in the
following locations, respectively:

  • /etc/initramfs-tools/hooks/ro_root.
  • /etc/initramfs-tools/scripts/init-bottom/ro_root.

They should both have the execute permission bit set.

After copying the files into place, regenerate your initramfs with
update-initramfs:

update-initramfs -u

The -u switch tells update-initramfs to update the initramfs for the most recent
kernel on the system. I assume that that is the kernel that you are running.
For most embedded or other single-purpose machines, there is typically only one
kernel installed.

Booting

The system should appear to boot as it would without the changes we made.
However, once it is finished booting, you can confirm that:

  • /ro contains the read-only base filesystem.
  • /rw contains the read/write layer, and will usually have some new files
    there immediately following boot (/var/run, etc.).
  • If you create a file and then reboot, the file will be gone.

Of course, a system like this has a few caveats:

  • The contents of /etc/mtab are likely not correct, so the output of the mount
    command is probably missing some information. There are steps we can take to
    correct /etc/mtab, but I won’t cover those in detail here.
  • No runtime state is preserved. Don’t forget that and save a file, expecting
    it to be around after a reboot!
  • Subtle semantic differences between aufs, tmpfs, and traditional filesystems
    may cause problems with some applications. Most applications won’t notice,
    but those that leverage more advanced filesystem features or rely on
    filesystem implementation details might run into errors or, worse, fail
    subtley. I believe most of these kinds of issues are now a thing of the
    past, but if you find yourself troubleshooting mysterious failures, keep it
    in mind.

This kind of system customization really demonstrates the power and flexibility
of the initramfs-tools configuration infrastructure. This architectural style
is common in Debian and Ubuntu, making these distributions ideal choices for
embedded and applied computing projects.

I hope this has been helpful. As always, I look forward to comments and
questions.

Improvements

[Section added 2009-02-23, updated 2009-04-28.]

The following updated script incorporates some improvements that helped with some problems that commenters ran into:

  • Boot with normally-mounted read/write root filesystem when the user requests single user mode (a.k.a. recovery mode).
  • Prevent /etc/init.d/checkroot.sh from running when booting into the read-only system.
  • Use mount –move instead of mount –bind when moving the ro and rw mount points into the new root.
#!/bin/sh

PREREQ=''

prereqs() {
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

# Boot normally when the user selects single user mode.
if grep single /proc/cmdline >/dev/null; then
  exit 0
fi

ro_mount_point="${rootmnt%/}.ro"
rw_mount_point="${rootmnt%/}.rw"

# Create mount points for the read-only and read/write layers:
mkdir "${ro_mount_point}" "${rw_mount_point}"

# Move the already-mounted root filesystem to the ro mount point:
mount --move "${rootmnt}" "${ro_mount_point}"

# Mount the read/write filesystem:
mount -t tmpfs root.rw "${rw_mount_point}"

# Mount the union:
mount -t aufs -o "dirs=${rw_mount_point}=rw:${ro_mount_point}=ro" root.union "${rootmnt}"

# Correct the permissions of /:
chmod 755 "${rootmnt}"

# Make sure the individual ro and rw mounts are accessible from within the root
# once the union is assumed as /.  This makes it possible to access the
# component filesystems individually.
mkdir "${rootmnt}/ro" "${rootmnt}/rw"
mount --move "${ro_mount_point}" "${rootmnt}/ro"
mount --move "${rw_mount_point}" "${rootmnt}/rw"

# Make sure checkroot.sh doesn't run.  It might fail or erroneously remount /.
rm -f "${rootmnt}/etc/rcS.d"/S[0-9][0-9]checkroot.sh

Comments (62)

  1. Julien Rhodes
    January 28, 2009

    Hey, nice post! I was recently reading https://help.ubuntu.com/community/aufsRootFileSystemOnUsbFlash which discusses a very similar procedure.

    I just put together a neat setup which mounts a readonly nfs mount but is a fully functional and locally writable – but i had to change one line in your init-bottom script. The mount line for me reads:
    mount -t aufs -o “br:${rw_mount_point}=rw:${ro_mount_point}=ro” aufs /root

  2. Philip
    February 20, 2009

    Clear, elegant, attractive idea. This could be nice on my netbook, but I’d feel more confident about trying it if I knew three more things. They’re all related.
    1. Precisely which boot files will this procedure alter?
    2. How can I save and restore my working boot setup in case something goes wrong while trying to build this one?
    3. Some day I may want to permanently upgrade a package on my read-only system using standard Debian/Ubuntu tools. What’s the easy way to do this? (Can I make a boot-time selection between a writable-root installation and the setup described here? Answers to questions 1-2 should make this easy.)

    Thanks for the nice writeup.

  3. Forest Bond
    Forest
    February 20, 2009

    Hi Philip,

    The added files affect the contents of generated initramfs images. These sit in /boot, usually.

    update-initramfs makes a backup before writing a new image, but you’ll lose your known-good image if you run update-initramfs twice. For this reason, you might want to make a backup like this:

    sudo cp -a /boot/initrd.img-$(uname -r) /boot/initrd-img-$(uname -r).old
    

    update-initramfs uses a .bak extension, so using .old ensures that update-initramfs won’t overwrite our backup.

    As long as you have GRUB set up to provide a boot menu with editing capabilities, you can use that to boot with your backup whenever you’d like. Just modify the initrd field and add “.old” to the end of the existing entry.

    You probably want to confirm that your GRUB setup allows editing at boot time before beginning the initramfs customization process. Generally speaking, you’ll want to make sure that /boot/grub/menu.lst has a timeout value greater than 0 (or no timeout at all).

    Now, if you want to occasionally be able to make permanent changes to the underlying filesystem, I’d suggest modifying the script such that the system boots normally (with no union) when you select recovery mode from the GRUB menu. Just modify the script by putting this after the case statement:

    if grep single /proc/cmdline >/dev/null; then
      exit 0
    fi
    

    When entering single user mode (recovery mode), the root filesystem will be left mounted read/write. You can log in as root and make permanent changes to the system.

    Hope this helps.

    -Forest

  4. Philip
    February 22, 2009

    Very informative. Thank you.

  5. Philip
    February 22, 2009

    I tried this, and it almost worked. I’m glad I made a backup first, though. I’m using Ubuntu 8.10 on an Acer Aspire One netbook. The kernel is 2.6.27-12-generic. With the suggested scripts installed, the system works as expected when the device boots on battery power and fails to boot when the AC adapter is plugged in. Interesting! The boot problems usually start like this:

    * Checking root file system…
    fsck 1.41.3 (12-Oct-2008)
    fsck.ext2: No such device or address while trying to open /lib/init/rw/rootdev
    Possibly non-existent or swap device?
    fsck died with exit status 8

    When the laptop boots on battery power, the init process chooses to skip the fsck step. Booting then proceeds normally, and everything I have tried seems to work. (I have not tested it extensively.)

    Comments are welcome. I’m reluctant to turn this comment service into a support thread, though. Is there a more appropriate place to discuss these issues? (Or should I just go away?) Tks.

  6. V. Schneider
    February 23, 2009

    Wow, what a lot of work. A simple USB flash drive system with a conventional linux file system on it really does the job. Even if it burns out once in six months, and I think the burnout interval is much longer than that, using it like a hard drive is so simple it makes no sense to go to all that trouble.

    I am writing this from firefox 3.06 running on a 2GB usb flash drive, conventional linux file system.

    (And, the flash drive has a “dual boot” 32-64 bit feature, depending on whether it is running on my 32-bit laptop (external power supply) or my 945GC MB underpowered intel dual processor.

  7. V. Schneider
    February 23, 2009

    Here is the culprit Linux system that will turn your usb flash into a hard drive file system.

    http://www.tux.org/pub/people/kent-robotti/looplinux/rip/

    You download the “bootable iso” image, around 90 megabytes, burn it to a CD, boot it up as a live system, _mount_ the CD, go to the /boot directory on the CD, and run the mkextlin.sh script for, say, /dev/sdb, which is your flash drive.

    It is a “live” system, residing in memory when in CD mode, but mkextlin.sh can even install onto a conventional hard drive, /dev/sda or /dev/hda, as long as you don’t want any other system on that drive.

    The system can be extended by downloading compatible packages from other systems.

  8. Forest Bond
    Forest
    February 23, 2009

    Philip:

    Sorry to hear you had problems. I tested on Ubuntu 8.04. Looks like Ubuntu 8.10 has slight differences with how it identifies the root device for the purpose of running fsck. I’ll have a look and let you know what I find.

    V. Schneider:

    I don’t think that copying two files and running update-initramfs is really that much work.

    Regardless, a read-only system has useful qualities beyond reduced writes to disk. For instance, there is a security benefit associated with prevention of permanent changes to the system.

    Further, using a USB flash drive might suit your particular situation well, but it would likely be inconvenient with a mobile device and insecure with a publicly-deployed system.

  9. Forest Bond
    Forest
    February 23, 2009

    Philip:

    I’ve looked into this a little more closely.

    I didn’t run into any problems because my /etc/fstab was empty (the system I’m testing with is on a disk image and was not created using the standard installer).

    /etc/init.d/checkroot.sh parses /etc/fstab to find the root device and then compares to see if it matches with the device that appears to be mounted at /. This causes problems for us, of course. If / is not listed in the fstab, checkroot.sh doesn’t bother checking anything. However, it does remount /, which seems to work fine despite the fact that / is a unionfs mount.

    The easiest way to deal with this is probably to add the following to the end of the script:

    touch "${rootmnt}/fastboot"
    

    checkroot.sh will not check the root filesystem if /fastboot exists.

    Alternatively, you could add this to the script:

    rm -f "${rootmnt}/etc/rcS.d"/S[0-9][0-9]checkroot.sh
    

    This would prevent the checkroot.sh script from running at all.

    I’ll update the post to include this modification, as well as the recovery mode modification mentioned above.

  10. Forest Bond
    Forest
    February 23, 2009

    Note: I corrected a few bugs in the script:

    • The script called chmod with / as argument instead of “${rootmnt}”.
    • The script called rm with /etc/rcS.d/S[0-9][0-9]checkroot.sh as argument instead of “${rootmnt}/etc/rcS.d”/S[0-9][0-9]checkroot.sh.
  11. Will Godfrey
    February 23, 2009

    Just wondered if this would work on a vanilla debian lenny install.

    Would any other changes be needed?

  12. Forest Bond
    Forest
    February 24, 2009

    Will:

    I can’t think of any reason it wouldn’t work. Both Debian and Ubuntu use initramfs-tools to manage the initramfs image by default, as far as I am aware. I haven’t tested it, though.

    Be sure to make a backup, and let us know how things go.

  13. Luigi
    March 9, 2009

    I tried this guide but it don’t work with ubuntu 8.10, can you help me to serolv my problem?

  14. Forest Bond
    Forest
    March 9, 2009

    Hi Luigi,

    What problems are you having?

    Thanks,
    Forest

  15. Dub Dublin
    March 10, 2009

    Two words: Puppy Linux. If you want all this goodness (which can be pretty complex) already sorted out for you in a lightweight distro that will boot and behave correctly (from CD, CF/SD/etc. flashcards, HDD, or USB flashdrives), try Puppy. Booting RO is difficult, but booting from flash correctly is even harder, and Barry and the other Puppy devs have done a great job of sorting it out: http://puppylinux.com

    This is the only Linux distro I know of that actually treats flash in a way that’s suitable for long-term embedded use. If you’ve got it configured correctly (which can happen nearly automagically in some cases) writes get gathered and written only periodically to avoid using up your flash memory’s precious write cycles (unless you’re smart enough to use an Intel SSD, in which case their brilliant embedded controller makes such gymnastics unnecessary.)

  16. Luigi
    March 11, 2009

    when try to:
    mount -t aufs -o “dirs=${rw_mount_point}=rw:${ro_mount_point}=ro” root.union “${rootmnt}”

    return the error:
    mount: Mounting root.union on /root failed: No such device

  17. Luigi
    March 12, 2009

    resolved
    I recompiled the kernel and there was no support for aufs
    with the generic kernel works fine

  18. Steve
    March 14, 2009

    Hi,

    Thanks for the great article. However, I applied the changes and my microwave oven stopped working.

    Any suggestions?? I can’t cook anything now.

    Thanks,
    Steve

  19. Forest Bond
    Forest
    March 17, 2009

    @Steve

    Sorry to hear about your troubles. Please be advised, this change should not be applied to 64-bit microwave ovens. It will only work on 32-bit models.

    You should unplug your microwave and leave it unplugged for at least 10 minutes. That should resolve the issue. Contact the manufacturer if you require further assistance.

    -Forest

  20. Craig Rae
    March 20, 2009

    Handy info, that.

    Vaguely related question time – has anyone tried out the MSI Atom boards that became available recently (MS-9830 and/or MS-9832) and tried chucking a Mini PCI-E flash disk at ’em (such as Crucial’s N100). I say Crucial as I’d trust them long before some of the other brands I see out there, and they do a short-length card for various Netbooks that looks like it’ll fit.

    I’m itching to give this a go, assuming I can ever lay my hands on one…32GB (or 64GB) of space for a very reasonable price, and presumably they do pretty decent wear levelling if the Netbook crowd run with these mounted read/write all the time.

    On a not-really-related note, has anyone else managed to get all of the RS232 ports going on either of the MSI Atoms under Linux? I have ttyS0 and 1 running with no problems, but I can only get 2 and 3 going under IRQ 0 (i.e. software polled) and 4 and 5 are a wash right now.

  21. April 28, 2009

    Thanks for your work on the script. I’m currently using it on an ALIX embedded computer as a router/firewall. I made one small change.

    I changed:

    mount –bind “${ro_mount_point}” “${rootmnt}/ro”
    mount –bind “${rw_mount_point}” “${rootmnt}/rw”

    to

    mount –move “${ro_mount_point}” “${rootmnt}/ro”
    mount –move “${rw_mount_point}” “${rootmnt}/rw”

    I found this prevents having extra entries in mtab and prevents errors from showing up in df.

  22. Forest Bond
    Forest
    April 28, 2009

    Hi Mark,

    That’s definitely the right way to do it. I’m not sure why I didn’t do it that way in the first place.

    I’ll include your change here in the improved version.

    Thanks for posting.

    -Forest

  23. Demian
    May 4, 2009

    Another distribution that is designed from the ground up to run from flash and live mostly in ram is Voyage Linux, http://linux.voyage.hk/ It can be extended easily with applications, has a very small footprint and was designed for appliances more than desktop computing. It has a simple way to shift to writable, add stuff and switch back to read only.

  24. Mathias
    May 7, 2009

    That is just great! I applied hook and script to a debian lenny system, and after only 15 minutes work, everything was running as expected. Never thought this kind of task could ever be that simple.

    Thanks a lot!!

  25. Chris
    May 12, 2009

    Hi Forest,

    I’d like to get this solution working since I’m running Ubuntu off of a CF card. Is there any reason why it wouldn’t work with Ubuntu 9.04 (UNR)? I tried it out, but when I get to the login screen, the keyboard doesn’t work.

  26. Chris
    May 12, 2009

    Me again. I almost got it working by switching from aufs to unionfs, but now for some reason, I can’t connect to my wireless network. Any ideas?

  27. Chris
    May 12, 2009

    Sorry for spamming, feel free to delete my comments. I’ve fixed my problem. Here’s what fixed it: http://ubuntuforums.org/showthread.php?t=1134892#postcount7138705

    I’m not really sure why using your script would cause that problem to emerge.

  28. TK
    May 16, 2009

    Anyone want to see the same thing for Fedora or CentOS 5.x? I’ve done several read-only systems from scratch with those OSs. I could probably also stand to have someone improve my method, I know I’ve got some quirks in it.

  29. Jack
    May 18, 2009

    This tutorial is excellent … very nice explanation of what we’re doing and why.

    A couple of things I found:

    1. I didn’t need a hook at all. I added unionfs and squashfs to /etc/initramfs-tools/modules and that added the modules to initrd for me. Also didn’t need to add chmod, chmod must already come with busybox.

    2. when initially debugging i found it helpful to add “break=bottom” to the kernel cmd line to stop the startup temporarily before the script so I could poke around and see what files/directories/environment variables are what.

    Again, a really really good write-up.

  30. Penpo
    June 27, 2009

    Using Debian I get the same error as “Luigi”:
    “mount: Mounting root.union on /root failed: No such device”
    Which he resolved by
    “I recompiled the kernel and there was no support for aufs
    with the generic kernel works fine”

    I haven’t managed to rebuild the kernel with aufs support, any pointers as to where to start?

  31. Penpo
    June 28, 2009

    apt-get install aufs-modules-2.6-amd64
    fixed it

    Thanks for an excellent guide!

  32. July 27, 2009

    Worked perfectly for me! Going to blog this and use it to roll out 6 full Ubuntu PCs in a public area.

  33. Penpo
    July 28, 2009

    Two thoughts.
    1. The system I did this one used a separate partition as swap, this partition resides on the flash device – not ideal I suppose (mainly since swap is used before the ram runs out).
    Possible solutions(?)
    * Turn off swap. I’ve read that that isn’t recommended, not sure of the reason though – theoretically that solution appeals to me.
    * Use a file or ramdrive as swap (seems like an ugly solution though, and won’t allow for much swap anyway – although after reading on the subject it might be sensible).
    * Echo 0 into /proc/sys/vm/swappiness, I don’t know what that actually results in so I don’t know whether it is sensible in this scenario.

    Basically I use the read-only solution on a server that acts as a router and file-server (mainly for backup and samba-storage for semi-large files). Unless an application leaks a lot of memory I don’t see how the 2 GB of ram would run out on this non-X machine.
    It’s often said that turning off swap will crash the OS if ram runs out, but won’t swap just postpone it?

    That said I’ve barely seen any traffic (iostat) to the swap-partition (then again I’m just experimenting with it so far), but since the machine will be out of my reach when it is deployed it would be quite annoying if the flash device was worn out.

    2. Wouldn’t it be possible to mount the flash-device and do something like:
    cp /rw/* /mnt/flash/ -ur to update all the files that have been changed in case you did some changes that you wanted to store permanently – without rebooting (the solutions to make permanent changes I’ve read about so far doesn’t really work that well to do remotely).

    Also, what decides how large the tmpfs should be?
    On my 2 GB system it takes 1004 MB but I’ve yet to see where that spaces comes from as ‘top’ etc. doesn’t seem to take into account that 1004 MB of the ram is locked away(?).

  34. YAM
    December 18, 2009

    I’ve read the articles carefully, and
    1. started from backing up kernel img
    2. edit menu.lst
    3. prepare 2 scripts
    4. reboot
    and it seems working very fine under kubuntu9.04 64 bit.
    log files are visible about only current session.
    i hope this usb os works forever!

    Thanks a lot.

  35. Read only root filesystem…

    The following references may be useful:…

  36. […] your linuxrc file (if you don’t know what this is, you’ll want to read the article from logicsupply) that mount your filesystem (at /mnt/root, for example) with something like […]

  37. Chris
    May 4, 2010

    Very very very good article – great quality of this post!

    Anybody else having trouble starting CUPS on the read-only side? It starts and works perfectly on the backend side, but when I run /etc/init.d/cups start in read-only mode I get the following:
    “* Starting Common Unix Printing System: cupsd
    /usr/svin/cupsd: error while loading shared libraries: libcupsmime.so.1: cannot open shared object file: No such file or directory ”

    Any ideas?

  38. Chris
    May 6, 2010

    “The best approach for
    dealing with this common use case is to have a tmpfs read/write layer and then
    mount some writable media on an arbitrary mount point like /var/local/data (for
    example).”

    Any commands on this and where to put them?

    Thanks!

  39. Forest Bond
    Forest
    May 6, 2010

    Chris,

    You should be able to add boot-time mounts to your fstab file like usual. The following link might be helpful:

    http://www.tuxfiles.org/linuxhelp/fstab.html

    -Forest

  40. Forest Bond
    Forest
    May 6, 2010

    Note that you must make changes to the fstab file on the actual on-disk filesystem (not the one in the union filesystem) so that the changes are permanent.

  41. Jon
    June 7, 2010

    Great post! However, what if i’m using a build that doesn’t utilize initramfs, such as Red Hat?

  42. rxd
    September 11, 2010

    I have done it by modifying the casper script inside initrd.lz to get read-only system so your rw layer is visible on the system. One good advantage of having this system is you can now merge squashfs file to your root at real time without rebooting the system. Prior to apt-get of any application make a copy of what you have on the /rw, then apt-get any application you want then generate the mksquashfs -ef excluded-files. Once you have say the apache2.squashfs now your next reboot you can just mount -o remount,append:/apache2 /.

  43. Andrew
    September 14, 2010

    Great article.

    However I’m having a problem when certain processes get denied access to files now that I’ve implemented aufs.

    Specifically when running “ifup eth0” I get an error from dhclient3 reporting it cannot find libc.so – but it does exists in the location returned from ldd.

    If I tail dmesg the kernel reports something along the lines:
    denided mask =’r’ operation=’open’ /ro/etc/ld.so.cache
    denided mask =’r’ operation=’open’ /ro/lib/libc.2.11.1.so

    The network interfaces worked perfectly before. Do you have any suggestions?

  44. Andrew
    September 16, 2010

    Nm – disabling apparmor (by appending apparmor=0 to the grub conf) did the trick.

  45. Purusothaman
    September 23, 2010

    Good article,

    I have tried and i was able to successfully make the rootfs as read-only using unionfs.But i have following two issues:
    1.Unable to unmount the unionfs and mounted partitions during reboot using “umount -a” command(device or resorce busy).
    2.Creation of the file in the writable layer(rw) is got reflected in read-only filesystem(ro).

    I have used a seperate partition of my hard disk to mount the writable layer instead of using tmpfs.

    Any suggestions for the above issues?

  46. Rick S
    October 13, 2010

    You wrote: “The contents of /etc/mtab are likely not correct, so the output of the mount command is probably missing some information. There are steps we can take to correct /etc/mtab, but I won’t cover those in detail here.”

    Can you point me int he right direction for getting the ‘mount’ command to list properly?

  47. abakan
    November 23, 2010

    In Ubuntu 10.04 it`s scripts work, but In Ubuntu 10.10 it is not corectly working.

    Touble is:
    Some applications doesn`t run at all. I try to run wine, Gogole Chrome but nothing doesn’t happen.

    Any ideas?

  48. abakan
    November 23, 2010

    When I run wine in terminal:
    “wineserver: mkdir /tmp/.wine-1000 : Operation not permitted”

  49. Diogo Resende
    December 23, 2010

    Some notes:
    – don’t use a journaling system (use ext2, xfs won’t work);
    – apparmor might block some services (I had to change usr.sbin.mysqld profile to be able to access root.ro and root.rw)

  50. Nicola
    December 27, 2010

    how? with ubuntu 10.10?
    Help!

  51. Matt Anderson
    January 28, 2011

    This worked great for me on Ubuntu 10.04.01 LTS. Thanks for the great write-up.

    The only gotcha I ran into was the single user mode trick did not work for me. My system hung at a blank screen and was non-responsive on the network. I didn’t test this before making the RO root changes, so I’m not sure if this was an existing problem with the computer or not.

    Instead what I’ve been doing is once booted:
    # mount -o remount,ro /ro
    # cd /ro
    # chroot .

    That lets me edit my system in a persistent way. Once I’m done I simply:

    # exit (this gets me out of the chroot)
    # cd /
    # mount -o remount,ro /ro

    Usually I’ll throw a `sync` in there for good measure 🙂

  52. Francesco
    February 7, 2011

    everything seems work, I’ve also added an item to grub, to start linux in normal mode, but I’ve encountered a bug: “Bug#607879: System hangs up with mmap.c:873”. Do you have any suggestion?
    Thanks again for your work

  53. March 26, 2011

    As an FYI, this guide is really superb. I’m using it with a Ubuntu 10.04 LTS server, booted off of a compact flash, in an embedded server application.

    One gajillion thanks!

  54. sparsile
    March 29, 2011

    TK (or anyone else!), if you’re still listening, I would very much appreciate a tutorial on building a RO linux system for CentOS 5.

  55. Johnny
    October 17, 2011

    This works except for one found one bug in Ubuntu 10.04:

    dhclient breaks.
    this is kind of a weird error. It reports that the libraries aren’t there, but they are.

    #dhclient: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
    # which dhclient
    /sbin/dhclient
    # ldd /sbin/dhclient
    linux-vdso.so.1 => (0x00007fffa8fff000)
    libc.so.6 => /lib/libc.so.6 (0x00007f4a3b820000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f4a3be1f000)
    # ls -al /lib | grep libc.so.6
    lrwxrwxrwx 1 root root 14 2011-10-10 20:05 libc.so.6 -> libc-2.11.1.so
    root@Archives:/lib# ls -al /lib | grep libc-2.11.1.so
    -rwxr-xr-x 1 root root 1572232 2011-01-21 14:23 libc-2.11.1.so
    lrwxrwxrwx 1 root root 14 2011-10-10 20:05 libc.so.6 -> libc-2.11.1.so

  56. FBP
    October 17, 2011

    I have a Seagate Dockstar (arm5 processor, 128 MB RAM, boots Debian squeeze from a usb flash drive). I have done this “Read-Only Linux System” procedure on a more “normal” computer (Intel Atom, Debian squeeze), and it works fine. But it fails on the Dockstar. It boots, it just isn’t read-only. I get the following in dmesg:

    [ 12.960395] aufs: module is from the staging directory, the quality is unknown, you have been warned.
    [ 12.993986] aufs 2-standalone.tree-32-20100125
    [ 13.003956] aufs test_add:218:mount[124]: unsupported filesystem, /root.ro (rootfs)

    I end up with no /rw or /ro mount points. Ideas?

  57. […] a NAS. I have been trying to make the boot flash drive read-only using the method described here: http://www.logicsupply.com/blog/2009…-linux-system/ The method entails moving the root filesystem to an aufs filesystem so that the boot drive is […]

  58. Forest Bond
    October 29, 2011

    Hi everyone,

    I’m no longer at Logic Supply, but if people still have questions I can do my best to get them answered. I’ve created a new thread for this purpose on the RapidRollout forums:

    Configuring Ubuntu for read-only storage

    Go ahead and post your questions there and I’ll see what I can do to help!

    -Forest

  59. August 17, 2012

    Hey There. I found your blog the usage of msn. This is a really neatly written article. I will be sure to bookmark it and come back to read more of your useful information. Thanks for the post. I’ll definitely return.

  60. OZ
    July 6, 2015

    Thanks for the How-To.

    I’ve done this on [L]ubuntu 14.04.

    Now – as it was supposed, all changes and/or new files are flushed on a reboot.
    That’s good.

    On the negative side:
    1) auto-mounting an USB Flash broke: reports that a device “/mount/%username%/%devicename%” is invalid;
    2) DHCP also broke – I have to assign settings manually.

    I was looking for something as easy as windows’ Enhanced Write Filter which has a manager program (ewfmgr.exe) – to be able to arbitrarily apply some changes (after OS or software update, etc). It is not clear how to do it with this environment.

    Please comment.
    Thank you in advance.

  61. OZ
    July 6, 2015

    correction – the error message is this:
    ‘The specified directory “/media/%username%/%volumename%” is not valid’.

  62. Darek Fanton
    Darek Fanton
    July 7, 2015

    Hi OZ, thank you for your comment. While Forest, the author of this post, is no longer with Logic Supply, he has a thread specifically designed for answering questions related to this tutorial here.

Leave a Comment

Your email address will not be published.