Skip to content

Read only root on Linux

Srijan Choudhary
2 min read

In many cases, it is required to run a system in such a way that it is tolerant of uncontrolled power losses, resets, etc. After such an event occurs, it should atleast be able to boot up and connect to the network so that some action can be taken remotely.

There are a few different ways in which this could be accomplished.

Mounting the root filesystem with read-only flags

Most parts of the linux root filesystem can be mounted read-only without much problems, but there are some parts which don't play well. This debian wiki page has some information about this approach. I thought this approach would not be very stable, so did not try it out completely.

Using aufs/overlayfs

aufs is a union file system for linux systems, which enables us to mount separate filesystems as layers to form a single merged filesystem. Using aufs, we can mount the root file system as read-only, create a writable tmpfs ramdisk, and combine these so that the system thinks that the root filesystem is writable, but changes are not actually saved, and don't survive a reboot.

I found this method to be most suitable and stable for my task, and have been using this for the last 6 months. This system mounts the real filesytem at mountpoint /ro with read-only flag, creates a writable ramdisk at mountpoint /rw, and makes a union filesystem using these two at mountpoint /.

The steps I followed for my implementation are detailed below. These are just a modified version of the steps in this ubuntu wiki page. I am using Debian in my implementation.

  1. Install debian using live cd or your preferred method.

  2. After first boot, upgrade and configure the system as needed.

  3. Install aufs-tools.

  4. Add aufs to initramfs and setup this script to start at init.

# echo aufs >> /etc/initramfs-tools/modules
# wget https://cdn.rawgit.com/srijan/383a8d7af6860de6f9de/raw/ -O /etc/initramfs-tools/scripts/init-bottom/__rootaufs
# chmod 0755 /etc/initramfs-tools/scripts/init-bottom/__rootaufs
  1. Remake the initramfs.
# update-initramfs -u
  1. Edit grub settings in /etc/default/grub and add aufs=tmpfs to GRUB_CMDLINE_LINUX_DEFAULT, and regenerate grub.
# update-grub
  1. Reboot

Making changes

To change something trivial (like a file edit), just remount the /ro mountpoint as read-write, edit the file, and reboot.

# mount -o remount,rw /ro

To do something more complicated (like install os packages), press e in grub menu during bootup, remove aufs=tmpfs from the kernel line, and boot using F10. The system will boot up normally once.

Another method could be to use a configuration management tool (puppet, chef, ansible, etc.) to make the required changes whenever the system comes online. The changes would be lost on reboot, but it would become much easier to manage multiple such systems.

Also, if some part of the system is required to be writable (like /var/log), that directory could be mounted separately as a read-write mountpoint.

devops

Related Posts

Advanced PostgreSQL monitoring using Telegraf, InfluxDB, Grafana

Introduction This post will go through my experience with setting up some advanced monitoring for PostgreSQL database using Telegraf, InfluxDB, and Grafana (also known as the TIG stack), the problems I faced, and what I ended up doing at the end. What do I mean by advanced? I liked this

Advanced PostgreSQL monitoring using Telegraf, InfluxDB, Grafana

Running docker jobs inside Jenkins running on docker

Jenkins [https://www.jenkins.io/] is a free and open source automation server, which is used to automate software building, testing, deployment, etc. I wanted to have a quick and easy way to run Jenkins inside docker, but also use docker containers to run jobs on the dockerized Jenkins. Using

Running docker jobs inside Jenkins running on docker

Telegraf: dynamically adding custom tags

Adding a custom tag to data coming in from an input plugin for telegraf