How to mount / in read-only mode by default (Stateless Linux) ?

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7

Issue

  • Trying to set up readonly / filesystem and have modified below parameters in /etc/sysconfig/readonly-root to the following:
    READONLY=yes
    TEMPORARY_STATE=yes
  • However when the server is rebooted, the root filesystem is mounted as rw and not ro as expected:
    /dev/xvda1 on / type xfs (rw,relatime,attr2,inode64,noquota)

Resolution

[1] Edited below file by changing the parameters as per your requirement. Minimal requirement is to edit READONLY to yes.

# cat /etc/sysconfig/readonly-root 

# Set to 'yes' to mount the system filesystems read-only.
READONLY=yes
# Set to 'yes' to mount various temporary state as either tmpfs
# or on the block device labelled RW_LABEL. Implied by READONLY
TEMPORARY_STATE=yes
# Place to put a tmpfs for temporary scratch writable space
RW_MOUNT=/var/lib/stateless/writable
# Label on local filesystem which can be used for temporary scratch space
RW_LABEL=stateless-rw
# Options to use for temporary mount
RW_OPTIONS=
# Label for partition with persistent data
STATE_LABEL=stateless-state
# Where to mount to the persistent data
STATE_MOUNT=/var/lib/stateless/state
# Options to use for persistent mount
STATE_OPTIONS=
# NFS server to use for persistent data?
CLIENTSTATE=
# Use slave bind-mounts
SLAVE_MOUNTS=yes

[2] Change fstab entry for / as ro by replacing defaults.

# cat /etc/fstab | grep root
/dev/mapper/rhel-root   /    xfs   ro     0 0

[3] Take a backup of current initramfs (to be on safer side).

# cp /boot/initramfs-3.10.0-229.el7.x86_64.img /boot/initramfs-3.10.0-229.el7.x86_64.img.bkp 

[4] Rebuild the initramfs for current kernel version.

# dracut -f -v

[5] Perform a reboot.

# reboot

[6] Verify if / is mounted in ro mode :

# cat /proc/mounts | grep root
rootfs / rootfs rw 0 0
/dev/mapper/rhel-root / xfs ro,seclabel,relatime,attr2,inode64,noquota 0 0

Root Cause

  • initramfs was not re-built.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

1 Comments

Hi,

I am very glad to see read-only root systems being addressed.

This current Solution in Progress could well break your servers if you do not plan adequately before.

I imagine steps missing to be: Move separate out your rootfs into separate mount points like / /etc/ /var /var/log and so on. Also, you should address patching via yum. On Debian and Ubunti we can use Pre and Post scripts to change mount options when apt-get is called. Does a similar method exist in RHEL?

Let's see if it's as easy to roll back. Luckily it is. See below.

@REDHAT - Would be very grateful if you would cover the missing steps for a working production ready server.

Here are some of the error messages.
LDAP BROKEN

<83>1 2018-10-25T09:50:21.740539+02:00 srv51 sudo - - - pam_ldap(sudo-i:auth): error reading from nslcd: Connection reset by peer

RSYSLOG partially BROKEN

<43>1 2018-10-25T09:53:32.477288+02:00 srv51 rsyslogd - - - imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Read-only file system [v8.24.0 try http://www.rsyslog.com/e/2013 ]

/VAR/LOG Most of the /var/log files disappeared

AUTOMOUNT

fails because cannot talk to LDAP <30>1 2018-10-25T09:37:57.471186+02:00 srv50 automount 1032 - - bind_ldap_simple: lookup(ldap): Unable to bind to the LDAP server: (default), error Can't contact LDAP server

VARIOUS NETWORKING PROBLEMS <6>1 2018-10-25T09:37:54.754433+02:00 srv50 kernel - - - IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready <28>1 2018-10-25T09:37:54.790432+02:00 srv50 NetworkManager 787 - - [1540453074.7895] dns-mgr: could not commit DNS changes: Failed to create file “/etc/resolv.conf.DJFWRZ”: Read-only file system <30>1 2018-10-25T09:37:55.231389+02:00 srv50 systemd - - - PID file /var/run/nslcd/nslcd.pid not readable (yet?) after start. <30>1 2018-10-25T09:37:55.270455+02:00 srv50 systemd - - - PID file /var/run/rhnsd.pid not readable (yet?) after start. <18>1 2018-10-25T09:37:55.697655+02:00 srv50 postfix 1118 - - fatal: open lock file /var/lib/postfix/master.lock: cannot open file: Read-only file system <27>1 2018-10-25T09:37:54.328992+02:00 srv50 nrpe 767 - - Unable to open config file '/u/nagios/conf/nrpe_local.cfg' for reading <30>1 2018-10-25T09:37:54.366425+02:00 srv50 gssproxy - - - gssproxy[781]: Failed to create Unix Socket! (98:Address already in use) <83>1 2018-10-25T09:37:54.366704+02:00 srv50 gssproxy 781 - - Failed to create Unix Socket! (98:Address already in use)

EMC NETWORKER

<81>1 2018-10-25T05:20:08.869874+02:00 srv50 sudo - - - addm : command not allowed ; TTY=pts/1 ; PWD=/u/addm ; USER=root ; COMMAND=/bin/test -f /nsr/res/servers <81>1 2018-10-25T05:20:09.031247+02:00 srv50 sudo - - - addm : command not allowed ; TTY=pts/1 ; PWD=/u/addm ; USER=root ; COMMAND=/bin/test -f /nsr/res/servers -a -r /nsr/res/servers <14>1 2018-10-25T09:33:30.586053+02:00 srv50 audispd - - - node=srv50 type=PATH msg=audit(1540452795.470:9506): item=1 name="nsr" inode=723716 dev=fd:01 mode=040755 ouid=0 ogid=0 rdev=00:00 objtype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 <30>1 2018-10-25T09:37:56.020350+02:00 srv50 networker - - - 90311:nsrexecd: Unable to read NetWorker product configuration file: Read-only file system <30>1 2018-10-25T09:37:56.146631+02:00 srv50 networker - - - 85846:nsrexecd: Failed to set permissions on nsr applogs directory: Read-only file system <30>1 2018-10-25T09:37:56.155174+02:00 srv50 networker - - - 85847:nsrexecd: Failed to create nsrexecd run file: Read-only file system

** How to roll back ** * Remove the ro mount option in /etc/fstab for root e.g

grep root /etc/fstab

/dev/mapper/bnpp_system-root / xfs defaults 0 0

  • Remount root read-write
mount -o remount,rw /
  • Revert the /etc/sysconfig/readonly-root file to the original settings e.g READONLY=no TEMPORARY_STATE=no

  • Rebuild the initramfs

dracut -f -v
  • reboot your server
reboot

Et voila, your server should be back.