path: root/labs/03-mounting.org

#+title: Mounting
#+date: 2026-03-17 Mon
#+author: W. Kosior
#+email: wkosior@agh.edu.pl
#+HTML_HEAD: <link rel="stylesheet" href="../org-dark-styles.css">

In these exercises we are going to configure an "ephemeral" system that keeps
all filesystem changes in RAM and forgets them when rebooted.  It's a suitable
setup for places where a kind of "guest mode" is needed and other solutions are
unfavored.

We are going to run the configured system inside a QEMU virtual machine and
automate the configuration with Ansible.

To be graded, send
1. the required screenshots,
2. your =playbook.yaml=, and
3. yout initramfs script
to @@html:<a href="email:wkosior@agh.edu.pl">wkosior@agh.edu.pl</a>@@ along with
your names.  You can work in teams of up to three people.

* Useful Resources

[[https://docs.ansible.com/][Ansible documentation]]

[[https://manpages.debian.org/testing/initramfs-tools-core/initramfs-tools.7.en.html][man initramfs-tools]]

[[https://www.yaml.info/learn/index.html][a YAML tutorial and reference]]

* Our Setup
We shall be using [[https://pluton.kt.agh.edu.pl/~wokosior/bso-vms/debian-13-nocloud-amd64.qcow2][the Debian VM image]] from [[../01-secure-booting.org][the first topic]].  Its credentials are
=root=​/​=security=.  It can be run with the following QEMU command.

#+begin_src shell-script
  qemu-system-x86_64 \
      -m 1G \
      -enable-kvm \
      -hda debian-13-nocloud-amd64.qcow2 \
      -net nic -net user,hostfwd=tcp::30022-:22 \
      -nographic # Replace with `-vga virtio' for a GUI.
#+end_src

If possible, choose one one your team's computers for running Ansible (PC "A")
and one for running QEMU (PC "Q").

Ansible is a tool that uses SSH connections to depoloy configuration to remote
machines.  To be able to use it, generate an SSH keypair on "A" if you haven't
already.  Then, enable key-based logins by copying the public part to the guest.

#+begin_src shell-script
  IP_OF_Q=10.20.30.40 # Fill in.
  ssh-keygen # If you do not yet posses a keypair under `~/.ssh/'.
  ssh-copy-id -p 30022 root@"$IP_OF_Q"
  ssh -p 30022 root@"$IP_OF_Q" 'echo "*hacker voice*: I'\''m in!"' # Verify.
#+end_src

Ansible shall benefit from a configuration file describing the host(s) it is
going to connect to.  Place the following inside =inventory.yaml= on "A".

#+begin_src yaml
  ungrouped:
    hosts:
      ephemeral:
        ansible_user: root
        ansible_host: 10.20.30.40 # Fill in with the IP address of "Q".
        ansible_port: 30022
#+end_src

Make sure Ansible is installed (e.g., through APT).

Verify with Ansible's =ping= module that it is able to connect to the host that
we've just named =ephemenral=.

#+begin_src shell-script
  ansible -i inventory.yaml -m ansible.builtin.ping ephemeral
#+end_src

Ansible operates on tasks that can be performed on one or more of our hosts.
The tasks are written in playbooks using YAML format.  Place the following in
=playbook.yaml= on "A".  This configuration consists of just one "Hello, World!"
task.  Perform the task using a command provided below.

#+begin_src yaml
  - name: Ansible playbook stub
    hosts: ephemeral
    tasks:
     - name: Debug
       ansible.builtin.debug:
         msg: Hello, World!
#+end_src

#+begin_src shell-script
  ansible-playbook -i inventory.yaml playbook.yaml
#+end_src

*Prepare screenshot 1: output of a successful =ansible-playbook= command with
the "Hello, World!" task.*

Note that you can speed up Ansible tasks by disabling fact gathering, for
example with the following environment variable.

#+begin_src shell-script
  export ANSIBLE_GATHERING=explicit
#+end_src

You might also want to explicitly specify the Python interpreter on the target
to avoid a warning from Ansible.  You can add the following to your inventory
config file.

#+begin_src yaml
  ansible_python_interpreter: /usr/bin/python3
#+end_src

* Hacking The Initramfs
As described in =man initramf-tools=, the initramfs image in Debian can be
customized by adding
- names of kernel modules and files to be included,
- *hooks* that execute during the generation process, and
- *scripts* that execute from within the initramfs.

Let's add a simple script that will spawn a shell for us to play with the
initramfs during boot.  Place the following in a file named
=spawn-initramfs-shell= on "A".

#+begin_src shell-script
#!/bin/sh

case $1 in
    prereqs)
        exit 0
        ;;
esac

/bin/sh -i # play around interactively
#+end_src

The =case= statement is there, because initramfs scripts are actually executed
twice.
- Once on the system, during generation, with argument =prereqs=, to allow
  ordering the scripts in case there are some dependencies between them.
- Once during boot, without the =prereqs= argument.

Distribution-shipped initramfs hooks and scripts reside under =/usr/share/=.
User-supplied scripts should be placed under =/etc/initramfs-tools/scripts/=.
Depending on when you want your script to be executed (before or after the new
root filesystem is mounted, etc.), you should place it in a different
subdirectory of =scripts/=.  We'd like =spawn-initramfs-shell= to be executed
after the root filesystem is already mounted, so we'll place it under
=scripts/local-bottom=.  Add the following =copy= task to your playbook.

#+begin_src yaml
  - name: Install `spawn-initramfs-shell'
    ansible.builtin.copy:
      src: spawn-initramfs-shell
      dest: /etc/initramfs-tools/scripts/local-bottom/
      mode: '744'
#+end_src

Then, use Ansible module =command= to allow updating target's =initrd.img= with
the playbook.  Add the following task after the last one.

#+begin_src yaml
  - name: Update Initramfs
    ansible.builtin.command:
      cmd: update-initramfs -c -k all
#+end_src

Run the playbook again.  Once the new initramfs is generated, use the =reboot=
module of Ansible.

#+begin_src shell-script
  ansible -i inventory.yaml -m ansible.builtin.reboot ephemeral
#+end_src

The guest shold reboot to a minimal shell available in the initramfs.  From "Q",
try poking around in the initramfs.

#+begin_src shell-script
  # Look around.
  ls /
  ls /bin/
  mount
  echo $rootmnt
  ls /root/
  lsmod # Command likely absent.
  chroot $rootmnt /bin/sh # ;)
#+end_src

*Prepare screenshot 2: output of the commands above when executed in the
initramfs shell.*

When you exit the shell, your initramfs script shall terminate and the system
shall continue booting normally.

#+begin_src shell-script
  exit
#+end_src

* Installing BusyBox
BusyBox is a popular a single-executable implementation of the most popular unix
commands.  Debian shall automatically include BusyBox (instead of the utilities
you've just been using) in an intramfs image.  This will give us more convenient
tooling for use in an initramfs shell.  Use the following playbook task (place
it somewhere before =Update Initramfs=).

#+begin_src yaml
  - name: Install BusyBox
    ansible.builtin.apt:
      name: busybox
#+end_src

Our goal is to overlay a RAM-based filesystem on top of the primary root
filesystem.  This requires the =overlay= kernel module to be present in the
initramfs.  Let's include it by appending the kernel module to
=/etc/initramfs-tools/modules=.  Use the following playbook task (also place it
somewhere before =Update Initramfs=).

#+begin_src yaml
  - name: Add overlayfs to initramfs
    ansible.builtin.shell:
      cmd: printf 'overlay\n' >> /etc/initramfs-tools/modules
#+end_src

Notice that we used Ansible module =shell= rather than =command=.  The latter
would not allow us to use shell redirection in the command.

After rebooting, in the (now BusyBox-based) initramfs shell, mount a RAM-based
filesystem.

#+begin_src shell-script
  mkdir /ovlay
  mount -t tmpfs -o size=100% my-root-overlay-tmpfs /ovlay
#+end_src

The =size=100%= option tells the kernel not to limit the amount of data stored
in this filesystem in any way.

Now, use the =mount -o move= to re-attach an already mounted root filesystem
under =/ovlay/lower=.

#+begin_src shell-script
  mkdir /ovlay/upper /ovlay/lower /ovlay/work
  mount -o move "${rootmnt}" /ovlay/lower
#+end_src

Now, mount the overlayfs.  Use =/ovlay/upper= for the actual added & modified
files.  Use =/ovlay/work= for additional files used by the driver.  Have the
overlayed root filesystem appear right where it was previously mounted without
overlay.

#+begin_src shell-script
  mount -t overlay \
        -o lowerdir=/ovlay/lower,upperdir=/ovlay/upper,workdir=/ovlay/work \
        my-root-overlayfs "${rootmnt}"
#+end_src

To see how overlayfs works, try adding a file to the overlay and see that it
does not get created in the lower directory.

#+begin_src shell-script
  touch "${rootmnt}"/my-file
  ls "${rootmnt}"
  ls /ovlay/lower
#+end_src

*Prepare screenshot 3: output of the commands above with =my-file= present in
the overlay and absent under =/ovlay/lower=.*

After exitig the shell and booting, check that the current root filesystem is
indeed the overlayfs.

#+begin_src shell-script
  mount | grep ' on / '
#+end_src

You can even play around and to some ephemeral damage :)

#+begin_src shell-script
  rm -rf /usr/bin/
#+end_src

* Optimizing the Playbook
=Update Initramfs= takes a lot of time.  And it is only needed if there are
actual changes to our initramfs configuration.  Optimize the playbook.  First,
add the following before the =Add overlayfs to initramfs= task.

#+begin_src yaml
    - name: Check if initramfs has overlayfs
      ansible.builtin.command:
        cmd: grep -qE ^overlay /etc/initramfs-tools/modules
      register: overlayfs_in_initramfs
      ignore_errors: true
#+end_src

Add the following to the =Add overlayfs to initramfs= to have it execute only
when needed.

#+begin_src yaml
  when: overlayfs_in_initramfs is failed
  register: enlist_overlayfs_module
#+end_src

add the following to the =Install BusyBox= task, to allow checking for changes.

#+begin_src yaml
  register: install_busybox
#+end_src

add the following to the =Install `spawn-initramfs-shell'= task to allow
checking for changes.

#+begin_src yaml
  register: install_initramfs_shell_spawner
#+end_src

Add the following to the =Update Initramfs= task to have it execute only when
needed.  Note that this YAML construct makes lines after "=>=" treated as a
single-line string value under the key =when=.

#+begin_src yaml
  when: >
    install_busybox.changed or
    enlist_overlayfs_module.changed or
    install_initramfs_shell_spawner.changed
#+end_src

Execute the playbook again.  It should not be trying to execute the lengthy
=Update Initramfs= task now, unless we change something this task relies on.

*Prepare screenshot 4: output of the =ansible-playbook= command with =Update
Initramfs= task skipped.*

* Initramfs Script for Root Overlay
We are now going to put the overlay mounting code in a script.  It shall run
from initramfs and make the system ephemeral provided that an =overlay-root=
kernel command line argument is passed.

Reboot the VM if you're still booted with the overlay.  Create a new initramfs
script with a name of your choice.  Inside it, place the following.

#+begin_src shell-script
  #!/bin/sh

  case $1 in
      prereqs)
          exit 0
          ;;
  esac

  # Check if `overlay-root' kernel command-line arg was passed.
  DO_OVERLAY=
  for ARG in $(cat /proc/cmdline); do
      if [ overlay-root = "$ARG" ]; then
          DO_OVERLAY=something
      fi
  done

  # Only proceed if `overlay-root' was present.
  if [ -z "$DO_OVERLAY" ]; then
    exit 0
  fi
#+end_src

In this script, after the conditional, add the commands previously used to mount
the overlay on top of the new root filesystem.

Add a playbook task analogous to the =Install `spawn-initramfs-shell'‍=.  The
task should install your new script.  Remember to update the =when= key of the
=Update Initramfs= task so that it gets run if just the new script contents get
updated at some point.

Now, remove
- the =spawn-initramfs-shell= script (in the VM!),
- the task that installs it, and
- the related part of the =when= condition of =Update Initramfs= task.

Run the playbook.

Note: at this point, if you
- fail to remove =spawn-initramfs-shell= in the VM,
- fail to update the initramfs after removal, or
- do so when booted to our ephemeral system,
you shall still get the initramfs shell upon each reboot.  Either properly
re-generate the initramfs without =spawn-initramfs-shell= or bear this
inconvenience 😉.

Reboot *with a QEMU GUI* (unless you feel brave and eager to use Emacs key
bindings instead of arrow keys).  In GRUB menu, press =e= to edit the boot menu
entry.  Add =overlay-root= somewhere in the Linux command line.  Then, boot the
edited entry.

Verify that the currently running system has an RAM-based overlay on the root
filesystem.

#+begin_src shell-script
  cat /proc/cmdline
  mount | grep ' on / '
#+end_src

*Prepare screenshot 5: output of the commands above indicating that our script
has properly mounted the overlay.*

* Permanently Adding a Kernel Command Line Argument
Reboot (so that the system is once again modifiable).

Add the following tasks.  They shall update the bootloader config to
automatically pass =overlay-root= to the kernel when booting.

#+begin_src yaml
    - name: Check if GRUB command line includes `overlay-root'
      ansible.builtin.shell:
        cmd: |
          grep -qE 'GRUB_CMDLINE_LINUX_DEFAULT=".*overlay-root' \
               /etc/default/grub
      register: overlay_in_grub_cmdline
      ignore_errors: true

    - name: Add `overlay-root' to GRUB command line
      ansible.builtin.shell:
        cmd: |
          sed -i 's/\(GRUB_CMDLINE_LINUX_DEFAULT="\).*"/\1 overlay-root"/' \
              /etc/default/grub
      when: overlay_in_grub_cmdline is failed

    - name: Update GRUB & its configuration
      ansible.builtin.command:
        cmd: update-grub
      when: overlay_in_grub_cmdline is failed
#+end_src

Run the playbook and then look for =overlay= in = /etc/default/grub= and in
=/boot/grub/grub.cfg= to see the effect.

#+begin_src shell-script
  cat /etc/default/grub
  less /boot/grub/grub.cfg
#+end_src

Reboot (in text mode, without GUI).  Changes to the system are now by default
not persisted.

* Kexec
/For this task, make sure you are booted in text mode and see guest's serial
output in a terminal on "Q"./

We might want to sometimes reboot the machine in non-ephemeral mode, for example
to perform updates or change some configuration.  The kexec functionality of the
Linux kernel shall allow us to achieve this easily.  Install =kexec-tools=
package using Ansible module =apt= (put this at the very beginning of your
playbook).

#+begin_src yaml
  - name: Install kexec
    ansible.builtin.apt:
      name: kexec-tools
#+end_src

After running the playbook, see how we can easily obtain the (current or target)
kernel command line and the the filename of our kernel image.

#+begin_src shell-script
  cat /proc/cmdline
  cat /proc/cmdline | sed 's/overlay-root//'
  ls /boot/vmlinuz-$(uname -r)
#+end_src

Use =kexec= command to prepare the system for booting again, with the same
kernel and initramfs, but with the =overlay-root= parameter removed.

#+begin_src shell-script
  kexec --load \
        --append="$(cat /proc/cmdline | sed 's/overlay-root//')" \
        --initrd=/boot/initrd.img-$(uname -r) \
        /boot/vmlinuz-$(uname -r)
#+end_src

A reboot can be triggered by either =kexec --exec= or =reboot= command.  The
latter is preferred in our case as it makes sure that system's all devices are
shut down before the reboot.

#+begin_src shell-script
  reboot # Automatically picks up and uses a kexec-loaded kernel.
#+end_src

*Prepare screenshot 6: Part of VM's serial output with phrases =Rebooting with
kexec= and =kexec_core: Starting new kernel= visible.*

After rebooting, verify that there is no overlayfs in use.

#+begin_src shell-script
  cat /proc/cmdline # After reboot.
  mount | grep ' on / '
#+end_src

* Integrating Ansible Playbook with Kexec
We can now make the playbook automatically reboot the system into non-ephemeral
mode before deploying any configuration to it.

At the beginning of the playbook (after the =Install kexec= task and before all
the others) create a new task using =ansible.builtin.shell=.  Name it =Load
kexec kernel without `overlay-root' arg=.  Have the task execute the kexec
command you have just used.

Add another task named =Reboot to non-ephemeral= immediately after.  Use
=ansible.builtin.reboot= (without any other keys).

Change task name =Install BusyBox= to =Install packages=, and change its
property =name: busybox= to =name: ['busybox', 'kexec-tools']=.  This is needed
to have Ansible install =kexec-tools= to the persistent system as well in case
it is not yet there.

At the very end of the playbook, add another two tasks that load a kernel with
kexec and reboot.  This time, *append* the =overlay-root= kernel argument
instead of removing it.

Run the playbook.

*Prepare screenshot 7: The output of successfully executed playbook using kexec.*

* Adding a User and Preventing Login in Non-Ephemeral Mode
An ephemeral system like ours would typically have an account for guests.  Add
such account using this playbook task (place it in an appropriate place in the
task list).

#+begin_src yaml
  - name: Add `student' user
    ansible.builtin.user:
      name: student
      password: '$y$j9T$CEEtmktyUYCFMlaeO0PIJ0$BoKJOYSU1eJrigIHcTyTIs1tN3NUYHDFbXBSe4PS4jC'
#+end_src

Run the playbook and see the effect.  Try logging in as the new user (the
account's credentials are =student=​/​=student=).

Now, we shall prevent this user from logging in when the system is booted
without overlay.  We do not want the guest to be able to make any persistent
changes (be it changes to files in user's home directory or default shell change
achieved through =chsh=).

Update the playbook to use [[https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/user_module.html][an appropriate parameter of ansible.builtin.user]] to
make the password locked by default.

Then, update your initramfs script to automatically unlock the password when
booting in our ephemeral mode.  To achieve this, in the script you can either
- bind-mount the overlayed =${rootmnt}= to =/etc= in initramfs and have
  initramfs' BusyBox' =passwd= command operate on it, or
- temporaily chroot into the overlayed =${rootmnt}= and use system's usual
  =passwd= executable for the task.

Whichever method you choose, here are usage messages of the relevant commands in
BusyBox.

#+begin_example
~ # mount --help
BusyBox v1.37.0 (Debian 1:1.37.0-6+b5) multi-call binary.

Usage: mount [OPTIONS] [-o OPT] DEVICE NODE

Mount a filesystem. Filesystem autodetection requires /proc.

        -a              Mount all filesystems in fstab
        -f              Dry run
        -i              Don't run mount helper
        -r              Read-only mount
        -t FSTYPE[,...] Filesystem type(s)
        -T FILE         Read FILE instead of /etc/fstab
        -O OPT          Mount only filesystems with option OPT (-a only)
-o OPT:
        loop            Ignored (loop devices are autodetected)
        [a]sync         Writes are [a]synchronous
        [no]atime       Disable/enable updates to inode access times
        [no]diratime    Disable/enable atime updates to directories
        [no]relatime    Disable/enable atime updates relative to modification time
        [no]dev         (Dis)allow use of special device files
        [no]exec        (Dis)allow use of executable files
        [no]suid        (Dis)allow set-user-id-root programs
        [r]shared       Convert [recursively] to a shared subtree
        [r]slave        Convert [recursively] to a slave subtree
        [r]private      Convert [recursively] to a private subtree
        [un]bindable    Make mount point [un]able to be bind mounted
        [r]bind         Bind a file or directory [recursively] to another location
        move            Relocate an existing mount point
        remount         Remount a mounted filesystem, changing flags
        ro              Same as -r

There are filesystem-specific -o flags.
#+end_example

#+begin_example
~ # chroot --help
BusyBox v1.37.0 (Debian 1:1.37.0-6+b5) multi-call binary.

Usage: chroot NEWROOT [PROG ARGS]

Run PROG with root directory set to NEWROOT
#+end_example

*Prepare screenshot 8: The contents of =/etc/shadow= with =student='s password
locked.*

Do not forget to include your initramfs script in the email!

* Extras (for Bonus Points!)
Optimize the playbook to only perform the final reboot if the current kernel has
not been booted with the =overlay-root= argument.

During boot we see some error messages related to systemd trying to perform some
operations on the root filesystem (which has its *original* entry still present
in =/etc/fstab=).  Find a way to avoid these errors.  You can try modifying the
fstab from your initramfs script or changing the configuration of systemd
services.