How to restore a Snapper root snapshot on an unbootable system

FAQ and Tutorials
1

Hi everyone,

I have been running CachyOS since August and absolutely loving it. I came from almost 5 years running Manjaro and I’m daily asking myself why I didn’t jump that ship earlier to get closer to a base Arch installation.

Anyway, so I just updated my system 2 days ago (hadn’t done so for about a week so there were like 200+ updates and afterwards my proton vpn wasn’t working. It was late at night when I did the update so I just decided to just rollback my system using BTRFS Assistant because I just didn’t want to mess with figuring out what the issue was right then and there.

So for the first time the system no longer booted when I rolled back to my pre-update snapshot. I’ve rolled back updates in the past using BTRFS Assistant and there has never been an issue but for whatever reason after rolling back to my pre update snapshot. Instead when I booted I was presented with the following message:

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

Press Enter for maintenance
(or press Control-D to continue):

So I guess this is what I get for being lazy and not figuring out the issue with my vpn client lol,
Anyway I now had to figure out how to manually restore a root snapshot so I could get my system working again so I dove into the arch wiki entry on snapper which gives and example of restoring home directory from a snapper snapshot so not really root but I figured I’d give the step by step here since it took a little playing around to wrap my head around how snapper lays out subvolumes for performing root snapshots which can be a little confusing if you aren’t familiar with BTRFS subvolumes. The most important thing that can be hard to conceptualize especially when subvolumes are located in sub-directories is that snapshotting is not recursive, so a subvolume or a snapshot is effectively a barrier and no files in the nested subvolumes appear in the snapshot.

So here is the step by step instructions to recovery using the CachyOS installation ISO to restore a root snapshot to a system that no longer boots. So for those of you using the CachyOS defaults of Systemd bootloader and setup Snapper here is how you recover from an update that makes your system no longer boot to your DE.

First things first, boot off the CachyOS installation USB

  1. Open up dolphin and select your BTRFS drive if you did a default CachyOS installation it should be named root, otherwise select the appropriate drive and this will automatically mount it when you click it.
  2. Click the > at the top of the dolphin window to see where the drive was mounted at, which will be (/run/media/liveuser/UUID) make a note of the first couple of digits of the UUID
  3. launch Alacritty terminal
  4. type the following:

cd /run/media/liveuser/UUID

(basically type the first couple of digits of the UUID then press tab to complete it unless you just really want to type out the whole UUID :wink: )

  1. view your directory to make sure you are in the correct place as the sample commands I am using are assuming you are in the root directory of your BTRFS drive.

sudo ls -al

This will give you a directory of the actual root of your BTRFS file system if you changed your directory correctly. You should see the various subvolumes such as @ (which gets mounted as your current root directory, along with a few others like @Home, @log, etc…) Also note if like in my case I’m recovering from a BTRFS Assistant snapshot root restore you’ll notice the “backup” it made of root before it restored your snapshot. This will be named @backup<date/time><description_ you_ gave_if_you_gave_one>

  1. if you don’t know the snapshot you want to restore you can list out the snapshot directory to see them with the date/time stamp as follows:

sudo ls -al @/.snapshots

  1. If you want to see the snapper metadata for a particular snapshot (i.ie. description snapper applied to it) then type:

sudo cat ./@/.snapshots/<snapshot#>/info.xml

(note the <snapshot#> is the number of the snapshot gotten from listing the directory in step 6.

  1. Once you have determined which snapshot you want to restore (for the purposes of this tutorial we will say the snapshot you want to restore is snapshot# 123) then type the following:

sudo mv @ @.broken

This will rename your existing root subvolume that isn’t booting to a temporary name. It is VERY important that you don’t delete this subvolume yet. While I don’t think the BTRFS filesystem would let you delete it since it has the subvolume ./snapshots buried in it I am not going to test this theory, if you are able to delete it before moving your snapshot subvolume out of it you would lose ALL of your root snapshots.

  1. All Snapper snapshots are Read-Only by default so next we need to make a RW snapshot of the snapshot we want to restore as root using snapshot 123 as our sample the command would be:

sudo btrfs subvolume snapshot @.broken/.snapshots/123/snapshot @

  1. The final step is to move your snapshot subvolume from the broken root directory into your new root snapshot as follows:

sudo mv @.broken/.snapshots @

This will move the .snapshots subvolume into your new Root directory

  1. Final step is to reboot your system and you will boot into the restore snapshot.
3

This guide is absolutely fantastic! :tada:
As a user of Btrfs and Snapper, I truly appreciate the clarity and logical structure of the steps provided.
The detailed explanation of working with subvolumes and commands makes the entire process straightforward, even for less experienced users. :computer::sparkles:

The emphasis on not deleting the original root subvolume and highlighting potential risks is incredibly valuable and adds an essential layer of safety. :rotating_light:
The way the guide explains everything – from working with UUIDs to restoring snapshots – is both intuitive and practical. :hammer_and_wrench:

The clarity and attention to detail make this guide an outstanding resource for the community. Such guides are exactly what’s needed to make efficient use of Btrfs and Snapper.

:star2: Amazing work! :clap:

4

I specifically made this account to thank you for this, this guide worked flawless to help me recover the os. Thank you!

5

I can’t thank you enough for your guide. The only thing I would note, is for those of us who don’t know, the snapshot number is the number at the end of the line in the list at step 6 vs the number at the beginning of the line. I didn’t know but a couple trial and error’s and it was apparent.

Thank you Thank you Thank you!!!
And thanks Cachy for a great OS!!!

Now I’ma figure out how to install a bootloader with snapshot entires already there versus the “speedier” bootloader I used. In cachy’s defense I was warned by the cachy installer.

6

Glad it was helpful.

The CachyOS team added the Limine bootloader in the March release of the ISO and actually defaults with it fully setup for snapshot rollback out of the gate.

I’m looking forward to checking it out when I have the chance on one of my systems. Sounds like it very well may be the best of both worlds (“speedier” bootloader like systemd boot with extra features like snapper snapshot rollback without the extra baggage a legacy bootloader like grub carries with it).

7

Earlier this year I used this guide to successfully restore from a snapshot. This time around I get the following error on step 9:
ERROR: Could not statfs: No such file or directory
When browsing to root/@.broken/@/ I see that the .snapshots directory is locked. Permissions says ownership: root
I also do not have a @.broken/.snapshots directory. What am I doing wrong?

Edit: The fix was to move @ back into the UUID directory and this time use mv to rename it @.broken2 and continue from there.

8

Yeah, I guess I should have discussed cleanup afterwards.

The purpose of moving the existing @ snapshot to @.broken is so that you could undo what you were doing if trying out a particular snapshot made things worse or also didn’t work.

I named the snapshot @.broken so that I would know that was the snapshot that didn’t work and once things did work I could remove it so it wasn’t taking up space anymore.

So I guess the final step after you have verified everything is working with your rolled back snapshot you should remove the @.broken snapshot from your system (or in your case now the @.broken and the @.broken2) or else the next time you try to do this the older broken root snapshots will still be around and won’t really serve any purpose other than potentially taking up extra space as time goes on.

I can’t add that final step to the original post but It’ll be here in the thread for future reference.

If you want to do it through the BTRFS Assistant go to the subvolumes tab and you will find the @.broken subvolume there. You may want to check the include Timeshift and Snapper Snapshots box at the bottom just to make sure that the /.snapshots/ subvolume is not located in that directory tree still before removing it.

9

Thanks for a useful guide!

I decided to try Limine as it’s easier to configure than Grub2. Was impressed with how well it integrated with snapshots and how the GUI told me to restore the snapshot it detected me booted into. And then I rebooted and no more Limine, just a black screen of death. Pity, it was going so well!

Edit: After using the live USB to boot and look at /boot/limine.conf - I decided to revert the “quiet: yes” back to “no” (I like a quiet boot and a timeout of 1 so most of the time it’s transparent unless I hammer the down arrow key). Boom, working again. Something about restoring a snapshot doesn’t like quiet mode.

10

I made this account just to thank you

I have installed cachyos few days ago

And i haven’t faced any problems but your tutorial is a life saves for beginners like me

I will surely need it in the future

11

A really Important thing is, don’t forget to do
limine-mkinitcpio after you’ve done everything, otherwise, you’ll be stuck in the terminal view for some time

12

My BTRFS Assistant says “failed to delete @.broken subvolume” any idea what’s causing that?

Also thanks so much for the excellent guide.

Also, (not trying to be sarcastic, legitimately asking) what is snapper supposed to be for if rolling back a snapshot from 10 minutes ago to undo an update renders the system unbootable. I’m willing to accept that I did something wrong, or that this tool is for advanced users and I’m not one of them. But it doesn’t seem to be talked about like that in general. So what’s the point if using it how most people would think to use it breaks the whole system?

13

Good to know. I’m new to CachyOS (and Arch-based distros in general), love to tinker, and wanted to get a jump on system recovery. :laughing: Thanks for that.