Hi everyone,
I have been running CachyOS since August and absolutely loving it. I came from almost 5 years running Manjaro and I’m daily asking myself why I didn’t jump that ship earlier to get closer to a base Arch installation.
Anyway, so I just updated my system 2 days ago (hadn’t done so for about a week so there were like 200+ updates and afterwards my proton vpn wasn’t working. It was late at night when I did the update so I just decided to just rollback my system using BTRFS Assistant because I just didn’t want to mess with figuring out what the issue was right then and there.
So for the first time the system no longer booted when I rolled back to my pre-update snapshot. I’ve rolled back updates in the past using BTRFS Assistant and there has never been an issue but for whatever reason after rolling back to my pre update snapshot. Instead when I booted I was presented with the following message:
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
Press Enter for maintenance
(or press Control-D to continue):
So I guess this is what I get for being lazy and not figuring out the issue with my vpn client lol,
Anyway I now had to figure out how to manually restore a root snapshot so I could get my system working again so I dove into the arch wiki entry on snapper which gives and example of restoring home directory from a snapper snapshot so not really root but I figured I’d give the step by step here since it took a little playing around to wrap my head around how snapper lays out subvolumes for performing root snapshots which can be a little confusing if you aren’t familiar with BTRFS subvolumes. The most important thing that can be hard to conceptualize especially when subvolumes are located in sub-directories is that snapshotting is not recursive, so a subvolume or a snapshot is effectively a barrier and no files in the nested subvolumes appear in the snapshot.
So here is the step by step instructions to recovery using the CachyOS installation ISO to restore a root snapshot to a system that no longer boots. So for those of you using the CachyOS defaults of Systemd bootloader and setup Snapper here is how you recover from an update that makes your system no longer boot to your DE.
First things first, boot off the CachyOS installation USB
- Open up dolphin and select your BTRFS drive if you did a default CachyOS installation it should be named root, otherwise select the appropriate drive and this will automatically mount it when you click it.
- Click the > at the top of the dolphin window to see where the drive was mounted at, which will be (/run/media/liveuser/UUID) make a note of the first couple of digits of the UUID
- launch Alacritty terminal
- type the following:
cd /run/media/liveuser/UUID
(basically type the first couple of digits of the UUID then press tab to complete it unless you just really want to type out the whole UUID )
- view your directory to make sure you are in the correct place as the sample commands I am using are assuming you are in the root directory of your BTRFS drive.
sudo ls -al
This will give you a directory of the actual root of your BTRFS file system if you changed your directory correctly. You should see the various subvolumes such as @ (which gets mounted as your current root directory, along with a few others like @Home, @log, etc…) Also note if like in my case I’m recovering from a BTRFS Assistant snapshot root restore you’ll notice the “backup” it made of root before it restored your snapshot. This will be named @backup<date/time><description_ you_ gave_if_you_gave_one>
- if you don’t know the snapshot you want to restore you can list out the snapshot directory to see them with the date/time stamp as follows:
sudo ls -al @/.snapshots
- If you want to see the snapper metadata for a particular snapshot (i.ie. description snapper applied to it) then type:
sudo cat ./@/.snapshots/<snapshot#>/info.xml
(note the <snapshot#> is the number of the snapshot gotten from listing the directory in step 6.
- Once you have determined which snapshot you want to restore (for the purposes of this tutorial we will say the snapshot you want to restore is snapshot# 123) then type the following:
sudo mv @ @.broken
This will rename your existing root subvolume that isn’t booting to a temporary name. It is VERY important that you don’t delete this subvolume yet. While I don’t think the BTRFS filesystem would let you delete it since it has the subvolume ./snapshots buried in it I am not going to test this theory, if you are able to delete it before moving your snapshot subvolume out of it you would lose ALL of your root snapshots.
- All Snapper snapshots are Read-Only by default so next we need to make a RW snapshot of the snapshot we want to restore as root using snapshot 123 as our sample the command would be:
sudo btrfs subvolume snapshot @.broken/.snapshots/123/snapshot @
- The final step is to move your snapshot subvolume from the broken root directory into your new root snapshot as follows:
sudo mv @.broken/.snapshots @
This will move the .snapshots subvolume into your new Root directory
- Final step is to reboot your system and you will boot into the restore snapshot.