Update broke my system and snapshot restore doesn't seem to work

[turwaith]

Okay so this morning, a pacman -Syu seemed to have broken my system. Fair enough. Upon boot, it entered emergency mode, but in a loop. It keeps saying “Press Enter to Continue”. If I press enter, it will load for minutes, and then fall back to that prompt again.

Now I have limine as boot loader, snapper installed and everything, so every pacman transaction creates a snapshot. After failing to boot, I selected a snapshot, booted from it, perfectly fine. I then pacman -Syu’s again, hoping maybe the failing package has been hot fixed already. It hadn’t, and boot failed again. But now, when I wanted to boot into a snapshot, they pretty much all disappeared. Now this is probably snapper cleaning up because of storage, but now my snapshot entries are gone.

So I boot a live usb, mount the file system, and all most snapshots are there, as /@/.snapshots. Cool, so I renamed @ to @.bak and restored the working snapshot from before with

sudo btrfs subvolume snapshot /mnt/@.broken/.snapshots/12/snapshot /mnt/@

I 100% know that this snapshot works, but even after restoring it from the live usb and then trying to boot again, it will again fall into the error and not boot.

The error I’m encountering says

Cannot open access to console, the root account is locked

and then after pressing enter

Reloading system manager configuration
Starting initrd.target

It will stay like this for about 2-3 minutes and then go from the start again, wanting me to press enter to continue.

I’m out of ideas. How can I restore my system?

[Zesko]

@turwaith

So I boot a live usb, mount the file system, and all most snapshots are there, as /@/.snapshots. Cool, so I renamed @ to @.bak and restored the working snapshot from before with

sudo btrfs subvolume snapshot /mnt/@.broken/.snapshots/12/snapshot /mnt/@

That alone is not enough. Restoring a Btrfs snapshot with Limine requires a few more steps:

1. Boot from the live USB.
2. Move child subvolumes (such as .snapshots) from @.broken back to @.
3. arch-chroot or cachy-chroot into your system.
4. Run limine-update to reinstall the kernels.

---

@turwaith

After failing to boot, I selected a snapshot, booted from it, perfectly fine. I then pacman -Syu’s again, hoping maybe the failing package has been hot fixed already. It hadn’t, and boot failed again.

Did you see a “Restore now” popup after booting the snapshot?
Which desktop environment do you use?

@turwaith

But now, when I wanted to boot into a snapshot, they pretty much all disappeared. Now this is probably snapper cleaning up because of storage, but now my snapshot entries are gone.

The snapshot boot entries should not disappear. Snapper itself refuses any action on a read-only snapshot with a writable overlay.
Not sure what triggered all snapshot boot entries to be removed, since limine-snapper-sync has safeguards to prevent accidentally deleting all snapshot boot entries when running from an invalid Btrfs layer.

@turwaith

I 100% know that this snapshot works, but even after restoring it from the live usb and then trying to boot again, it will again fall into the error and not boot.

The snapshot itself is fine. The problem is that only the Btrfs snapshot was restored, while the matching kernels stored outside Btrfs were not. That results in a kernel mismatch.
limine-snapper-restore is the only CLI that restores both the snapshot and the matching kernels.

[turwaith]

First of all, thanks for your answer, definitely helpful for the next time something happens and I have to restore from a live system. After burning through all my available tokens on claude opus I could resolve the issues and boot again. Probably only worked in the end because I refused a lot of destructive actions claude suggested. I’m back on a working system, but I have to re-init snapper because the process killed my snapshots subvolume, or I just lost it.

@Zesko

That alone is not enough. Restoring a Btrfs snapshot with Limine requires a few more steps:

1. Boot from the live USB.
2. Move child subvolumes (such as .snapshots) from @.broken back to @.
3. arch-chroot or cachy-chroot into your system.
4. Run limine-update to reinstall the kernels.

I did not do that, might have worked from the start if I did. I’ll write that down in my notes.

@Zesko

Did you see a “Restore now” popup after booting the snapshot?
Which desktop environment do you use?

Yes I executed the restore every time it popped up. I’m on kde plasma wayland.

@Zesko

The snapshot boot entries should not disappear. Snapper itself refuses any action on a read-only snapshot with a writable overlay.
Not sure what triggered all snapshot boot entries to be removed, since limine-snapper-sync has safeguards to prevent accidentally deleting all snapshot boot entries when running from an invalid Btrfs layer.

Yeah that’s also still a mystery to me. I might have configures something wrong when I set up the snapper stack? No idea how it happened, but the only available snapshot in limine was the latest ‘after’ snapshot, which is basically at the same state as the current running system.

@Zesko

The snapshot itself is fine. The problem is that only the Btrfs snapshot was restored, while the matching kernels stored outside Btrfs were not. That results in a kernel mismatch.
limine-snapper-restore is the only CLI that restores both the snapshot and the matching kernels.

Yep that was probably my issue after initially restoring the snapshot. But I then started to manually edit the /boot/limine.conf file, and did mkinitcpio -P, reinstalled the kernel via pacman etc… and in the end, there were a duplicate root UUID in /etc/kernel/cmdline. No idea how it got there, but that’s pretty certainly my fault.

Man I fucking hate LLMs, the amount of times I read “Bingo, now we got it. Do xxx to resolve all issues now” yesterday. And the actions that followed probably made it worse.

What I have to do now is clean up the snapshot mess, make sure I’m not actually on a snapshot anymore (there was no popup to restore, but how can I be sure?) and re-init snapper, because on login it complained that it could not find the snapshot subvolume.