Hi! I’m a new CachyOS fan but sadly I’m also a bit of a zfs nerd. With @cscs’s help on Friday I got CachyOS installed onto a dedicated nvme stick and I’m really impressed. It’s so good!!
But of course now I want more…
[First, some feedback on the zfs installer- This is a laptop machine so I opted for enabling zfs encryption. It everything worked fine except in my case I ended up with /boot pointing to a partition that appears to be not only EFI bootable but also holds the initramfs and vmlinuz bootstrap files.
This is clever! If I’m understanding correctly (??) it’s a way to boot the kernel without having to load zfs keys first via zfsbootmenu or whatever. However it also leaves these boot files highly vulnerable outside the encryption envelope. I’m not sure is this is actually a good thing; I don’t think it’s the way zfsbootmenu is supposed to work (disclaimer, I’m a noob)]
On to my ask: My eventual goal is I’d like to have CachyOS live alongside other bootable distros within zfs. ZFSBootMenu supports this but at this point CachyOS apparently doesn’t but I think it easily could. This seems like the future to me.
All the installer needs to do is allow zfs to be selected without the erase-disk option and then ask where to be installed in zfs. In my case that would be zroot/ROOT/cos on my other nvme stick.
Of course I tried to zfs send/receive zpcachyos/ROOT/cos/* into zroot/ROOT/cos which is when I realized /boot was empty and copied the files from the efi partition in. But the thing won’t boot (it was a longshot…), it panics unable to load the root fs. I think initramfs isn’t configured correctly for that. But I’m at the edge of my understanding here. I’m new to arch and even in my comfort-zone (which is debian. sorry!) all of this booting stuff is challenging.
[My sales pitch to the CachyOS developers for this feature: Heyy, wouldn’t it be nice to be able to have multiple versions of CachyOS installed as easily supported by zbm? You could flip back and forth between distros and without all that fixed partitioning stuff!]
But in the absence of direct support for this feature in the installer, does anyone know of a guide to install “zfs root on CachyOS”? I can’t even find an official guide for Arch! Or maybe some hints on how to fix that broken copy that I moved?
You can get around that with a semi-manual install. And yes you want your boot files in the zfs root for zbm to work properly and for security. What I did for encrypted zfs root with zfsbootmenu
Picked refind for boot manager
Chose manual partitioning rather than automatic and set my EFS partition to /boot/efi
Use the rest for zfs mounted at /, encrypted
(The installer correctly warned that this might not be bootable, but I didn’t care…)
Before reboot I mount the system partition and manually copy one of tbe ZBM binaries there
Need to set a couple of filesystem properties
sudo zfs set org.zfsbootmenu:rootprefix="root=ZFS=" zpcachyos (without this it misdetects the root prefix and won’t boot)
sudo zfs set org.zfsbootmenu:commandline="rw quiet loglevel=4" zpcachyos and any other kernel parameters (if you don’t set this the default is read only…)
reboot - on the first boot only hit “ctrl-d” on your boot pool to mark it as the boot filesystem. alternatively you can use zpool set on the command line to do this. Snapshots won’t work until the bootfs is set.
Make sure zfs-utils is installed pacman -S zfs-utils
the zfs pacmanhook is in aur so install (paru pacman-zfs-hook) It’s important that the zfs-utils is installed and boot pool is marked first - if not it will break pacman…)
sudo zfs set keylocation=file:///etc/zfs/zroot.key zpcachyos You need to rebuild the intramfs (mkinitpcio -S, or will happen automatically when you install a new kernel) to add the key for this to become effective, but if you don’t it should just keep asking for your password twice at boot.
You can optionally save the key on a different filesystem that you specify with the org.zfsbootmenu:keysource property
Thanks so much for the super helpful info @mattsteg - Now that I think of it, I don’t think the system is booting thru zbm like it should. I didn’t even think of that until now. It’s just going from refind into cachy.
The system was installed from a custom ISO @cscs built for me that solved some hardware incompatibility issues I was having, but did display a warning (I ignored) it was an old distro. I also noticed the ZFS install options were different & less complete - the zfs dataset path setting optoins weren’t there. Maybe the installer version is old and it didn’t set up zbm correctly? I might try to overlay the new installer os kernel from @cscs on the official installer image and see what happens. My ideal config is refind → zbm → cachy so I can also multi-boot into Windows on the second nvme when necessary. With your clues I can either transplant cachy again or try a (semi-)manual install like you guys suggest. I was hoping to keep the thing in a maintainable state (ie mostly stock config with minimal bells & whistles) by transplanting it but that may not be possible.
I’ll update here if there’s anything interesting to report.
Thinking about it, zfsbootmenu itself lives outside the encrypted envelope and so it’s also vulnerable to being trojaned with a password stealer or whatever. So that makes my point about that issue moot- the vulnerability is just moved around. I guess without secure-boot (which I’m not a big fan of) this is just a reality.
Just to clarify about the image previously provided..
I did not make any removals.
In fact I made sure to include linux-cachyos-zfs despite it technically being out of scope, because the module had obviously been included with the already present linux-cachyos-lts-zfs. I was trying to retain/duplicate all features but just add the current (6.16) kernel.
I made no changes to the installer.
I also do not know where the ‘old distro warning’ could have come from or what that was in reference to. The image created the other day would technically have been more up to date than anything officially available.
If you care enough to encrypt your data you should probably set up secure boot…ideally without microsoft keys if your hardware allows (but proceed with caution)
Easy to set up
There aren’t dataset path selections. Well there are in a config file I guess but nothing exposed to the user.
You can just drop refind into the setup I described. Indeed I shoehorned cachy into a dual boot with an existing small windows EFS
I regret implying anything negative here. However just so we’re clear I took a moment to start the installer as supplied and here’s a phone camera shot of the old version warning I referred to
I’ll try to post a shot of the ‘missing’ zfs features I referred to if/when I see it again
update- Ok, I’m not seeing the zfs features fields that I claimed to earlier. Of course the machine froze several times while taking a look. This hardware certainly doesn’t like old kernels. I’ll try to retrofit new kernel into the 250828 ISO on my lunch break and report back if I ever see them again although I don’t expect to get support for such a hacked setup.
Please do - they sounded like features I’d potentially like but have never seen
Real ZBM support would be nice and seems likely to be pretty minimal work, but the default dataset layout is reasonable enough and adding ZBM is easy enough.
Hmm, as I recall it I thought at the time the design problem was the UI flow- selecting zfs requires the user choose Erase Disk first. (??) To achieve zbm multiboot with these phantom zfs dataset fields all that would be needed is not having to erase the disk, etc. but I don’t know how that would be cleanly shoehorned in to the UI, especially as zfs users are probably rather uncommon right now.
Just FYI as I recall it, I think Wednesday before I received the updated installer, these zfs datasets field, it was just a basic listbox in the install wizard that appeared when erase disk + zfs was selected. It was pre-populated with the default zpcachyos/ROOT/cos, zpcachyos/ROOT/cos/home, (etc) entries and a way to edit them that I don’t recall exactly. At the time I was impressed but didn’t change anything. Anyway I couldn’t get thru the install due to the machine freezing.
Disclaimer: My recollection could easily be wrong here, I was trying a lot of things last week and it’s all blurred together at this point.
Sorry to go off topic but squeezed for time,thought you might want to know, poking around in the 202508 ISO I see shellx64.iso on the root. this is usually (??) an artifact of xorriso, it’s a copy of the efi partition and can probably be safely removed. ymmv, imho & all the usual disclaimers its just fyi
It’s worth noting that you get some differences in the installer depending on the bootloader that you choose, so it’s possible there’s an inconsistency there driving things. It’s not in the UI flow that I encounter.
With your help I got CachyOS installed into my primary zpool.
I ended up doing it the lazy way- using zfs send/receive -r from zpcachyos/ROOT/cos to zroot/ROOT/cos and changing the rootprefix and commandline as you suggested. I was fearing landing the OS in a weird state where the post-install steps were missed ( )
There’s just one problem- zfs appears to be mounting /home to the previous zroot/home instead of Cachy’s zroot/ROOT/cos/home, which means the home directory is shared across installs. Luckily I did snapshot the home dataset so no harm done. Currently I have canmount=noauto for cachy and =on for the other distro which I think is the problem but I’m not sure how to fix it.
Can you suggest a way to separate the /home mounts? I fear setting both to =on will create an overlay mount and both to noauto will cause
@mattsteg posting on this thread just to avoid polluting the forum with new threads with trivial stuff since there doesn’t seem to be a zfs-specific category. (Are you responsible for the zfs portion of the installer?? What’s best-practice for non-bug/tuning-suggestion feedback like the following?)
I just wanted to point out that /sys/module/zfs/parameters/zfs_arc_max by default is 0 which means consume all system ram. The zfs slab allocator will respond to memory pressure, but slowly, which can cause problems. I suggest this parameter be set to 50% of system ram for workstation-target spins.
(edit) hmm, im new to arch and not sure how zfs arc will interact with zram cache. might be good, might be bad… (??) I think I’ll leave it alone for now since performance on this laptop is superb. Might circle back to this though as my goal is for cachy to act as a vm host and will can expect to get hammered with load from guests (development environments mostly) soon. This machine only has 32GB so it should be a good stress-test.
I just prefer zfs, zbm, and cachy. I’m not responsible for anything.
Historically ‘0’ on linux meant half ram. The docs still say this. There was a commit a while back to bring the BSD-style “RAM-1GB” limit. I’m not sure why that change isn’t reflected in the docs.
That’s been the case on my systems as well. Defaults seem fine. Highest routine RAM stressor is a 16GB vm on a 32GB system.
Luckily I did snapshot the home dataset so no harm done. Currently I have canmount=noauto for cachy and =on for the other distro which I think is the problem but I’m not sure how to fix it.