Console claims no space left on device

Issues & Assistance
1

System claims “no space left on device” constantly no matter the package size:

error: failed to create temporary download directory /var/cache/pacman/pkg/download-S00FMk: No space left on device
error: failed to commit transaction (failed to initialize download)

even though sudo df -h shows that there is clearly space available

Filesystem      Size  Used Avail Use% Mounted on
overlay          16G   20M   16G   1% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G   92M   16G   1% /dev/shm
efivarfs        128K   41K   83K  34% /sys/firmware/efi/efivars
tmpfs           6.1G   19M  6.1G   1% /run
none            1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
none            1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service
tmpfs            16G   18M   16G   1% /tmp
/dev/nvme0n1p6   30G   22G  8.0G  73% /root
/dev/nvme0n1p6   30G   22G  8.0G  73% /srv
/dev/nvme0n1p6   30G   22G  8.0G  73% /var/cache
/dev/nvme0n1p6   30G   22G  8.0G  73% /var/log
/dev/nvme0n1p6   30G   22G  8.0G  73% /var/tmp
/dev/nvme0n1p5  4.1G  2.5G  1.6G  61% /boot
/dev/nvme0n1p7  793G  563G  229G  72% /home
tmpfs           3.1G   96K  3.1G   1% /run/user/1000
/dev/nvme0n1p6   30G   22G  8.0G  73% /run/lss
2

That should not normally say overlay. Nor 20 MB used.

Are you running an old btrfs snapshot without restoring it perhaps? Some other unusual filesystem setup?

3

My current snapshot did completely snap and boots into emergency mode so currently I am using an older snapshot which I’ve tried to restore but it fails due to same issue of “no space left”:

image

4

It’s a virtual machine.

5

What is the output of duf?

6

image
image999×487 75.6 KB

7

Can you show output of: btrfs filesystem df /

You probably really are running out of space and/or meta data space. Your root partition is extremely small.

df / duf is not as reliable for btrfs as other filesystems, it doesn’t see the full picture.

kaede 8

image

9

Yeah, looks like it’s out of room for metadata, and the rest is already allocated to data.

It’s not actually using all of the space allocated to data though. You may be able to sort it out by just rebalancing. Try this:

btrfs balance start -dusage=10 /

It may help to first delete some old snapshots you don’t need anymore. But it might work without it.

Check the output of these commands before and after:

btrfs filesystem df /
btrfs filesystem usage /
10

So, i’ve rebooted the PC in the meantime to get a fresh .iso on a pendrive in case i need to reinstall and now / is no longer recognised as btrfs again:

Filesystem      Size  Used Avail Use% Mounted on
overlay          16G   14M   16G   1% /

which causes this error to show up:

❯ btrfs balance start -dusage=10 /

ERROR: not a btrfs filesystem: /
11
12

I have tried remounting and this is the result:

mount: /: fsconfig() failed: overlay: No changes allowed in reconfigure.
       dmesg(1) may have more information after failed mount system call.
13

Mm… it’s going to be temperamental with the metadata so full. Big “I’m not sure any of this will work” disclaimer.

Did you boot a snapshot again? Or just regular and it showed up as overlay anyway?

If the former, you could try booting normally, and hopefully it will succeed long enough to fix things.

If the latter, I suspect it auto remounted itself in read-only to prevent corruption. But we need root mounted as a proper btrfs filesystem in read-write to do anything about this.

If rebooting doesn’t fix /, you could try this:

# Remount root in read-write,
# skipping auto-balancing in case that's the reason it failed
sudo mount -o remount,rw,clear_cache,skip_balance /

If either rebooting or remounting works, and you have a proper / again, we need to stop any pending rebalance before doing anything else:

btrfs fi balance show /
btrfs fi balance cancel /

Next, we need to give btrfs some temporary space to grow into, so we can safely rebalance:

# Make a 5GB loopback device in memory
dd if=/dev/zero of=/tmp/rescue_space.img bs=1G count=5
losetup -v -f /tmp/rescue_space.img

# Give the 5GB of temp space to btrfs
# NOTE: `loopX` needs to be replaced with loop device created above
# losetup should print the correct device, if not use `df` to find it
# Careful not to use the wrong one!
btrfs device add /dev/loopX /

# Try rebalancing now 
# Start with -dusage=0, increment by 5 and try again if nothing changes
btrfs balance start -dusage=0 /

# At this point, it might also be good to delete some old snapshots
# And then maybe balance again

# Check these as you go
# In particular, you want to see `Data: total=...` decrease
# And `Metadata: total=...` go to 2.00 GB
btrfs filesystem df /
btrfs filesystem usage /

# Once confident space has been reclaimed, and FS now takes up less space
# Remove and delete temp space
# NOTE: Replace `loopX` as above
btrfs device delete /dev/loopX /
losetup -d /tmp/rescue_space.img
rm /tmp/rescue_space.img

See also:

14

Yeah, I was worried about that, it’s not just mounted read-only, but as overlay. So remounting won’t work. Not sure what mechanism is causing that though.

Are you sure you didn’t boot a snapshot?

15

So a quick update and I guess end to the thread - I did boot a snapshot, as I’ve mentioned, maybe was not clear enough which is on me and I apologise if it caused any confusion, the main system completely dropped, was booting right into emergency mode and all I could operate on were the snapshots I had from which every single one mounted the / as overlay.

Since this took more time energy than it was worth it I nuked the drive and reinstalled Cachy from scratch (this time nuking windows partition as well and allowing the installer to do its thing as previously I’ve set up the partitions manually following some probably too old tutorial).

So thanks for the effort, 99% chance I got myself into a point of no return (or close to one) through mishandling of the initial problems and their troubleshooting.

16

It might have been salvageable, but it’s iffy and would not have been simple.

The main reason for the issue was the tiny 30 GB / partition, coupled with Cachy’s default of keeping up to 50 snapshots. If you went with defaults this time, / and /home should be combined, so you’ll likely never run into this again.

Do keep an eye on btrfs fi df / and btrfs fi usage / if the disk starts approaching anywhere near full. The gui btrfs assistant also has a visual overview of that, and some buttons to do balance and scrub for you.

Consider lowering max snapshots to 20 or so in btrfs assistant.