Slow Boot and Crashing After Upgrade

Hello, cachyos was working perfectly 1 week ago, after i did some updates it is taking such a long time to boot, before it was 4-5 seconds, now it is taking 1-3 minutes.

And i just had an instance of after 10 minutes booted the GPU/System i dunno just crashed and i got a black screen and could not do anything.

I tried to read the

sudo dmesg

Here is the output for my

sudo cachyos-bugreport.sh
❯ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @1min 13.529s
└─multi-user.target @1min 13.529s
  └─docker.service @1min 12.367s +1.162s
    └─network-online.target @1min 12.366s
      └─NetworkManager-wait-online.service @1min 6.366s +5.999s
        └─NetworkManager.service @1min 6.041s +323ms
          └─basic.target @1min 6.040s
            └─dbus-broker.service @1min 6.021s +17ms
              └─dbus.socket @1min 6.007s
                └─sysinit.target @1min 6.006s
                  └─systemd-resolved.service @1min 5.954s +51ms
                    └─run-credentials-systemd\x2dresolved.service.mount @1min 5.969s

Please lat me know if there is missing info or if i need to provide more details or logs.

Thanks.

Some ideas:

  1. The BIOS has an update available (F40d)

  2. It looks like something with USB is going on?

    [   10.707541] usb 1-1.1.3: device descriptor read/64, error -110
    

    That gets repeated further on with seconds in between them.
    Is there something plugged in that’s not working correctly or maybe a cable issue?

  3. This looks weird:

    [   21.941494] systemd[1]: systemd-journald.service: start operation timed out. Terminating.
    

    and

    [   32.191849] systemd[1]: systemd-journald.service: Killing process 448 (systemd-journal) with signal SIGKILL.
    

    Is that the systemd journal service getting killed? :face_with_raised_eyebrow:

  4. Both can also be seen in the journal:

    nov 18 17:12:53 MarioDesktop kernel: usb 1-1.1.3: device descriptor read/64, error -110
    nov 18 17:12:53 MarioDesktop systemd[1]: systemd-journald.service: start operation timed out. Terminating.
    nov 18 17:12:53 MarioDesktop kernel: usb 1-1.1.3: device descriptor read/64, error -110
    nov 18 17:12:53 MarioDesktop systemd[1]: systemd-journald.service: State 'stop-sigterm' timed out. Killing.
    nov 18 17:12:53 MarioDesktop kernel: usb 1-1.1.3: device descriptor read/64, error -110
    nov 18 17:12:53 MarioDesktop systemd[1]: systemd-journald.service: Processes still around after SIGKILL. Ignoring.
    nov 18 17:12:53 MarioDesktop kernel: usb 1-1.1.3: device descriptor read/64, error -110
    nov 18 17:12:53 MarioDesktop systemd[1]: systemd-journald.service: State 'final-sigterm' timed out. Killing.
    
  5. Maybe this still works to get the log of the failing service:

    $ journalctl --unit systemd-journald.service
    
  6. Does

    $ systemd-analyze blame
    

    show a stalling unit? Times should be below 1 second, usually.

1 Like

Thanks @deex

I will download and update the BIOS.

Regarding the USB i will remove some and try to boot with only the monitor and add ony by one to see which one is giving this error.

Output of journalctl --unit systemd-journald.service

Output of systemd-analyze blame

9.561s systemd-journald.service
5.999s NetworkManager-wait-online.service
1.162s docker.service

Seems like the issue are with systemd journal and the network manager

Seems like every boot journald tries to move the log file out of the way:

File /run/log/journal/781556cc27f54311af2f9671f579c7bd/system.journal corrupted or uncleanly shut down, renaming and replacing.

Here it’s freshly created:

Runtime Journal (/run/log/journal/781556cc27f54311af2f9671f579c7bd) is 16M, max 1.5G, 1.5G free.

And here it corrupts again, in the same boot:

Received client request to flush runtime journal.
File /var/log/journal/781556cc27f54311af2f9671f579c7bd/system.journal corrupted or uncleanly shut down, renaming and replacing.

So I wonder what’s going on there.

Could you check the filesystem?

You could also try to remove all logs - make sure you do not need those

But now I’m starting to guess.

9.561s systemd-journald.service
5.999s NetworkManager-wait-online.service
1.162s docker.service

Shows journald taking its good time trying to sight the killing.
Network has strange durations sometimes.
Docker also is pretty heavy, so it’s okay.

I am also having issues with a really long boot time. Bazzite and Mint both take less than a minute. CachyOS can take a handful of minutes.

systemd-analyze blame

I get the following:

989ms systemd-binfmt.service
941ms systemd-resolved.service
927ms systemd-timesyncd.service
442ms NetworkManager.service
279ms dev-nvme0n1p2.device
191ms dev-zram0.swap
131ms user@1000.service
119ms systemd-udev-trigger.service
 83ms upower.service
 78ms plymouth-start.service
 57ms systemd-zram-setup@zram0.service
 56ms systemd-udevd.service
 56ms polkit.service
 56ms systemd-journald.service
 53ms systemd-tmpfiles-setup.service
 52ms lvm2-monitor.service
 47ms udisks2.service
 46ms systemd-hostnamed.service
 45ms systemd-tmpfiles-setup-dev-early.service
 44ms accounts-daemon.service
 44ms systemd-fsck@dev-disk-by\x2duuid-7622\x2d4A45.service
 41ms systemd-logind.service
 41ms systemd-tmpfiles-clean.service
 37ms power-profiles-daemon.service
 37ms systemd-boot-random-seed.service
 34ms systemd-journal-flush.service
 29ms dbus-broker.service
 27ms systemd-vconsole-setup.service
 27ms boot.mount
 24ms systemd-journal-catalog-update.service
 24ms plymouth-quit-wait.service
 24ms plymouth-quit.service
 22ms systemd-tmpfiles-setup-dev.service
 21ms modprobe@dm_mod.service
 21ms modprobe@loop.service
 20ms wpa_supplicant.service
 20ms systemd-remount-fs.service
 20ms systemd-modules-load.service
 19ms home.mount
 18ms systemd-rfkill.service
 17ms avahi-daemon.service
 17ms systemd-userdbd.service
 16ms root.mount
 16ms systemd-user-sessions.service
 15ms srv.mount
 14ms user-runtime-dir@1000.service
 14ms dev-hugepages.mount
 14ms proc-sys-fs-binfmt_misc.mount
 14ms dev-mqueue.mount
 13ms plymouth-read-write.service
 13ms sys-kernel-debug.mount
 13ms sys-kernel-tracing.mount
 12ms kmod-static-nodes.service
 11ms modprobe@configfs.service
 11ms jupiter-biosupdate.service
 11ms modprobe@drm.service
 11ms modprobe@fuse.service
 10ms rtkit-daemon.service
  9ms systemd-random-seed.service
  9ms systemd-udev-load-credentials.service
  9ms systemd-sysctl.service
  7ms jupiter-controller-update.service
  6ms systemd-update-utmp.service
  5ms var-cache.mount
  4ms systemd-update-done.service
  4ms var-log.mount
  4ms sys-kernel-config.mount
  4ms var-tmp.mount
  3ms sys-fs-fuse-connections.mount
  3ms tmp.mount

I also ran a sudo cachyos-bugreport.sh and systemd-analyze critical-chain and this is what I got for them.

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @2.876s
└─multi-user.target @2.875s
  └─plymouth-quit.service @2.850s +24ms
    └─systemd-user-sessions.service @2.832s +16ms
      └─network.target @2.830s
        └─wpa_supplicant.service @2.809s +20ms
          └─basic.target @2.363s
            └─dbus-broker.service @2.332s +29ms
              └─dbus.socket @2.329s
                └─sysinit.target @2.328s
                  └─systemd-resolved.service @1.387s +941ms
                    └─run-credentials-systemd\x2dresolved.service.mount @2.272s

~

Hey @jonstump my problem was related to the USB and maybe the BIOS version, after i updated my BIOS and removed a USB HUB from my PC the boot time is as fast as before

1 Like

Updating the bios has no change. Still a 3+ min boot time.

I don’t have anything that I can remove from my setup usb wise.

I see my motherboard’s splash screen sometimes 3+ times while CachyOS is trying to boot. Again this is not typical behavior for my computer with any other linux distro. I’m very confused by this as well.

Some observations:

  • You are using the linux-cachyos-deckify kernel, which, according to CachyOS Kernel is geared towards the Steam Deck. Do you have one of those? Maybe try using the default kernel linux-cachyos.
  • Make sure you did not install the CachyOS Handheld Edition on a normal computer
  • Service startup-times look good, so I’d wage a guess it’s not related to a systemd service stalling
  • The journalling service has problems but seems to catch himself later on:
    [   78.130947] systemd-journald[428]: File /var/log/journal/b6c81b801b9f42f6aed127ae7cd3ae67/system.journal corrupted or uncleanly shut down, renaming and replacing.
    
  • Steam helper is issuing a command which is not understood by your CPU:
    [  905.763921] traps: steamwebhelper[20060] trap invalid opcode ip:75bea8acabcf sp:7ffc58cbdad0 error:0 in libcef.so[66c9bcf,75bea47ed000+a36d000]
    
    Maybe an v4 or otherwise incompatible package? Your AMD Ryzen 5 5600G is Zen 3. Maybe just sloppy programming from their side. Maybe the file is corrupted. Maybe just unrelated but it’s probably crashing.
  • There is another process segfaulting:
    [ 1814.697888] vesktop[29502]: segfault at 1 ip 000073997ede6dcc sp 00007ffdd3d2b000 error 6 in gameoverlayrenderer.so[19dcc,73997ede3000+33000] likely on CPU 0 (core 0, socket 0)
    
    Again something from Steam.
  • USB device 1-9 has some problems, I could not figure out what it is:
    [   17.659372] usb 1-9: device descriptor read/64, error -110
    [   17.733408] hid-generic 0003:2DC8:5201.0007: input,hidraw5: USB HID v1.11 Keyboard [8BitDo 8BitDo Retro Keyboard Receiver] on usb-0000:09:00.3-1.3/input2
    [   17.735628] elecom 0003:056E:010C.0004: Fixing up Elecom mouse button count
    [   17.735745] input: ELECOM TrackBall Mouse HUGE TrackBall as /devices/pci0000:00/0000:00:08.1/0000:09:00.3/usb3/3-1/3-1.1/3-1.1:1.0/0003:056E:010C.0004/input/input9
    [   17.786421] elecom 0003:056E:010C.0004: input,hiddev99,hidraw6: USB HID v1.11 Mouse [ELECOM TrackBall Mouse HUGE TrackBall] on usb-0000:09:00.3-1.1/input0
    [   33.531371] usb 1-9: device descriptor read/64, error -110
    [   33.813325] usb 1-9: new full-speed USB device number 5 using xhci_hcd
    [   49.403369] usb 1-9: device descriptor read/64, error -110
    [   65.275369] usb 1-9: device descriptor read/64, error -110
    [   65.384356] usb usb1-port9: attempt power cycle
    [   65.825321] usb 1-9: new full-speed USB device number 6 using xhci_hcd
    [   70.651598] usb 1-9: Device not responding to setup address.
    [   70.881000] usb 1-9: Device not responding to setup address.
    [   71.086322] usb 1-9: device not accepting address 6, error -71
    [   71.086410] usb 1-9: WARN: invalid context state for evaluate context command.
    [   71.261322] usb 1-9: new full-speed USB device number 7 using xhci_hcd
    [   76.087717] usb 1-9: Device not responding to setup address.
    [   76.320998] usb 1-9: Device not responding to setup address.
    [   76.526322] usb 1-9: device not accepting address 7, error -71
    [   76.526409] usb 1-9: WARN: invalid context state for evaluate context command.
    

I want my gaming PC to boot into game mode and I did not see an option to do that with the standard desktop version of catchyos. Hence why I installed the handheld version. Is there no way I can install cachyos and have it behave like a steam deck?

What do you mean with “boot into game mode” ?

CachyOS

Which is intended for handhelds, and explicitly marked as experimental and it is explicitly stated you can expect broken things: Readme.

It also has firmware and quirks for several handheld gaming consoles.
This edition is not intended for desktops and not tailored for them.
It is for handhelds…

Well, what exactly do you desire here?


The gamemode stuff can be installed with:

sudo pacman -S gamemode

The scx schedulers can be installed with:

sudo pacman -S scx-scheds

and started with e.g.:

sudo scx_lavd

Available are currently:

scx_bpfland   scx_flash     scx_lavd      scx_loader    scx_pair      scx_rlfifo    scx_rusty     scx_userland  
scx_central   scx_flatcg    scx_layered   scx_nest      scx_qmap      scx_rustland  scx_simple  

More info here.

You can also make them autostart via systemd units.

What do I mean by have my computer act like a steam deck?

I’m a little surprised by this question. A Steam Deck boots into Valve’s game mode, that is not a desktop environment. That is what I want. My gaming pc is first and foremost for gaming. I don’t want to boot into a DE and then boot into big picture mode. I want to boot my PC and boot straight into gaming mode just like the steam deck. I want it to recognize the keyboard shorts, for gaming mode just like a steam deck. Which CachyOS also doesn’t do.

I just want my gaming PC to be a giant steam deck.

Pretty sure the handheld image won’t work like that on a normal desktop. As its not meant for that.

Okay, I understand but I am sorry to say I can not help you any further.

I do not game on a console and have never even had a gaming console, which includes the Steam Deck so I have no clue how their UI or UX is supposed to look like.

My world of gaming consists of a lightweight desktop environment (not even gamemode or gamescope), wine (not even proton) and that’s it.

With that I fire up Star Citizen, Elite Dangerous, Stalker, Cyberpunk and probably most other heavy and modern stuff.

I have no Steam account (GOG and direct customer only) so I’m out of this, too.

It looks like what you want is exactly what I try my hardest to avoid.
No hard feelings, ask me anything you want, I’m just out of my expertise. :beers:

1 Like

One small, last addition:
I don’t know if you read my analysis above at all, but your CPU does not seem to even be compatible at the hardware level with the Handheld image and its repositories, as processes segfault and execute illegal opcodes on it.

I briefly tried the normal desktop version. I immediately swapped when I saw no way to default to game mode with it. It also took 3+ minutes to start. It can’t recall everything it hung on, but it hangs on a lot of stuff during boot.

I guess what I’m getting out of this is that CachyOS is not the distro that it seems to be when I read about it, which was “basically bazzite but arch and mutable”

Well, every distro has its target audience.
There is more than one set of trousers in the world :slight_smile:
Everyone finds their style.
That does not mean CachyOS is not what it says it is - it is different from what you thought it is for you.

I might describe it as a desktop distro with a leaning toward supporting gaming. It is not a Gameboy/PS4/Xbox in Disguise.

That is Peters statement.

1 Like

I wanted to mention that I didn’t mean to imply that CachyOS itself describes it this way, this was more the vibe I got from reading and watching stuff about CachyOS when doing research. In fact…

Here’s an example in this very thread. Which basically matches what I’ve consumed about other’s experiences with CachyOS. However I wouldn’t describe it as such and I agree that CachyOS doesn’t either from what I’ve actually seen on their website and now experienced.

I’d describe CachyOS as Arch but with packages a little bit more performant because of the platform optimizations.

Not a gaming distro, not a convenience distro, not an exactly user friendly distro.

It is Arch. Arch is hard (1). With Arch you need to know your way around.
The documentation is awesome and detailed to understand how stuff works.

Arch is not a distribution off the shelf but a very modular, complex machine, where you put stuff together yourself. You break it, you keep it and you have to build it in the first place.

While CachyOS, and Arch, to an extend, have become much much more user-friendly and have gotten rid of many footguns and are more approachable than ever, this meme still holds true:


(1) Until you try Linux from Scratch :wink:

1 Like