How to make your system crash debugging easier!

FAQ and Tutorials
1

Here i am making debugging hassle free (atleast for myself)

The Prelude

With us (Cachy User) living in bleeding edge version of all the package, we often encounter system crash, kernel panic, etc and in most case we used journalctl -b -1 p 3 or equivalent after the crash and reboot, here i came up with an idea, to use a single command everytime system boot, aka sudo dmesg -w -l err what it did is output error kernel message to terminal, but now it’s often pain to run it every single time boot.

The Idea

Can i just dump it to the .log files ??
YES, we can just use

sudo dmesg -w -l err >> log/dmesg-err.log 2>&1

to output the error kernel message to log/dmesg-err.log, and now we can just left it for good (running the terminal) then forget (do your normal daily task / gaming session)
then i just remember, can i convert it into script / service instead, so it’s hassle free for me?
YES i can!

The problem

Now lie the problem, with dmesg by default required sudo to run, we need to have workaround for this if we’re using script

  1. is modify kernel param that allows dmesg run with non-root, this was what Ai recommend, but after further check, it was kinda security risk, so i ditched it
  2. create system wide service, i guess this is the way for me

The solution

Since i decided to create system wide service, here is the step needed (do note you might require your favourite terminal text editor, mine usually use nano so i’ll use it for the step below)

  1. create a system service file first for this
sudo nano /etc/systemd/system/dmesg-monitor.service
  1. then put this script inside
[Unit]
Description=Monitoring your dmesg error and logging it to your specific dir
After=multi-user.target

[Service]
Type=simple
ExecStart=/bin/bash -c '\
    TARGET_DIR="/media/media/log/dmesg"; \
    mkdir -p $TARGET_DIR; \
    chown -R mine:mine $TARGET_DIR; \
    /usr/bin/dmesg -w -l err,crit,alert,emerg >> "$TARGET_DIR/dmesg-err-$(date +%%Y%%m%%d-%%H%%M).log" '
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
  1. now reload the daemon
sudo systemctl daemon-reload
  1. then enable the service
sudo systemctl enable --now dmesg-monitor.service

that’s all, now every time i boot my CachyOS, it will log all the kernel error or any attention required to my directory at /media/media/log/dmesg
and i can just check the log there if i encountered system crash or something, no more need to worry about running dmesg after the crash later on, and try to reproduce the crash, which is sometimes PAIN, if your encountered bug was intermittent :+1:

The breakdown

here i break some command that you might wondering (incase you’re totally don’t know the syntax is)

TARGET_DIR="/media/media/log/dmesg"

this is your intended directory for the log to be stored, DO NOTE, if the location didn’t exit root will creating the folder automatically later with next command, and the ownership is ofc will be root, if you want to avoid root ownership, you need to create the directory by yourself first, use absolute path

mkdir -p $TARGET_DIR

this command will create directory if the target didn’t exist, as previous msg said, if the folder is not created it will create with root ownership

chown -R mine:mine $TARGET_DIR

this will fix ownership to user mine (change this to your username) the log folder (do not this is only the log folder, if you use parent folder it will created with root ownership like i said in 2 previous msg), it will fix the last log after you crash, the newer ones or the current boot will always have root permission

/usr/bin/dmesg

this is the system command to check the kernel log

-w

his is parameter to make it continously monitoring the kernel log

-l err,crit,alert,emerg

this is parameter to filter the error,critical,alert and emergency type only

$TARGET_DIR/dmesg-err-$(date +%%Y%%m%%d-%%H%%M).log

this the target path that your log will be created with datestamp, example output filename is like dmesg-err-20260621-1219.log which is equal to created at 2026-06-21 at 12:19

Final word

now we already in the bottom guide, i hope with this guide (for those who use it) can easily debug your system, because from my experiences, even using journalctl sometimes not reliable for me.

Bonus

if you noticed the timestamp in dmesg-err.log was something like this [ 628.205130] and i know it might confuse you to read it, there is a workaround for this

  1. you can use -T parameter to get it shown with human readable time, so the command should be something like this /usr/bin/dmesg -w -T -l err,crit,alert,emerg but the downside of this method is, if you’re machine goes sleep, the time stamp inside the log might drift / become weird but if you don’t ever go with sleep, it’s fine
  2. convert it to human readable time with this command (do note i use Ai to get this command, if you have a better or shorter command for this, let me know)
date -d "@$(echo "$(awk '/btime/ {print $2}' /proc/stat) + [put your numerical in here]" | bc)"

so for example, date -d "@$(echo "$(awk '/btime/ {print $2}' /proc/stat) + 628.205130" | bc)" will translated to Sun Jun 21 12:28:23 PM WIB 2026

Again, thanks for reading, and

HAPPY DEBUGGING!