Lots of available memory and processes dying by OOM

My CachyOS has worked flawlessly, but lately, I’m experiencing this problem:

❯ LANG=en-US free -h
               total        used        free      shared  buff/cache   available
Mem:            28Gi        13Gi       860Mi       549Mi        15Gi        14Gi
Swap:           54Gi       2.1Gi        52Gi

I have a lot of available memory (32 Gb of physical RAM)

I have zram0 swap enabled.

Some apps like ungoogled chromium and k3s are dying because of low memory.

k3s is a kubernetes cluster run locally.

In dmseg I can see some messages like this:

[15476.990366] node invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=1000
[15476.990375] CPU: 13 UID: 0 PID: 126762 Comm: node Not tainted 6.14.0-4-cachyos #1 016ff1edc592562531da7158aff85145022cf30c
[15476.990378] Hardware name: Trigkey S5/S5, BIOS 5800H506 03/29/2023
[15476.990380] Call Trace:
[15476.990382]  <TASK>
[15476.990386]  dump_stack_lvl+0x71/0x90
[15476.990391]  dump_header+0x50/0x1f0
[15476.990394]  oom_kill_process+0x1db/0x2c0
[15476.990397]  out_of_memory+0x4ab/0x5f0
[15476.990399]  __alloc_pages_slowpath+0x729/0x1050
[15476.990405]  ? __alloc_frozen_pages_noprof.cold+0x96/0xf9
[15476.990409]  ? vma_alloc_folio_noprof+0x12a/0x210
[15476.990412]  ? do_pte_missing+0xde/0x12e0
[15476.990414]  ? srso_alias_return_thunk+0x5/0xfbef5
[15476.990415]  ? __mod_memcg_lruvec_state.llvm.1293502753125321087+0x8e/0x190
[15476.990418]  ? srso_alias_return_thunk+0x5/0xfbef5
[15476.990420]  ? handle_mm_fault+0x536/0x850
[15476.990425]  ? do_user_addr_fault+0x1c3/0x410
[15476.990428]  ? exc_page_fault+0x7d/0x95
[15476.990431]  ? asm_exc_page_fault+0x26/0x30
[15476.990436]  </TASK>
[15476.990437] Mem-Info:
[15476.990441] active_anon:2255074 inactive_anon:737983 isolated_anon:0
                active_file:506056 inactive_file:2961309 isolated_file:0
                unevictable:117 dirty:1927 writeback:0
                slab_reclaimable:328230 slab_unreclaimable:108811
                mapped:664268 shmem:127372 pagetables:44954
                sec_pagetables:1205 bounce:0
                kernel_misc_reclaimable:0
                free:45370 free_pcp:8402 free_cma:0

swap config:

❯ swapon -s
Nome de arquivo                         Tipo            Tamanho         Usado   Prioridade
/dev/zram0                              partition       29680636        2187000         100
/dev/nvme0n1p1                          partition       27339772        34316           -2

Kernel parameters:

❯ cat /boot/refind_linux.conf
"Boot with standard options"    "quiet zswap.enabled=0 nowatchdog splash splash rw rootflags=subvol=/@ root=UUID=29b63aea-88f6-4bc6-b3cb-698f1a36d4d7 resume=UUID=df119baa-39a0-4654-a9ef-51f1e03f05f3"
"Boot to single-user mode"    "quiet zswap.enabled=0 nowatchdog splash splash rw rootflags=subvol=/@ root=UUID=29b63aea-88f6-4bc6-b3cb-698f1a36d4d7" single
"Boot with minimal options"   "ro root=/dev/nvme0n1p3"

I have taken some measures like reducing the size of tmpfs to 2 Gb, but no luck in resolving my problem.

More info:

  • I’m using BTRFS, encrypted over LUKS.
  • Other apps like Firefox that are more memory hungry, don’t never have problems with memory.

Do you have systemd-oomd enabled?

Thanks for your reply.
I had never heard about it.

I think I don’t have it:

❯ sudo journalctl -u systemd-oomd
-- No entries --
❯ pacman -Q|grep oom
(empty output)

Related unixstackexchange question.

There was another report of OOM and apparently it was caused by scx. Do you have it enabled and running?

Thanks for your valuable suggestion.

I don’t even know what is scx. I haven’t tuned my kernel.

I’ve discovered that all my memory problems are related with chromium.

Only chromium complains about the lack of memroy (outside or inside containers).

After a container that runs chromium dies, chromium outside a container dies contantly.

Full dmesg output:

[11436.139927] node invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=1000
[11436.139935] CPU: 12 UID: 0 PID: 117116 Comm: node Not tainted 6.14.0-4-cachyos #1 016ff1edc592562531da7158aff85145022cf30c
[11436.139938] Hardware name: Trigkey S5/S5, BIOS 5800H506 03/29/2023
[11436.139939] Call Trace:
[11436.139942]  <TASK>
[11436.139945]  dump_stack_lvl+0x71/0x90
[11436.139951]  dump_header+0x50/0x1f0
[11436.139954]  oom_kill_process+0x1db/0x2c0
[11436.139957]  out_of_memory+0x4ab/0x5f0
[11436.139960]  __alloc_pages_slowpath+0x729/0x1050
[11436.139967]  ? __alloc_frozen_pages_noprof.cold+0x96/0xf9
[11436.139971]  ? vma_alloc_folio_noprof+0x12a/0x210
[11436.139974]  ? do_pte_missing+0xde/0x12e0
[11436.139976]  ? srso_alias_return_thunk+0x5/0xfbef5
[11436.139979]  ? handle_mm_fault+0x536/0x850
[11436.139985]  ? do_user_addr_fault+0x1c3/0x410
[11436.139987]  ? exc_page_fault+0x7d/0x95
[11436.139991]  ? asm_exc_page_fault+0x26/0x30
[11436.139996]  </TASK>
[11436.139997] Mem-Info:
[11436.140001] active_anon:3058275 inactive_anon:538971 isolated_anon:0
                active_file:741187 inactive_file:1887471 isolated_file:0
                unevictable:131 dirty:3795 writeback:2
                slab_reclaimable:302381 slab_unreclaimable:108505
                mapped:676793 shmem:78652 pagetables:68529
                sec_pagetables:1225 bounce:0
                kernel_misc_reclaimable:0
                free:45399 free_pcp:6876 free_cma:0
[11436.140005] Node 0 active_anon:12233100kB inactive_anon:2155884kB active_file:2964748kB inactive_file:7549884kB unevictable:524kB isolated(anon):0kB isolated(file):0kB mapped:2707172kB dirty:15180kB writeback:8kB shmem:314608kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:1781760kB writeback_tmp:0kB kernel_stack:63392kB pagetables:274116kB sec_pagetables:4900kB all_unreclaimable? no
[11436.140010] Node 0 DMA free:11264kB boost:0kB min:32kB low:44kB high:56kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[11436.140015] lowmem_reserve[]: 0 3151 28970 28970 28970
[11436.140020] Node 0 DMA32 free:110104kB boost:0kB min:7240kB low:10412kB high:13584kB reserved_highatomic:0KB active_anon:1120540kB inactive_anon:241532kB active_file:295352kB inactive_file:906172kB unevictable:0kB writepending:56kB present:3342412kB managed:3226760kB mlocked:0kB bounce:0kB free_pcp:12316kB local_pcp:972kB free_cma:0kB
[11436.140025] lowmem_reserve[]: 0 0 25819 25819 25819
[11436.140030] Node 0 Normal free:60228kB boost:0kB min:60304kB low:86740kB high:113176kB reserved_highatomic:6144KB active_anon:11112232kB inactive_anon:1914680kB active_file:2669412kB inactive_file:6643712kB unevictable:524kB writepending:15128kB present:26971136kB managed:26439136kB mlocked:440kB bounce:0kB free_pcp:15188kB local_pcp:808kB free_cma:0kB
[11436.140034] lowmem_reserve[]: 0 0 0 0 0
[11436.140039] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
[11436.140053] Node 0 DMA32: 294*4kB (ME) 346*8kB (UME) 481*16kB (UME) 453*32kB (UME) 365*64kB (UME) 158*128kB (UE) 48*256kB (UE) 24*512kB (UME) 15*1024kB (UM) 0*2048kB 0*4096kB = 109656kB
[11436.140071] Node 0 Normal: 483*4kB (UE) 255*8kB (UME) 161*16kB (UME) 644*32kB (UME) 80*64kB (UME) 14*128kB (UME) 2*256kB (UM) 3*512kB (M) 23*1024kB (UM) 0*2048kB 0*4096kB = 59668kB
[11436.140089] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[11436.140090] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[11436.140092] 2708146 total pagecache pages
[11436.140093] 891 pages in swap cache
[11436.140094] Free swap  = 51719708kB
[11436.140095] Total swap = 57020408kB
[11436.140096] 7582386 pages RAM
[11436.140097] 0 pages HighMem/MovableOnly
[11436.140098] 162072 pages reserved
[11436.140099] 0 pages cma reserved
[11436.140100] 0 pages hwpoisoned
[11436.140101] Tasks state (memory values in pages):
[11436.140102] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[11436.140112] [   1369]    81  1369     2621     1266      411      855         0    53248      128          -900 dbus-broker-lau
[11436.140130] [   1378]     0  1378     4058     1815       80     1735         0    77824      160             0 systemd-logind
[11436.140132] [   1389]   972  1389     1615      320       36      284         0    57344       32             0 avahi-daemon
...
[11436.140179] [   3044]  1000  3044   312138     8443      822     7621         0   499712     4000             0 kwalletd6
[11436.140181] [   3045]  1000  3045    15589     2817       96     2721         0    94208      128             0 sddm-helper-sta
[11436.140184] [   3046]  1000  3046   306524    22453     6539    10338      5576   577536     4480             0 Xorg
[11436.140186] [   3066]  1000  3066    57616     4831       64     4767         0   229376     1184             0 startplasma-x11
[11436.140189] [   3081]  1000  3081     2025      758        0      758         0    61440       64           200 dbus-broker-lau
[11436.140191] [   3083]  1000  3083     2136     1482      921      561         0    61440       32           200 dbus-broker
[11436.140194] [   3123]  1000  3123    67445     8195      640     7555         0   208896      928           200 kglobalacceld
[11436.140196] [   3126]  1000  3126 67172964    87089        0    87089         0  3948544    18912           200 baloo_file
[11436.140198] [   3131]  1000  3131   275237     4030      320     3710         0   217088      448           200 xdg-desktop-por
[11436.140201] [   3155]  1000  3155    77269     1367       32     1335         0   106496       64           200 xdg-permission-
[11436.140203] [   3160]  1000  3160   154050     1581      128     1453         0   139264       32           200 xdg-document-po
[11436.140206] [   3167]  1000  3167      659      481        0      481         0    53248        0           200 fusermount3
[11436.140208] [   3177]  1000  3177   310830     9706     2073     7633         0   483328     2656           200 ksmserver
[11436.140210] [   3179]  1000  3179   464594    18049     3763    14286         0   614400     4672           200 kded6
[11436.140213] [   3180]  1000  3180   699795    35608    12209    23399         0  1032192    10718           200 kwin_x11
[11436.140216] [   3187]     0  3187   117763     3985      500     3485         0   143360      352             0 udisksd
[11436.140218] [   3198]     0  3198    80117     2864      192     2672         0   122880      384             0 upowerd
[11436.140220] [   3248]  1000  3248  1670890    94327    60538    33789         0  2043904    12279           200 plasmashell
[11436.140223] [   3256]  1000  3256    41065     1314       64     1250         0    73728       64           200 dconf-service
[11436.140225] [   3303]  1000  3303   147909     7971     1026     6945         0   258048      800           200 kactivitymanage
[11436.140228] [   3306]  1000  3306    57859     4636      160     4476         0   147456      352           200 gmenudbusmenupr
[11436.140230] [   3307]  1000  3307   310689     9557     2153     7404         0   491520     2528           200 kaccess
[11436.140233] [   3308]  1000  3308   310842     9733     1932     7801         0   483328     2816           200 polkit-kde-auth
[11436.140235] [   3309]  1000  3309   173215     8594      768     7826         0   303104     1024           200 org_kde_powerde
[11436.140237] [   3310]  1000  3310   372585    22844     5102    17742         0   577536     1632           200 xdg-desktop-por
[11436.140240] [   3311]  1000  3311    57726     4544       96     4448         0   139264      384           200 xembedsniproxy
[11436.140242] [   3312]  1000  3312    28858     3625     1356     2107       162   114688     1244           200 pipewire
[11436.140244] [   3313]  1000  3313   121563     3292      608     2684         0   159744      416           200 wireplumber
[11436.140247] [   3314]  1000  3314    34267     7320     4904     2265       151   151552     1792           200 pipewire-pulse
[11436.140250] [   3363]  1000  3363     2075      864        0      864         0    57344       64           200 xsettingsd
[11436.140252] [   3365]  1000  3365    57441     4574      160     4414         0   143360      416           200 kscreen_backend
[11436.140254] [   3613]  1000  3613   321960    11021     2820     8201         0   540672     3008           200 kdeconnectd
[11436.140257] [   3686]     0  3686    78000     1529       64     1465         0   102400       64             0 power-profiles-
[11436.140259] [   5587]  1000  5587    78432     2074       96     1978         0   102400       64           200 agent
[11436.140262] [   5598]  1000  5598   414211    22712     6194    15534       984   622592     2400           200 konsole
[11436.140264] [   5600]  1000  5600   489607    24604     4104    17436      3064   675840     5792           200 krusader
[11436.140266] [   5602]  1000  5602  5387826   184948    94255    83047      7646  4014080    52021           200 firefox
[11436.140269] [   5604]  1000  5604  2938620    54752    20721    33923       108  1875968    29950           200 cachy-browser
[11436.140271] [   5607]  1000  5607   439573    15918     5065    10853         0   835584     7712           200 kalendarac
[11436.140274] [   5613]  1000  5613   596356    33326    13227    19093      1006  1019904     7744           200 korganizer
[11436.140276] [   5935]  1000  5935   181047    15812     6060     9400       352   593920     4352           200 sublime_text
[11436.140279] [   5999]  1000  5999    95256     1629       32     1597         0   102400       96           200 at-spi-bus-laun
[11436.140281] [   6007]  1000  6007     1993      814       32      782         0    57344       32           200 dbus-broker-lau
[11436.140283] [   6010]  1000  6010     1059      545       32      513         0    49152        0           200 dbus-broker
[11436.140286] [   6014]  1000  6014    38063      492        0      492         0    61440       32           200 crash_handler
[11436.140288] [   6120]  1000  6120    42205     1698       96     1602         0    90112       64           200 at-spi2-registr
[11436.140291] [   6143]  1000  6143     3922     2581     1020     1561         0    61440      128           200 zsh
[11436.140294] [   6157]     0  6157   309667     3924     1203     2721         0   106496        0             1 containerd-shim
[11436.140296] [   6163]  1000  6163     3883     2085      699     1386         0    57344      448           200 zsh
[11436.140298] [   6185]  1000  6185     3880     2168      687     1481         0    73728      414           200 zsh
[11436.140301] [   6188] 65535  6188      243      109        0      109         0    40960        0          -998 pause
[11436.140303] [   6305]  1000  6305    65958     7367        0     7302        65   237568     2112           200 Socket Process
[11436.140306] [   6309]  1000  6309     3228      589       61      528         0    53248      512           200 zsh
[11436.140308] [   6310]  1000  6310     3218      604       61      543         0    69632      512           200 zsh
[11436.140310] [   6311]  1000  6311     3227      645       92      553         0    53248      512           200 zsh
[11436.140313] [   6387]  1000  6387    14663      665       15      649         1   131072     4896           200 plugin_host-3.3
[11436.140315] [   6390]  1000  6390    30587     1436      129     1306         1   221184    12640           200 plugin_host-3.8
[11436.140318] [   6416]     0  6416   309667     4357     1665     2692         0   102400        0             1 containerd-shim
[11436.140320] [   6467] 65535  6467      243      109        0      109         0    40960        0          -998 pause
[11436.140322] [   6594]  1000  6594   311131     8276      336     7940         0   487424     4320           200 akonadi_control
[11436.140325] [   6649]  1000  6649   240902    26132     3762    22141       229  1032192    13814           200 GPU Process
[11436.140327] [   6744]  1000  6744   496432     7625      934     6691         0   438272     1088           200 akonadiserver
[11436.140330] [   6862]  1000  6862   601737     6995     2745     4250         0   843776    66368           200 mysqld
[11436.140332] [   6886]     0  6886   309667     3977     1269     2708         0   102400        0             1 containerd-shim
[11436.140335] [   6928]     0  6928   309667     4042     1334     2708         0   102400        0             1 containerd-shim
[11436.140337] [   6954] 65535  6954      243      109        0      109         0    40960        0          -998 pause
[11436.140339] [   6972] 65535  6972      243      109        0      109         0    40960        0          -998 pause
[11436.140342] [   6973]  1000  6973    66434     7508      160     7280        68   212992     2144           200 Socket Process
[11436.140344] [   7010]     0  7010   309683     4068     1308     2760         0   102400        0             1 containerd-shim
[11436.140347] [   7059] 65535  7059      243      109        0      109         0    40960        0          -998 pause
[11436.140349] [   7066]  1000  7066     3706      766      246      520         0    57344      672           200 zsh
[11436.140351] [   7067]  1000  7067     3702      709      292      417         0    57344      640           200 zsh
[11436.140354] [   7070]  1000  7070     1705      309        0      309         0    49152       32           200 gitstatusd-linu
[11436.140356] [   7082]  1000  7082     3707      641      101      540         0    73728      832           200 zsh
[11436.140358] [   7091]  1000  7091     3703      534      101      433         0    73728      832           200 zsh
[11436.140361] [   7118]  1000  7118     1705      353        0      353         0    49152       32           200 gitstatusd-linu
[11436.140363] [   7169]  1000  7169     3705      853      323      530         0    57344      640           200 zsh
[11436.140365] [   7171]  1000  7171     3701      734      291      443         0    57344      640           200 zsh
[11436.140368] [   7177]  1000  7177     1705      328       32      296         0    49152       32           200 gitstatusd-linu
[11436.140370] [   7216]     0  7216   309683     4377     1649     2728         0   106496        0             1 containerd-shim
[11436.140372] [   7251] 65535  7251      243      109        0      109         0    40960        0          -998 pause
[11436.140375] [   7308]     0  7308   309667     4446     1770     2676         0   102400        0             1 containerd-shim
[11436.140377] [   7342] 65535  7342      243      109        0      109         0    40960        0          -998 pause
[11436.140380] [   7393]     0  7393   309667     4428     1699     2729         0   102400        0             1 containerd-shim
[11436.140382] [   7428] 65535  7428      243      109        0      109         0    40960        0          -998 pause
[11436.140384] [   7446]  1000  7446   646187    20522     2713    17692       117   716800     4928           167 Privileged Cont
[11436.140387] [   7454]  1000  7454    98327     7227      193     6969        65   360448     3072           200 RDD Process
[11436.140389] [   7495]  1000  7495   634157    36780     5874    30545       361   720896     2434           167 Privileged Cont
[11436.140392] [   7503]  1000  7503   191899    21326      912    19738       676   671744     7566           200 RDD Process
[11436.140394] [   7544]     0  7544   309667     4400     1659     2741         0   106496        0             1 containerd-shim
[11436.140396] [   7578] 65535  7578      243      109        0      109         0    40960        0          -998 pause
[11436.140399] [   7602]     0  7602   309667     4311     1636     2675         0   102400        0             1 containerd-shim
[11436.140401] [   7645] 65535  7645      243       96        0       96         0    40960        0          -998 pause
[11436.140404] [   7663]     0  7663   309667     4301     1618     2683         0   102400        0             1 containerd-shim
[11436.140406] [   7704] 65535  7704      243      109        0      109         0    40960        0          -998 pause
[11436.140409] [   7719]  1000  7719   785531    70481    42401    27715       365  3510272    14659           100 Isolated Web Co
[11436.140411] [   7725]  1000  7725   665106    19715     1623    18018        74  1171456    19712           167 Isolated Web Co
[11436.140414] [   7740]  1000  7740   679628    21503     2584    18820        99  1425408    26144           167 Isolated Web Co
[11436.140416] [   7746]  1000  7746   792388    63954    43707    20067       180  3457024    23808           167 Isolated Web Co
[11436.140419] [   7813]  1000  7813   676230    64312    35076    28951       285  1720320     5368           167 Isolated Web Co
[11436.140421] [   7819]  1000  7819   741097    58814    18156    26112     14546  2142208    43680           167 Isolated Web Co
[11436.140424] [   7824]  1000  7824  4017689   171976   128481    39314      4181  5197824    20310           167 Isolated Web Co
[11436.140427] [   7841]  1000  7841  2977076   165227   128560    34013      2654  5894144    34272           167 Isolated Web Co
[11436.140429] [   7905]  1000  7905  4966091    73040    39175    30709      3156  2600960    16643           100 Isolated Web Co
[11436.140431] [   7928]  1000  7928  4993839    66568    48088    18464        16  3252224    13112           100 WebExtensions
[11436.140434] [   7941]  1000  7941   690075    47630    20651    26787       192  1765376    29270           167 Isolated Web Co
[11436.140436] [   7960]  1000  7960   707128    77595    44688    32559       348  2138112     5854           167 Isolated Web Co
[11436.140439] [   8081]  1000  8081    98583     7581       52     7464        65   372736     3136           200 Utility Process
[11436.140441] [   8096]  1000  8096   764914    43098    23703    19254       141  2719744    28615           167 Isolated Web Co
[11436.140444] [   8104]  1000  8104   620900    10905      192    10570       143   487424     3392           233 Web Content
[11436.140446] [   8262]  1000  8262   654630    30976     4893    25847       236  1019904     2598           100 WebExtensions
[11436.140463] [   8347]     0  8347   309731     4305     1729     2576         0   106496        0             1 containerd-shim
[11436.140467] [   8389]     0  8389   309731     3957     1271     2686         0   106496        0             1 containerd-shim
[11436.140470] [   8397] 65535  8397      243      109        0      109         0    40960        0          -998 pause
[11436.140473] [   8427] 65535  8427      243      109        0      109         0    40960        0          -998 pause
[11436.140476] [   8454]  1000  8454   313523     8648      521     8127         0   499712     4192           200 akonadi_birthda
[11436.140480] [   8475]  1000  8475   313074     8665      475     8190         0   503808     4192           200 akonadi_contact
[11436.140484] [   8477]     0  8477   318790    10007     2819     7188         0   135168        0          -997 local-path-prov
[11436.140487] [   8484]  1000  8484   317751    18030     4892    13138         0   548864     4032           200 akonadi_ical_re
[11436.140491] [   8493]  1000  8493   315060    14081      920    13161         0   520192     4192           200 akonadi_indexin
[11436.140494] [   8494]  1000  8494   312879     8662      417     8245         0   499712     4224           200 akonadi_maildir
[11436.140496] [   8504]  1000  8504   313028     8744      525     8219         0   503808     4256           200 akonadi_maildis
[11436.140499] [   8505]  1000  8505   313585     8651      438     8213         0   499712     4224           200 akonadi_migrati
[11436.140502] [   8507]  1000  8507   317690     8816      483     8333         0   532480     4352           200 akonadi_newmail
[11436.140504] [   8524]     0  8524   309667     4384     1702     2682         0   102400        0             1 containerd-shim
[11436.140507] [   8558] 65535  8558      243      109        0      109         0    40960        0          -998 pause
[11436.140509] [   8726]  1000  8726   303141     9773      551     9222         0   524288     4512           200 plasma-browser-
[11436.140546] [  11438]    70 11438    51738     2276      501     1075       700   [11436.140661] [ 101581]  1000 101581     2453     1708       96     1612         0    49152        0           200 less
[11436.140664] [ 101609] 65532 101609   324102    16982     4597    12385         0   233472        0           998 coredns
[11436.140666] [ 101617]     0 101617      404      174        0      174         0    49152        0          1000 sleep
[11436.140669] [ 101686]   101 101686       54        0        0        0         0    24576        0           997 dumb-init
[11436.140672] [ 101699]   101 101699   318183    14034     5647     8387         0   196608        0           997 nginx-ingress-c
[11436.140674] [ 101725]   101 101725    37046     9557     7625     1772       160   184320        0           997 nginx
[11436.140677] [ 101730]   101 101730    40044    10612     9417     1163        32   172032        0           997 nginx
[11436.140680] [ 101731]   101 101731    40076    10714     9470     1174        70   180224        0           997 nginx
[11436.140682] [ 101732]   101 101732    40044    10659     9470     1157        32   172032        0           997 nginx
[11436.140685] [ 101733]   101 101733    40044    10648     9438     1178        32   172032        0           997 nginx
[11436.140687] [ 101734]   101 101734    40044    10579     9438     1109        32   172032        0           997 nginx
[11436.140690] [ 101735]   101 101735    40044    10631     9438     1161        32   172032        0           997 nginx
[11436.140692] [ 101736]   101 101736    40044    10644     9438     1174        32   172032        0           997 nginx
[11436.140695] [ 101737]   101 101737    40044    10597     9443     1122        32   172032        0           997 nginx
[11436.140697] [ 101738]   101 101738    40044    10610     9438     1140        32   172032        0           997 nginx
[11436.140700] [ 101739]   101 101739    40044    10626     9438     1156        32   172032        0           997 nginx
[11436.140702] [ 101740]   101 101740    40044    10612     9406     1174        32   172032        0           997 nginx
[11436.140705] [ 101741]   101 101741    40044    10644     9438     1174        32   172032        0           997 nginx
[11436.140707] [ 101742]   101 101742    40044    10617     9438     1147        32   172032        0           997 nginx
[11436.140710] [ 101759]   101 101759    40044    10624     9456     1136        32   172032        0           997 nginx
[11436.140712] [ 101831]   101 101831    40044    10676     9470     1174        32   172032        0           997 nginx
[11436.140715] [ 102079]   101 102079    40044    10679     9502     1145        32   172032        0           997 nginx
[11436.140717] [ 102137]   101 102137    36533     7597     7207      390         0   135168        0           997 nginx
[11436.140720] [ 104686]     0 104686      622      352        0      352         0    49152        0          1000 sleep
[11436.140722] [ 105050]     0 105050   755498    39711    24404    15307         0   913408        0          1000 minio
[11436.140725] [ 105107]     0 105107     1064      840      160      616        64    57344        0          1000 httpd
[11436.140728] [ 105133]    82 105133     2105      519      208      311         0    61440        0          1000 httpd
[11436.140730] [ 105134]    82 105134     2105      521      208      313         0    61440        0          1000 httpd
[11436.140732] [ 105135]    82 105135     2105      518      208      310         0    61440        0          1000 httpd
[11436.140735] [ 105233]     0 105233     2595     1886      384     1406        96    57344        0          1000 httpd
[11436.140737] [ 105260]    82 105260     3604      833      482      351         0    61440        0          1000 httpd
[11436.140740] [ 105261]    82 105261     3604      801      450      351         0    61440        0          1000 httpd
[11436.140742] [ 105262]    82 105262     3604      801      450      351         0    61440        0          1000 httpd
[11436.140745] [ 105615]  1000 105615  6695227   132600   126164     6436         0  1826816        0          1000 java
[11436.140749] [ 113782]     0 113782     3928     1945      224     1721         0    65536        0             0 systemd-userwor
[11436.140751] [ 113783]     0 113783     3928     2070      224     1846         0    69632        0             0 systemd-userwor
[11436.140754] [ 114066]  1000 114066 67295596   143930     2393   141537         0  8478720   124000           200 baloo_file_extr
[11436.140757] [ 116853]  1000 116853  1760496    89411    46409    42008       994  2007040        0           167 Isolated Web Co
[11436.140760] [ 116930]   968 116930   158282     7797     2560     5237         [11436.140784] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=cri-containerd-88542c5e302300b874ce38080261be8c4dd353f9c8c6a46f005b543de565d6ac.scope,mems_allowed=0,global_oom,task_memcg=/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a5003a_6a1b_4195_bc90_2ffa1a0f5f83.slice/cri-containerd-88542c5e302300b874ce38080261be8c4dd353f9c8c6a46f005b543de565d6ac.scope,task=node,pid=117108,uid=0
[11436.140819] Out of memory: Killed process 117108 (node) total-vm:13639172kB, anon-rss:2490160kB, file-rss:41516kB, shmem-rss:0kB, UID:0 pgtables:72352kB oom_score_adj:1000
[11436.140881] Tasks in /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a5003a_6a1b_4195_bc90_2ffa1a0f5f83.slice/cri-containerd-88542c5e302300b874ce38080261be8c4dd353f9c8c6a46f005b543de565d6ac.scope are going to be killed due to memory.oom.group set
[11436.140919] Out of memory: Killed process 117030 (npm start) total-vm:706600kB, anon-rss:21012kB, file-rss:44208kB, shmem-rss:0kB, UID:0 pgtables:948kB oom_score_adj:1000
[11436.140955] Out of memory: Killed process 117058 (node) total-vm:673712kB, anon-rss:14776kB, file-rss:39656kB, shmem-rss:0kB, UID:0 pgtables:960kB oom_score_adj:1000
[11436.140993] Out of memory: Killed process 117085 (npm run serve) total-vm:702264kB, anon-rss:16628kB, file-rss:43052kB, shmem-rss:0kB, UID:0 pgtables:808kB oom_score_adj:1000
[11436.141031] Out of memory: Killed process 117086 (npm test) total-vm:702536kB, anon-rss:18936kB, file-rss:43256kB, shmem-rss:0kB, UID:0 pgtables:816kB oom_score_adj:1000
[11436.141051] Out of memory: Killed process 117107 (webpack) total-vm:34322376kB, anon-rss:2202060kB, file-rss:41024kB, shmem-rss:0kB, UID:0 pgtables:62944kB oom_score_adj:1000
[11436.141062] Out of memory: Killed process 117108 (node) total-vm:13639172kB, anon-rss:2490160kB, file-rss:41516kB, shmem-rss:0kB, UID:0 pgtables:72352kB oom_score_adj:1000
[11436.141070] Out of memory: Killed process 117111 (node) total-vm:34322376kB, anon-rss:2202060kB, file-rss:41024kB, shmem-rss:0kB, UID:0 pgtables:62944kB oom_score_adj:1000
[11457.202071] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=64412 DF PROTO=2 
[11497.306362] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=1405 DF PROTO=2 
[11537.225758] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=4232 DF PROTO=2 
[11570.948659] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth07955477 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948669] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth12f61a89 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948675] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth01eb9f12 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948681] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth688c7cfd MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948686] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth8e0ee0ef MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948692] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=vethcb32fe49 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948697] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth1ab373df MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948702] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth13aef1e6 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948708] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=vetha5dcaa34 MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11570.948713] [UFW BLOCK] IN=cni0 OUT=cni0 PHYSIN=veth13aef1e6 PHYSOUT=veth2c8f3adf MAC=33:33:00:00:00:02:16:a1:ce:f7:a9:63:86:dd SRC=fe80:0000:0000:0000:14a1:ceff:fef7:a963 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=133 CODE=0 
[11577.239810] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=7161 DF PROTO=2 
[11617.309724] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=8099 DF PROTO=2 
[11657.255018] [UFW BLOCK] IN=wlan0 OUT= MAC=e0:2e:0b:90:25:54:d4:86:60:e7:45:f6:08:00 SRC=192.168.1.1 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=10922 DF PROTO=2 

systemctl status systemd-oomd nothing?

If CachyOS doesn’t enable this by default, the preset for this is disabled. And OP doesn’t seem to have enabled it themselves.

The issue is caused by OOM Killer which is enabled in the kernel. I don’t pretend to know much about it but I have got interested in it lately so I am looking at some documents on and off to learn more.

OOM Killer starts killing processes based of a score that the kernel assigns to each process. The range of this score varies from -1000 to 1000 where that latter is assigned to those processes that most likely will perish.

The score is caclculated based on several factors: memory usage, cpu usage, process importance, process lifespan etc.

I can see in the output OP has posted several processes with the score 1000 which have been killed.

[11436.140919] Out of memory: Killed process 117030 (npm start) total-vm:706600kB, anon-rss:21012kB, file-rss:44208kB, shmem-rss:0kB, UID:0 pgtables:948kB oom_score_adj:1000
[11436.140955] Out of memory: Killed process 117058 (node) total-vm:673712kB, anon-rss:14776kB, file-rss:39656kB, shmem-rss:0kB, UID:0 pgtables:960kB oom_score_adj:1000
[11436.140993] Out of memory: Killed process 117085 (npm run serve) total-vm:702264kB, anon-rss:16628kB, file-rss:43052kB, shmem-rss:0kB, UID:0 pgtables:808kB oom_score_adj:1000
[11436.141031] Out of memory: Killed process 117086 (npm test) total-vm:702536kB, anon-rss:18936kB, file-rss:43256kB, shmem-rss:0kB, UID:0 pgtables:816kB oom_score_adj:1000
[11436.141051] Out of memory: Killed process 117107 (webpack) total-vm:34322376kB, anon-rss:2202060kB, file-rss:41024kB, shmem-rss:0kB, UID:0 pgtables:62944kB oom_score_adj:1000
[11436.141062] Out of memory: Killed process 117108 (node) total-vm:13639172kB, anon-rss:2490160kB, file-rss:41516kB, shmem-rss:0kB, UID:0 pgtables:72352kB oom_score_adj:1000
[11436.141070] Out of memory: Killed process 117111 (node) total-vm:34322376kB, anon-rss:2202060kB, file-rss:41024kB, shmem-rss:0kB, UID:0 pgtables:62944kB oom_score_adj:1000

I think OP should monitor those processes and the values listed in the output to get to the root of the problem.

I leave here a few links for some reading material that I have just started to look at also.

https://www.oracle.com/technical-resources/articles/it-infrastructure/dev-oom-killer.html

Finally I’ve end up reinstalling my system.
I couldn’t work.
Now I’m trying Manjaro.