May 2025 - Qlustar General - Qlustar Mailing Lists

Re: Users job processes are not killable by OOM
by Rolandas Naujikas 02 Jun '25

02 Jun '25

diff -u /usr/sbin/ql-adjust-oomscore ql-adjust-oomscore --- /usr/sbin/ql-adjust-oomscore 2025-04-15 11:41:22.000000000 +0300 +++ ql-adjust-oomscore 2025-05-28 15:52:27.993037862 +0300 @@ -32,6 +32,7 @@ systemd Xorg sssd rsyslogd dbus-daemon lightdm mysqld \ qluman-execd.py qlumand.py qluman-dhcpscanner.py qluman-router.py \ slurmctld slurmd" +USERS="root,_rpc,statd,messagebus,systemd-timesync,ganglia" DEFAULTS=/etc/default/oomscore function set_oom() { @@ -42,7 +43,7 @@ SCORE="$COMPAT_SCORE" fi for proc in $procs; do - pids=$(pgrep -f $proc) + pids=$(pgrep -u $USERS -f $proc) for i in $pids; do oom_adj=/proc/$i/${target_file} if [ -f $oom_adj ]; then On 2025-05-28 14:15, rolnas(a)gmail.com wrote: > Hi, > > Finally found a source of problem. In Qlustar 13 distribution there is a periodic (hourly) script to make some processes unkillable for OOM, but pattern to choose processes by they command line is too wide and process 182580 were matching it (because of slurmd in name). > > Regards > Rolandas > > P.S. Script ql-adjust-oomscore. > > On 2025-05-28 11:29, rolnas(a)gmail.com wrote: >> Hi, >> >> We are using SLURM 24.11.5 and maintaining HPC cluster. >> Several times we had nodes stuck after user jobs. >> Investigation lead to OOM situations. >> Deeper investigation found in some strange situations users jobs oom_score_adj is left at -1000 as slurmstepd process. >> >> For e.g. (some information is changed for privacy): >> root 182565 0.0 0.0 947952 30128 ? SLl May27 0:04 slurmstepd: [177126.batch] >> hpc_user 182580 0.0 0.0 1845524 70696 ? Sl May27 0:04 \_ / scratch/lustre/home/hpc_user/miniforge-pypy3/envs/snakemake/bin/ python3.12 /var/lib/slurm-llnl/slurmd/job177126/slurm_script -p all -- use-conda --rerun-incomplete >> hpc_user 241662 99.9 7.2 30762668 28598376 ? R May27 1418:09 \_ ClonalFrameML ... >> >> cat /proc/{182565,182580,241662}/oom_score_adj >> -1000 >> -1000 >> -1000 >> >> I'm trying to replicate that situation and no luck. Every time I'm getting users jobs with oom_score_adj=0 what it should be. >> >> Regards >> Rolandas >> >> P.S. Command to launch this job was: >> sbatch -A alloc_2a21c_test -c 12 --time=100:00:00 snakemake -p all -- use-conda --rerun-incomplete

2 3

Connection error using Qluman singularity images
by Beethoven Santos 28 May '25

28 May '25

Dear All, I cannot use the Qluman singularity images to connect to a small cluster I am managing. When I try to connect, Qluman refuses to do so with the following message: non-network local connections being added to access control list qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in "" QStandardPaths: XDG_RUNTIME_DIR points to non-existing path '/run/user/1000', please create it with 0700 permissions. Qt: Session management error: None of the authentication protocols specified are supported 2025-05-27 06:08:12,776 ERROR qlunet.Node router for '******' is not responding to pings: reconnecting I had to replace sensitive information with *'s. Some additional info: Qluman singularity image version: 13.7.2 Version of qluman-* pkgs in the cluster: 13.5.6 Please help me with this problem. Sincerely, Beethoven Santos

2 1

Qlustar 12 is EOL
by no-reply-list＠qlustar.org 16 May '25

16 May '25

With the latest security release, Qlustar 12 has reached its EOL and will receive no further updates. Please upgrade to Qlustar 13 as soon as possible in case your cluster is still running Qlustar 12. See https://docs.qlustar.com/Qlustar/13/ClusterOS/release-notes.html#_update_in… for instructions.

1 0

Security/Bug fix releases 14.0.1, 13.3.1, 12.0.3.12
by no-reply-list＠qlustar.org 16 May '25

16 May '25

The Qlustar releases 14.0.1, 13.3.1 and 12.0.3.12 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0515251 https://qlustar.com/qsa/2025/0515252 https://qlustar.com/qsa/2025/0515253 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.3 * Fix AttributeError in HostnameWidget save method * RemoteManagement: - Implement clone button - Save changes in redfish CheckBox

1 0

Qlustar 14 is out with fully revamped monitoring
by no-reply-list＠qlustar.org 07 May '25

07 May '25

The Qlustar team is pleased to announce the immediate availability of Qlustar 14 for download [1]. It updates Qlustar's core platform to current Ubuntu 24.04 LTS. The AlmaLinux [2] edge platform remains based on 8.10. The highlight of this release is the new state-of-the-art Qlustar Monitoring stack. It is now based on Grafana [3] and Prometheus [4]. Both of them are tightly integrated with the other Qlustar components and can be controlled with a powerful new configuration interface [5] of the QluMan GUI. This release also includes the following individual major HPC core stack component updates: - Spack version 0.23.1 [6] - Slurm version 24.11.3 [7] - RDMA-Core version 55.0 [8] - HWloc version 2.11.2 [9] - UCX version 1.18.1 [10] Please consult the release notes [11] for more details. [1] https://qlustar.com/download/ [2] https://almalinux.org/ [3] https://grafana.com/grafana/ [4] https://prometheus.io/ [5] https://docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config-Classes.h… [6] https://spack.readthedocs.io/en/latest/ [7] https://www.schedmd.com/slurm/ [8] https://github.com/linux-rdma/rdma-core [9] https://www.open-mpi.org/projects/hwloc/ [10] https://openucx.org/ [11] https://docs.qlustar.com/Qlustar/14/ClusterOS/release-notes.html

3 2