- Qlustar General - Qlustar Mailing Lists

Re: Users job processes are not killable by OOM
by Rolandas Naujikas 02 Jun '25

02 Jun '25

diff -u /usr/sbin/ql-adjust-oomscore ql-adjust-oomscore --- /usr/sbin/ql-adjust-oomscore 2025-04-15 11:41:22.000000000 +0300 +++ ql-adjust-oomscore 2025-05-28 15:52:27.993037862 +0300 @@ -32,6 +32,7 @@ systemd Xorg sssd rsyslogd dbus-daemon lightdm mysqld \ qluman-execd.py qlumand.py qluman-dhcpscanner.py qluman-router.py \ slurmctld slurmd" +USERS="root,_rpc,statd,messagebus,systemd-timesync,ganglia" DEFAULTS=/etc/default/oomscore function set_oom() { @@ -42,7 +43,7 @@ SCORE="$COMPAT_SCORE" fi for proc in $procs; do - pids=$(pgrep -f $proc) + pids=$(pgrep -u $USERS -f $proc) for i in $pids; do oom_adj=/proc/$i/${target_file} if [ -f $oom_adj ]; then On 2025-05-28 14:15, rolnas(a)gmail.com wrote: > Hi, > > Finally found a source of problem. In Qlustar 13 distribution there is a periodic (hourly) script to make some processes unkillable for OOM, but pattern to choose processes by they command line is too wide and process 182580 were matching it (because of slurmd in name). > > Regards > Rolandas > > P.S. Script ql-adjust-oomscore. > > On 2025-05-28 11:29, rolnas(a)gmail.com wrote: >> Hi, >> >> We are using SLURM 24.11.5 and maintaining HPC cluster. >> Several times we had nodes stuck after user jobs. >> Investigation lead to OOM situations. >> Deeper investigation found in some strange situations users jobs oom_score_adj is left at -1000 as slurmstepd process. >> >> For e.g. (some information is changed for privacy): >> root 182565 0.0 0.0 947952 30128 ? SLl May27 0:04 slurmstepd: [177126.batch] >> hpc_user 182580 0.0 0.0 1845524 70696 ? Sl May27 0:04 \_ / scratch/lustre/home/hpc_user/miniforge-pypy3/envs/snakemake/bin/ python3.12 /var/lib/slurm-llnl/slurmd/job177126/slurm_script -p all -- use-conda --rerun-incomplete >> hpc_user 241662 99.9 7.2 30762668 28598376 ? R May27 1418:09 \_ ClonalFrameML ... >> >> cat /proc/{182565,182580,241662}/oom_score_adj >> -1000 >> -1000 >> -1000 >> >> I'm trying to replicate that situation and no luck. Every time I'm getting users jobs with oom_score_adj=0 what it should be. >> >> Regards >> Rolandas >> >> P.S. Command to launch this job was: >> sbatch -A alloc_2a21c_test -c 12 --time=100:00:00 snakemake -p all -- use-conda --rerun-incomplete

2 3

Connection error using Qluman singularity images
by Beethoven Santos 28 May '25

28 May '25

Dear All, I cannot use the Qluman singularity images to connect to a small cluster I am managing. When I try to connect, Qluman refuses to do so with the following message: non-network local connections being added to access control list qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in "" QStandardPaths: XDG_RUNTIME_DIR points to non-existing path '/run/user/1000', please create it with 0700 permissions. Qt: Session management error: None of the authentication protocols specified are supported 2025-05-27 06:08:12,776 ERROR qlunet.Node router for '******' is not responding to pings: reconnecting I had to replace sensitive information with *'s. Some additional info: Qluman singularity image version: 13.7.2 Version of qluman-* pkgs in the cluster: 13.5.6 Please help me with this problem. Sincerely, Beethoven Santos

2 1

Qlustar 12 is EOL
by no-reply-list＠qlustar.org 16 May '25

16 May '25

With the latest security release, Qlustar 12 has reached its EOL and will receive no further updates. Please upgrade to Qlustar 13 as soon as possible in case your cluster is still running Qlustar 12. See https://docs.qlustar.com/Qlustar/13/ClusterOS/release-notes.html#_update_in… for instructions.

1 0

Security/Bug fix releases 14.0.1, 13.3.1, 12.0.3.12
by no-reply-list＠qlustar.org 16 May '25

16 May '25

The Qlustar releases 14.0.1, 13.3.1 and 12.0.3.12 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0515251 https://qlustar.com/qsa/2025/0515252 https://qlustar.com/qsa/2025/0515253 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.3 * Fix AttributeError in HostnameWidget save method * RemoteManagement: - Implement clone button - Save changes in redfish CheckBox

1 0

Qlustar 14 is out with fully revamped monitoring
by no-reply-list＠qlustar.org 07 May '25

07 May '25

The Qlustar team is pleased to announce the immediate availability of Qlustar 14 for download [1]. It updates Qlustar's core platform to current Ubuntu 24.04 LTS. The AlmaLinux [2] edge platform remains based on 8.10. The highlight of this release is the new state-of-the-art Qlustar Monitoring stack. It is now based on Grafana [3] and Prometheus [4]. Both of them are tightly integrated with the other Qlustar components and can be controlled with a powerful new configuration interface [5] of the QluMan GUI. This release also includes the following individual major HPC core stack component updates: - Spack version 0.23.1 [6] - Slurm version 24.11.3 [7] - RDMA-Core version 55.0 [8] - HWloc version 2.11.2 [9] - UCX version 1.18.1 [10] Please consult the release notes [11] for more details. [1] https://qlustar.com/download/ [2] https://almalinux.org/ [3] https://grafana.com/grafana/ [4] https://prometheus.io/ [5] https://docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config-Classes.h… [6] https://spack.readthedocs.io/en/latest/ [7] https://www.schedmd.com/slurm/ [8] https://github.com/linux-rdma/rdma-core [9] https://www.open-mpi.org/projects/hwloc/ [10] https://openucx.org/ [11] https://docs.qlustar.com/Qlustar/14/ClusterOS/release-notes.html

3 2

Home File-System
by ignas.bunikis＠scilifelab.uu.se 08 Apr '25

08 Apr '25

Hej, Would you mind giving a short guide how to correctly setup Home File-System on NFS? Documentation just informs that: "If you choose Other, then no configuration for a home file-system will be done. In this case, you won’t be able to add any cluster users without further manual steps to setup a home file-system." Cheers, Iggy

2 7

Qlustar HPC Core Stack 25.03
by no-reply-list＠qlustar.org 22 Mar '25

22 Mar '25

The Qlustar team is pleased to announce the availability of the HPC Core Stack 25.03 for Qlustar 13. It was published together with the latest security bug fix update and can be installed by updating the Qlustar 13 chroots on the cluster. The HPC CORE STACK [1] was introduced with the release of Qlustar 13 as a new component definition within Qlustar. It is related to the introduction of Spack. This release includes the following individual major HPC core stack component updates: - Spack version 0.23.1 [2] - Slurm version 24.11.3 [3] - RDMA-Core version 55.0 [4] - HWloc version 2.11.2 [5] - UCX version 1.18.1 [6] The new version of Spack allows easy and direct access to the most recent compilers, MPIs and hundreds of other applications relevant to HPC. Amongst others, it includes gcc 14.2.0 and OpenMPI 5.0.5 as new default versions. Minor version updates for other Qlustar 13 components are also included, in particular Nvidia graphics drivers (550.144.03), Lustre (2.15.6) and gdrcopy (2.4.4) for use with Nvidia GPUs. [1] https://qlustar.com/qlustar/hpc-stack/hpc-core-stack/ [2] https://spack.readthedocs.io/en/latest/ [3] https://www.schedmd.com/slurm/ [4] https://github.com/linux-rdma/rdma-core [5] https://www.open-mpi.org/projects/hwloc/ [6] https://openucx.org/

1 0

Security/Bug fix releases 13.3.0, 12.0.3.11
by no-reply-list＠qlustar.org 22 Mar '25

22 Mar '25

The Qlustar releases 13.3.0 and 12.0.3.11 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0321251 https://qlustar.com/qsa/2025/0321252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Qlustar HPC Core Stack 03/25 update (details to be announced separately)

1 0

Denied permission when restoring a database in qluman-qt
by ldettmann＠web.de 25 Feb '25

25 Feb '25

Qlustar version: 13.2 Qluman version: 13.7.2 I am trying to restore a qluman database from another server using "Manage Cluster" > "Global Configs" > "Restore database" and encountered an error: Permission denied. However, I also tried it with a backup, which I created within the current session, and encountered the same error. What can I do? Best regards, Lorenz

3 4

QluMan configuration for single sign on
by João Manuel Rodrigues 19 Feb '25

19 Feb '25

Documentation says: "b) QluMan may be configured to allow for a single sign-on setup where users can be imported from and authenticated against an Active Directory (AD) server." <https://docs.qlustar.com/Qlustar/13/ClusterOS/qluman-guide/components/ldap.…> How can I configure QluMan this way? I have configured sssd on the head node to connect to our AD (ldap + kerberos), but in QluMan, "Components > LDAP > Import Users > From AD" is disabled (gray). Thanks, João

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General