- Qlustar General - Qlustar Mailing Lists

QluMan GUI running natively on Windows
by no-reply-list＠qlustar.org 01 Dec '25

01 Dec '25

With the release of WSL 2 [1] it has become straight-forward to run the QluMan GUI directly on Windows workstations. We have now added this constellation into our portfolio of tested, verified and supported components. This means that cluster admins working on windows can now follow the corresponding setup guide [2] and enjoy a much improved user experience when working on Qlustar clusters. [1] https://learn.microsoft.com/en-us/windows/wsl/about#what-is-wsl-2 [2] https://docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/qluman-guide.htm…

1 0

Qlustar enters Top 100
by no-reply-list＠qlustar.org 21 Nov '25

21 Nov '25

This week at SC 25 in St. Louis, a recently implemented Qlustar 14 AI Kubernetes cluster entered the Top 500 list at rank 90 with a measured Linpack performance of *19.48 PFlop/s*. This huge success establishes Qlustar as a proven platform for even the most demanding work-loads in the AI and HPC universe. The cluster worker nodes are HGX machines each of them hosting 8 NVIDIA H100 (80GB HBM3) GPUs which are networked in a rail-optimized fat-tree NDR Infiniband fabric. Qlustar's new NVIDIA image modules support this hardware out-of-the-box with optimal performance. The *Qlustar Kubernetes stack* powering this cluster will become GA and be officially introduced in an upcoming Qlustar 14 feature release. [1] https://top500.org/system/180424/

1 0

Qlustar Feature Update 11/25
by no-reply-list＠qlustar.org 21 Nov '25

21 Nov '25

A large bunch of new features were completed and published since the Qlustar 14 release, both in QluMan as well as for Nvidia Enterprise GPU support. Please consult the comprehensive overview at https://qlustar.com/news/2025/2025-11-21-feature-update-11-25

1 0

ssh server and oom_score_adj
by Rolandas Naujikas 13 Nov '25

13 Nov '25

Hi, Recently I found that our login nodes are abused with users running heavy nodejs sessions (from visual studio code). I setup some memory limits for users and OOM is coming, but cannot kill users processes, because all of them are with oom_score_adj=-1000. It is coming from parent sshd server also with oom_score_adj=-1000. As a workaround I'm stopping traditional ssh server and starting ssh.socket at boot (via rc.boot script). Now oom_score_adj=0 as it should be. That is also important for computing nodes with ssh connection enabled. Regards Rolandas P.S. We are still on Qlustar 13, so I don't know situation on Qlustar 14, but it could be different because ssh connection in Ubuntu 24.04 LTS is activated via ssh socket already.

2 3

Add users from command line
by Rolandas Naujikas 11 Nov '25

11 Nov '25

Hi, Do there is some way to add/manage users in Qlustar 12/13/14 from command line ? Some admins have difficulties to run qluman GUI. Could it be done via Qluman router API ? Regards Rolandas

2 1

Security/Bug fix releases 14.0.4, 13.3.4
by no-reply-list＠qlustar.org 18 Oct '25

18 Oct '25

The Qlustar releases 14.0.4 and 13.3.4 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/1017251 https://qlustar.com/qsa/2025/1017252 https://qlustar.com/qsa/2025/1017253 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.7 * Remove DB backup script and cron entry. It is now done via stuff in qluman-misc-tools package. * Slurm: Detect HA setup and restart slurmctld via HA system when writing the config. * Alerts: - Drop node_interrupts_total metric, to reduce output. - Add scrapping kubelet-cadvisor. * Power Management: Implement power status, cycle and reset for redfish. * New Hosts Widget: Fix Import MACs feature. * Remote Management: - Allow removing and reordering BIOS boot devices. - Check BIOS boot order on write and sync. - Move configured BIOS boot devices to the front of boot order if they exist. - Store settings checkmarks in DB. * Add qlumand debug options for redfish access and BIOS settings. - core module * Monitoring stack: - Further trim down node-exporter metrics. - kubernetes module * Add kubectl bash shell completion support.

1 0

Security/Bug fix releases 14.0.3, 13.3.3
by no-reply-list＠qlustar.org 28 Aug '25

28 Aug '25

The Qlustar releases 14.0.3 and 13.3.3 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0826251 https://qlustar.com/qsa/2025/0826252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.6 * qlumand - Add Depends and Provides to qlustar-module infos * New GUI feature: Implement network proxy to the managed clusters allowing firefox (embedded in singularity image now) to easily connect to a BMCs web interface via a button in the GUI. * RemoteManagement - Add filter for BIOS Settings - Improve on the display of BIOS options: Show the real BIOS text rather than cryptic option keys. - Add BootDevices to BIOS read and change writing boot order * Monitoring - Add direct proxied connections to prometheus / prometheus-alert-manager - Add interface to handle silencing of alerts * HA - Fix resources start command getting executed twice in parallel due to race condition with state updates - Catch monitoring being done on resources with an action due to update races * LDAP user-/group management - Pass ldap group name as 5th argument to add/remove user hook * NewHostWidget - Add ignored MACs feature - Increase default height so more than 1 Unknwon MAC is shown * LDAP - CreateUsersDialog + Enable Retry button for slurm account creation when done with errors + Count slurmd offline instances as errors + Color code table entries for pending / done / error + Handle errors when slurmdb is offline - Add Run / Run without Slurm / Save and Sync Node buttons - Fix exception in generating result text when deleting groups - Fix exceptions with new filtering when importing from passwd file - Sort tab order of UserManagementWidget * Slurm: - Export slurmd status in slurmd component (GUI) - Change calculation of available RAM for Slurm to make it change less often after updates due to varying kernel and image size - Add slurm daemon restart menu in slurmd LED. Allows to restart slurmctld, slurmdbd and qluman-slurmd. - Better handle situations where slurm head-node components are unavailable. * Execd: fix exception when diffing file that doesn't yet exist * Fix DHCP scanner to understand new syslog timestamps * Alerts: - Include config for prometheus scraper - Handle VirtualGL Remote Desktop nodes as separate group * Internal change: Use improved common exec_os_cmd function everywhere. - core module * Monitoring stack: - Trim down node-exporter metrics to generally needed ones - Add IPMI sensors metrics - nvidia modules * Add nvidia-570 module with 570 driver series and switch to open GPU kernel modules * Add nvidia-extra module that contains everything that doesn't depend on a specific driver version (e.g. container toolkit and kubernetes device plugin) * Images with the kubernetes or containers modules should include nvidia-extra to support Nvidia GPUs.

1 0

Security/Bug fix releases 14.0.2, 13.3.2
by no-reply-list＠qlustar.org 05 Jul '25

05 Jul '25

The Qlustar releases 14.0.2 and 13.3.2 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0703251 https://qlustar.com/qsa/2025/0703252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.4 * Add '# GPUs' HW property and /etc/qlustar/options.d/gpu-count * NewHostsWidget: Catch error looking up owner of Mac addresses * Improve AD Import Users dialog * Redesign prometheus label logic depending on host role/functionality * Fix bug where LDAP UIDs where multiply added to internal search indices * LDAP: Handle LDAP connection failing on start * LDAP ConfigWidget: Add tooltips and units for password policy settings * Set 'sudo_provider = none' in generated ssd.conf to prevent slapd errors * qluman-cli: Save user certificate in original PWD if save path is relative * Copy config files listed by image modules to compute nodes on boot rather than having them pulled into images during the build phase. * Implement binary support for file transfer to execd. Allows binary files in root-fs customizations. - core module * Activate ACL support for ZFS based /var and /scratch * Add lshw - container module * Add Nvidia CDI support for containerd/kubernetes - nvidia module * Add nvidia fabric manager to support NVSwitch systems * Add nvidia datacenter GPU manager * Add nvidia-bug-report.sh * Add firmware link so that firmware is always found by the kernel module - Kubernetes module * Update to 1.32.5 * Add helm * Add nvidia-device-plugin to support Nvidia GPU workloads - Slurm module * Add correct path for slurm dynamic loading of libpmix For Qlustar 13 and 14: - core module * Add systemd oomscore dropins to replace old mechanism for lowering oomscore of important system processes

1 0

Re: Users job processes are not killable by OOM
by Rolandas Naujikas 02 Jun '25

02 Jun '25

diff -u /usr/sbin/ql-adjust-oomscore ql-adjust-oomscore --- /usr/sbin/ql-adjust-oomscore 2025-04-15 11:41:22.000000000 +0300 +++ ql-adjust-oomscore 2025-05-28 15:52:27.993037862 +0300 @@ -32,6 +32,7 @@ systemd Xorg sssd rsyslogd dbus-daemon lightdm mysqld \ qluman-execd.py qlumand.py qluman-dhcpscanner.py qluman-router.py \ slurmctld slurmd" +USERS="root,_rpc,statd,messagebus,systemd-timesync,ganglia" DEFAULTS=/etc/default/oomscore function set_oom() { @@ -42,7 +43,7 @@ SCORE="$COMPAT_SCORE" fi for proc in $procs; do - pids=$(pgrep -f $proc) + pids=$(pgrep -u $USERS -f $proc) for i in $pids; do oom_adj=/proc/$i/${target_file} if [ -f $oom_adj ]; then On 2025-05-28 14:15, rolnas(a)gmail.com wrote: > Hi, > > Finally found a source of problem. In Qlustar 13 distribution there is a periodic (hourly) script to make some processes unkillable for OOM, but pattern to choose processes by they command line is too wide and process 182580 were matching it (because of slurmd in name). > > Regards > Rolandas > > P.S. Script ql-adjust-oomscore. > > On 2025-05-28 11:29, rolnas(a)gmail.com wrote: >> Hi, >> >> We are using SLURM 24.11.5 and maintaining HPC cluster. >> Several times we had nodes stuck after user jobs. >> Investigation lead to OOM situations. >> Deeper investigation found in some strange situations users jobs oom_score_adj is left at -1000 as slurmstepd process. >> >> For e.g. (some information is changed for privacy): >> root 182565 0.0 0.0 947952 30128 ? SLl May27 0:04 slurmstepd: [177126.batch] >> hpc_user 182580 0.0 0.0 1845524 70696 ? Sl May27 0:04 \_ / scratch/lustre/home/hpc_user/miniforge-pypy3/envs/snakemake/bin/ python3.12 /var/lib/slurm-llnl/slurmd/job177126/slurm_script -p all -- use-conda --rerun-incomplete >> hpc_user 241662 99.9 7.2 30762668 28598376 ? R May27 1418:09 \_ ClonalFrameML ... >> >> cat /proc/{182565,182580,241662}/oom_score_adj >> -1000 >> -1000 >> -1000 >> >> I'm trying to replicate that situation and no luck. Every time I'm getting users jobs with oom_score_adj=0 what it should be. >> >> Regards >> Rolandas >> >> P.S. Command to launch this job was: >> sbatch -A alloc_2a21c_test -c 12 --time=100:00:00 snakemake -p all -- use-conda --rerun-incomplete

2 3

Connection error using Qluman singularity images
by Beethoven Santos 28 May '25

28 May '25

Dear All, I cannot use the Qluman singularity images to connect to a small cluster I am managing. When I try to connect, Qluman refuses to do so with the following message: non-network local connections being added to access control list qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in "" QStandardPaths: XDG_RUNTIME_DIR points to non-existing path '/run/user/1000', please create it with 0700 permissions. Qt: Session management error: None of the authentication protocols specified are supported 2025-05-27 06:08:12,776 ERROR qlunet.Node router for '******' is not responding to pings: reconnecting I had to replace sensitive information with *'s. Some additional info: Qluman singularity image version: 13.7.2 Version of qluman-* pkgs in the cluster: 13.5.6 Please help me with this problem. Sincerely, Beethoven Santos

2 1

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General