- Qlustar General - Qlustar Mailing Lists

Add users from command line
by Rolandas Naujikas 29 Oct '25

29 Oct '25

Hi, Do there is some way to add/manage users in Qlustar 12/13/14 from command line ? Some admins have difficulties to run qluman GUI. Could it be done via Qluman router API ? Regards Rolandas

1 0

Security/Bug fix releases 14.0.4, 13.3.4
by no-reply-list＠qlustar.org 18 Oct '25

18 Oct '25

The Qlustar releases 14.0.4 and 13.3.4 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/1017251 https://qlustar.com/qsa/2025/1017252 https://qlustar.com/qsa/2025/1017253 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.7 * Remove DB backup script and cron entry. It is now done via stuff in qluman-misc-tools package. * Slurm: Detect HA setup and restart slurmctld via HA system when writing the config. * Alerts: - Drop node_interrupts_total metric, to reduce output. - Add scrapping kubelet-cadvisor. * Power Management: Implement power status, cycle and reset for redfish. * New Hosts Widget: Fix Import MACs feature. * Remote Management: - Allow removing and reordering BIOS boot devices. - Check BIOS boot order on write and sync. - Move configured BIOS boot devices to the front of boot order if they exist. - Store settings checkmarks in DB. * Add qlumand debug options for redfish access and BIOS settings. - core module * Monitoring stack: - Further trim down node-exporter metrics. - kubernetes module * Add kubectl bash shell completion support.

1 0

Security/Bug fix releases 14.0.3, 13.3.3
by no-reply-list＠qlustar.org 28 Aug '25

28 Aug '25

The Qlustar releases 14.0.3 and 13.3.3 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0826251 https://qlustar.com/qsa/2025/0826252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.6 * qlumand - Add Depends and Provides to qlustar-module infos * New GUI feature: Implement network proxy to the managed clusters allowing firefox (embedded in singularity image now) to easily connect to a BMCs web interface via a button in the GUI. * RemoteManagement - Add filter for BIOS Settings - Improve on the display of BIOS options: Show the real BIOS text rather than cryptic option keys. - Add BootDevices to BIOS read and change writing boot order * Monitoring - Add direct proxied connections to prometheus / prometheus-alert-manager - Add interface to handle silencing of alerts * HA - Fix resources start command getting executed twice in parallel due to race condition with state updates - Catch monitoring being done on resources with an action due to update races * LDAP user-/group management - Pass ldap group name as 5th argument to add/remove user hook * NewHostWidget - Add ignored MACs feature - Increase default height so more than 1 Unknwon MAC is shown * LDAP - CreateUsersDialog + Enable Retry button for slurm account creation when done with errors + Count slurmd offline instances as errors + Color code table entries for pending / done / error + Handle errors when slurmdb is offline - Add Run / Run without Slurm / Save and Sync Node buttons - Fix exception in generating result text when deleting groups - Fix exceptions with new filtering when importing from passwd file - Sort tab order of UserManagementWidget * Slurm: - Export slurmd status in slurmd component (GUI) - Change calculation of available RAM for Slurm to make it change less often after updates due to varying kernel and image size - Add slurm daemon restart menu in slurmd LED. Allows to restart slurmctld, slurmdbd and qluman-slurmd. - Better handle situations where slurm head-node components are unavailable. * Execd: fix exception when diffing file that doesn't yet exist * Fix DHCP scanner to understand new syslog timestamps * Alerts: - Include config for prometheus scraper - Handle VirtualGL Remote Desktop nodes as separate group * Internal change: Use improved common exec_os_cmd function everywhere. - core module * Monitoring stack: - Trim down node-exporter metrics to generally needed ones - Add IPMI sensors metrics - nvidia modules * Add nvidia-570 module with 570 driver series and switch to open GPU kernel modules * Add nvidia-extra module that contains everything that doesn't depend on a specific driver version (e.g. container toolkit and kubernetes device plugin) * Images with the kubernetes or containers modules should include nvidia-extra to support Nvidia GPUs.

1 0

Security/Bug fix releases 14.0.2, 13.3.2
by no-reply-list＠qlustar.org 05 Jul '25

05 Jul '25

The Qlustar releases 14.0.2 and 13.3.2 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0703251 https://qlustar.com/qsa/2025/0703252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.4 * Add '# GPUs' HW property and /etc/qlustar/options.d/gpu-count * NewHostsWidget: Catch error looking up owner of Mac addresses * Improve AD Import Users dialog * Redesign prometheus label logic depending on host role/functionality * Fix bug where LDAP UIDs where multiply added to internal search indices * LDAP: Handle LDAP connection failing on start * LDAP ConfigWidget: Add tooltips and units for password policy settings * Set 'sudo_provider = none' in generated ssd.conf to prevent slapd errors * qluman-cli: Save user certificate in original PWD if save path is relative * Copy config files listed by image modules to compute nodes on boot rather than having them pulled into images during the build phase. * Implement binary support for file transfer to execd. Allows binary files in root-fs customizations. - core module * Activate ACL support for ZFS based /var and /scratch * Add lshw - container module * Add Nvidia CDI support for containerd/kubernetes - nvidia module * Add nvidia fabric manager to support NVSwitch systems * Add nvidia datacenter GPU manager * Add nvidia-bug-report.sh * Add firmware link so that firmware is always found by the kernel module - Kubernetes module * Update to 1.32.5 * Add helm * Add nvidia-device-plugin to support Nvidia GPU workloads - Slurm module * Add correct path for slurm dynamic loading of libpmix For Qlustar 13 and 14: - core module * Add systemd oomscore dropins to replace old mechanism for lowering oomscore of important system processes

1 0

Re: Users job processes are not killable by OOM
by Rolandas Naujikas 02 Jun '25

02 Jun '25

diff -u /usr/sbin/ql-adjust-oomscore ql-adjust-oomscore --- /usr/sbin/ql-adjust-oomscore 2025-04-15 11:41:22.000000000 +0300 +++ ql-adjust-oomscore 2025-05-28 15:52:27.993037862 +0300 @@ -32,6 +32,7 @@ systemd Xorg sssd rsyslogd dbus-daemon lightdm mysqld \ qluman-execd.py qlumand.py qluman-dhcpscanner.py qluman-router.py \ slurmctld slurmd" +USERS="root,_rpc,statd,messagebus,systemd-timesync,ganglia" DEFAULTS=/etc/default/oomscore function set_oom() { @@ -42,7 +43,7 @@ SCORE="$COMPAT_SCORE" fi for proc in $procs; do - pids=$(pgrep -f $proc) + pids=$(pgrep -u $USERS -f $proc) for i in $pids; do oom_adj=/proc/$i/${target_file} if [ -f $oom_adj ]; then On 2025-05-28 14:15, rolnas(a)gmail.com wrote: > Hi, > > Finally found a source of problem. In Qlustar 13 distribution there is a periodic (hourly) script to make some processes unkillable for OOM, but pattern to choose processes by they command line is too wide and process 182580 were matching it (because of slurmd in name). > > Regards > Rolandas > > P.S. Script ql-adjust-oomscore. > > On 2025-05-28 11:29, rolnas(a)gmail.com wrote: >> Hi, >> >> We are using SLURM 24.11.5 and maintaining HPC cluster. >> Several times we had nodes stuck after user jobs. >> Investigation lead to OOM situations. >> Deeper investigation found in some strange situations users jobs oom_score_adj is left at -1000 as slurmstepd process. >> >> For e.g. (some information is changed for privacy): >> root 182565 0.0 0.0 947952 30128 ? SLl May27 0:04 slurmstepd: [177126.batch] >> hpc_user 182580 0.0 0.0 1845524 70696 ? Sl May27 0:04 \_ / scratch/lustre/home/hpc_user/miniforge-pypy3/envs/snakemake/bin/ python3.12 /var/lib/slurm-llnl/slurmd/job177126/slurm_script -p all -- use-conda --rerun-incomplete >> hpc_user 241662 99.9 7.2 30762668 28598376 ? R May27 1418:09 \_ ClonalFrameML ... >> >> cat /proc/{182565,182580,241662}/oom_score_adj >> -1000 >> -1000 >> -1000 >> >> I'm trying to replicate that situation and no luck. Every time I'm getting users jobs with oom_score_adj=0 what it should be. >> >> Regards >> Rolandas >> >> P.S. Command to launch this job was: >> sbatch -A alloc_2a21c_test -c 12 --time=100:00:00 snakemake -p all -- use-conda --rerun-incomplete

2 3

Connection error using Qluman singularity images
by Beethoven Santos 28 May '25

28 May '25

Dear All, I cannot use the Qluman singularity images to connect to a small cluster I am managing. When I try to connect, Qluman refuses to do so with the following message: non-network local connections being added to access control list qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in "" QStandardPaths: XDG_RUNTIME_DIR points to non-existing path '/run/user/1000', please create it with 0700 permissions. Qt: Session management error: None of the authentication protocols specified are supported 2025-05-27 06:08:12,776 ERROR qlunet.Node router for '******' is not responding to pings: reconnecting I had to replace sensitive information with *'s. Some additional info: Qluman singularity image version: 13.7.2 Version of qluman-* pkgs in the cluster: 13.5.6 Please help me with this problem. Sincerely, Beethoven Santos

2 1

Qlustar 12 is EOL
by no-reply-list＠qlustar.org 16 May '25

16 May '25

With the latest security release, Qlustar 12 has reached its EOL and will receive no further updates. Please upgrade to Qlustar 13 as soon as possible in case your cluster is still running Qlustar 12. See https://docs.qlustar.com/Qlustar/13/ClusterOS/release-notes.html#_update_in… for instructions.

1 0

Security/Bug fix releases 14.0.1, 13.3.1, 12.0.3.12
by no-reply-list＠qlustar.org 16 May '25

16 May '25

The Qlustar releases 14.0.1, 13.3.1 and 12.0.3.12 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0515251 https://qlustar.com/qsa/2025/0515252 https://qlustar.com/qsa/2025/0515253 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - QluMan 14.0.3 * Fix AttributeError in HostnameWidget save method * RemoteManagement: - Implement clone button - Save changes in redfish CheckBox

1 0

Qlustar 14 is out with fully revamped monitoring
by no-reply-list＠qlustar.org 07 May '25

07 May '25

The Qlustar team is pleased to announce the immediate availability of Qlustar 14 for download [1]. It updates Qlustar's core platform to current Ubuntu 24.04 LTS. The AlmaLinux [2] edge platform remains based on 8.10. The highlight of this release is the new state-of-the-art Qlustar Monitoring stack. It is now based on Grafana [3] and Prometheus [4]. Both of them are tightly integrated with the other Qlustar components and can be controlled with a powerful new configuration interface [5] of the QluMan GUI. This release also includes the following individual major HPC core stack component updates: - Spack version 0.23.1 [6] - Slurm version 24.11.3 [7] - RDMA-Core version 55.0 [8] - HWloc version 2.11.2 [9] - UCX version 1.18.1 [10] Please consult the release notes [11] for more details. [1] https://qlustar.com/download/ [2] https://almalinux.org/ [3] https://grafana.com/grafana/ [4] https://prometheus.io/ [5] https://docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config-Classes.h… [6] https://spack.readthedocs.io/en/latest/ [7] https://www.schedmd.com/slurm/ [8] https://github.com/linux-rdma/rdma-core [9] https://www.open-mpi.org/projects/hwloc/ [10] https://openucx.org/ [11] https://docs.qlustar.com/Qlustar/14/ClusterOS/release-notes.html

3 2

Home File-System
by ignas.bunikis＠scilifelab.uu.se 08 Apr '25

08 Apr '25

Hej, Would you mind giving a short guide how to correctly setup Home File-System on NFS? Documentation just informs that: "If you choose Other, then no configuration for a home file-system will be done. In this case, you won’t be able to add any cluster users without further manual steps to setup a home file-system." Cheers, Iggy

2 7

Qlustar HPC Core Stack 25.03
by no-reply-list＠qlustar.org 22 Mar '25

22 Mar '25

The Qlustar team is pleased to announce the availability of the HPC Core Stack 25.03 for Qlustar 13. It was published together with the latest security bug fix update and can be installed by updating the Qlustar 13 chroots on the cluster. The HPC CORE STACK [1] was introduced with the release of Qlustar 13 as a new component definition within Qlustar. It is related to the introduction of Spack. This release includes the following individual major HPC core stack component updates: - Spack version 0.23.1 [2] - Slurm version 24.11.3 [3] - RDMA-Core version 55.0 [4] - HWloc version 2.11.2 [5] - UCX version 1.18.1 [6] The new version of Spack allows easy and direct access to the most recent compilers, MPIs and hundreds of other applications relevant to HPC. Amongst others, it includes gcc 14.2.0 and OpenMPI 5.0.5 as new default versions. Minor version updates for other Qlustar 13 components are also included, in particular Nvidia graphics drivers (550.144.03), Lustre (2.15.6) and gdrcopy (2.4.4) for use with Nvidia GPUs. [1] https://qlustar.com/qlustar/hpc-stack/hpc-core-stack/ [2] https://spack.readthedocs.io/en/latest/ [3] https://www.schedmd.com/slurm/ [4] https://github.com/linux-rdma/rdma-core [5] https://www.open-mpi.org/projects/hwloc/ [6] https://openucx.org/

1 0

Security/Bug fix releases 13.3.0, 12.0.3.11
by no-reply-list＠qlustar.org 22 Mar '25

22 Mar '25

The Qlustar releases 13.3.0 and 12.0.3.11 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0321251 https://qlustar.com/qsa/2025/0321252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Qlustar HPC Core Stack 03/25 update (details to be announced separately)

1 0

Denied permission when restoring a database in qluman-qt
by ldettmann＠web.de 25 Feb '25

25 Feb '25

Qlustar version: 13.2 Qluman version: 13.7.2 I am trying to restore a qluman database from another server using "Manage Cluster" > "Global Configs" > "Restore database" and encountered an error: Permission denied. However, I also tried it with a backup, which I created within the current session, and encountered the same error. What can I do? Best regards, Lorenz

3 4

QluMan configuration for single sign on
by João Manuel Rodrigues 19 Feb '25

19 Feb '25

Documentation says: "b) QluMan may be configured to allow for a single sign-on setup where users can be imported from and authenticated against an Active Directory (AD) server." <https://docs.qlustar.com/Qlustar/13/ClusterOS/qluman-guide/components/ldap.…> How can I configure QluMan this way? I have configured sssd on the head node to connect to our AD (ldap + kerberos), but in QluMan, "Components > LDAP > Import Users > From AD" is disabled (gray). Thanks, João

1 0

Security/Bug fix releases 13.2.4, 12.0.3.10
by no-reply-list＠qlustar.org 05 Feb '25

05 Feb '25

The Qlustar releases 13.2.4 and 12.0.3.10 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2025/0205251 https://qlustar.com/qsa/2025/0205252 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Update to QluMan 13.7.2 + Slurm: void exception when User Management is opened at startup + HA: - fix exception when mail can't be send - change koton command for ipmitool syntax + LoginData: add missing imports for converting v1 safe to v2 + qluman-qt: Add support for running on wayland

1 0

BeeFGS setup help
by aninramesh＠gmail.com 25 Jan '25

25 Jan '25

Hello, I'm currently attempting to establish a BeeFGS file system on qlustar. Thus far, I've been following the qlustar documentation to assist with the setup process. However, I've encountered a problem following the step "+ Their beegfs role is assigned to the nodes in /etc/qlustar/common/image-files/softgroups. Change the node names there according to your naming scheme," that is found in the [BeeGFS on Qlustar - Quick Start] https://docs.qlustar.com/Qlustar/13/DataStack/quick-start-guide.html. Despite integrating the client and server name in accordance with my qlustar setup, the mount point does not appear on the client side. I attempted to restart the nodes, but this was unsuccessful. Is it possible for you to provide some guidance on this issue? I also want to point out that an error occurs when I input the `beegfs-ctl` command on the storage node. Even though the configuration file indicates that connDisableAuthentication is set to true, the error message states, "Unrecoverable error: No connAuthFile configured. Using BeeGFS without connection authentication is considered insecure and is not recommended. If you really want or need to run BeeGFS without connection authentication, please set connDisableAuthentication to true." I have also made an attempt to mount a drive via the 'Network FS Mounts' dialogue box in Qluman, but to no avail. I followed the instructions provided in section 7.5 - Network Filesystem Exports/Mounts. I welcome any suggestions or thoughts you may have on this issue. Additionally, I believe I have identified an error in the `https://docs.qlustar.com/Qlustar/13/DataStack/quick-start-guide.html`. Within section 1.2.1 (Create storage server instance), isn’t the command `beegfs-setup-storage -p /beegfs/stor -s ${HOSTNAME##-} -i ${HOSTNAME##*-}01 -m beosrv-c` as opposed to `beegfs-setup-storage -p /beegfs/stor -s ${HOSTNAME##*-} -i ${HOSTNAME##*-}01 -m beosrv-c`? Anin

2 1

Error in LoginData.py
by Yaroslav Boyko 10 Jan '25

10 Jan '25

Good afternoon ! After the last update I got the following error in qluman-qt: 2024-12-10 11:16:49,539 ERROR client.qt.ConnectClusterDialog loading ClusterModel failed: Traceback (most recent call last): File “/usr/lib/python3/dist-packages/qluman-13/client/qt/ConnectClusterDialog.py”, line 187, in __init__ self.undo() File “/usr/lib/python3/dist-packages/qluman-13/client/qt/ConnectClusterDialog.py”, line 232, in undo self.root = Node.from_safe(data) File “/usr/lib/python3/dist-packages/qluman-13/common/LoginData.py”, line 50, in from_safe t = ClusterInfo.from_safe(child) File “/usr/lib/python3/dist-packages/qluman-13/common/LoginData.py”, line 197, in from_safe ip=IPv4Address(data[“ip”]), NameError: name 'IPv4Address' is not defined To fix it, I inserted from ipaddress import IPv4Address in /usr/lib/python3/dist-packages/qluman-13/common/LoginData.py

2 1

Security/Bug fix releases 13.2.3, 12.0.3.9
by no-reply-list＠qlustar.org 07 Dec '24

07 Dec '24

The Qlustar releases 13.2.3 and 12.0.3.9 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2024/1206241 https://qlustar.com/qsa/2024/1206242 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Update to QluMan 13.7.1 + GUI: Add missing import of zmq so new keypairs can be created + Some improvements for the repair functionality of QluMan HA

1 0

MD5 and SHA256 hashes for the QluMan 12.0.7.5 Singularity image don't match those shown in the Download page
by Beethoven Santos 30 Nov '24

30 Nov '24

Hi, Whenever I download the file `qluman-12.0.7.5-singularity.sqsh` and use the `md5sum` and `sha256sum` commands to check its MD5 and SHA256 hashes, I find that they do not match the hash values shown on the [Download page](https://qlustar.com/download/) for this version (all others are ok). For this reason, I believe that using this file is not safe and I should avoid it. Is this something on Qlustar's side? Could anyone confirm this issue? Best B. Santos

2 2

How to add GrestTypes to slum.conf?
by uchida＠ulgs.jp 28 Nov '24

28 Nov '24

Hi, Now I'm using, qlustar management interface, but GPU wizard do delete the GresTypes=gpu to GresTypes=, then it caused an error as follows, ### beosrv-c ### writing /etc/qlustar/common/slurm-llnl/slurm.conf writing /etc/qlustar/common/slurm-llnl/gres.conf writing /etc/qlustar/common/slurm-llnl/cgroup.conf postcmd: service slurmctld restart; echo ' * Waiting for slurmd to be ready...'; sleep 5; scontrol reconfigure Job for slurmctld.service failed because the control process exited with error code. See "systemctl status slurmctld.service" and "journalctl -xeu slurmctld.service" for details. * Waiting for slurmd to be ready... scontrol: error: Parse error in file /etc/qlustar/common/slurm-llnl/slurm.conf line 169: "GresTypes=" scontrol: fatal: Unable to process configuration file 1 How to fix it and how to add GressTypes? Can I set as GresTypes=gpu,mps ? Regards, Morihisa

2 1

qluman-qt can't work
by uchida＠ulgs.jp 28 Nov '24

28 Nov '24

Hi, singularity image of qluman 13.7.0 and qluman-qt can't work to access at first time. It shows, File "/usr/lib/python3/dist-packages/qluman-13/client/qt/ConnectClusterDialog.py", line 846, in on_okButton_clicked cluster = ClusterInfo(alias=self.aliasLineEdit.text(), File "/usr/lib/python3/dist-packages/qluman-13/common/LoginData.py", line 169, in __init__ (self.client_pubkey, self.client_seckey) = zmq.curve_keypair() NameError: name 'zmq' is not defined Is there any workaround? Regards,

4 10

A new approach to Cluster High-Availability
by no-reply-list＠qlustar.org 20 Nov '24

20 Nov '24

The Qlustar team is pleased to announce the first public release of QluMan HA as the new driver of the Qlustar HA stack. Implemented on top of general Qlustar functionality, QluMan HA was *specifically designed for HPC, AI and Storage cluster solutions* where a scalable number of compute and storage nodes are controlled and managed by a pair of redundant head-nodes. Read the full announcement at https://qlustar.com/news/2024/2024-11-19-qluman-ha

1 0

Security/Bug fix releases 13.2.2, 12.0.3.8
by no-reply-list＠qlustar.org 19 Oct '24

19 Oct '24

The Qlustar releases 13.2.2 and 12.0.3.8 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2024/1019241 https://qlustar.com/qsa/2024/1019242 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Update to QluMan 13.7.0 + GUI improvement: Save open windows for each cluster tab and reopen them when reconnecting + Catch errors parsing LDAP user data and handle users without gecos field

1 0

Problem after upgrade to latest Qlustar 13
by Rolandas Naujikas 24 Aug '24

24 Aug '24

Hi, There are problem to create jammy chroot via qluman (3.6.0) GUI. As a workaround I tried to do that from cli with manage-qlustar-chroot. The problem is with version field. In qluman I can select only 13.1 or 13.2 - both don't work. Only 13.0 from cli works. I rechecked with freshly installation from 13.2-2 iso - the same problem. At least qlustar-initial-install script creates jammy chroot successfully. Regards Rolandas

2 1

Security/Bug fix releases 13.2.1, 12.0.3.7
by no-reply-list＠qlustar.org 24 Aug '24

24 Aug '24

The Qlustar releases 13.2.1 and 12.0.3.7 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2024/0822241 https://qlustar.com/qsa/2024/0822242 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 13 only: - Update to QluMan 13.6.3 + Fix jammy chroot creation + Make argument replacement in Remote Management Configs more powerful to also support IPMI addresses in networks where OS *AND* IPMI adapters are present.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General