- Qlustar General - Qlustar Mailing Lists

AlmaLinux 8 substitutes CentOS 8
by Qlustar List 21 Feb '22

21 Feb '22

https://qlustar.com/news/2022/2022-02-21-almalinux-8

1 0

How to generate a image for tesla m2075 ?
by Gerhard W. Recher 21 Feb '22

21 Feb '22

Dear members, I just test transition from rocks cluster to qlustar. one point is our tesla m2075 nodes wont boot with nvidia driver due to not supportet by ubuntou-focal-nvidea. <snip> [ 60.417821] NVRM: The NVIDIA Tesla M2075 GPU installed in this system is NVRM: supported through the NVIDIA 390.xx Legacy drivers. Please NVRM: visit http://www.nvidia.com/object/unix.html for more NVRM: information. The 470.74 NVIDIA driver will ignore NVRM: this GPU. Continuing probe... [ 60.418062] NVRM: No NVIDIA GPU found. [ 60.418406] nvidia-nvlink: Unregistered the Nvlink Core, major device number 245 [ 65.107363] nvidia-nvlink: Nvlink Core is being initialized, major device number 245 <snap> so how can i achieve this on head-node to assemble a new boot image with older driver ? fail with this one: NVIDIA-Linux-x86_64-352.79.run nvidia-installer log file '/var/log/nvidia-installer.log' creation time: Sun Feb 20 13:33:54 2022 installer version: 352.79 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin nvidia-installer command line: ./nvidia-installer Unable to load: nvidia-installer ncurses user interface Using built-in stream user interface -> Detected 4 CPUs online; setting concurrency level to 4. WARNING: You do not appear to have an NVIDIA GPU supported by the 352.79 NVIDIA Linux graphics driver installed in this system. For further details, please see the appendix SUPPORTED NVIDIA GRAPHICS CHIPS in the README available on the Linux driver download page at www.nvidia.com. -> License accepted. -> Installing NVIDIA driver version 352.79. -> Performing CC sanity check with CC="/usr/bin/cc". -> Kernel source path: '/lib/modules/5.4.167-ql-generic-12.0-13/build' -> Kernel output path: '/lib/modules/5.4.167-ql-generic-12.0-13/build' -> Performing rivafb check. -> Performing nvidiafb check. -> Performing Xen check. -> Performing PREEMPT_RT check. -> Cleaning kernel module build directory. executing: 'cd ./kernel; /usr/bin/make clean'... -> Building NVIDIA kernel module: executing: 'cd ./kernel; /usr/bin/make module SYSSRC=/lib/modules/5.4.167-ql-generic-12.0-13/build SYSOUT=/lib/modules/5.4.167-ql-generic-12.0-13/build -j4 NV_BUILD_MODULE_INSTANCES='... NVIDIA: calling KBUILD... make[1]: Entering directory '/usr/src/linux-headers-5.4.167-ql-generic-12.0-13' /usr/bin/make -f ./Makefile syncconfig /usr/bin/make -f ./scripts/Makefile.build obj=scripts/basic rm -f .tmp_quiet_recordmcount /usr/bin/make -f ./scripts/Makefile.build obj=scripts/kconfig syncconfig scripts/kconfig/conf --syncconfig Kconfig Kconfig:34: can't open file "Documentation/Kconfig" make[3]: *** [scripts/kconfig/Makefile:73: syncconfig] Error 1 make[2]: *** [Makefile:590: syncconfig] Error 2 make[1]: *** [Makefile:696: include/config/auto.conf.cmd] Error 2 make[1]: Leaving directory '/usr/src/linux-headers-5.4.167-ql-generic-12.0-13' NVIDIA: left KBUILD. nvidia.ko failed to build! make: *** [nvidia-modules-common.mk:245: module] Error 1 -> Error. ERROR: Unable to build the NVIDIA kernel module. ERROR: Installation has failed. Please see the file '/var/log/nvidia-installer.log' for details. You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com. -- Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 8191 4283888 +49 171 4802507

1 0

tcsh segfault in Ubuntu 20.04 with external LDAP
by a.gabbana＠tue.nl 17 Feb '22

17 Feb '22

Dear Roland, we observe the following reproducible problem: - A fresh installation of qlustar 12 - A chroot with Ubuntu 20.04 - An external (fresh) ldap server (connection via shared conf files under /etc/qlustar/common/image-files) Calling tcsh returns segfault. The problem seems to be connected to libgssapi_krb5.so.2. The problem does not occur with a Ubuntu 18.04 chroot. Moreover, the problem does not occur with a fresh installation of Ubuntu 20.04 (without qlustar) connected to the same ldap Do you have any clue what could be triggering the problem / or can you suggest a possible workaround? best Alessandro

2 1

Cannot create a new chroot
by a.gabbana＠tue.nl 27 Jan '22

27 Jan '22

With Qlustar V12.0.6.0 I get the following error when trying to create a new chroot (similar thing happens if I try to clone/delete chroots): 2022-01-21 12:59:43,087 ERROR qlunet.Node Channel[('zmq_version_info', 6)].do_recv(): exception in request generator Traceback (most recent call last): File "/usr/lib/python3/dist-packages/qlunet/Node.py", line 151, in do_recv args = self._generator.send(msg) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/ManageChroots/ManageChrootsWidget.py", line 370, in create self.create_exec() File "/usr/lib/python3/dist-packages/qluman-12/client/qt/ManageChroots/ManageChrootsWidget.py", line 393, in create_exec ExecuteWidget.create_chroot(cluster=self.cluster, name=name, desc=desc, server=server, File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteWidget.py", line 625, in create_chroot ExecuteWidget.generic_execute_request(cluster=cluster, title=title, request=request) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteWidget.py", line 608, in generic_execute_request t = ExecuteTabWidget.create_request_execute(cluster, request, w, hidden=hidden) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteTabWidget.py", line 639, in create_request_execute t = request["write_file"].hostlist KeyError: 'write_file'

4 4

Qlustar Feature Update 01/22
by Qlustar List 10 Jan '22

10 Jan '22

A number of new Qlustar features were completed recently and published together with the latest security updates. QluMan 12.0.6.0 has the following new capabilities: - QLUMAN TIMERS [1] present a new Config class assignable to nodes. They allow for time-based execution of commands on nodes and are easily configured via the GUI. - ENHANCEMENTS OF THE SLURM GUI COMPONENT [2]: * Added on-the-fly job filtering and job grouping. * Significantly improved mechanism to transfer job data. The QluMan GUI is now able to smoothly handle thousands of jobs. - HARDWARE DETECTION INFO for nodes in the GUI Enclosure view [3] now transparently highlights misconfigurations, helping admins to track down faulty settings. - FILTERS IN USER/GROUP MANAGEMENT [4] have been added, making LDAP management a lot clearer and easier to follow. Other new features: - MELLANOX OFED [5] drivers and tools have been included for optimized support of clusters with newest Infiniband (IB) technology. Qlustar now automatically detects whether IB adapters supported by the current Mellanox OFED release are present and activates its drivers if that's the case. For all other IB hardware, the in-kernel drivers are used. - GPU DIRECT RDMA [6] support has been added for nodes running Nvidia GPUs together with IB adapters supported by Mellanox OFED. It provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all GPU-GPU communications across the network. - The VIRTUALGL image module is now production ready. When used in a node image, Qlustar can provide 3D remote visualization on nodes with the required hardware. [1] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/Config-Classes… [2] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/components/slu… [3] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/Adding-Hosts.h… [4] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/components/lda… [5] https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed [6] https://www.mellanox.com/products/GPUDirect-RDMA

1 0

Security/Bug fix releases 12.0.0.8-b546f1425 / 11.0.1.12-b543f1424
by Qlustar List 28 Dec '21

28 Dec '21

The Qlustar releases 12.0.0.8-b546f1425 and 11.0.1.12-b543f1424 are ready including a number of security/bug fixes and improvements. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2021/1221211 https://qlustar.com/qsa/2021/1221212 The following non-security related enhancements/bug fixes are included: For 12.0 only: - QluMan 12.0.6.0 (GUI update to 12.0.6.0 needed as well) * Major overhaul of slurm component - Monitor slurm data only when a GUI client needs it - Remove duplicate job properties before transmitting to client - Encode run time of jobs so it doesn't need to be transmitted every 5 sec * Further improve Slurm JobManagement interface - Add better column titles for new job properties - Allow grouping of jobs by columns - Show number of jobs shown after filter / total jobs - Run time of running jobs now updates every second * Add reminder about syncing users and groups with nodes upon LDAP changes * Switch to unlimited lifetime of LDAP certs (previously expired after 1 year). Add option '--update-certs' to qluman-ldap-cli. * Set DHCP lease time to 1 hour * EnclosureView: Several fixes/improvements with enclosure handling * NetworksWidget: Allow routed networks to be master for slave networks * WriteFiles/ssh - Drop files in /etc/qlustar/common/image-files and include authorized_keys in preview - replace /etc/ssh/ssh_known_hosts + shosts.equiv symlinks with files on online nodes * Fix wrong timestamps in message viewer

1 0

Stack Smashing Detected During qlustar-image-reconfigure
by hereiam＠mit.edu 25 Nov '21

25 Nov '21

Hi everyone, I've got a new one for you all... I was updating my chroot image successfully using qlustar-image-reconfigure, and all was well. However, when I updated qlustar using apt-get upgrade as usual (I haven't done so since July), now I get this error whenever I try to reconfigure the chroot image: 0 root@geb /var/lib/qlustar/images/Standard-focal # qlustar-image-reconfigure Standard-focal Reconfiguring image Standard-focal focal/12.0 - creating squashfs for download using kernel flavour ql-generic-* *** stack smashing detected ***: terminated /usr/sbin/qlustar-image-create: line 661: 8766 Aborted DEBIAN_FRONTEND=noninteractive chroot "$IMAGE_DIR" pam-auth-update --package --force Building image Standard-focal focal/12.0 failed. Any idea what might have happened? I'm all ears, and can't seem to figure it out on my own. Thanks! -Mike

1 0

[Qlustar-General] Node Replacement Issue - Infinite DNSMasq Lease in /var/lib/misc/dnsmasq.leases
by hereiam＠mit.edu 25 Nov '21

25 Nov '21

Hi everyone, I've run into what may be an uncommon issue when replacing nodes in Qluman. Say I have a node all configured, then I completely remove that machine and replace it with a new one (may or may not be different specs). I have noticed that Qluman allows one to make the new node in the configs, but it never boots Qlustar. I also found the fix - the file /var/lib/misc/dnsmasq.leases doesn't appear to get updated *only when deleting a node.* Adding new nodes works perfectly, as does changing MAC addresses. However, I've had to go in and manually delete the corresponding entries from the file on the headnode, after which Qluman works perfectly adding a New Host with the old IP address. Can anyone else reproduce this issue, and if so, would an addition to the scripts to the tune of deleting the line matching the hostname or MAC address in /var/lib/misc/dnsmasq.leases be required for the next round of updates? It makes sense that leases should have infinite lifetime in a cluster (nodes should sit tight for years), though in this rare situation it requires a bit of extra manual tweaking. Thanks, -Mike

4 5

Security/Bug fix releases 12.0.0.7-b542f1400 / 11.0.1.11-b543f1399
by Qlustar List 19 Oct '21

19 Oct '21

The Qlustar releases 12.0.0.7-b542f1400 and 11.0.1.11-b543f1399 are ready including a number of security/bug fixes and improvements. Please check the following web pages for details about all security fixes and special update instructions: https://qlustar.com/qsa/2021/1019211 https://qlustar.com/qsa/2021/1019212 The following non-security related enhancements/bug fixes are included: For 12.0 only: - QluMan 12.0.5.0 (GUI update to 12.0.5.0 needed as well) * Improve Slurm JobManagement interface - Add quick-search filter - Make server <-> client job updates much more efficient - Improve job filters - Allow actions (e.g. kill job) on jobs with different states - Fix jobs not getting removed * Improve Slurm NodeState Management GUI dialogs - name enclosures after their hosts - filter enclosure view items to show only slurm nodes - fix exception sorting enclosures * qluman-slurmd: Avoid memory leak in pyslurm creating data classes * FilesystemExports Widget - Correct network priorities when changing servers - fix broken priorities when an entry is displayed * FilesystemMounts - select server name for exports in the network of the server with a gateway - try to match NIC types for gateways * Add GlobalProp SearchDomains by default, change seperator to " " add DNS search domains to /etc/resolv.conf - Add cleanup for stale unionfs files upon boot - Update Nvidia driver to 470.74 - slurm module * Update to 20.02.7 * Fix a memory leak in the job structure

1 0

A few Qlustar 12.0 questions
by david.whipp＠helsinki.fi 20 Sep '21

20 Sep '21

Hi, First off, many thanks Roland for your earlier help with the Qlustar 12.0 installer. We've been up and running for a few months now and have almost everything working as desired. I'm writing because I have a few questions both about some issues we have not been able to resolve and some introduced in the latest security updates (QSA-0903211 and QSA-0903212). Latest issues first: 1. Since installing the latest security updates a few days ago I have noticed that Slurm has stopped working and the qluman-qt 'Components' option is grayed out (see https://imgur.com/a/W1Ejsvn). I'm not totally sure what the issue might be, but I noticed that the slurmdbd and slurmctld services are not running on the head node, and in fact are masked, which I understand will stop them from starting, right? You can see the output from 'service --status-all' and 'systemctl list-unit-files --state=masked' below. Any suggestions about how to re-enable Slurm? service --status-all: 0 root@geo-hpcc ~ # service --status-all [ - ] apache-htcacheclean [ + ] apache2 [ - ] console-setup.sh [ + ] cpufrequtils [ + ] cron [ + ] dbus [ + ] dnsmasq [ ? ] ganglia-monitor [ ? ] gmetad [ - ] grub-common [ - ] hwclock.sh [ - ] ipmievd [ - ] jobmonarch-jobmond [ - ] keyboard-setup.sh [ + ] kmod [ + ] loadcpufreq [ - ] lvm2 [ - ] lvm2-lvmpolld [ + ] munge [ + ] mysql [ + ] nagios4 [ - ] networking [ - ] nfs-common [ + ] nfs-kernel-server [ + ] ntp [ + ] postfix [ + ] procps [ + ] qluman-dhcpscanner [ + ] qluman-execd [ + ] qluman-router [ + ] qluman-server [ - ] qluman-slurmd [ + ] resolvconf [ - ] rng-tools [ + ] rpcbind [ - ] rsync [ + ] rsyslog [ + ] saslauthd [ + ] schroot [ - ] scsitools-pre.sh [ - ] scsitools.sh [ + ] slapd [ - ] slurmctld [ - ] slurmdbd [ + ] ssh [ + ] sssd [ + ] sysstat [ + ] udev [ - ] x11-common systemctl list-unit-files --state=masked: 0 root@geo-hpcc ~ # systemctl list-unit-files --state=masked UNIT FILE STATE VENDOR PRESET cryptdisks-early.service masked enabled cryptdisks.service masked enabled hwclock.service masked enabled jobmonarch-jobmond.service masked enabled lvm2.service masked enabled nfs-common.service masked enabled qluman-slurmd.service masked enabled rc.service masked enabled rcS.service masked enabled scsitools-pre.service masked enabled scsitools.service masked enabled slurmctld.service masked enabled slurmdbd.service masked enabled sudo.service masked enabled x11-common.service masked enabled 15 unit files listed. 2. In addition to the Slurm issue, I have also not been able to create new user accounts because the 'Components' option in qluman-qt is grayed out. I have previously migrated our old NIS user accounts to LDAP, but I am lackign experience with LDAP and unsure how to explore this issue. And finally, one older issue we have not resolved: 3. How do you change user passwords for LDAP users in Qlustar 12? I had been able to generate new passwords in qluman, but have not been able to figure out how what users should do to update their passwords. It seems perhaps there is a lifetime to the autogenerated passwords, so I have manually reset passwords when users needed them. Obviously, this is not something that is good in the long run. Do you have a suggestion? And sorry for the compilation of questions above. Hopefully the fixes are obvious to you and not too much trouble. Best, Dave

3 5

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General