- Qlustar General - Qlustar Mailing Lists

users created with LDAP in QluMan can't connect to head node with ssh
by saeed.s.jahromi＠gmail.com 22 Feb '22

22 Feb '22

Greetings I have installed the qlustar on a small cluster with one head node and 15 compute nodes. I managed to add all nodes to the cluster and configure slurm on the cluster via QluMan. As the last step to make the cluster available to users, I started creating new groups and users with QluMan. after creating new users and syncing I tried to connect with users to the head node by ssh. However, all the users that I create can not connect to the head node because it says the user does not have premission. I also chose to create a virtual demo cluster during the installation and the demo user which is created during the installation also does not have permission to ssh login to the head node. I checked on the head node and all user's home directory is created in the /data/home/ directory but they can not connect to the cluster. I would be so thankful if you guide me on how should I resolve this issue or where in the qlustar config files should I look for the source of the problem. PS, this cluster was already using an older version of the qlustar from 6 years ago and we just updated it to the latest qlustar 12. Thanks in advance for your assistance. Saeed

3 5

AlmaLinux 8 substitutes CentOS 8
by Qlustar List 21 Feb '22

21 Feb '22

https://qlustar.com/news/2022/2022-02-21-almalinux-8

1 0

How to generate a image for tesla m2075 ?
by Gerhard W. Recher 21 Feb '22

21 Feb '22

Dear members, I just test transition from rocks cluster to qlustar. one point is our tesla m2075 nodes wont boot with nvidia driver due to not supportet by ubuntou-focal-nvidea. <snip> [ 60.417821] NVRM: The NVIDIA Tesla M2075 GPU installed in this system is NVRM: supported through the NVIDIA 390.xx Legacy drivers. Please NVRM: visit http://www.nvidia.com/object/unix.html for more NVRM: information. The 470.74 NVIDIA driver will ignore NVRM: this GPU. Continuing probe... [ 60.418062] NVRM: No NVIDIA GPU found. [ 60.418406] nvidia-nvlink: Unregistered the Nvlink Core, major device number 245 [ 65.107363] nvidia-nvlink: Nvlink Core is being initialized, major device number 245 <snap> so how can i achieve this on head-node to assemble a new boot image with older driver ? fail with this one: NVIDIA-Linux-x86_64-352.79.run nvidia-installer log file '/var/log/nvidia-installer.log' creation time: Sun Feb 20 13:33:54 2022 installer version: 352.79 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin nvidia-installer command line: ./nvidia-installer Unable to load: nvidia-installer ncurses user interface Using built-in stream user interface -> Detected 4 CPUs online; setting concurrency level to 4. WARNING: You do not appear to have an NVIDIA GPU supported by the 352.79 NVIDIA Linux graphics driver installed in this system. For further details, please see the appendix SUPPORTED NVIDIA GRAPHICS CHIPS in the README available on the Linux driver download page at www.nvidia.com. -> License accepted. -> Installing NVIDIA driver version 352.79. -> Performing CC sanity check with CC="/usr/bin/cc". -> Kernel source path: '/lib/modules/5.4.167-ql-generic-12.0-13/build' -> Kernel output path: '/lib/modules/5.4.167-ql-generic-12.0-13/build' -> Performing rivafb check. -> Performing nvidiafb check. -> Performing Xen check. -> Performing PREEMPT_RT check. -> Cleaning kernel module build directory. executing: 'cd ./kernel; /usr/bin/make clean'... -> Building NVIDIA kernel module: executing: 'cd ./kernel; /usr/bin/make module SYSSRC=/lib/modules/5.4.167-ql-generic-12.0-13/build SYSOUT=/lib/modules/5.4.167-ql-generic-12.0-13/build -j4 NV_BUILD_MODULE_INSTANCES='... NVIDIA: calling KBUILD... make[1]: Entering directory '/usr/src/linux-headers-5.4.167-ql-generic-12.0-13' /usr/bin/make -f ./Makefile syncconfig /usr/bin/make -f ./scripts/Makefile.build obj=scripts/basic rm -f .tmp_quiet_recordmcount /usr/bin/make -f ./scripts/Makefile.build obj=scripts/kconfig syncconfig scripts/kconfig/conf --syncconfig Kconfig Kconfig:34: can't open file "Documentation/Kconfig" make[3]: *** [scripts/kconfig/Makefile:73: syncconfig] Error 1 make[2]: *** [Makefile:590: syncconfig] Error 2 make[1]: *** [Makefile:696: include/config/auto.conf.cmd] Error 2 make[1]: Leaving directory '/usr/src/linux-headers-5.4.167-ql-generic-12.0-13' NVIDIA: left KBUILD. nvidia.ko failed to build! make: *** [nvidia-modules-common.mk:245: module] Error 1 -> Error. ERROR: Unable to build the NVIDIA kernel module. ERROR: Installation has failed. Please see the file '/var/log/nvidia-installer.log' for details. You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com. -- Gerhard W. Recher net4sec UG (haftungsbeschränkt) Leitenweg 6 86929 Penzing +49 8191 4283888 +49 171 4802507

1 0

tcsh segfault in Ubuntu 20.04 with external LDAP
by a.gabbana＠tue.nl 17 Feb '22

17 Feb '22

Dear Roland, we observe the following reproducible problem: - A fresh installation of qlustar 12 - A chroot with Ubuntu 20.04 - An external (fresh) ldap server (connection via shared conf files under /etc/qlustar/common/image-files) Calling tcsh returns segfault. The problem seems to be connected to libgssapi_krb5.so.2. The problem does not occur with a Ubuntu 18.04 chroot. Moreover, the problem does not occur with a fresh installation of Ubuntu 20.04 (without qlustar) connected to the same ldap Do you have any clue what could be triggering the problem / or can you suggest a possible workaround? best Alessandro

2 1

Cannot create a new chroot
by a.gabbana＠tue.nl 27 Jan '22

27 Jan '22

With Qlustar V12.0.6.0 I get the following error when trying to create a new chroot (similar thing happens if I try to clone/delete chroots): 2022-01-21 12:59:43,087 ERROR qlunet.Node Channel[('zmq_version_info', 6)].do_recv(): exception in request generator Traceback (most recent call last): File "/usr/lib/python3/dist-packages/qlunet/Node.py", line 151, in do_recv args = self._generator.send(msg) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/ManageChroots/ManageChrootsWidget.py", line 370, in create self.create_exec() File "/usr/lib/python3/dist-packages/qluman-12/client/qt/ManageChroots/ManageChrootsWidget.py", line 393, in create_exec ExecuteWidget.create_chroot(cluster=self.cluster, name=name, desc=desc, server=server, File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteWidget.py", line 625, in create_chroot ExecuteWidget.generic_execute_request(cluster=cluster, title=title, request=request) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteWidget.py", line 608, in generic_execute_request t = ExecuteTabWidget.create_request_execute(cluster, request, w, hidden=hidden) File "/usr/lib/python3/dist-packages/qluman-12/client/qt/RxEngine/ExecuteTabWidget.py", line 639, in create_request_execute t = request["write_file"].hostlist KeyError: 'write_file'

4 4

Qlustar Feature Update 01/22
by Qlustar List 10 Jan '22

10 Jan '22

A number of new Qlustar features were completed recently and published together with the latest security updates. QluMan 12.0.6.0 has the following new capabilities: - QLUMAN TIMERS [1] present a new Config class assignable to nodes. They allow for time-based execution of commands on nodes and are easily configured via the GUI. - ENHANCEMENTS OF THE SLURM GUI COMPONENT [2]: * Added on-the-fly job filtering and job grouping. * Significantly improved mechanism to transfer job data. The QluMan GUI is now able to smoothly handle thousands of jobs. - HARDWARE DETECTION INFO for nodes in the GUI Enclosure view [3] now transparently highlights misconfigurations, helping admins to track down faulty settings. - FILTERS IN USER/GROUP MANAGEMENT [4] have been added, making LDAP management a lot clearer and easier to follow. Other new features: - MELLANOX OFED [5] drivers and tools have been included for optimized support of clusters with newest Infiniband (IB) technology. Qlustar now automatically detects whether IB adapters supported by the current Mellanox OFED release are present and activates its drivers if that's the case. For all other IB hardware, the in-kernel drivers are used. - GPU DIRECT RDMA [6] support has been added for nodes running Nvidia GPUs together with IB adapters supported by Mellanox OFED. It provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all GPU-GPU communications across the network. - The VIRTUALGL image module is now production ready. When used in a node image, Qlustar can provide 3D remote visualization on nodes with the required hardware. [1] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/Config-Classes… [2] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/components/slu… [3] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/Adding-Hosts.h… [4] https://docs.qlustar.com/Qlustar/12.0/ClusterOS/qluman-guide/components/lda… [5] https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed [6] https://www.mellanox.com/products/GPUDirect-RDMA

1 0

Security/Bug fix releases 12.0.0.8-b546f1425 / 11.0.1.12-b543f1424
by Qlustar List 28 Dec '21

28 Dec '21

The Qlustar releases 12.0.0.8-b546f1425 and 11.0.1.12-b543f1424 are ready including a number of security/bug fixes and improvements. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2021/1221211 https://qlustar.com/qsa/2021/1221212 The following non-security related enhancements/bug fixes are included: For 12.0 only: - QluMan 12.0.6.0 (GUI update to 12.0.6.0 needed as well) * Major overhaul of slurm component - Monitor slurm data only when a GUI client needs it - Remove duplicate job properties before transmitting to client - Encode run time of jobs so it doesn't need to be transmitted every 5 sec * Further improve Slurm JobManagement interface - Add better column titles for new job properties - Allow grouping of jobs by columns - Show number of jobs shown after filter / total jobs - Run time of running jobs now updates every second * Add reminder about syncing users and groups with nodes upon LDAP changes * Switch to unlimited lifetime of LDAP certs (previously expired after 1 year). Add option '--update-certs' to qluman-ldap-cli. * Set DHCP lease time to 1 hour * EnclosureView: Several fixes/improvements with enclosure handling * NetworksWidget: Allow routed networks to be master for slave networks * WriteFiles/ssh - Drop files in /etc/qlustar/common/image-files and include authorized_keys in preview - replace /etc/ssh/ssh_known_hosts + shosts.equiv symlinks with files on online nodes * Fix wrong timestamps in message viewer

1 0

Stack Smashing Detected During qlustar-image-reconfigure
by hereiam＠mit.edu 25 Nov '21

25 Nov '21

Hi everyone, I've got a new one for you all... I was updating my chroot image successfully using qlustar-image-reconfigure, and all was well. However, when I updated qlustar using apt-get upgrade as usual (I haven't done so since July), now I get this error whenever I try to reconfigure the chroot image: 0 root@geb /var/lib/qlustar/images/Standard-focal # qlustar-image-reconfigure Standard-focal Reconfiguring image Standard-focal focal/12.0 - creating squashfs for download using kernel flavour ql-generic-* *** stack smashing detected ***: terminated /usr/sbin/qlustar-image-create: line 661: 8766 Aborted DEBIAN_FRONTEND=noninteractive chroot "$IMAGE_DIR" pam-auth-update --package --force Building image Standard-focal focal/12.0 failed. Any idea what might have happened? I'm all ears, and can't seem to figure it out on my own. Thanks! -Mike

1 0

[Qlustar-General] Node Replacement Issue - Infinite DNSMasq Lease in /var/lib/misc/dnsmasq.leases
by hereiam＠mit.edu 25 Nov '21

25 Nov '21

Hi everyone, I've run into what may be an uncommon issue when replacing nodes in Qluman. Say I have a node all configured, then I completely remove that machine and replace it with a new one (may or may not be different specs). I have noticed that Qluman allows one to make the new node in the configs, but it never boots Qlustar. I also found the fix - the file /var/lib/misc/dnsmasq.leases doesn't appear to get updated *only when deleting a node.* Adding new nodes works perfectly, as does changing MAC addresses. However, I've had to go in and manually delete the corresponding entries from the file on the headnode, after which Qluman works perfectly adding a New Host with the old IP address. Can anyone else reproduce this issue, and if so, would an addition to the scripts to the tune of deleting the line matching the hostname or MAC address in /var/lib/misc/dnsmasq.leases be required for the next round of updates? It makes sense that leases should have infinite lifetime in a cluster (nodes should sit tight for years), though in this rare situation it requires a bit of extra manual tweaking. Thanks, -Mike

4 5

Security/Bug fix releases 12.0.0.7-b542f1400 / 11.0.1.11-b543f1399
by Qlustar List 19 Oct '21

19 Oct '21

The Qlustar releases 12.0.0.7-b542f1400 and 11.0.1.11-b543f1399 are ready including a number of security/bug fixes and improvements. Please check the following web pages for details about all security fixes and special update instructions: https://qlustar.com/qsa/2021/1019211 https://qlustar.com/qsa/2021/1019212 The following non-security related enhancements/bug fixes are included: For 12.0 only: - QluMan 12.0.5.0 (GUI update to 12.0.5.0 needed as well) * Improve Slurm JobManagement interface - Add quick-search filter - Make server <-> client job updates much more efficient - Improve job filters - Allow actions (e.g. kill job) on jobs with different states - Fix jobs not getting removed * Improve Slurm NodeState Management GUI dialogs - name enclosures after their hosts - filter enclosure view items to show only slurm nodes - fix exception sorting enclosures * qluman-slurmd: Avoid memory leak in pyslurm creating data classes * FilesystemExports Widget - Correct network priorities when changing servers - fix broken priorities when an entry is displayed * FilesystemMounts - select server name for exports in the network of the server with a gateway - try to match NIC types for gateways * Add GlobalProp SearchDomains by default, change seperator to " " add DNS search domains to /etc/resolv.conf - Add cleanup for stale unionfs files upon boot - Update Nvidia driver to 470.74 - slurm module * Update to 20.02.7 * Fix a memory leak in the job structure

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General