- Qlustar General - Qlustar Mailing Lists

Spack DB visibility delay on compute nodes — NFSv3 mount session never refreshes (mount -o remount is a no-op)
by garroyo＠ifimar-conicet.gob.ar 31 Aug '26

31 Aug '26

Hi all, We run Qlustar 14 with the standard Spack "HPC Core Stack" setup (softadm group, /srv/apps NFSv3 export, diskless netboot compute nodes). We ran into a reproducible issue that we tracked down fully and wanted to share, since it doesn't seem to be documented anywhere and may affect other sites with the same standard layout. ## Symptom After `spack install <pkg>` completes on the head node, the new package is correctly visible via `spack find` on the head node immediately, but stays INVISIBLE from `spack find` on compute nodes for an unpredictable amount of time — anywhere from hours to several days. Raw reads of `.spack-db/index.json` on the affected compute node (via `grep`/`cat`) show the file IS up to date at the filesystem level (correct mtime/size/content), but `spack find` on that same node, at that same moment, still returns the stale package list. So it isn't a filesystem staleness problem in the usual sense — the data is there and correct, but the client's NFS session serving it isn't being refreshed. ## Root cause found The compute nodes mount `/srv/apps` (NFSv3, hard, tcp) at boot. This mount is NOT in `/etc/fstab` in our default netboot image — it's set up directly during boot by the image/initramfs, outside of fstab. We confirmed via `/proc/self/mountstats` that the mount's `age` field stays constant (multiple days) even immediately after running `mount -o remount /apps/local` as root. In other words, **`-o remount` does not reset the underlying NFS session at all** — it's a superficial VFS-level operation. We initially believed a remount was fixing the visibility issue (it appeared to work in a few manual tests), but a follow-up sweep showed previously "fixed" nodes degraded again within hours — which matches: the apparent fix was never a real session refresh, just coincidental timing. The actual fix is a genuine `umount` + `mount` cycle (not `-o remount`). After that, `age` in mountstats correctly resets to 0, and `spack find` immediately reflects the current DB. ## Fix validated in production (18 compute nodes) Interesting operational detail: since we deliberately avoid `-f`/`-l` on the `umount`, this is safe to run even on nodes with active jobs. On nodes where a job had open file handles under the mount, `umount` simply fails cleanly ("device is busy") without touching anything — but the subsequent `mount` still succeeds, stacking a fresh NFS session on top of the busy one underneath. Processes with the old mount already open keep working unaffected; new path lookups get the fresh session. We validated this across all 18 compute nodes in one pass, without filtering for idle state: 18/18 converged to the correct package count, zero jobs affected. ## Questions for the list - Has anyone else hit this specific NFSv3-session-staleness-vs-remount behavior with Spack on Qlustar's standard /srv/apps layout? - Is there a recommended/supported way to trigger a real per-node NFS session refresh after Spack installs (e.g. a hook in the install workflow, or a documented cron pattern), rather than each site reinventing this? - Is NFSv4 (with its integrated locking/session model) considered a supported alternative for /srv/apps in a future Qlustar release, given /srv was reportedly chosen specifically to keep that door open? Happy to share our diagnostic scripts / the umount+mount wrapper we use if useful to anyone else. Thanks, Gonzalo Arroyo IFIMAR-UNMDP / CONICET HPC cluster

2 1

Security/Bug fix releases 14.1.11, 13.4.10
by no-reply-list＠qlustar.org 29 Aug '26

29 Aug '26

The Qlustar releases 14.1.11 and 13.4.10 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0829261 https://qlustar.com/qsa/2026/0829262 No non-security related updates/enhancements/bug fixes are included.

1 0

Security/Bug fix releases 14.1.10, 13.4.9
by no-reply-list＠qlustar.org 22 Aug '26

22 Aug '26

The Qlustar releases 14.1.10 and 13.4.9 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0822261 https://qlustar.com/qsa/2026/0822262 No non-security related updates/enhancements/bug fixes are included.

1 0

Qlustar 14, cannot use lustre-2.15 module
by Rolandas Naujikas 12 Aug '26

12 Aug '26

Hi, The following packages have unmet dependencies: qlustar-module-lustre-2.15-noble-amd64-14 : Depends: qlustar-module-core-noble-amd64-14 (= 14.0.5-b589f1605) but 14.1.9-b589f1641 is to be installed So I forced to use lustre-2.17, but I would like to update first from Qlustar 13 with lustre-2.15 to the same version of Lustre, so I could go back in case of problems. Regards Rolandas

2 3

Security/Bug fix releases 14.1.9, 13.4.8
by no-reply-list＠qlustar.org 08 Aug '26

08 Aug '26

The Qlustar releases 14.1.9 and 13.4.8 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0808261 https://qlustar.com/qsa/2026/0808262 No non-security related updates/enhancements/bug fixes are included.

1 0

Security/Bug fix releases 14.1.8, 13.4.7
by no-reply-list＠qlustar.org 01 Aug '26

01 Aug '26

The Qlustar releases 14.1.8 and 13.4.7 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0801261 https://qlustar.com/qsa/2026/0801262 The following non-security related updates/enhancements/bug fixes are included: For Qlustar 14 only: - Add mstflint 4.36.0 (Nvidia/Mellanox firmware management tools) Debian package and also add the tools to the ofed qlustar image module. - On AlmaLinux 8 only: Fix the Qlustar nvidia prometheus scraper and make sure the qlustar-prometheus-node-metrics systemd timer is activated at boot.

1 0

Qlustar Feature Update 07/26
by no-reply-list＠qlustar.org 31 Jul '26

31 Jul '26

A large bunch of new features were completed and published since the last Qlustar 14 feature release 11/25. Please consult the comprehensive overview at https://qlustar.com/news/2026/2026-07-31-feature-update-07-26

1 0

Security/Bug fix releases 14.1.7, 13.4.6
by no-reply-list＠qlustar.org 27 Jul '26

27 Jul '26

The Qlustar releases 14.1.7 and 13.4.6 are ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0727261 https://qlustar.com/qsa/2026/0727262 Qlustar 13.4.6/Ubuntu includes a fix for GhostLock (CVE-2026-43499). No non-security related updates/enhancements/bug fixes are included.

1 0

Security/Bug fix release 14.1.6
by no-reply-list＠qlustar.org 22 Jul '26

22 Jul '26

The Qlustar release 14.1.6 is ready including a number of security/bug fixes. Please check the following web pages for details about security fixes and special update instructions: https://qlustar.com/qsa/2026/0721261 https://qlustar.com/qsa/2026/0721262 Please note: Due to the increased threat level by the high frequency of discovered and exploitable security bugs, we are now shortening our usual security update release cycle and will couple it mostly to newly released kernel.org kernel versions. Since the kernel.org update frequency of the Qlustar 13 kernel 5.15.x is a lot less than that for the 6.12.x Qlustar 14 kernel, this means that we will also have security releases covering only Qlustar 14. This is an example. No non-security related updates/enhancements/bug fixes are included.

1 0

Stange Qluman quirks
by João Rodrigues 21 Jul '26

21 Jul '26

I recently upgraded our cluster from Qlustar 13 to 14. Since yesterday I noticed strange behaviors in Qluman GUI: 1. When using Hardware or GPU wizards, no nodes are shown in the "selected" list, even though I selected one in the "Enclosure view" window. 2. On the "Slurm Config" window, when creating a new "Gres group", I select a Type of GPU, but when pressing enter or tab, it is cleared. 3. Today, the "Enclosure view" is completely blank. Since I noticed these problems yesterday, I have avoided writing changes to files. What could be causing this? João Rodrigues

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

Qlustar General