Hi all,
We run Qlustar 14 with the standard Spack "HPC Core Stack" setup (softadm group, /srv/apps NFSv3 export, diskless netboot compute nodes). We ran into a reproducible issue that we tracked down fully and wanted to share, since it doesn't seem to be documented anywhere and may affect other sites with the same standard layout.
## Symptom
After `spack install <pkg>` completes on the head node, the new package is correctly visible via `spack find` on the head node immediately, but stays INVISIBLE from `spack find` on compute nodes for an unpredictable amount of time — anywhere from hours to several days. Raw reads of `.spack-db/index.json` on the affected compute node (via `grep`/`cat`) show the file IS up to date at the filesystem level (correct mtime/size/content), but `spack find` on that same node, at that same moment, still returns the stale package list. So it isn't a filesystem staleness problem in the usual sense — the data is there and correct, but the client's NFS session serving it isn't being refreshed.
## Root cause found
The compute nodes mount `/srv/apps` (NFSv3, hard, tcp) at boot. This mount is NOT in `/etc/fstab` in our default netboot image — it's set up directly during boot by the image/initramfs, outside of fstab.
We confirmed via `/proc/self/mountstats` that the mount's `age` field stays constant (multiple days) even immediately after running `mount -o remount /apps/local` as root. In other words, **`-o remount` does not reset the underlying NFS session at all** — it's a superficial VFS-level operation. We initially believed a remount was fixing the visibility issue (it appeared to work in a few manual tests), but a follow-up sweep showed previously "fixed" nodes degraded again within hours — which matches: the apparent fix was never a real session refresh, just coincidental timing.
The actual fix is a genuine `umount` + `mount` cycle (not `-o remount`). After that, `age` in mountstats correctly resets to 0, and `spack find` immediately reflects the current DB.
## Fix validated in production (18 compute nodes)
Interesting operational detail: since we deliberately avoid `-f`/`-l` on the `umount`, this is safe to run even on nodes with active jobs. On nodes where a job had open file handles under the mount, `umount` simply fails cleanly ("device is busy") without touching anything — but the subsequent `mount` still succeeds, stacking a fresh NFS session on top of the busy one underneath. Processes with the old mount already open keep working unaffected; new path lookups get the fresh session. We validated this across all 18 compute nodes in one pass, without filtering for idle state: 18/18 converged to the correct package count, zero jobs affected.
## Questions for the list
- Has anyone else hit this specific NFSv3-session-staleness-vs-remount behavior with Spack on Qlustar's standard /srv/apps layout? - Is there a recommended/supported way to trigger a real per-node NFS session refresh after Spack installs (e.g. a hook in the install workflow, or a documented cron pattern), rather than each site reinventing this? - Is NFSv4 (with its integrated locking/session model) considered a supported alternative for /srv/apps in a future Qlustar release, given /srv was reportedly chosen specifically to keep that door open?
Happy to share our diagnostic scripts / the umount+mount wrapper we use if useful to anyone else.
Thanks, Gonzalo Arroyo IFIMAR-UNMDP / CONICET HPC cluster