Hi Roland,
I installed singularity in the chroot-xenial environment. During the installation a directory singularity is recreated in /var/lib which is required to run any image as user.
I started to debug the problem and it turned out that the directory /var from the chroot environment is not merged with /var from the image file. However when I boot a compute node as diskless everything works fine.
Thats why I come to the conclusion that something should go wrong in /lib/qlustar/disk-auto-setup
init_var() { local var_device=$1 fstype_opt=""
if ! $diskless; then grep -q $var_device /etc/fstab && \ fstype_opt="-t $(awk '$2 == "/var" {print $3}' /etc/fstab)" if ! mount $fstype_opt $var_device /mnt ; then echo "Unable to mount /mnt on disk partition $var_device" >&2 remove_start_links return 1 fi
# Copy ramdisk var directory on new var filesystem if $force_var_copy ; then cp -p -R /var/* /mnt else # Make sure that certain files of /var in the image # are always copied to the /var on disk # var device is now mounted on /mnt contents_file=/etc/qlustar/contents handle_contents ${contents_file} /var /mnt [ -d /var/yp ] && cp -a /var/yp /mnt fi
My guess is that here only the ramdisk is copied which is normal. However after I dont see that unionfs again is initialized for /var.
as a result I get the following when I try to singularity: $singularity shell container.img ERROR : Failed to resolve path to /var/lib/singularity/mnt/container: No such file or directory ABORT : Retval = 255