Hello Ansgar Esztermann,
I'm sorry to hear about the boot failure. Unfortunately the logs you provided don't shed any light on the cause. But let me explain what I see in the logs:
router.log ========
The "Zap request" is ZeroMQs way of authenticating peers. So this showing up at all means the node can talk to the router. Next the "Asking master" shows the router forwarding the authentication request to the qlumand and "Authenticated peer" shows the qlumand has responded. The peer was correctly identified and authenticated as an execd. So far everything is going right.
Even the last line, that the exec has expired might or might not be normal. During boot the node requests the configuration for the node and then the qluman-execd process restarts with the new configuration. From the router log it's impossible to say if the disconnect is due to a failure to get the configuration or the planned restart. From what I can tell the router did it's job fine, it routed all the traffic correctly.
That leaves 2 other components in the mix:
1) The qlumand running on the headnode.
Please have another look in the /var/log/qluman/qlumand.log and try to capture everything between the time you turn on a node and the node failing with the message that it failed to request the node config.
2) The qluman-execd running on the node itself.
If you can please include the execd.log. Depending on the qlustar version the log should be either in /root/log or /run/log. Alternatively you can run execd manualy from the shell you get after the timeout using:
timeout -s TERM 120 /usr/sbin/qluman-execd --write-conf 2>&1 | tee execd.log
trace.ksh ========
"Lost connection to MySQL server during query" means exactly that. The qluman-server lost the connection to the database where all the configurations are stored. But you are correct that this isn't correlated to the nodes booting. "scan_apt_database" is invoked on updates to detect new versions of qlustar image modules. If at the same time the mysql package is updated then the mysql server is shut down for restart causing the connection to be dropped.
So this exception is not totaly unexpected. You can check if mysqld is running but if the error does not repeat itself then it is nothing to worry about. The qlumand automatically reconnects to mysqld when it has restarted.
I hope this brings us closer to finding the problem.
Yours, Goswin von Brederlow