Missing dashboards in Grafana after upgrade
Hi, I managed to upgrade from 13 to 14 following the upgrade guide, but am missing all predefined dashboard of Grafana. Is there an alternative way to obtain these dashboards? Are the json files somewhere available? Thanks, Wouter
Hej, Have similar observation after upgrading to 14. It seems that /etc/prometheus/prometheus.yml is missing targets for job 'node' and therefor no metrics gets collected. Adding targets manually fixes it: - job_name: 'node' static_configs: - targets: ['beosrv-c:9100'] - targets: - comp-01:9100' - 'comp-02:9100' etc But I guess this should be generated automatically by qluman? Cheers, Iggy
On 2/16/26 16:31, Nicolai, Wouter via Qlustar General wrote:
Hi,
I managed to upgrade from 13 to 14 following the upgrade guide, but am missing all predefined dashboard of Grafana. Is there an alternative way to obtain these dashboards? Are the json files somewhere available?
Thanks,
Wouter
_______________________________________________ Qlustar General mailing list --qlustar-general@qlustar.org To unsubscribe send an email toqlustar-general-leave@qlustar.org
Hi, the predefined Grafana dashboards are created dynamically when writing the Alerts Config config class. The predefined dashboards are customized to what is monitored for alerts in the cluster. So if slurm isn't used or no slurm node is monitored then the slurm part of the dashboards are excluded. Similar for GPUs or Kubernetes. And if no nodes are monitored for alerts then no predefined dashboards. At a minimum you should create 2 Alert Groups in Qluman-qt: Head Node and Compute Node. Open the Alert Configs widget via the Manage Configs / Alert Configs menu entry and click the New button to create a new group. Both groups can be left empty. Next the Alert Groups need to be assigned to the nodes that should be monitored. For the Head Node group open the Enclosure View and select the beosrv-c node (in case of HA select the 2 physical headnodes instead). Then open the context menu and select: Set Config / Alerts / Head Node to assign the config. For the other nodes you probably have a Host Template that is used and is a more convenient way to add the Alerts Config. Open the Config Sets widget via the Manage Sets / Config Sets menu entry. Add the Compute Node alert group to the FE VM Node, Demo VM Node and Standard node Config Sets or whatever sets you have in use. Last click the Write Files button at the bottom of the Qluman-qt window (or Manage Cluster / Write Files menu entry). Writing the Alerts entry (or just write all) will update the Prometheus config to gather metrics from any node with an Alert Group assigned. It will also update the predefined Gafana dashboards to match the nodes Prometheus now monitors. You can read more about the alerts in the Qluman guide: https://docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config-Classes.ht... Hope that helps. Regards, Goswin von Brederlow
Hi, I did everything as you advised. Head node appeared in grafana dashboard, but compute nodes - no. In prometheus.yml there are no references to compute nodes after "Write Files". Manually adding compute node entries to prometheus.yml works. Regards Rolandas On 2/24/26 13:43, Goswin von Brederlow via Qlustar General wrote:
On 2/16/26 16:31, Nicolai, Wouter via Qlustar General wrote:
Hi,
I managed to upgrade from 13 to 14 following the upgrade guide, but am missing all predefined dashboard of Grafana. Is there an alternative way to obtain these dashboards? Are the json files somewhere available?
Thanks,
Wouter
_______________________________________________ Qlustar General mailing list --qlustar-general@qlustar.org To unsubscribe send an email toqlustar-general-leave@qlustar.org
Hi,
the predefined Grafana dashboards are created dynamically when writing the Alerts Config config class. The predefined dashboards are customized to what is monitored for alerts in the cluster. So if slurm isn't used or no slurm node is monitored then the slurm part of the dashboards are excluded. Similar for GPUs or Kubernetes. And if no nodes are monitored for alerts then no predefined dashboards.
At a minimum you should create 2 Alert Groups in Qluman-qt: Head Node and Compute Node. Open the Alert Configs widget via the Manage Configs / Alert Configs menu entry and click the New button to create a new group. Both groups can be left empty.
Next the Alert Groups need to be assigned to the nodes that should be monitored. For the Head Node group open the Enclosure View and select the beosrv-c node (in case of HA select the 2 physical headnodes instead). Then open the context menu and select: Set Config / Alerts / Head Node to assign the config. For the other nodes you probably have a Host Template that is used and is a more convenient way to add the Alerts Config. Open the Config Sets widget via the Manage Sets / Config Sets menu entry. Add the Compute Node alert group to the FE VM Node, Demo VM Node and Standard node Config Sets or whatever sets you have in use.
Last click the Write Files button at the bottom of the Qluman-qt window (or Manage Cluster / Write Files menu entry). Writing the Alerts entry (or just write all) will update the Prometheus config to gather metrics from any node with an Alert Group assigned. It will also update the predefined Gafana dashboards to match the nodes Prometheus now monitors.
You can read more about the alerts in the Qluman guide: https:// docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config- Classes.html#sec-Monitoring
Hope that helps.
Regards,
Goswin von Brederlow
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
Please send a screenshot of the 'Properties and Configs' part of the Enclosure View when one of the compute nodes is selected. Additionally a screenshot of the fully unfolded Alerts node in the File Classes tree of the 'Write Files' window. On 2/25/26 12:24, Rolandas Naujikas via Qlustar General wrote:
Hi,
I did everything as you advised. Head node appeared in grafana dashboard, but compute nodes - no. In prometheus.yml there are no references to compute nodes after "Write Files". Manually adding compute node entries to prometheus.yml works.
Regards Rolandas
On 2/24/26 13:43, Goswin von Brederlow via Qlustar General wrote:
On 2/16/26 16:31, Nicolai, Wouter via Qlustar General wrote:
Hi,
I managed to upgrade from 13 to 14 following the upgrade guide, but am missing all predefined dashboard of Grafana. Is there an alternative way to obtain these dashboards? Are the json files somewhere available?
Thanks,
Wouter
_______________________________________________ Qlustar General mailing list --qlustar-general@qlustar.org To unsubscribe send an email toqlustar-general-leave@qlustar.org
Hi,
the predefined Grafana dashboards are created dynamically when writing the Alerts Config config class. The predefined dashboards are customized to what is monitored for alerts in the cluster. So if slurm isn't used or no slurm node is monitored then the slurm part of the dashboards are excluded. Similar for GPUs or Kubernetes. And if no nodes are monitored for alerts then no predefined dashboards.
At a minimum you should create 2 Alert Groups in Qluman-qt: Head Node and Compute Node. Open the Alert Configs widget via the Manage Configs / Alert Configs menu entry and click the New button to create a new group. Both groups can be left empty.
Next the Alert Groups need to be assigned to the nodes that should be monitored. For the Head Node group open the Enclosure View and select the beosrv-c node (in case of HA select the 2 physical headnodes instead). Then open the context menu and select: Set Config / Alerts / Head Node to assign the config. For the other nodes you probably have a Host Template that is used and is a more convenient way to add the Alerts Config. Open the Config Sets widget via the Manage Sets / Config Sets menu entry. Add the Compute Node alert group to the FE VM Node, Demo VM Node and Standard node Config Sets or whatever sets you have in use.
Last click the Write Files button at the bottom of the Qluman-qt window (or Manage Cluster / Write Files menu entry). Writing the Alerts entry (or just write all) will update the Prometheus config to gather metrics from any node with an Alert Group assigned. It will also update the predefined Gafana dashboards to match the nodes Prometheus now monitors.
You can read more about the alerts in the Qluman guide: https:// docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config- Classes.html#sec-Monitoring
Hope that helps.
Regards,
Goswin von Brederlow
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
Hi, sorry for the delay. This had us scratching our head for a bit. It looks like you did everything right and repeating the same steps on our test clusters and a recent update for a customers had this procedure working out of the box. Digging around in the code that generates the configuration I think I found the problem now. Nodes that have no boot config are ignored in the monitoring as that is how switches or power strips are added in the GUI historically. Since there is no prometheus on switches or power strips incuding them would just raise alerts all the time so they are skipped. To confirm check /var/log/qluman/qlumand.log for entries containing: no boot config, ignoring node If you add a boot config to the 'Standard node' Config Set even if it's 'default' then hopefully the compute nodes will finally appear in the prometheus config and Grafana. Regards, Goswin von Brederlow On 2/25/26 13:46, Rolandas Naujikas wrote:
Please see attached files.
For information: this small test cluster was upgraded from Qlustar 13 to 14 and now is upgraded to 14.1.0. All upgrade procedures was followed.
Rolandas
On 2/25/26 13:37, Roland Fehrenbacher via Qlustar General wrote:
Please send a screenshot of the 'Properties and Configs' part of the Enclosure View when one of the compute nodes is selected. Additionally a screenshot of the fully unfolded Alerts node in the File Classes tree of the 'Write Files' window.
On 2/25/26 12:24, Rolandas Naujikas via Qlustar General wrote:
Hi,
I did everything as you advised. Head node appeared in grafana dashboard, but compute nodes - no. In prometheus.yml there are no references to compute nodes after "Write Files". Manually adding compute node entries to prometheus.yml works.
Regards Rolandas
On 2/24/26 13:43, Goswin von Brederlow via Qlustar General wrote:
On 2/16/26 16:31, Nicolai, Wouter via Qlustar General wrote:
Hi,
I managed to upgrade from 13 to 14 following the upgrade guide, but am missing all predefined dashboard of Grafana. Is there an alternative way to obtain these dashboards? Are the json files somewhere available?
Thanks,
Wouter
_______________________________________________ Qlustar General mailing list --qlustar-general@qlustar.org To unsubscribe send an email toqlustar-general-leave@qlustar.org
Hi,
the predefined Grafana dashboards are created dynamically when writing the Alerts Config config class. The predefined dashboards are customized to what is monitored for alerts in the cluster. So if slurm isn't used or no slurm node is monitored then the slurm part of the dashboards are excluded. Similar for GPUs or Kubernetes. And if no nodes are monitored for alerts then no predefined dashboards.
At a minimum you should create 2 Alert Groups in Qluman-qt: Head Node and Compute Node. Open the Alert Configs widget via the Manage Configs / Alert Configs menu entry and click the New button to create a new group. Both groups can be left empty.
Next the Alert Groups need to be assigned to the nodes that should be monitored. For the Head Node group open the Enclosure View and select the beosrv-c node (in case of HA select the 2 physical headnodes instead). Then open the context menu and select: Set Config / Alerts / Head Node to assign the config. For the other nodes you probably have a Host Template that is used and is a more convenient way to add the Alerts Config. Open the Config Sets widget via the Manage Sets / Config Sets menu entry. Add the Compute Node alert group to the FE VM Node, Demo VM Node and Standard node Config Sets or whatever sets you have in use.
Last click the Write Files button at the bottom of the Qluman-qt window (or Manage Cluster / Write Files menu entry). Writing the Alerts entry (or just write all) will update the Prometheus config to gather metrics from any node with an Alert Group assigned. It will also update the predefined Gafana dashboards to match the nodes Prometheus now monitors.
You can read more about the alerts in the Qluman guide: https:// docs.qlustar.com/Qlustar/14/ClusterOS/qluman-guide/Config- Classes.html#sec-Monitoring
Hope that helps.
Regards,
Goswin von Brederlow
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
_______________________________________________ Qlustar General mailing list -- qlustar-general@qlustar.org To unsubscribe send an email to qlustar-general-leave@qlustar.org
participants (5)
-
Goswin von Brederlow -
ignas.bunikis＠scilifelab.uu.se -
Nicolai, Wouter -
Roland Fehrenbacher -
Rolandas Naujikas