node-exporter Error log Spam

[yannic-hamann-abb]

What happened: The service node-exporter is continuously writing ~3 error logs per minute.

What you expected to happen: No error logs.

How to reproduce it: For us, just spawn a new k8 node. We can observe this on every node.

Side note:
I was surprised to find two node-exporter processes running on the AKS nodes. One is the built-in version from AgentBaker, and the other is the one we deployed ourselves via the kube-prometheus-stack DaemonSet.
What's your recommendation for scraping host metrics? Should we keep using our own node-exporter, or is it better to rely on the built-in one from AKS?

Environment:

• AgentBaker version: AKSUbuntu-2204gen2containerd-202605.14.0
• Kubernetes version (use kubectl version): Server Version: v1.34.2
• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel (e.g. uname -a): Linux aks-xxx-yyyy-vmss000000 5.15.0-1111-azure #120-Ubuntu SMP Wed Apr 15 15:09:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

● node-exporter.service - Prometheus Node Exporter
     Loaded: loaded (/etc/systemd/system/node-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2026-05-26 07:29:13 UTC; 1 month 27 days ago
       Docs: https://github.com/prometheus/node_exporter
   Main PID: 7782 (node-exporter)
      Tasks: 6 (limit: 38497)
     Memory: 26.8M
        CPU: 42min 44.215s
     CGroup: /system.slice/node-exporter.service
             └─7782 /opt/bin/node-exporter --web.listen-address=10.240.11.6:19100 --no-collector.wifi --no-collector.hwmon --collector.cpu.info "--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run>

Jul 23 05:42:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:42:40.157Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
Jul 23 05:43:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:09.048Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:43:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:10.148Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:43:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:43:40.167Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:44:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:09.041Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
Jul 23 05:44:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:10.167Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=7.>
Jul 23 05:44:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:44:40.151Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:09 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:09.047Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:10 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:10.170Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.>
Jul 23 05:45:40 aks-default1342-19836111-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:45:40.155Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=9.>
lines 1-21/21 (END)

Jul 23 05:57:10 aks-xxxx-yyyyy-vmss000000 node-exporter-startup.sh[7782]: time=2026-07-23T05:57:10.165Z level=ERROR source=collector.go:168 msg="collector failed" name=infiniband duration_seconds=6.6539e-05 err="error obtaining InfiniBand class info: read /sys/class/infiniband/mana_0/ports/1/rate: invalid argument"

[chmill-zz]

Thanks for the detailed report. This is caused by the MANA RDMA driver in the Ubuntu Azure kernel. The upstream change RDMA/mana_ib: Implement port parameters addresses the underlying /sys/class/infiniband/mana_0/ports/1/rate read failure, but it is not present in the Ubuntu 22.04 5.15.0-1111-azure kernel used by this node image.

Ubuntu backported the change to its 6.8 Azure kernel starting with 6.8.0-1041-azure, tracked by LP #2125704. AgentBaker Ubuntu 24.04 images using 6.8.0-1041-azure or newer should contain the fix.

The exporter otherwise continues to work: the failure is isolated to the InfiniBand collector, so other host metrics are still emitted. The main impact is missing InfiniBand metrics and an error on each scrape; this does not indicate that MANA networking itself is failing. We are evaluating disabling the InfiniBand collector on affected MANA kernels as a mitigation.

Regarding the two exporters: the host-level exporter on port 19100 supports AKS/Azure Monitor managed Prometheus. If you operate your own Prometheus through kube-prometheus-stack, you should generally retain its node-exporter DaemonSet rather than depend directly on the AKS-managed endpoint.