🚨 Nothing quite wakes you up like an iLO email at 20:30 telling you a server just fell over. This post is the story of chasing that crash down a rabbit hole — from a scary-looking hardware log entry, through a real HPE bug advisory that turned out to be a dead end, to what’s now my actual leading suspect: an out-of-date management daemon quietly talking to exactly the chip that failed.
📧 The Email
iLO doesn’t do subtle. The subject line was “CRITICAL” and the body contained two back-to-back Integrated Management Log entries, both timestamped the same second:
EVENT: Uncorrectable PCI Express Error Detected. [UNKNOWN] 0
(Segment 0x0, Bus 0x1, Device 0x0, Function 0x2).
Uncorrectable Error Status: 0x14000
ACTION: Update the firmware of the failing device. If the issue persists, replace the device.
EVENT: Unrecoverable I/O Error has occurred. System Firmware will log
additional details in a separate IML message entry if possible.Translation for anyone who hasn’t stared at PCIe Advanced Error Reporting codes before: something on the PCI Express bus threw an error so bad the hardware itself couldn’t correct it, and the box just… stopped responding. One of my three Proxmox nodes had quietly face-planted overnight. 💀
🔍 Finding the Actual Chip
“Bus 1, Device 0, Function 2” doesn’t mean much on its own, so I went straight to the server’s own hardware inventory via the iLO Redfish API rather than guess. Every embedded PCI device on this box reports its exact Bus/Device/Function address:
BCM 5720 NIC port 1 -> Bus 2, Dev 0, Func 0
BCM 5720 NIC port 2 -> Bus 2, Dev 0, Func 1
Embedded SATA Ctrl -> Bus 0, Dev 23, Func 0
Embedded Video Ctrl -> Bus 1, Dev 0, Func 1 <-- same Bus+Device as the faultFunction 2 at that address isn’t separately listed, but it sits right next to the embedded video controller — which, on this generation of ProLiant, is actually implemented inside the iLO 5 chip itself, not a separate GPU. So the failing device is some function of iLO’s own silicon talking to the host over PCIe. Not “iLO is dead” (it was cheerfully emailing me about its own crash log, after all) — more like the internal PCIe link between iLO and the rest of the system had a bad moment. 🧩
📋 The HPE Advisory That Matched Perfectly (Almost)
A quick search turned up an HPE advisory that could not have matched better if I’d written it myself: “HPE ProLiant DL20/ML30 Gen10 Plus and ProLiant MicroServer Gen10 Plus v2 – Server May Experience ‘Uncorrectable PCI Express Error Detected’ Events.” Same server family, same error code (0x14000), same PCI address. The advisory describes a documented System ROM (BIOS) firmware bug and lists the fix as a ROM update. Case closed, right? 🎉
Except: I checked. BIOS, iLO, and the embedded video controller firmware were all already at the latest publicly available versions. The “just update the firmware” fix doesn’t apply here — I already have it, or at least whatever HPE currently publishes as latest. So either this specific erratum isn’t fully squashed by the current firmware, or something else entirely is going on. A textbook-matching bug report that doesn’t actually resolve when you follow its own advice is its own special kind of frustrating. 🤷
🕵️ Enter amsd — My Current Prime Suspect
Here’s where it got interesting. HPE ships a userspace daemon called AMS (Agentless Management Service, amsd) that runs on the host OS and feeds hardware telemetry — fan speeds, thermals, health data — back to iLO. Critically, it does this over an internal channel that rides the same embedded PCIe link as the chip that just faulted.
Turns out this particular box had been running amsd 3.6.0, a build explicitly compiled for Debian 12 — while Proxmox VE (which is what actually boots on the metal) is Debian 13. A version mismatch, quietly running for about a week before the crash, actively poking at exactly the hardware interface that failed. That’s a genuinely suspicious coincidence. 🔦
Since discovering this, I’ve updated amsd to the current Debian-13-native build (4.7.0) across all three hosts as a precaution.
⚖️ Being Honest About What I Don’t Actually Know
I’d love to end this post with “and that was definitely it.” I can’t, in good conscience. My other two servers — identical hardware, identical BIOS, identical firmware — were also running the same outdated, Debian-12-built amsd at the exact same time, and none of them crashed. So this isn’t a clean “old driver = guaranteed crash” story. My best working theory right now is that this is a combination of two half-culprits: a known-but-apparently-not-fully-fixed HPE platform erratum, possibly made more likely to trigger by a version-mismatched daemon hammering the same PCIe link — neither one alone being sufficient, both together being unlucky. Which is, annoyingly, exactly the kind of intermittent hardware-adjacent bug that never gives you a satisfying “aha, found it” moment. 🎯❌
What I’m left doing: watching the IML for any repeat occurrence, keeping every host’s firmware and management daemons current going forward, and quietly hoping this was a one-off electrical hiccup and not the first sign of a chip slowly dying. 🤞 If it happens again, that’s the point where “open an HPE support case” stops being optional.
