The other day I threw together a check_snmp_time.pl script that simply checks the remote’s time against the monitor host’s time. It’s fairly straight forward, and will alert you when host’s time drift too much, indicating that your ntpd configuration is bad. I had to do some custimizations to it, because not all hosts have the HOST-RESOURCES-MIB::hrSystemDate.0 option. For hosts that don’t it falls-back to the UCD-SNMP-MIB::versionCDate.0 option.
After putting it on all my monitors this evening, I started getting an alert from one of my hosts saying it has an offset of -50400 seconds (14 hours). It threw me for a few minutes, but it wasn’t long before I verified that the system time was correct, and it was snmpd that was at fault. It’s a bug where the remote snmp host reported the time as GMT+7, when it was supposed to be GMT-7, and affects CentOS 6’s net-snmp package.
I then found the bug reported on sourceforge, as well as the bug at redhat’s bugzilla. It’s fixed in RHEL 6.2, and since CentOS isn’t there yet, I ended up having to download the SRPM and build the package myself to correct the problem. But it’s fixed now, and if you ran into this problem, now you have some answers.