How do you guys monitor SolarWinds applications in general? What about the MSMQs? I came in this morning and thought everything was fine until I had to do a few rounds of server reboots, went to unmanage the servers, and no matter how many times I unmanaged, nothing would happen. After some fudging around, it'd eventually unmanage the node, I did my work, remanaged and same behavior. When it finally re-managed it'd take 15-20 minutes before it'd show the node was in a state other than unknown, which is usually immediate.
After some poking around, I found a hint in the Application event log...
Storage quota exceeded for MSMQ queue DIRECT=OS:myserver.pvt\private$\solarwinds/collector/processingqueue/solarwin. No more messages can be stored in the queue. You can increase a queue's storage quota by editing queue properties in Computer Management console. This event is logged at most once per 3600 seconds. To change this setting, set \HKLM\Software\Microsoft\MSMQ\Parameters\Event2182 registry value to desired time in seconds.
Okay, clearly an issue with MSMQ storage, and not enough space... So a quick look in the MSMQ interface...
Well there's the obvious problem, it's not processing any data. With 325k items in the responstetimestatus.icmp queue alone, it's a good sign that something is broken. The quick fix was to restart the SolarWinds services, and within about 30 minutes the queues were all back to zero, or fluctuating about there.
So the question is, how do we monitor for this? Obviously SolarWinds cannot monitor itself because it's not processing the queue data to alert on (we had a node go down during this queue issue and wasn't notified until it processed the queues). What other methods do people use to watch the watcher? How do we know at 0200 that SolarWinds stopped reading/processing MSMQ when we're all asleep?