π ΰ² _ΰ² in our monitoring system
The forever war: #monitoringsucks vs #monitoringlove
The forever war: #monitoringsucks vs #monitoringlove
I know I'm not the only one who learned Outlook rules because of Nagios...
In the dystopian future of monitoring, alerts have achieved sentience and spend their time arguing about which metrics matter most while your production system burns in the background.
"If a server crashes in the datacenter and no one is around to see the alert, did it really go down?"
β Zen and the Art of Alert Fatigue
π Monitoring Fun Facts
- Nagios taught a generation of sysadmins advanced email filtering techniques
- The first rule of monitoring: everything is broken until proven working
- The second rule of monitoring: even when it's working, it's probably still broken
- "Alert fatigue" is just the monitoring system's way of teaching you mindfulness
- False positives build character (and drinking problems)
- The best monitoring system is the one that alerts you to problems you didn't know you had
- ΰ² _ΰ² is the official facial expression of anyone looking at monitoring dashboards
π₯ The Monitoring Philosophy
True monitoring enlightenment comes when you realize that the real metrics were the friends we made along the way... who also happen to be on PagerDuty rotation.
