Be Careful with Reservoirs

Be Careful with Reservoirs

6 years ago
Anonymous $oIHRkISgaL

https://engineering.salesforce.com/be-careful-with-reservoirs-708884018daf

This post is part of the Observability 101 series. The series covers basic ideas behind monitoring and observability and, I hope, it will help engineers to make their products better prepared for a long life in production.

We found an interesting problem recently with some of the metrics in our product: we noticed pretty long intervals of absolutely flat lines on charts for a high percentile of a latency metric. We also saw the same pattern on a few other charts.