
Be Careful with Reservoirs
https://engineering.salesforce.com/be-careful-with-reservoirs-708884018daf
This post is part of the Observability 101 series. The series covers basic ideas behind monitoring and observability and, I hope, it will help engineers to make their products better prepared for a long life in production.
We found an interesting problem recently with some of the metrics in our product: we noticed pretty long intervals of absolutely flat lines on charts for a high percentile of a latency metric. We also saw the same pattern on a few other charts.