https://coder.tw/en/categories/sre/Recent content in SRE on Lin's BlogHugoen-USMon, 19 Oct 2026 22:19:33 +0800https://coder.tw/en/posts/opensearch-memory-alert-old-generation/Mon, 19 Oct 2026 22:19:33 +0800https://coder.tw/en/posts/opensearch-memory-alert-old-generation/<p>I manage close to a hundred OpenSearch clusters. A while ago, the heap alert started firing, but only on a few clusters, not all of them. The alert fires when heap usage goes above 75%. On those few clusters, it would recover by itself a few minutes later, and then fire again.</p> <p>At first I was worried that a node was in trouble, so I checked the cluster for problems:</p> <ul> <li><strong>Cluster status</strong>: Green, no unassigned shards</li> <li><strong>Writes are healthy</strong>: Fluentd&rsquo;s buffer was not piling up and it was not retrying constantly</li> <li><strong>No OOM kills</strong>: the pod restart count did not increase, and <code>Last State</code> was not OOMKilled</li> </ul> <p>All three were fine, so I opened the heap metric. Heap usage was a sawtooth: it kept climbing, dropped, then climbed again. The alert fired right when it reached the peak, and recovered a few minutes later when it dropped.</p> <p>My first thought was to raise the 75% threshold. But since not every cluster fired, it did not look like a threshold problem alone. What confused me more: the other clusters had the same sawtooth, but they dropped at around 60%, while this one climbed to 80% before dropping. Was its configuration different? Was it busier and short on memory?</p> <p>To answer these questions, I first had to understand how the JVM reclaims memory, especially Young GC.</p>