JVM

I manage close to a hundred OpenSearch clusters. A while ago, the heap alert started firing, but only on a few clusters, not all of them. The alert fires when heap usage goes above 75%. On those few clusters, it would recover by itself a few minutes later, and then fire again. At first I was worried that a node was in trouble, so I checked the cluster for problems: Cluster status: Green, no unassigned shards Writes are healthy: Fluentd’s buffer was not piling up and it was not retrying constantly No OOM kills: the pod restart count did not increase, and Last State was not OOMKilled All three were fine, so I opened the heap metric. Heap usage was a sawtooth: it kept climbing, dropped, then climbed again. The alert fired right when it reached the peak, and recovered a few minutes later when it dropped. My first thought was to raise the 75% threshold. But since not every cluster fired, it did not look like a threshold problem alone. What confused me more: the other clusters had the same sawtooth, but they dropped at around 60%, while this one climbed to 80% before dropping. Was its configuration different? Was it busier and short on memory? To answer these questions, I first had to understand how the JVM reclaims memory, especially Young GC. ...