Fix PGC edenSurvivalRate overflow issues in CopyForward

The edenSurvivalRate represents the fraction of Eden regions that
survived a PGC (nominally in the range [0.0, 1.0]).  An overflow
(rate > 1.0) can cause incorrect heap/Eden size adjustments, producing
unexpected allocation spikes and, in extreme cases, out-of-memory
errors.  The issue is not a regression but was exposed by a recent
Eden-resize change.

Four overflow sources are addressed:

1. Incorrect edenCountBeforeCollect (denominator)
   During a PGC the heap may expand, triggering an Eden size
   recalculation.  The newly calculated Eden size can differ
   significantly from the size at collection start, making the survival
   rate inaccurate.  Fix: use copyForwardStats->_edenEvacuateRegionCount
   as the denominator instead of getCurrentEdenSizeInRegions(env).

2. Semantic mismatch between edenSurvivorCount and edenEvacuateCount
   In the first few PGCs after a GMP, previous Eden survivors are
   ADDRESS_ORDERED_MARKED regions with logicalAge == 0.  They enter the
   evacuate set as non-Eden regions, but their surviving objects spill
   into age-0 survivor regions that are counted as Eden survivors.
   This can cause _edenSurvivorRegionCount > _edenEvacuateRegionCount.
   Fix: add _hasEdenContent to MM_CopyScanCacheVLHGC and
   MM_HeapRegionDescriptorVLHGC::_copyForwardData.  The flag is set
   during copy() when the source region isEden() (ADDRESS_ORDERED),
   propagated from cache to region in stopCopyingIntoCache(), and used
   in postProcessRegions() to classify survivor regions accurately.

3. Over-counting of Eden survivor regions (shared fresh survivors)
   Because Eden and age-0 non-Eden regions share the same compact group,
   they can fill the same fresh survivor regions, causing
   _edenSurvivorRegionCount to be over-counted.  Fix: redistribute
   edenSurvivorCount and nonEdenSurvivorCount using _copyBytesEden as a
   ceiling-division estimate of the true Eden survivor region count.

4. _scanBytesEden double-counting (cap)
   During abort recovery, objects that were partially copied (forwarding
   pointer already installed) are scanned again in the source region,
   double-counting their bytes in _scanBytesEden.  Combined with high
   survival rates, alignment waste, or NUMA/hybrid no-evacuation cases,
   this can push edenSurvivorCount above edenCountBeforeCollect.
   Fix: cap edenSurvivorCount at edenCountBeforeCollect and add an
   assertion that thisSurvivalRate <= 1.0.

Signed-off-by: lhu <linhu@ca.ibm.com>

Author: LinHu2016

runtime/gc_vlhgc/CopyForwardScheme.cpp

@@ -493,6 +493,8 @@ MM_CopyForwardScheme::preProcessRegions(MM_EnvironmentVLHGC *env)
 	while (NULL != (region = regionIterator.nextRegion())) {
 		region->_copyForwardData._survivor = false;
 		region->_copyForwardData._freshSurvivor = false;
+		region->_copyForwardData._hasEdenContent = false;
+
 		_regionShouldMark[_regionManager->mapDescriptorToRegionTableIndex(region)] = region->_markData._shouldMark;
 		if (region->containsObjects()) {
 			region->_copyForwardData._initialLiveSet = true;
@@ -541,8 +543,13 @@ MM_CopyForwardScheme::postProcessRegions(MM_EnvironmentVLHGC *env)
 				static_cast<MM_CycleStateVLHGC *>(env->_cycleState)->_vlhgcIncrementStats._copyForwardStats._nonEdenEvacuateRegionCount += 1;
 			}
 		} else if (region->isFreshSurvivorRegion()) {
-			/* check Eden Survivor Regions */
-			if (0 == region->getLogicalAge()) {
+			/* Classify the survivor region by the Eden-ness of its source objects.
+			 * _hasEdenContent is set during copy when sourceRegion->isEden() (ADDRESS_ORDERED type),
+			 * which is the same predicate used on the evacuate side.  This avoids the previous bug
+			 * where logicalAge==0 was used instead, which incorrectly included survivors from
+			 * non-Eden regions at compact group 0 (type ADDRESS_ORDERED_MARKED).
+			 */
+			if (region->_copyForwardData._hasEdenContent) {
 				static_cast<MM_CycleStateVLHGC *>(env->_cycleState)->_vlhgcIncrementStats._copyForwardStats._edenSurvivorRegionCount += 1;
 			} else {
 				static_cast<MM_CycleStateVLHGC *>(env->_cycleState)->_vlhgcIncrementStats._copyForwardStats._nonEdenSurvivorRegionCount += 1;
@@ -582,6 +589,7 @@ MM_CopyForwardScheme::postProcessRegions(MM_EnvironmentVLHGC *env)
 		region->_copyForwardData._requiresPhantomReferenceProcessing = false;
 		region->_copyForwardData._survivor = false;
 		region->_copyForwardData._freshSurvivor = false;
+		region->_copyForwardData._hasEdenContent = false;
 
 		if (region->_copyForwardData._evacuateSet) {
 			Assert_MM_true(region->_sweepData._alreadySwept);
@@ -708,6 +716,7 @@ MM_CopyForwardScheme::reinitCache(MM_EnvironmentVLHGC *env, MM_CopyScanCacheVLHG
 
 	Assert_MM_true(compactGroup < _compactGroupMaxCount);
 	cache->_compactGroup = compactGroup;
+	cache->_hasEdenContent = false;
 	Assert_MM_true(0.0 == cache->_allocationAgeSizeProduct);
 
 	MM_HeapRegionDescriptorVLHGC * region = (MM_HeapRegionDescriptorVLHGC *)_regionManager->tableDescriptorForAddress(cache->cacheBase);
@@ -1859,6 +1868,12 @@ MM_CopyForwardScheme::stopCopyingIntoCache(MM_EnvironmentVLHGC *env, uintptr_t c
 
 		MM_HeapRegionDescriptorVLHGC * region = (MM_HeapRegionDescriptorVLHGC *)_regionManager->tableDescriptorForAddress(copyCache->cacheBase);
 
+		/* propagate Eden-content flag: a region is an Eden-content survivor if any cache that filled it received Eden objects */
+		if (copyCache->_hasEdenContent) {
+			region->_copyForwardData._hasEdenContent = true;
+		}
+		copyCache->_hasEdenContent = false;
+
 		/* atomically add (age * usedBytes) product from this cache to the regions product */
 		double newAllocationAgeSizeProduct = region->atomicIncrementAllocationAgeSizeProduct(copyCache->_allocationAgeSizeProduct);
 		region->updateAgeBounds(copyCache->_lowerAgeBound, copyCache->_upperAgeBound);
@@ -2094,6 +2109,7 @@ MM_CopyForwardScheme::copy(MM_EnvironmentVLHGC *env, MM_AllocationContextTarok *
 					env->_copyForwardCompactGroups[sourceCompactGroup]._edenStats._liveBytes += objectCopySizeInBytes;
 					env->_copyForwardCompactGroups[destinationCompactGroup]._edenStats._copiedObjects += 1;
 					env->_copyForwardCompactGroups[destinationCompactGroup]._edenStats._copiedBytes += objectCopySizeInBytes;
+					copyCache->_hasEdenContent = true;
 				} else {
 					env->_copyForwardCompactGroups[sourceCompactGroup]._nonEdenStats._liveObjects += 1;
 					env->_copyForwardCompactGroups[sourceCompactGroup]._nonEdenStats._liveBytes += objectCopySizeInBytes;

runtime/gc_vlhgc/CopyScanCacheVLHGC.hpp

@@ -53,6 +53,7 @@ class MM_CopyScanCacheVLHGC : public MM_CopyScanCache
 	uint64_t _lowerAgeBound; /**< lowest possible age of any object in this copy cache */
 	uint64_t _upperAgeBound; /**< highest possible age of any object in this copy cache */
 	uintptr_t _arraySplitIndex; /**< The index within the array in scanCurrent to start scanning from (meaningful is OMR_COPYSCAN_CACHE_TYPE_SPLIT_ARRAY is set) */
+	bool _hasEdenContent; /**< true if at least one object copied into this cache originated from an Eden (ADDRESS_ORDERED) source region */
 
 	/* Members Function */
 private:
@@ -88,6 +89,7 @@ class MM_CopyScanCacheVLHGC : public MM_CopyScanCache
 		, _lowerAgeBound(U_64_MAX)
 		, _upperAgeBound(0)
 		, _arraySplitIndex(0)
+		, _hasEdenContent(false)
 	{}
 };
 

runtime/gc_vlhgc/HeapRegionDescriptorVLHGC.cpp

@@ -83,6 +83,7 @@ MM_HeapRegionDescriptorVLHGC::initialize(MM_EnvironmentBase *env, MM_HeapRegionM
 	_copyForwardData._survivor = false;
 	_copyForwardData._freshSurvivor = false;
 	_copyForwardData._nextRegion = NULL;
+	_copyForwardData._hasEdenContent = false;
 	_copyForwardData._previousRegion = NULL;
 
 #if defined (J9VM_GC_MODRON_COMPACTION)

runtime/gc_vlhgc/HeapRegionDescriptorVLHGC.hpp

@@ -71,6 +71,7 @@ class MM_HeapRegionDescriptorVLHGC : public MM_HeapRegionDescriptor
 		bool _freshSurvivor;
 		MM_HeapRegionDescriptorVLHGC *_nextRegion;  /**< Region list link for compact group resource management during a copyforward operation */
 		MM_HeapRegionDescriptorVLHGC *_previousRegion;  /**< Region list link for compact group resource management during a copyforward operation */
+		bool _hasEdenContent;  /**< true if at least one object copied into this survivor region originated from an Eden (ADDRESS_ORDERED) source region */
 	} _copyForwardData;
 #if defined (J9VM_GC_MODRON_COMPACTION)
 	MM_HeapRegionDataForCompactVLHGC _compactData;

runtime/gc_vlhgc/SchedulingDelegate.cpp

@@ -372,7 +372,15 @@ MM_SchedulingDelegate::partialGarbageCollectCompleted(MM_EnvironmentVLHGC *env,
 	bool globalSweepHappened = _globalSweepRequired;
 	_globalSweepRequired = false;
 	/* copy out the Eden size of the previous interval (between the last PGC and this one) before we recalculate the next one */
-	uintptr_t edenCountBeforeCollect = getCurrentEdenSizeInRegions(env);
+	/* During GC, the heap may expand to allow the current collection to continue.
+	 * A heap resize triggers an eden size recalculation without allowing further
+	 * heap expansion (heap reconfiguration). As a result, the newly calculated
+	 * eden size can differ significantly from the eden size at the start of the
+	 * collection, causing the SurvivalRate calculation to become inaccurate.
+	 * Therefore, set copyForwardStats->_edenEvacuateRegionCount (instead of getCurrentEdenSizeInRegions(env)) to
+	 * edenCountBeforeCollect.
+	 */
+	uintptr_t edenCountBeforeCollect = copyForwardStats->_edenEvacuateRegionCount;
 
 	Trc_MM_SchedulingDelegate_partialGarbageCollectCompleted_stats(env->getLanguageVMThread(),
 			copyForwardStats->_edenEvacuateRegionCount,
@@ -389,16 +397,44 @@ MM_SchedulingDelegate::partialGarbageCollectCompleted(MM_EnvironmentVLHGC *env,
 		uintptr_t edenSurvivorCount = copyForwardStats->_edenSurvivorRegionCount;
 		uintptr_t nonEdenSurvivorCount = copyForwardStats->_nonEdenSurvivorRegionCount;
 
+		/* Adjust estimated edenSurvivorCount and nonEdenSurvivorCount based on _copyBytesEden
+		 * due to fresh survivor region could be shared between eden evacuateRegions and age0 nonEden evacuateRegions.
+		 */
+		if (copyForwardStats->_copyBytesEden > 0) {
+			uintptr_t neededEdenSurvivorCount = (copyForwardStats->_copyBytesEden + regionSize - 1) / regionSize;
+			if (neededEdenSurvivorCount != edenSurvivorCount) {
+				/* avoid nonEdenSurvivorCount underflow in case using age0 non-eden regions as survivors */
+				if (neededEdenSurvivorCount > (nonEdenSurvivorCount + edenSurvivorCount)) {
+					edenSurvivorCount += nonEdenSurvivorCount;
+					nonEdenSurvivorCount = 0;
+				} else {
+					nonEdenSurvivorCount = nonEdenSurvivorCount + edenSurvivorCount - neededEdenSurvivorCount;
+					edenSurvivorCount = neededEdenSurvivorCount;
+				}
+			}
+		}
 		/* estimate how many more regions we would have needed to avoid abort */
 		Assert_MM_true( (0 == copyForwardStats->_scanBytesEden) || copyForwardStats->_aborted || (0 != copyForwardStats->_nonEvacuateRegionCount));
 		Assert_MM_true( (0 == copyForwardStats->_scanBytesNonEden) || copyForwardStats->_aborted || (0 != copyForwardStats->_nonEvacuateRegionCount));
 		edenSurvivorCount += (copyForwardStats->_scanBytesEden + regionSize - 1) / regionSize;
+		if (edenSurvivorCount > edenCountBeforeCollect) {
+			/* when abort is in progress, every Eden object scanned in abort-recovery contributes to _scannedBytes.
+			 * But an object that was partially copied before abort (the copy succeeded,
+			 * the forwarding pointer was installed) is also counted again during the abort scan of the source region.
+			 * So _scanBytesEden might be double-counted, then cause edenSurvivorCount is bigger than edenCountBeforeCollect.
+			 * And High survival + alignment/TLH waste(numa case and Hybrid noEvacuation, too)
+			 * for these cases, set edenSurvivorCount = edenCountBeforeCollect.
+			 */
+			edenSurvivorCount = edenCountBeforeCollect;
+		}
 		nonEdenSurvivorCount += (copyForwardStats->_scanBytesNonEden + regionSize - 1) / regionSize;
 
 		/* Eden count could be 0 in special case, after compaction if there is still no free region for scheduling eden(eden count = 0),
 		   will skip update Survival Rate */
 		if (0 != edenCountBeforeCollect) {
+			/* SurvivalRate should never be over 1.0 */
 			double thisSurvivalRate = (double)edenSurvivorCount / (double)edenCountBeforeCollect;
+			Assert_MM_true(1.0 >= thisSurvivalRate);
 			updateSurvivalRatesAfterCopyForward(thisSurvivalRate, nonEdenSurvivorCount);
 		}