GC tracing with DRC collector creates excessively large allocations

[alexcrichton]

This issue is borne out of an OSS-Fuzz discovered issue, https://oss-fuzz.com/testcase-detail/6137295404335104, which is an OOM when fuzzing. Looking at the test case it has a pattern where:

• A few small GC objects are created
• One massive GC array is then created
• A few more small GC values are created
• Everything is live and a GC happens which tries to trace things

The final step, during tracing of the array, pushes all entries in the array into a local Vec within the DRC allocator. Specifically this allocation is what is quite large.

The heap is bounded by a 1G limit while fuzzing, but this stack array can relatively easily more-or-less contain the entire heap meaning it's doubling the size of RSS for the given program.

We probably want to be more optimal about this, for example pushing "array with N elements" onto the tracing stack and then using that as an iterator-of-sorts. That avoids pushing O(heapsize) on the stack and instead it's just O(some-other-metric-related-to-gc-reference-depth-sort-of-I-dont-know).

[fitzgen]

One thing we can do (perhaps only for arrays above a certain size?) is instead of pushing their VMGcRef elements directly to the mark stack, we can have a second mark stack specifically for array elements that contains something like (VMArrayRef, usize) rather than VMGcRef.

I guess this is basically what you say here:

We probably want to be more optimal about this, for example pushing "array with N elements" onto the tracing stack and then using that as an iterator-of-sorts. That avoids pushing O(heapsize) on the stack and instead it's just O(some-other-metric-related-to-gc-reference-depth-sort-of-I-dont-know).

The other idea would be to put the mark stack into the GC heap as a linked list in VMDrcHeader, similar to the over-approximate stack roots list. This would probably be a fairly big slow down though.