unable to allocate aligned OS memory directly, fall back to over-allocation

[immno]

#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;

fn main() {
    let mut list = Vec::new();
    for _ in 0..300000000 {
        list.push(DataValue::Double(66.66));
    }
}

#[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum DataValue {
    #[default]
    Null,
    Bool(bool),
    Long(i64),
    Double(f64),
    String(String)
}

Using MiMalloc in this scenario will crash the program, using about 8GB of memory, but using std will run fine.

mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x40800000 bytes, address: 0x7FBD9084A000, alignment: 0x2000000, commit: 1)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x80800000 bytes, address: 0x7FBD0F800000, alignment: 0x2000000, commit: 1)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x100800000 bytes, address: 0x7FBC0D800000, alignment: 0x2000000, commit: 1)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x200800000 bytes, address: 0x7FBA0B800000, alignment: 0x2000000, commit: 1)
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x400800000 bytes, address: 0x7FB609800000, alignment: 0x2000000, commit: 1)
已杀死

How can I optimize it so that there is no OOM when the memory is sufficient? This is a big problem for me. I have a real-time data processing program that processes rdkafka. Sometimes OOM occurs at 3.7GB (the limit is 4GB). Now I have to increase the memory to 8GB to ensure stability.

[nathaniel-daniel]

Well, the largest variant in DataValue is a String, so let's optimistically say that the size of DataValue is the size of a String. On 64 bit computers, this is 24 (ptr, len, capacity). You are creating a Vec with 300,000,000 elements. The number of elements multiplied by the size of each elements is 24 * 300,000,000 = 7,200,000,000. This is 7.2GB. Furthermore, a Vec will allocate exponentially to avoid needing to reallocate as much.
As a result, an 8GB memory limit seems perfectly reasonable for 64 bit.

Assuming that you use 32bit, you'll still need (4 * 3) * 300,000,000 = 3600000000 bytes of memory, or 3.6GB. I don't think 32 bit programs can even create an allocation this large.

I'm not sure how you got the std allocator to work while keeping the memory limit below 4GB. That's not physically possible with that many elements in a 64 bit program. Maybe the allocation gets optimized out when using the std allocator, or maybe you aren't measuring the memory use of the program correctly.

You can also see microsoft/mimalloc#640 to look into solving those warnings, but I don't think it will save you the amount of memory that you want. Also, you probably want to use with_capacity on that Vec to avoid such frequent, large reallocations.