How to control data placement?

[Xiaoyang-Lu]

Hello,

I am using Cramsim to model HBM, the aim is to build a heterogeneous memory system, HBM and DRAM are combined into a large flat memory. I want to know, can I use Cramsim or other components to control the data placement by myself in this memory system? For example, for an application, I want to place some data structure on HBM and others on DRAM.

Thanks!

[hughes-c]

You can use address ranges to segment the memory. You can also use Opal with Samba to provide hints for page allocation.

[Xiaoyang-Lu]

@hughes-c I found from the Opal document that a process can call malloc with hints to decide to allocate from local or remote memory. However, I am confused about how to add hints. Suppose there is a simple disaggregate memory system like the one in the Opal paper, it has local memory and remote memory pool. How to using malloc with hints in application to allocate pages?

Maybe using mlm_malloc(size_t size, int level) function can work, the parameter level here is hint?
For example, 0 means local memory, 1 means remote? I am not sure.

If you can provide some examples, it will help me a lot, thank you.

[amroawad2]

Hi Xiaoyang, Opal is still under development. There has been some divergence between our internally developed model and what is now pushed into SST. We recommend you to wait until Opal is officially merged with SST. Cheers, Amro

…

On Jun 28, 2019, at 4:38 PM, Xiaoyang Lu ***@***.***> wrote: @hughes-c I found from the Opal document that a process can call malloc with hints to decide to allocate from local or remote memory. However, I am confused about how to add hints. Suppose there is a simple disaggregate memory system like the one in the Opal paper, it has local memory and remote memory pool. How to using malloc with hints in application to allocate pages? Maybe using mlm_malloc(size_t size, int level) function can work, the parameter level here is hint? For example, 0 means local memory, 1 means remote? I am not sure. If you can provide some examples, it will help me a lot, thank you. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Xiaoyang-Lu]

@amroawad2 Thank you. So now I cannot use the Opal or mlm_malloc function with hints to control the allocation? Or do you have any other suggestions?

[amroawad2]

I would wait until fully supported.

…

On Jun 28, 2019, at 5:21 PM, Xiaoyang Lu ***@***.***> wrote: @amroawad2 Thank you. So now I cannot use the Opal or mlm_malloc function with hints to control the allocation? Or do you have any other suggestions? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[Xiaoyang-Lu]

@amroawad2 Do you know how long it will take?

[Xiaoyang-Lu]

Is opal fully supported in the latest release?

[hughes-c]

@Xiaoyang-Lu @amroawad2 We have not officially released Opal but it is passing our nightly tests with the latest master/devel branches.

[Xiaoyang-Lu]

Thank you @hughes-c
I cloned the latest project source on the master branches:
SST-Core Version (-dev, git branch : master, SHA: 40d7176bcbdeb1eef6ee7abde5aee458c8187f79)

I have several questions:

1. I found that the test code from /src/sst-elements/src/sst/elements/Opal/tests/basic_1node_1smp.py has both local memory and shared memory, the Opal configuration shows "node0.allocation_policy" : 1, could you tell me the meaning of 1 here? From Opal's paper, 1 is Local memory first policy, 2 is Alternate allocation policy, 3 is Round-robin allocation policy, 4 is Proportional allocation policy. I don't know if they are consistent with the currently released version.

2. I think the "allocation_policy" is the only way to control memory allocation. I checked the application opal_test.c from src/sst-elements/src/sst/elements/Opal/tests/app/opal_test.c, three arrays a,b,c use malloc directly to allocate the requested memory. Let us take the same application as an example, If here I plan to allocate array b to the local memory, a and c to the shared memory, can we implement it with Opal? Or how can I provide hints for page allocation? This question has puzzled me for a long time. Any examples, codes or suggestions would be helpful.

Thank you very much.