Issue with MIG gpus on huggingface ZeroGPU:

[WillSH97]

seems to be a problem with the autogenerated MIG gpu_ids for Huggingface ZeroGPU

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 256, in thread_wrapper
res = future.result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/user/app/carbon_footprint_gradio.py", line 39, in llama32_3b_chat
with EmissionsTracker() as tracker:
File "/usr/local/lib/python3.10/site-packages/codecarbon/emissions_tracker.py", line 323, in init
ressource_tracker.set_CPU_GPU_ram_tracking()
File "/usr/local/lib/python3.10/site-packages/codecarbon/core/resource_tracker.py", line 222, in set_CPU_GPU_ram_tracking
self.set_GPU_tracking()
File "/usr/local/lib/python3.10/site-packages/codecarbon/core/resource_tracker.py", line 191, in set_GPU_tracking
self.tracker._gpu_ids: List[int] = parse_gpu_ids(self.tracker._gpu_ids)
File "/usr/local/lib/python3.10/site-packages/codecarbon/core/config.py", line 64, in parse_gpu_ids
return list(map(int, str_ids))
ValueError: invalid literal for int() with base 10: 'MIGf1e1ec43bc015b91af5f69b64be2b8c9'

[WillSH97]

skimming your code, it looks like you don't need to have the static typing of "List[int]" in line 191 of resource_tracker.py?

[inimaz]

Thanks for using codecarbon @WillSH97 !
I think this is because so far in codecarbon, gpu_ids accepts only comma-separated numbers. But I see in the docs that CUDA_VISIBLE_DEVICES accepts as well a MIG id. This has not been implemented in codecarbon, but I agree it would be a good enhancement!

[cianc]

I wouldn't mind taking a crack at this, but I'd like to make sure my reading of the code is correct first:

1. Users can pass gpu_id values via the API, config, or the CUDA_VISIBLE_DEVICES or CODECARBON_GPU_IDS env variables.
2. We currently expect only integers. If the parsing code sees a string it attempts to convert it to a list of integers which is where the above error is coming from.
3. GPU ids are currently expected to be indices over the number of GPUs on the system, here and here
4. But GPUs also have an explicit index

So maybe we could switch to using the explicit GPU index in hardware.py, then we could accept arbitrary strings fo gpu_id.

One other thing, test_config.py checks that parse_gpu_ids can handle strings and lists, but also a bare id ((1,1)), but since we only ever pass a list or a string, I'm going to assume I can just remove this test case.

[cianc]

@WillSH97 do you have instructions for a simple repro by any chance (I have not used huggingface before)? I have a possible fix, but I'd like to check the id provided by CUDA_VISIBLE_DEVICES vs the device uuid.

[WillSH97]

hey sure - this issue is a bit stale and I ended up abandoning this because we moved away from ZeroGPU (plus the ZeroGPU container doesn't have sufficient permissions to access nvidia-smi or whatever), but it'd be generally good to fix for local MIG deployments that have sufficient access, which is likely what we'll use for our eventual local deployment.

off the top of my head, my local hotfix for this was to just loosen the type hinting requirements for the device list since upstream it should still return a list of strings for device ids, which was compatible with the rest of the code.

[cianc]

I'm not sure removing the type hinting will fix this bug. The issue is that we try to use the gpu id as an index into the list of GPUs in multiple locations in hardware.py, so something like 'MIGf1e1ec43bc015b91af5f69b64be2b8c9' will just be ignored as invalid. My potential fix adds logic to also check passed in gpu ids as prefixes of gpu uuids as detected by pynvml, but I'd like to check my assumptions against a real example in case I'm misreading the docs

[cianc]

I have a draft PR that I'd appreciate some feedback on : #923