Dynamic Vram

[owayej]

With dynamic vram enable. It takes too much time to generate and image or whatever it is. with disable it takes only few min. Then why won't we can not get the option to disable it? I'm using rtx 4070s. It's so annoying to use with dynamic vram on.

[rattus128]

If dyn vram is that much slower its a bug and you can report it. If all we get is an annecdotal account that its slower, we cant do anything with that.

[MoeMonsuta]

I'm actually terrified they will remove disable dynamic VRAM during a random update. It's the only way I can use ComfyUI..comfortably anymore.

Tons of people ARE reporting it and ZERO resolutions are offered sadly. It's EXTREMELY easy to reproduce since it DUMPS THE MODEL from RAM after EVERY SINGLE generation if you had to OFFLOAD to CPU...

#12927

There are many more, just search dynamic VRAM. The addition of dynamic VRAM has been catastrophic to NORMAL VRAM owners since release. :/

---

EDIT: Issue was caching, NOT dynamic VRAM. Running with --cache-classic solved the problem entirely. 6 second generations now compared to 250+ seconds before.

[Moussack]

0.24.0 still has unload issue for me, every gen is loading the loras and TE or clip (i didnt make any change on the workflow), making every gen unnecessary longer.

how to fix this on the latest build?? 0.21.0 doesnt have this issue.

[cdmusic2019]

dynamic vram

I also have an RTX 4070s. When using "dynamic vram", the "SamplerCustomAdvanced" node freezes during the second rendering, with GPU usage at 100%, and it takes a long time, usually 30-60 minutes, to recover normally. Sometimes it takes even longer.

[rattus128]

@cdmusic2019 I made a diagnostic --debug-hang on the latest master. If you can run it into that bad state and ctrl-C and paste the log I can debug but I have no reproducer of that reported hang to do anything about it.

Your definately not alone, its just I cant wholesale revert features for bugs ive never seen on scattered reports and I cant do anything until I get enough system info to reproduce or a user who I can work with to smash it out.

[rattus128]

@Moussack a working theory is this issue is actually related to the recent cache changes. Are you able to try with --cache-classic? In theory it should optimize but if that differentiates the performance on your system and workflow we can go from there.

[rattus128]

I'm actually terrified they will remove disable dynamic VRAM during a random update.
It's the only way I can use ComfyUI..comfortably anymore. Tons of people ARE reporting it and ZERO resolutions are offered sadly. It's EXTREMELY easy to reproduce since it DUMPS THE MODEL from RAM after EVERY SINGLE generation if you had to OFFLOAD to CPU...

@MoeMonsuta It just does not do this. Please see this video of 2x runs of Zimage BF16 on a 5060 (so offloaded) with task manager disk tab and the visualizer.

offload-demo.mp4

Memory usage visualizer here if you want to see whats up for your own:

https://github.com/kijai/ComfyUI-MemoryVisualization

Be wary of custom node packs out their monitoring VRAM and implementing panic model dumps. Some of them you don't even have to have a node in the workflow, just having the pack is enough.

Please try starting fresh with a bug report, or just follow the full bug report process here in thread.

#12927

That OPs report is about step rate and there have been several fixes since, and that one had a few tangentials. You seem to be reporting a model dump-out which is very different. Yes dyn vram has multiple bugs and that one is the early days. It probably still has bugs and things may fall through the cracks but your chance of getting a fix for you bug increases 1000% with a full bug report.

There are many more, just search dynamic VRAM. The addition of dynamic VRAM has been catastrophic to NORMAL VRAM owners since release. :/

Not sure what NORMAL is. But same system and workflow with --disable-dynamic-vram:

...
[INFO] Unloaded partially: 3539.22 MB freed, 1687.50 MB remains loaded, 253.12 MB buffer reserved, lowvram patches: 0
[INFO] loaded completely; 323.38 MB usable, 159.87 MB loaded, full load: True
[INFO] Prompt executed in 156.05 seconds
[INFO] got prompt
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
[INFO] Requested to load ZImageTEModel_
[INFO] loaded partially; 5547.68 MB usable, 5307.25 MB loaded, 2365.00 MB offloaded, 237.50 MB buffer reserved, lowvram patches: 0
[INFO] model weight dtype torch.bfloat16, manual cast: None
[INFO] model_type FLOW
[INFO] Requested to load Lumina2
[INFO] loaded partially; 5449.37 MB usable, 5224.37 MB loaded, 6515.17 MB offloaded, 225.00 MB buffer reserved, lowvram patches: 0
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:14<00:00,  1.85s/it]
[INFO] Requested to load AutoencodingEngine
[INFO] loaded completely; 2010.88 MB usable, 159.87 MB loaded, full load: True
[INFO] Prompt executed in 134.78 seconds

It is non dynamic actually doing the thing and reloading from disk on the second run while trashing the pagefile.

[cdmusic2019]

@cdmusic2019 I made a diagnostic --debug-hang on the latest master. If you can run it into that bad state and ctrl-C and paste the log I can debug but I have no reproducer of that reported hang to do anything about it.

Your definately not alone, its just I cant wholesale revert features for bugs ive never seen on scattered reports and I cant do anything until I get enough system info to reproduce or a user who I can work with to smash it out.

I have reproduced the issue again. While working, I have Photoshop open simultaneously. Sometimes when I click "Run" in ComfyUI and then switch to Photoshop, the GPU usage spikes to 100%. At this point, ComfyUI's nodes remain paused, and if I don't do anything, they may stay paused indefinitely. However, if I open a browser and browse any webpage, ComfyUI immediately resumes, the GPU usage drops from 100%, and the program runs normally. It doesn't seem to be a VRAM overflow issue; rather, the code seems to be waiting for some instruction to free up the VRAM. I have uploaded the runtime logs below.

This is my log file.

LOG.txt

[Moussack]

@Moussack a working theory is this issue is actually related to the recent cache changes. Are you able to try with --cache-classic? In theory it should optimize but if that differentiates the performance on your system and workflow we can go from there.

Yup It worked, "--cache-classic" flag fixed the unload issue every gen, thx man.

[Razunter]

Managed to render an image, but failed to load VAE.
debug_hang.txt

Flux Klein 9B stuck on "model initializing"
debug_hang2.txt

After multiple restarts of ComfyUI in a row Flux Klein 9B worked perfectly. Then I've switched to Ideogram 4 and it hangs...

Ideogram 4 render with memory visualization images

Started like this.

Then became this and started rendering, but extremely slow and becoming faster on every step.

Switch to VAE, slow, high GPU use

And stuck in this state forever.

debug_hang3.txt

[rattus128]

@Razunter Those logs are very helpful. That flux2-klein bug is a very un-expected lockup could be on any level of the stack, if you reproduce that form consistently the verbose aimdo log might have critical info. To get that log edit comfy with:

--- a/main.py
+++ b/main.py
@@ -242,6 +242,7 @@ if args.enable_dynamic_vram or (enables_dynamic_vram() and comfy.model_managemen
             comfy_aimdo.control.set_log_warning()
         else: #INFO
             comfy_aimdo.control.set_log_info()
+        comfy_aimdo.control.set_log_verbose()
 
         comfy.model_patcher.CoreModelPatcher = comfy.model_patcher.ModelPatcherDynamic
         comfy.memory_management.aimdo_enabled = True

(This needs to be a switch as its getting more and more useful - but for the moment its a code edit)

Was VRAM reported as full or close to full at the time by either task manager or visualizer?

Can I get your software stack versions for Cuda and pytorch (or a portable version number if that's your way) ?

The others are shared VRAM spills which should also not be happening, but I have other reports of something very similar on 4090 with some users having slow (in your case practially inifinate) recovery and will send something to try for that shortly.

[rattus128]

@Moussack can I get your system specs? What I am especially interested in is if you have two disks. Do you have a setup where the OS and model library are on different disks and can I get your disk specifications? Also RAM, GPU and OS version.

I do want to avoid people having to pick flags to get the best experience so if I can get this info I can figure out why RAM cache was a lose for you.

[owayej]

You guys see. most of the people don't want it. it just takes too much time. I'm not alone who don't like this dynamic vram stuff. it is really annoying.