用A100推理，只支持Eager模式，不只支持FlashInfer模式吗？

[biubiobiu]

用4张A100 直接运行：sh run_demo_instruct.sh
发现 混元Image3.0 instruct 在A100显卡下，运行不了FlashInfer模式，只能运行Eager模式。Eager模型下 速度较慢，请问一下有什么时候支持FlashInfer模式呢？

问题2：混元Image3.0 instruct 有没有其他的加速框架啊？比如：vLLM、Sglang、。。。

[tulvgengenr]

请问A100显卡是报什么错呢

[Bigfield77]

flashinfer fused moe requires cuda backend 9.0 min (a100 is 8.0)

I opened an issue for backend 8.6 (3090) but it does not seem possible:
flashinfer-ai/flashinfer#1941

[biubiobiu]

请问A100显卡是报什么错呢

报错信息是：
Assistant:
do_sample: True
max_new_tokens: 2048
top_k: 1024
top_p: 0.95
temperature: 0.6
repetition_penalty: 1.0

Traceback (most recent call last):
File "/code/HunyuanImage-3.0/run_image_gen.py", line 232, in
main(args)
File "/code/HunyuanImage-3.0/run_image_gen.py", line 203, in main
cot_text, samples = model.generate_image(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 3331, in generate_image
outputs = self.generate(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 3176, in generate
samples = super().generate(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/generation/utils.py", line 2564, in generate
result = decoding_method(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/generation/utils.py", line 2784, in _sample
outputs = self(**model_inputs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 2207, in forward
outputs = self.model(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 1633, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 1457, in forward
hidden_states = self.mlp(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/HunyuanImage-3.0/hunyuan_image_3/modeling_hunyuan_image_3.py", line 1190, in forward
_ = flashinfer.fused_moe.cutlass_fused_moe( # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/core.py", line 818, in cutlass_fused_moe
return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/core.py", line 290, in get_cutlass_fused_moe_module
raise ValueError(f"Invalid backend: {backend}")
ValueError: Invalid backend: 80

[tulvgengenr]

Thanks to @Bigfield77 for his answer. Currently, only eager mode is supported on Ampere series graphics cards. We will release pruned lightweight models in the future. Stay tuned.

[thorneliu]

@biubiobiu You may check this #93 to get a faster text generation in eager mode

[biubiobiu]

@biubiobiu You may check this #93 to get a faster text generation in eager mode

大佬🐂啊👍🏻👍🏻👍🏻