[Usage]: import ucm_custom_ops报错ModuleNotFoundError: No module named 'ucm_custom_ops'

[Jia-Xueyu]

Your current environment

ERROR 03-11 02:58:23 [multiproc_executor.py:487]     import ucm_custom_ops
ERROR 03-11 02:58:23 [multiproc_executor.py:487] ModuleNotFoundError: No module named 'ucm_custom_ops'
ERROR 03-11 02:58:23 [multiproc_executor.py:487] WorkerProc failed to start.
ERROR 03-11 02:58:23 [multiproc_executor.py:487] Traceback (most recent call last):
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 461, in worker_main
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     worker = WorkerProc(*args, **kwargs)
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 357, in __init__
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     self.worker.init_device()
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/vllm-workspace/vllm/vllm/worker/worker_base.py", line 606, in init_device
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     self.worker.init_device()  # type: ignore
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 140, in init_device
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     self._init_worker_distributed_environment()
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 340, in _init_worker_distributed_environment
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     ensure_ucm_sparse_initialized(self.vllm_config)
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/workspace/unified-cache-management/ucm/sparse/state.py", line 52, in ensure_ucm_sparse_initialized
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     _UCM_SPARSE_AGENT = UcmSparseFactory.create_sparse_method(
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/workspace/unified-cache-management/ucm/sparse/factory.py", line 39, in create_sparse_method
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     sparse_method_cls = cls._registry[sparse_method_name]()
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/workspace/unified-cache-management/ucm/sparse/factory.py", line 25, in loader
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     module = importlib.import_module(module_path)
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/usr/local/python3.10.17/lib/python3.10/importlib/__init__.py", line 126, in import_module
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     return _bootstrap._gcd_import(name[level:], package, level)
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ERROR 03-11 02:58:23 [multiproc_executor.py:487]   File "/workspace/unified-cache-management/ucm/sparse/gsa_on_device/gsa_on_device.py", line 9, in <module>
ERROR 03-11 02:58:23 [multiproc_executor.py:487]     import ucm_custom_ops
ERROR 03-11 02:58:23 [multiproc_executor.py:487] ModuleNotFoundError: No module named 'ucm_custom_ops'

跑的是0.3.0-release分支，按官网文档安装和编译的，跑gsa_on_device用例报错ucm_custom_ops，是编译时候什么依赖项没加吗，编译添加了export ENABLE_SPARSE=TRUE环境变量，编译也没报错。
麻烦解答，谢谢

How would you like to use ucm.

I want to run inference of a [specific model](put link here). I don't know how to integrate it with unified.

[yuanzhg078]

你可以用当前develop分支尝试下，之前release0.3.0分支没有上传gsa_on_device算子。

[Jia-Xueyu]

你可以用当前develop分支尝试下，之前release0.3.0分支没有上传gsa_on_device算子。

能提个pr到0.3.0-release吗，想在一个稳定分支上开发

[yuanzhg078]

你可以用当前develop分支尝试下，之前release0.3.0分支没有上传gsa_on_device算子。

能提个pr到0.3.0-release吗，想在一个稳定分支上开发

还涉及其他修改，还是建议用develop尝试，gsa_on_device的功能会相对完备。

[Jia-Xueyu]

你可以用当前develop分支尝试下，之前release0.3.0分支没有上传gsa_on_device算子。

能提个pr到0.3.0-release吗，想在一个稳定分支上开发

还涉及其他修改，还是建议用develop尝试，gsa_on_device的功能会相对完备。

[ERROR] TBE(3626,python3):2026-03-12-09:18:20.986.224 [../../../opc_tool/op_compilation.py:338][__single_op_compile] Exception occured, An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.186 [../../../opc_tool/opc_common.py:231][_log_error] Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/op_compilation.py", line 330, in __single_op_compile
json_file_path = single_op_obj.op_compile()
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/single_op_compile.py", line 231, in op_compile
json_file_path = self.__call_op()
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/single_op_compile.py", line 142, in __call_op
op_func(*inputs, *outputs, *new_attrs, self.__op_info.get(OpcOptions.KERNEL_NAME), **kwargs)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/common/register/operation_func_mgr.py", line 214, in wrapper
return func(*args, **kwargs)
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.257 [../../../opc_tool/opc_common.py:231][_log_error] File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/common/utils/para_check.py", line 543, in _in_wrapper
return func(*args, **kwargs)
File "/workspace/unified-cache-management/ucm/sparse/gsa_on_device/csrc/ascend/build/binary/ascend910b/src/reshape_and_cache_bnsd/ReshapeAndCacheBnsd.py", line 205, in reshape_and_cache_bnsd
compile_op(src, origin_func_name, op_info, options, code_channel, '{}')
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/compile_op.py", line 1083, in compile_op
infered_info_from_ifile = KernelInfoInfer.get_tiling_key_list_and_simple_infer_code_channel(cce_file,
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.313 [../../../opc_tool/opc_common.py:231][_log_error] File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/compile_op.py", line 457, in get_tiling_key_list_and_simple_infer_code_channel
CommonUtility.run_cmd_inner(pre_compile_cmd, CompileStage.PRECOMPILE, compile_log_path)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/ascendc_common_utility.py", line 315, in run_cmd_inner
CommonUtility.dump_build_log(output.decode(), cmds, stage, ret_code)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/ascendc_common_utility.py", line 480, in dump_build_log
raise Exception("An error occurred during compile phases of {}, msg is {}".
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.366 [../../../opc_tool/opc_common.py:231][_log_error] Exception: An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'

[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.421 [../../../opc_tool/op_compilation.py:211][record_compile_error_info] Op[ReshapeAndCacheBnsd] of index[0] compile failed, kernelName: ReshapeAndCacheBnsd_69cf40a071c531b49b5be27bfaa4fab0.
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.989.641 [../../../opc_tool/opc_common.py:524][opc_log_full] [op_compilation.py:213][record_compile_error_info] reason is:[An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'
].
[ERROR] TBE(3626,python3):2026-03-12-09:18:20.990.335 [../../../opc_tool/opc.py:721][main] Opc tool compile failed.
Opc tool start working now, please wait for a moment.
[2026-3-12 9:18:20] [ERROR] [ascend910b] ReshapeAndCacheBnsd_69cf40a071c531b49b5be27bfaa4fab0 bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'
/workspace/unified-cache-management/ucm/sparse/gsa_on_device/csrc/ascend/build/binary/ascend910b/bin/reshape_and_cache_bnsd/ReshapeAndCacheBnsd_69cf40a071c531b49b5be27bfaa4fab0.json not generated!
gmake[2]: *** [CMakeFiles/reshape_and_cache_bnsd_ascend910b_1.dir/build.make:73: binary/ascend910b/gen/reshape_and_cache_bnsd_ascend910b_1.done] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1288: CMakeFiles/reshape_and_cache_bnsd_ascend910b_1.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.979.678 [../../../opc_tool/op_compilation.py:338][__single_op_compile] Exception occured, An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.334 [../../../opc_tool/opc_common.py:231][_log_error] Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/op_compilation.py", line 330, in __single_op_compile
json_file_path = single_op_obj.op_compile()
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/single_op_compile.py", line 231, in op_compile
json_file_path = self.__call_op()
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/opc_tool/single_op_compile.py", line 142, in __call_op
op_func(*inputs, *outputs, *new_attrs, self.__op_info.get(OpcOptions.KERNEL_NAME), **kwargs)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/common/register/operation_func_mgr.py", line 214, in wrapper
return func(*args, **kwargs)
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.397 [../../../opc_tool/opc_common.py:231][_log_error] File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/common/utils/para_check.py", line 543, in _in_wrapper
return func(*args, **kwargs)
File "/workspace/unified-cache-management/ucm/sparse/gsa_on_device/csrc/ascend/build/binary/ascend910b/src/hamming_dist_top_k/HammingDistTopK.py", line 229, in hamming_dist_top_k
compile_op(src, origin_func_name, op_info, options, code_channel, '{}')
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/compile_op.py", line 1083, in compile_op
infered_info_from_ifile = KernelInfoInfer.get_tiling_key_list_and_simple_infer_code_channel(cce_file,
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.450 [../../../opc_tool/opc_common.py:231][_log_error] File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/compile_op.py", line 457, in get_tiling_key_list_and_simple_infer_code_channel
CommonUtility.run_cmd_inner(pre_compile_cmd, CompileStage.PRECOMPILE, compile_log_path)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/ascendc_common_utility.py", line 315, in run_cmd_inner
CommonUtility.dump_build_log(output.decode(), cmds, stage, ret_code)
File "/usr/local/Ascend/ascend-toolkit/8.1.RC1/python/site-packages/tbe/tikcpp/ascendc_common_utility.py", line 480, in dump_build_log
raise Exception("An error occurred during compile phases of {}, msg is {}".
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.500 [../../../opc_tool/opc_common.py:231][_log_error] Exception: An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'

[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.552 [../../../opc_tool/op_compilation.py:211][record_compile_error_info] Op[HammingDistTopK] of index[0] compile failed, kernelName: HammingDistTopK_d80ff005b9d8dec2b179b923e99799e2.
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.983.761 [../../../opc_tool/opc_common.py:524][opc_log_full] [op_compilation.py:213][record_compile_error_info] reason is:[An error occurred during compile phases of CompileStage.PRECOMPILE, msg is bisheng: error: unsupported option '--op_relocatable_kernel_binary=true'
].
[ERROR] TBE(3631,python3):2026-03-12-09:18:20.984.411 [../../../opc_tool/opc.py:721][main] Opc tool compile failed.
Opc tool start working now, please wait for a moment.
develop分支编译报错，这个是什么原因

[Infinite666]

你用的是什么版本的镜像？需要vllm-ascend:v0.11.0，这个镜像里CANN版本应该是8.3.RC2