[Bug]: 0.11.0.RC0版本拉起bge-m3服务化，拉起成功，推理报错 #3462
Description
Your current environment
The output of `python collect_env.py`
Collecting environment information... PyTorch version: 2.7.1+cpu Is debug build: FalseOS: openEuler 24.03 (LTS-SP2) (aarch64)
GCC version: (GCC) 12.3.1 (openEuler 12.3.1-98.oe2403sp2)
Clang version: Could not collect
CMake version: version 4.1.0
Libc version: glibc-2.38
Python version: 3.11.13 (main, Aug 29 2025, 04:31:33) [GCC 12.3.1 (openEuler 12.3.1-97.oe2403sp2)] (64-bit runtime)
Python platform: Linux-5.10.0-60.18.0.50.oe2203.aarch64-aarch64-with-glibc2.38
CPU:
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Vendor ID: HiSilicon
BIOS Vendor ID: HiSilicon
Model name: Kunpeng-920
BIOS Model name: HUAWEI Kunpeng 920 5250 To be filled by O.E.M. CPU @ 2.6GHz
BIOS CPU family: 280
Model: 0
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 4
Stepping: 0x1
BogoMIPS: 200.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs
L1d cache: 12 MiB (192 instances)
L1i cache: 12 MiB (192 instances)
L2 cache: 96 MiB (192 instances)
L3 cache: 192 MiB (8 instances)
NUMA node(s): 8
NUMA node0 CPU(s): 0-23
NUMA node1 CPU(s): 24-47
NUMA node2 CPU(s): 48-71
NUMA node3 CPU(s): 72-95
NUMA node4 CPU(s): 96-119
NUMA node5 CPU(s): 120-143
NUMA node6 CPU(s): 144-167
NUMA node7 CPU(s): 168-191
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] pyzmq==27.1.0
[pip3] torch==2.7.1+cpu
[pip3] torch_npu==2.7.1.dev20250724
[pip3] torchvision==0.22.1
[pip3] transformers==4.56.2
[conda] Could not collect
vLLM Version: 0.11.0rc3
vLLM Ascend Version: 0.11.0rc0
ENV Variables:
ATB_OPSRUNNER_KERNEL_CACHE_LOCAL_COUNT=1
ATB_STREAM_SYNC_EVERY_RUNNER_ENABLE=0
ATB_OPSRUNNER_SETUP_CACHE_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_GLOBAL=0
ATB_DEVICE_TILING_BUFFER_BLOCK_NUM=32
ATB_STREAM_SYNC_EVERY_KERNEL_ENABLE=0
ATB_OPSRUNNER_KERNEL_CACHE_GLOABL_COUNT=5
ATB_HOME_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
ATB_COMPARE_TILING_EVERY_KERNEL=0
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
LD_LIBRARY_PATH=/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling:/usr/local/Ascend/driver/lib64/common/:/usr/local/Ascend/driver/lib64/driver/:
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_STREAM_SYNC_EVERY_OPERATION_ENABLE=0
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
ATB_MATMUL_SHUFFLE_K_ENABLE=1
ATB_WORKSPACE_MEM_ALLOC_ALG_TYPE=1
ATB_HOST_TILING_BUFFER_BLOCK_NUM=128
ATB_SHARE_MEMORY_NAME_SUFFIX=
TORCH_DEVICE_BACKEND_AUTOLOAD=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
NPU:
+------------------------------------------------------------------------------------------------+
| npu-smi 25.2.0 Version: 25.2.0 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B4-1 | OK | 91.0 37 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 3400 / 65536 |
+===========================+===============+====================================================+
| 1 910B4-1 | OK | 90.3 37 0 / 0 |
| 0 | 0000:C2:00.0 | 0 0 / 0 3398 / 65536 |
+===========================+===============+====================================================+
| 2 910B4-1 | OK | 95.3 39 0 / 0 |
| 0 | 0000:81:00.0 | 0 0 / 0 3397 / 65536 |
+===========================+===============+====================================================+
| 3 910B4-1 | OK | 93.6 37 0 / 0 |
| 0 | 0000:82:00.0 | 0 0 / 0 3397 / 65536 |
+===========================+===============+====================================================+
| 4 910B4-1 | OK | 94.4 40 0 / 0 |
| 0 | 0000:01:00.0 | 0 0 / 0 3395 / 65536 |
+===========================+===============+====================================================+
| 5 910B4-1 | OK | 90.9 42 0 / 0 |
| 0 | 0000:02:00.0 | 0 0 / 0 3394 / 65536 |
+===========================+===============+====================================================+
| 6 910B4-1 | OK | 94.1 43 0 / 0 |
| 0 | 0000:41:00.0 | 0 0 / 0 3395 / 65536 |
+===========================+===============+====================================================+
| 7 910B4-1 | OK | 97.3 42 0 / 0 |
| 0 | 0000:42:00.0 | 0 0 / 0 3394 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| No running processes found in NPU 0 |
+===========================+===============+====================================================+
| No running processes found in NPU 1 |
+===========================+===============+====================================================+
| No running processes found in NPU 2 |
+===========================+===============+====================================================+
| No running processes found in NPU 3 |
+===========================+===============+====================================================+
| No running processes found in NPU 4 |
+===========================+===============+====================================================+
| No running processes found in NPU 5 |
+===========================+===============+====================================================+
| No running processes found in NPU 6 |
+===========================+===============+====================================================+
| No running processes found in NPU 7 |
+===========================+===============+====================================================+
CANN:
package_name=Ascend-cann-toolkit
version=8.2.RC1
innerversion=V100R001C22SPC001B231
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21],[V100R001C23]
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.2.RC1/aarch64-linux
🐛 Describe the bug
服务化拉起过程：
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
export VLLM_USE_V1=1
vllm serve /home/XXXXXXX/bge-m3 --enforce-eager
——————————————————————————————————
请求动作：
curl -w "time_total:%{time_total}" -X POST -H "Content-Type: application/json" -d '{"model": "/XXXXXX/bge-m3", "input": ["how are you"]}' http://localhost:8000/v1/embeddings
客户端报错：
{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"BadRequestError","param":null,"code":400}}time_total:0.025061
——————————————————————————————————
服务端具体报错：
INFO 10-15 02:15:18 [init.py:36] Available plugins for group vllm.platform_plugins:
INFO 10-15 02:15:18 [init.py:38] - ascend -> vllm_ascend:register
INFO 10-15 02:15:18 [init.py:41] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 10-15 02:15:18 [init.py:207] Platform plugin ascend is activated
WARNING 10-15 02:15:22 [_custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:24 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:25 [model_runner_v1.py:2627] Starting to load model /home/w30042044/bge-m3...
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.19s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.19s/it]
(EngineCore_DP0 pid=3442)
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:28 [default_loader.py:267] Loading weights took 2.93 seconds
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:29 [model_runner_v1.py:2661] Loading model weights took 1.0559 GB
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:29 [core.py:210] init engine (profile, create kv cache, warmup model) took 0.02 seconds
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:30 [config.py:839] Found sentence-transformers tokenize configuration.
(EngineCore_DP0 pid=3442) WARNING 10-15 02:15:31 [core.py:112] Using configured V1 scheduler class vllm_ascend.core.scheduler.AscendScheduler. This scheduler interface is not public and compatibility may not be maintained.
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:31 [core.py:121] Disabling chunked prefill for model without KVCache
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:31 [init.py:381] Cudagraph is disabled under eager mode
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:31 [init.py:440] Only "last" pooling supports chunked prefill and prefix caching; disabling both.
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:31 [platform.py:141] Non-MLA LLMs forcibly disable the chunked prefill feature,as the performance of operators supporting this feature functionality is currently suboptimal.
(EngineCore_DP0 pid=3442) INFO 10-15 02:15:31 [platform.py:179] Compilation disabled, using eager mode by default
(APIServer pid=3304) INFO 10-15 02:15:31 [loggers.py:147] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 1
(APIServer pid=3304) INFO 10-15 02:15:31 [api_server.py:1634] Supported_tasks: ['encode', 'embed']
(APIServer pid=3304) INFO 10-15 02:15:31 [api_server.py:1912] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:34] Available routes are:
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /docs, Methods: HEAD, GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /redoc, Methods: HEAD, GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /health, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /load, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /ping, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /ping, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /tokenize, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /detokenize, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/models, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /version, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/responses, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/chat/completions, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/completions, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/embeddings, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /pooling, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /classify, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /score, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/score, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/audio/translations, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /rerank, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v1/rerank, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /v2/rerank, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /invocations, Methods: POST
(APIServer pid=3304) INFO 10-15 02:15:31 [launcher.py:42] Route: /metrics, Methods: GET
(APIServer pid=3304) INFO: Started server process [3304]
(APIServer pid=3304) INFO: Waiting for application startup.
(APIServer pid=3304) INFO: Application startup complete.
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] Traceback (most recent call last):
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] engine_core.run_busy_loop()
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 728, in run_busy_loop
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] self._process_engine_step()
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 283, in step
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] scheduler_output = self.scheduler.schedule()
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] File "/vllm-workspace/vllm-ascend/vllm_ascend/core/scheduler.py", line 209, in schedule
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] blocks = new_computed_blocks.blocks[0]
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
(EngineCore_DP0 pid=3442) ERROR 10-15 02:15:55 [core.py:710] IndexError: tuple index out of range
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] AsyncLLM output_handler failed.
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] Traceback (most recent call last):
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 439, in output_handler
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] outputs = await engine_core.get_output_async()
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 846, in get_output_async
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] raise self._format_exception(outputs) from None
(APIServer pid=3304) ERROR 10-15 02:15:55 [async_llm.py:480] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=3304) INFO: 172.17.0.1:38816 - "POST /v1/embeddings HTTP/1.1" 400 Bad Request
(EngineCore_DP0 pid=3442) Process EngineCore_DP0:
(EngineCore_DP0 pid=3442) Traceback (most recent call last):
(EngineCore_DP0 pid=3442) File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=3442) self.run()
(EngineCore_DP0 pid=3442) File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=3442) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=3442) raise e
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP0 pid=3442) engine_core.run_busy_loop()
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 728, in run_busy_loop
(EngineCore_DP0 pid=3442) self._process_engine_step()
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP0 pid=3442) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=3442) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 283, in step
(EngineCore_DP0 pid=3442) scheduler_output = self.scheduler.schedule()
(EngineCore_DP0 pid=3442) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=3442) File "/vllm-workspace/vllm-ascend/vllm_ascend/core/scheduler.py", line 209, in schedule
(EngineCore_DP0 pid=3442) blocks = new_computed_blocks.blocks[0]
(EngineCore_DP0 pid=3442) ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
(EngineCore_DP0 pid=3442) IndexError: tuple index out of range
(APIServer pid=3304) INFO: Shutting down
(APIServer pid=3304) INFO: Waiting for application shutdown.
(APIServer pid=3304) INFO: Application shutdown complete.
(APIServer pid=3304) INFO: Finished server process [3304]
(EngineCore_DP0 pid=3442) [rank0]: Traceback (most recent call last):
(EngineCore_DP0 pid=3442) [rank0]: File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py", line 60, in get_py_exception_dict
(EngineCore_DP0 pid=3442) [rank0]: if isinstance(err_msg, tuple) and err_msg[0] and isinstance(err_msg[0], dict):
(EngineCore_DP0 pid=3442) [rank0]: ~~~~~~~^^^
(EngineCore_DP0 pid=3442) [rank0]: IndexError: tuple index out of range
(EngineCore_DP0 pid=3442) [ERROR] 2025-10-15-02:15:58 (PID:3442, Device:0, RankID:-1) ERR99999 UNKNOWN applicaiton exception