\receiveddate

XX Month, XXXX \reviseddateXX Month, XXXX \accepteddateXX Month, XXXX \publisheddateXX Month, XXXX \currentdateXX Month, XXXX \doiinfoOJITS.2022.1234567

\authornote

^∗Ji Zhou and Yilin Ding contributed equally to this work. The code of this work is available via the following link: https://github.com/ftgTUGraz/LLM-SOTIF.

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

Ji Zhou Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Yilin Ding Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Yongqi Zhao Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Jiachen Xu Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Dong Bi Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Johannes Betz Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Arno Eichberger Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Affiliation: Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria (e-mail: ji.zhou@student.tugraz.at; yilin.ding@student.tugraz.at; yongqi.zhao@tugraz.at; arno.eichberger@tugraz.at) Affiliation: Independent Researcher, 200000 Shanghai, China (e-mail: jcxu97@gmail.com) Affiliation: School of Intelligent Connected Vehicle, Hubei University of Automotive Technology, 442002 Shiyan, China (e-mail: dbi@huat.edu.cn) Affiliation: Professorship of Autonomous Vehicle Systems, Technical University of Munich, 85748 Garching, Germany (e-mail: johannes.betz@tum.de)

Abstract

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against two specialized detectors: the anchor-based YOLOv5 and the transformer-based RT-DETRv4. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3) surpass the YOLOv5 in recall by over 25% and closely match RT-DETRv4 under natural visual degradation, while specialized detectors retain an advantage in geometric precision for handcrafted perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.

Index Terms:

Automated driving, Safety of the Intended Functionality, Vision-Language Models, Perception failure.

^†^†corresponding: CORRESPONDING AUTHOR: Yongqi Zhao (e-mail: yongqi.zhao@tugraz.at).

I Introduction

Automated vehicles rely heavily on onboard perception systems to sense and interpret their surroundings, with vision-based sensing playing a crucial role in environment understanding [1]. Beyond failure caused by hardware or software malfunctions, safety risks may also arise when correctly functioning perception algorithms encounter conditions that exceed their operational capabilities, such as severe glare or low visibility. To address such non-malfunction-related safety risks, Safety of the Intended Functionality (SOTIF), formalized in ISO 21448, focuses on hazards caused by functional insufficiency rather than component faults [2]. For example, as shown in Figure 1, severe glare from oncoming traffic can significantly degrade visibility. In such scenarios, an automated vehicle may fail to detect a pedestrian or another vehicle even though the perception systems operate as specified. Such failure modes highlight systematic perception limitations that are not attributable to component malfunctions.

Refer to caption — Fig. 1: An example of potential perception functional insufficiency under severe glare from oncoming traffic, as provided in the PeSOTIF dataset [3].

Conventional object detection benchmarks in autonomous driving, such as KITTI [4], nuScenes [5] and BDD100K [6], are primarily designed around common traffic scenarios and standard detection conditions, and therefore often underrepresent perception failures in adverse environments. This discrepancy motivates SOTIF-oriented datasets that explicitly target non-malfunction-related perception limitations. To this end, PeSOTIF dataset [3] was introduced to curate challenging scenarios, such as the one illustrated in Figure 1, characterized by adverse conditions and atypical objects, which are prone to inducing perception failures. By systematically capturing these corner cases, PeSOTIF serves as a dedicated testbed for identifying the operational boundaries of perception systems and evaluating their robustness against functional insufficiencies.

Motivated by the SOTIF-oriented perception failures discussed above, existing algorithmic efforts for object detection can be broadly grouped into two paradigms. One line of research enhances conventional detectors by explicitly enlarging safety margins under perceptual uncertainty, thereby improving robustness within the paradigm of detector-centric architectures. For example, Peng et al. [7] incorporate probabilistic inference mechanisms into the YOLO architecture to quantify uncertainty and identify perception risk in complex SOTIF scenarios. Similarly, Wang et al. [8] augment YOLO [9] with enhanced geometric constraints, achieving improved robustness and higher detection accuracy on KITTI.

In parallel, the emergence of Large Vision-Language Models (LVLMs) offers a new perspective: leveraging their vast world knowledge and semantic reasoning capabilities to handle open-set concepts and interpret complex scenarios. Representative studies focus on high-level reasoning, interpretation, or decision-making tasks using models such as GPT-4 and Video-LLaVA [10], or propose SOTIF-oriented frameworks for risk interpretation specifically under adverse conditions [11]. Other efforts employ LLMs as auxiliary tools, for instance by guiding diffusion-based data augmentation to improve few-shot object detection [12]. However, these studies predominantly operate at the semantic level. It remains unverified whether the high-level reasoning of LVLMs can be effectively translated into precise geometric localization, a prerequisite for safe motion planning. Consequently, it is unclear how their detection performance compares systematically against established detectors such as YOLO on dedicated benchmarks like PeSOTIF.

To bridge this gap, the presented work conducts a comprehensive benchmark of ten mainstream LVLMs for object detection under SOTIF conditions, employing established YOLOv5 and RT-DETRv4 [13] detectors as comparative baselines. The main contributions are summarized as follows:

1.

A unified evaluation pipeline is developed to enable LVLMs to perform 2D object detection via visual grounding under SOTIF conditions, effectively bridging the gap between high-level semantic reasoning and precise geometric localization without task-specific fine-tuning.
2.

A systematic benchmark of ten representative large foundation models is conducted for SOTIF-oriented object detection, using PeSOTIF as a dedicated testbed and two specialized detectors, YOLOv5 and RT-DETRv4, as reference baselines.
3.

A quantitative analysis of detection performance and failure patterns is provided, revealing a critical trade-off where LVLMs exhibit superior robustness in semantic recall but lag behind conventional detectors in geometric precision.

By establishing this benchmark, it is aimed to provide a quantitative reference for the safe deployment of foundation models in automated driving. The insights derived from the performance trade-offs are expected to guide the design of future hybrid perception systems, which combine the semantic robustness of LVLMs with the geometric precision of specialized detectors to address the safety challenges of SOTIF.

II Methodology

The overall workflow of the presented work is depicted in Figure 2. The process initiates with image preprocessing, where raw samples from the PeSOTIF dataset [3] are standardized through resizing, border annotation, and the superimposition of scale markers. Subsequently, the processed inputs are fed into ten LVLMs for object detection, with the generated responses parsed into a structured format consistent with the dataset annotations. In the final stage¹¹ 1 The metrics are explained in subsection II-E., performance evaluation is conducted by quantitatively comparing the predicted bounding boxes against ground-truth labels to measure comprehensive detection performance.

II-A PeSOTIF Dataset

This work employs the PeSOTIF dataset [3] as the testbed. As a dedicated benchmark for SOTIF-oriented perception, PeSOTIF comprises 1126 frames capturing long-tail traffic scenarios. Distinct from conventional datasets, it prioritizes scenes characterized by perception-degrading conditions and atypical road anomalies, which serve as stress tests for vision-based systems.

As depicted in Figure 3, PeSOTIF is structured into two subsets. The environment subset categorizes visual degradation into natural conditions (e.g., rain) and handcrafted perturbations (e.g., synthetic glare). Complementing this, the object subset distinguishes between common vehicles and atypical anomalies (e.g., overturned trucks).

II-B Image Preprocessing

II-B1 Image Resize

As shown in the first stage of the preprocessing pipeline (Figure 2), all images are resized using a standardized procedure to ensure a consistent input format across the dataset and to facilitate LVLM inference. Each raw image $I$ , with original width $W_{\text{orig}}$ and height $H_{\text{orig}}$ , was scaled to fit within bounds $W_{\text{max}}=800$ and $H_{\text{max}}=600$ pixels, thereby providing a uniform coordinate space for subsequent evaluation. The resizing operation preserved the original aspect ratio to avoid geometric distortion of traffic objects. A global scaling factor $S$ was computed as

S=\min\left(\frac{W_{\max}}{W_{\text{orig}}},\frac{H_{\max}}{H_{\text{orig}}}\right).

(1)

The resized dimensions were then obtained by $W_{\text{new}}=S\cdot W_{\text{orig}}$ and $H_{\text{new}}=S\cdot H_{\text{orig}}$ . To reduce aliasing²² 2 Aliasing describes distortions introduced during image resizing or downsampling, where fine spatial details are not correctly preserved. during downsampling and maintain visual quality, the Lanczos resampling filter [14] was applied.

II-B2 Draw Border

After resizing, a clear visual boundary was added to define the spatial extent of the input. Specifically, a continuous rectangular border with a linewidth of 2 pixels was drawn along the image perimeter, spanning from $(0,0)$ to $(W-1,H-1)$ . This boundary serves two purposes: it provides an explicit visual cue that delineates the Region of Interest (ROI) for the LVLM, and it helps separate scene content from the canvas edges, which may reduce coordinate estimation errors near image boundaries.

II-B3 Draw Scale

Because LVLMs may be limited in precise spatial localization³³ 3 This limitation is closely related to the concept of visual grounding, which describes the alignment between language expressions and image regions in computer vision., an explicit visual coordinate reference was embedded into the image. A visual coordinate reference in the form of ruler ticks was overlaid along the top (horizontal) and left (vertical) margins of the image. The coordinate range was divided into ten equal intervals, with tick marks placed every 10% of the image width ( $W$ ) and height ( $H$ ), respectively. Normalized numeric labels (e.g., 0.1, 0.2, …, 0.9) were displayed next to each tick. This augmentation provides a direct spatial reference within the input. By acting as explicit visual landmarks, these ticks allow the model to “read” coordinates directly from the image rather than implicitly, thereby improving the precision of bounding-box regression in the normalized coordinate domain $[0,1]$ .

II-C LVLMs Under Test

II-C1 LVLMs Specification

To ensure a comprehensive evaluation, ten state-of-the-art foundation models were selected from major developers, including Google, OpenAI, xAI, Anthropic, Alibaba, ByteDance, and Tencent. These models represent the forefront of multimodal capabilities as of late 2025. The selection covers a broad spectrum of parameter scales where publicly disclosed, ranging from lightweight models to trillion-parameter giants. The candidate models include: Grok 2, Grok 4, Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 3, GPT-5, Claude 4.5, Qwen 3-Max, Doubao and Hunyuan 2.0. Key specifications, including developer, release note, and parameter count, are summarized in Table I.

TABLE I: Specifications of LVLMs under test

Model	Developer	Release Date	Parameter
Grok 2	xAI	Aug. 2024	270B
Grok 4	xAI	July 2025	1.7T
Gemini 2.5 Pro	Google	June 2025	128B
Gemini 2.5 Flash	Google	May 2025	5B
Gemini 3	Google	Nov. 2025	-
GPT-5	OpenAI	Aug. 2025	-
Qwen 3-Max	Alibaba	Sep. 2025	1T
Claude 4.5	Anthropic	Nov. 2025	-
Doubao	ByteDance	May 2024	-
Hunyuan 2.0	Tencent	Dec. 2025	32B

II-C2 Prompt Engineering

As depicted in Figure 4, the prompt is designed with a structured chain-of-thought [15] approach, comprising four strategic components to maximize inference reliability: (a) Role Definition establishes an expert persona for image-based perception, aiming to activate the model’s domain-specific latent knowledge; (b) Task Specification explicitly details the 11-class taxonomy and instructs the model to use the visual rulers grounding, thereby aligning semantic understanding with the normalized coordinate space $[0,1]$ ; (c) Output Format enforces a strict JSON schema for class IDs and bounding boxes, facilitating automated parsing; (d) Constraints restrict the response to the specified fields to ensure consistent, machine-readable outputs.

(a)

(b)

(c)

(d)

Fig. 4: Structure of the LLM prompt for 2D object detection, consisting of (a) role definition, (b) task specification with class taxonomy and coordinate reference, (c) output format specification in JSON, and (d) constraints to enforce format consistency.

II-D LVLM Output Parsing

The unstructured, verbose textual responses of LVLMs were transformed into structured data for quantitative analysis. Using regular expressions, the system extracted the JSON-formatted detection result from the raw outputs while removing additional text (e.g., explanations). The extracted JSON was then parsed to obtain the required object attributes, including class labels, normalized bounding-box coordinates, and safety-critical indicators. This procedure ensures that only valid and standardized detection records are passed to the downstream Intersection-Over-Union (IoU) evaluation module.

II-E Performance Evaluation

In this study, two specialized detectors are adopted as comparative baselines to span the two dominant paradigms of modern 2D object detection. First, YOLOv5 is selected as a representative of the single-stage, anchor-based convolutional detectors. While newer versions such as YOLOv11 exist, YOLOv5 is chosen for its recognized stability, extensive validation in safety-critical applications, and proven maturity as an industrial standard [16]. Specifically, the 60-epoch variant (YOLOv5_60e), reported in [3] is adopted as the representative baseline. Second, RT-DETRv4 [13] is introduced as a transformer-based, end-to-end detector that removes the hand-crafted non-maximum suppression step and achieves state-of-the-art speed-accuracy trade-offs on standard benchmarks. Including RT-DETRv4 alongside YOLOv5 enables the LVLMs to be compared against both the classical CNN lineage and the more recent DETR-style architectures that currently define the production frontier, thereby providing a more comprehensive reference for the benchmark. To assess LVLM performance in traffic-scene perception, standard object detection metrics based on IoU are adopted following the COCO benchmark protocols [17]. Detection precision is quantified using Mean Average Precision ( $\text{mAP}_{50}$ ) at an IoU threshold of 0.50, which is the mean of Average Precision (AP) across all $N_{c}$ object classes:

\text{mAP}_{50}=\frac{1}{N_{c}}\sum_{c=1}^{N_{c}}\text{AP}_{c}(0.50),

(2)

where $\text{AP}_{c}(0.50)$ denotes the average precision for class $c$ . Similarly, the detection completeness is measured using Mean Average Recall ( $\text{mAR}_{50}$ ), calculated as the mean of maximum recall values across classes:

\text{mAR}_{50}=\frac{1}{N_{c}}\sum_{c=1}^{N_{c}}\text{AR}_{c}(0.50),

(3)

where $\text{AR}_{c}(0.50)$ is the average recall for class $c$ . Finally, to evaluate localization robustness across diverse overlap requirements, the component-averaged Mean Average Precision ( $\text{mmAP}_{50:95}$ ) is computed by averaging the mAP over multiple IoU thresholds $\mathcal{T}=\{0.50,0.55,\dots,0.95\}$ :

\text{mmAP}_{50:95}=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\text{mAP}_{t}.

(4)

III Result

Figure 5visualizes the detection predictions across five representative SOTIF scenarios in the PeSOTIF dataset. As shown in the first three rows, general-purpose LVLMs demonstrate remarkable robustness under weather-induced degradation, effectively localizing objects even in rain, snow, and dense fog. Furthermore, these models exhibit strong semantic generalization in the last row, successfully grounding uncommon objects that challenge traditional detectors. These qualitative results suggest that LVLMs can translate high-level semantic understanding into reasonable spatial predictions, maintaining operation where traditional systems might fail.

III-A Overall Performance

Figure 6reports the overall object-detection performance of ten LVLMs compared to YOLOv5_60e and RT-DETRv4 [13]. RT-DETRv4 outperforms YOLOv5_60e across all three metrics. Nevertheless, Gemini 3 (⑤) achieves the highest mAP₅₀ score, followed by Doubao (⑨) and Gemini 2.5 Pro (③), with Gemini 3 surpassing both specialized detectors in mAP₅₀ and mAR₅₀, while falling marginally short of RT-DETRv4 in mmAP_50:95 (0.407 vs. 0.420). Moreover, the ranking is not fully explained by model parameter scale, as Gemini 2.5 Flash (④, 5B) outperforms larger models, such as Grok 4 (1.7T) and Qwen 3-Max (1T), in terms of mAP₅₀, highlighting the role of architecture and training in spatially grounded prediction.

III-B Performance in Environment Subset

Figure 7summarizes detection performance on the environment subset of PeSOTIF, covering perception-degrading conditions such as rain, snow, particulates, and challenging illumination. Overall, several LVLMs remain competitive under these conditions. Gemini 3 (⑤) achieves the best performance and surpasses both baselines in $mAP_{50}$ and $mAR_{50}$ , while falling marginally short of RT-DETRv4 in $mmAP_{50:95}$ . Doubao (⑨) and Gemini 2.5 Pro (③) also show strong results, outperforming YOLOv5_60e but remaining below RT-DETRv4, indicating that LVLM-based approaches can better tolerate environmental interference in long-tail traffic scenes while only the strongest LVLM matches the transformer-based detector.

III-B1 Performance in Natural Subset

Figure 8summarizes detection performance on the natural subset, which consists of real-world footage with complex and unstructured visual noise. The strongest LVLM outperforms both baselines in $mAP_{50}$ and $mAR_{50}$ . In particular, Gemini 3 (⑤) achieves the best $mAP_{50}$ and $mAR_{50}$ , exceeding YOLOv5_60e and RT-DETRv4; its $mmAP_{50:95}$ remains comparable to RT-DETRv4 and well above YOLOv5_60e. Doubao (⑨) and Gemini 2.5 Pro (③) also outperform YOLOv5_60e but remain below RT-DETRv4, indicating that several LVLMs maintain competitive detection quality in authentic driving scenes.

III-B2 Performance in Handcraft Subset

Figure 9reports the detection performance in the handcraft subset, which is composed of real-world images with manually constructed environment degradations. A different performance pattern is observed compared with the natural subset. The YOLOv5_60e baseline achieves the highest $mAP_{50}$ in this subset, outperforming all LVLMs and the RT-DETRv4 detector.

In contrast, recall and $mmAP_{50:95}$ are dominated by RT-DETRv4, which surpasses YOLOv5_60e and every LVLM on both metrics. Among the LVLMs, Gemini 3 (⑤) comes closest to the transformer-based detector in recall, while Doubao (⑨) still exceeds YOLOv5_60e in recall, suggesting fewer missed detections than the anchor-based baseline. Overall, these results point to a trade-off: YOLOv5_60e retains the strongest $mAP_{50}$ localization, whereas RT-DETRv4 and several LVLMs provide better object coverage on the handcraft subset.

III-C Performance in Object Subset

Figure 10illustrates the detection performance heatmap on the object subset. Gemini 3 (⑤) achieves the highest $mAP_{50}$ and $mAR_{50}$ across all evaluated models, while RT-DETRv4 attains a marginally higher $mmAP_{50:95}$ , indicating tighter bounding-box localization across stricter IoU thresholds. The YOLOv5_60e baseline exhibits competitive precision in terms of $mAP_{50}$ but substantially lower recall, highlighting its limited coverage under object-centric challenges. Gemini 3 (⑤) demonstrates the most balanced capability across localization accuracy and detection coverage in the object subset.

III-D Ablation on the Ruler Overlay

To substantiate the contribution of the ruler ticks introduced in the image preprocessing, an ablation study is conducted in which the ruler ticks are removed from the LVLM inputs while all other prompt components and image preprocessing operations remain unchanged. Grok 2 (①) is excluded from this ablation as the model was discontinued and unavailable at the time of the ablation experiment. Two baselines, YOLOv5_60e and RT-DETRv4, are unaffected by this ablation and are retained as references. The outcome is presented in Figure 11.

A significant degradation is observed across all evaluated LVLMs. Gemini 3 (⑤) remains the strongest LVLM but its $mAP_{50}$ drops from 0.695 to 0.525, and its $mmAP_{50:95}$ decreases from 0.407 to 0.307, indicating a clear loss in both detection coverage and localization tightness. Doubao (⑨) exhibits a similar trend, with $mAP_{50}$ falling from 0.535 to 0.391 and $mmAP_{50:95}$ from 0.243 to 0.145. The remaining LVLMs, including Gemini 2.5 Pro (③) and Gemini 2.5 Flash (④), experience even larger relative drops, with all of them below 0.25 in $mAP_{50}$ . In contrast, both baselines retain their original performance, and RT-DETRv4 [13] outperforms all LVLMs across all metrics.

These observations confirm that the ruler ticks are essential for accurate object detection in the proposed pipeline. Without them, LVLMs must rely on implicit visual grounding alone, which yields noticeably weaker detection performance. This establishes the ruler overlay as a lightweight, training-free mechanism for bridging the gap between semantic reasoning and precise localization under SOTIF conditions.

IV Conclusion and Discussion

An evaluation of ten LVLMs for 2D object detection within SOTIF-relevant traffic scenarios is presented in this study. Utilizing the PeSOTIF dataset and a visual-prompting methodology, we demonstrated that general-purpose LVLMs can effectively translate high-level semantic understanding into structured spatial output. Very importantly, without any task-specific fine-tuning on PeSOTIF dataset, the leading LVLMs already surpass the YOLOv5_60e baseline across all metrics, and exceed RT-DETRv4 in $mAP_{50}$ and $mAR_{50}$ , while remaining slightly below in $mmAP_{50:95}$ . The advantage is most pronounced in the natural subsets, where LVLMs exceed YOLOv5_60e in recall by over 25%, while closely matching RT-DETRv4 in the same subset, suggesting that their global contextual reasoning is more resilient to the degradation of local visual features than the feature-matching pipelines of YOLOv5_60e and RT-DETRv4.

The ablation study in Section III-III-D demonstrates the value of the ruler ticks as a core component of the proposed methodology. When the ruler is removed, the geometric accuracy of every evaluated LVLM degrades and RT-DETRv4 [13] regains the overall lead; when the ruler is provided, leading LVLMs move ahead of both baselines. This contrast establishes the ruler overlay as a lightweight, training-free prompt-level design that unlocks the geometric localization capability of pretrained LVLMs without any model modification. The ruler is therefore not an incidental trick but a principled and transferable interface between general-purpose LVLMs and precise geometric tasks.

Despite the promising detection capabilities, practical deployment faces challenges regarding computational efficiency, as the current processing speed of LVLMs generally lags behind the real-time requirements of automated driving. Consequently, LVLMs are currently best positioned as a high-level “safety validator” or a redundant perception branch to handle long-tail corner cases that challenge conventional sensors. Future research should target the distillation of these LVLMs into lightweight architectures and the refinement of end-to-end spatial alignment to further bridge the gap between semantic reasoning and real-time execution.

References

[1] E. Marti, M. A. de Miguel, F. Garcia, and J. Perez (2019) A review of sensor technologies for perception in automated driving. IEEE Intelligent Transportation Systems Magazine 11 (4), pp. 94–108. External Links: Document Cited by: §I.
[2] International Organization for Standardization (2022) Road vehicles — safety of the intended functionality. Standard Technical Report ISO 21448:2022, International Organization for Standardization, Geneva, Switzerland. Cited by: §I.
[3] L. Peng, J. Li, W. Shao, and H. Wang (2023) PeSOTIF: a challenging visual dataset for perception SOTIF problems in long-tail traffic scenarios. In 2023 IEEE Intelligent Vehicles Symposium (IV), pp. 1–8. Cited by: Fig. 1, §I, Fig. 3, §II-A, §II-E, §II.
[4] A. Geiger, P. Lenz, and R. Urtasun (2012) Are we ready for autonomous driving? the KITTI vision benchmark suite. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , pp. 3354–3361. External Links: Document Cited by: §I.
[5] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020) nuScenes: a multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I.
[6] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell (2020) BDD100K: a diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I.
[7] L. Peng, H. Wang, and J. Li (2021) Uncertainty evaluation of object detection algorithms for autonomous vehicles. Automotive Innovation 4 (3), pp. 241–252. Cited by: §I.
[8] S. Wang, Z. Wang, S. Hong, P. Wang, and S. Zhang (2025) Ensuring SOTIF: enhanced object detection techniques for autonomous driving. Accident Analysis & Prevention 218, pp. 108094. Cited by: §I.
[9] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi (2016) You only look once: unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I.
[10] S. Jain, S. Thapa, K. Chen, A. L. Abbott, and A. Sarkar (2024) Semantic understanding of traffic scenes with large vision language models. In 2024 IEEE Intelligent Vehicles Symposium (IV), Vol. , pp. 1580–1587. External Links: Document Cited by: §I.
[11] S. Huang, F. Shi, C. Sun, J. Zhong, M. Ning, Y. Yang, Y. Lu, H. Wang, and A. Khajepour (2025) DriveSOTIF: advancing SOTIF through multimodal large language models. IEEE Transactions on Vehicular Technology. Cited by: §I.
[12] Y. Jiang, S. Qiang, W. Li, and Y. Liang (2025) LLM-DiffAug: enhancing few-shot object detection via LLM-guided diffusion augmentation. Knowledge-Based Systems 326, pp. 114066. Cited by: §I.
[13] Z. Liao, Y. Zhao, X. Shan, Y. Yan, C. Liu, L. Lu, X. Ji, and J. Chen (2025) RT-DETRv4: painlessly furthering real-time object detection with vision foundation models. arXiv preprint arXiv:2510.25257. Cited by: §I, §II-E, Fig. 10, Fig. 6, Fig. 7, Fig. 8, Fig. 9, §III-A, §III-D, §IV.
[14] K. Turkowski (1990) Filters for common resampling tasks. In Graphics gems, pp. 147–165. Cited by: §II-B1.
[15] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 24824–24837. Cited by: §II-C2.
[16] N. Jegham, C. Y. Koh, M. Abdelatti, and A. Hendawi (2025) YOLO evolution: a comprehensive benchmark and architectural review of yolov12, yolo11, and their previous versions. External Links: 2411.00201, Link Cited by: §II-E.
[17] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft COCO: common objects in context. In Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.), Cham, pp. 740–755. External Links: ISBN 978-3-319-10602-1 Cited by: §II-E.