Skip to content

vllm.model_executor.models.transformers

Wrapper around transformers models

Modules:

  • base –

    Transformers modeling backend base class.

  • causal –

    Transformers modeling backend mixin for causal language models.

  • fuser –

    Fuser detection for the Transformers modeling backend.

  • fusers –

    Concrete fusers for the Transformers modeling backend.

  • fx_utils –

    fx tracing and forward-source rewriting for the Transformers backend fusers.

  • layers –

    Layer provider resolution for the Transformers modeling backend.

  • legacy –

    Transformers modeling backend mixin for legacy models.

  • moe –

    Transformers modeling backend mixin for Mixture of Experts (MoE) models.

  • multimodal –

    Transformers modeling backend mixin for multi-modal models.

  • pooling –

    Transformers modeling backend mixins for pooling models.

  • utils –

    Transformers modeling backend utilities.

Functions:

  • __getattr__ –

    Handle imports of non-existent classes with a helpful error message.

  • check_sinks –

    Fail loudly if the model applies a sink the attention layer will not.

__getattr__(name)

Handle imports of non-existent classes with a helpful error message.

Source code in vllm/model_executor/models/transformers/__init__.py 
190
191
192
193
194
195
196
197
198
def __getattr__(name: str):
    """Handle imports of non-existent classes with a helpful error message."""
    if name not in globals():
        raise AttributeError(
            "The Transformers modeling backend does not currently have a class to "
            f"handle the requested model type: {name}. Please open an issue at "
            "https://github.com/vllm-project/vllm/issues/new"
        )
    return globals()[name]

check_sinks(module, self_attn, s_aux)

Fail loudly if the model applies a sink the attention layer will not.

Only the attention impl can fold a sink into the softmax denominator, so a sink that never reached Attention is dropped and every softmax is subtly wrong.

Source code in vllm/model_executor/models/transformers/__init__.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
def check_sinks(
    module: "torch.nn.Module",
    self_attn: "Attention | MLAAttention",
    s_aux: "torch.Tensor | None",
):
    """Fail loudly if the model applies a sink the attention layer will not.

    Only the attention impl can fold a sink into the softmax denominator, so a sink
    that never reached `Attention` is dropped and every softmax is subtly wrong.
    """
    if s_aux is None or getattr(self_attn, "has_sink", False):
        return
    raise ValueError(
        f"{type(module).__name__} applies attention sinks, but they were not passed "
        f"to {type(self_attn).__name__}, so the output would be wrong. Either the "
        "Transformers modeling backend could not find the parameter holding them, or "
        "vLLM does not support sinks for this kind of attention. Please open an issue "
        "at https://github.com/vllm-project/vllm/issues/new"
    )