Pooling in Embedding

Pooling is typically used in natural language processing (NLP) to reduce the dimensionality of feature representations, making them more compact while retaining the most relevant information. In this context, it can be used to transform variable-length input sequences into fixed-size representations.

sample code

CLS Pooling

Extracts the first token (or embedding) from each sequence, which is often used in models like BERT where the first token summarizes the entire input. It’s useful when the first token has special meaning (e.g., classification in transformer models).

Mean pooling

Computes the average of all tokens in the sequence, summarizing the sequence by averaging all embeddings. Mean pooling treats all elements equally, giving a balanced summary of the sequence. It works well when all parts of the sequence are important.

Fixed-size output: No matter the length of the input sequence (whether it’s 5 words or 100 words), mean pooling will…