Beating Qwen3 LoRA with a Tiny PyTorch Encoder on the Large‑Scale Product Corpus

In this post I revisit product category classification on the Large Scale Product Corpus V2020 dataset with a fresh approach. Instead of adapting a large pretrained model, I train a small, task‑specific 80M parameters encoder with bidirectional attention end‑to‑end and compare it to my earlier Qwen3 0.8B LoRA baseline. The new encoder reaches 0.9315 macro‑F1 and 0.9487 accuracy on the test split in respect to baseline result: 0.8360 macro‑F1 and 0.8791 accuracy.

The new PyTorch encoder improves macro‑F1 by roughly 9.5 points, while using about one seventh of the parameters and being much easier to deploy. 80M encoder with batch-256 achieves latency around 0.5 ms per title versus roughly 3 ms per title for the larger Qwen3 LoRA model. All of the code, training scripts, and dataset utilities are open‑source on GitHub: github.com/ivan-digital/web-product-data.

For context, back in July I published Fine-Tuning Qwen3 Embeddings for Product Category Classification on the Large-Scale Product Corpus, where I took Qwen3‑Embedding‑0.6B, added a LoRA adapter, and fine‑tuned it on the same LSPC V2020 data.

What is the Large-Scale Product Corpus (LSPC)?

If you work on e‑commerce machine learning, the Large-Scale Product Corpus (LSPC) V2020 is an extremely useful dataset. It’s built from the Web Data Commons product subset: pages that contain structured product markup (titles, brands, GTINs, prices, categories and so on). There is exploration script.

Because merchants add this metadata for SEO, the dataset arrives almost fully self‑labelled. For this project, I focused on a simple, realistic product category classification task: Given a product title, predict one of six top‑level categories: Automotive, Baby, Books, Clothing, Jewelry, Shoes.

After preprocessing with the scripts in the repo, the dataset looks roughly like this:

• Training set: ~8.1 million product titles
• Validation set: ~480,000 titles
• Test set: ~950,000 titles

The corpus is heavily English‑skewed, but it’s multilingual. You’ll see German, French, Spanish, Japanese and other languages. That makes the LSPC dataset ideal for evaluating multilingual product classification models.

The Qwen3 Embeddings + LoRA baseline

In the previous post, I treated Qwen3-Embedding‑0.6B as a general‑purpose text classification model and adapted it with LoRA fine‑tuning.

The idea was:

• Start from Qwen3’s strong multilingual sentence embeddings.
• Add a LoRA adapter on top (low‑rank weight updates).
• Attach a simple classification head.
• Fine‑tune on the LSPC titles for product category classification.

Because Qwen3 Embeddings were originally trained on hundreds of millions of text pairs for retrieval and semantic similarity, they already captured a lot of useful structure: brand names, model numbers, domain‑specific jargon, multilingual patterns.

With a small LoRA patch and one epoch of training, the Qwen3 + LoRA model reached:

• Macro‑F1: 0.8360
• Accuracy: 0.8791

Latency was also decent: around 3.3–3.9 ms per title across batch sizes 16–128 on a single modern GPU, with throughput of about 300 titles per second at batch size 32.

Architectural choices and why they work for LSPC

The design is inspired by architectures like Qwen and LLaMA, but scaled down for short product titles and a 6‑class classification objective. Instead of just listing the components, it’s useful to spell out why each choice is there:

Qwen3 tokenizer reuse

I reuse the Qwen3 tokenizer, so tokenisation is identical to the LoRA baseline. That buys two things:

• All of Qwen’s pretraining work around splitting product‑style text into sensible tokens (brands, model numbers, units) is preserved, even though the encoder itself is new.

10 transformer‑style blocks, not 2 or 48

• Ten blocks give the model enough depth to build up from local patterns (“Air”, “Max”, “95”) to higher‑level structure (“Nike Air Max 95 running shoes”), without blowing up latency or memory. For short sequences and a small label space, more depth would mostly increase cost; less depth tends to hurt the ability to combine multiple weak signals in a title.

Hidden size 384 with 6 attention heads

A hidden size of 384 and 6 attention heads (64 dimensions per head) is a deliberate compromise:

• It’s large enough to store rich lexical and semantic information about millions of products and multiple languages.
• It keeps the total parameter count ~80M, which fits comfortably on a single GPU and makes inference cheap.
• The 64‑dim head size is a well‑behaved sweet spot: big enough for expressive attention, small enough to be efficient.

Grouped KV heads (6 Q heads share 2 K/V heads)

• Keys and values use grouped KV heads, where six query heads share two sets of keys and values. This reduces memory and compute overhead for K/V projections and attention without noticeably hurting quality on short sequences:
• You still get multiple attention patterns from the different Q heads.
• You pay for only two K/V “views” per layer instead of six, which matters when you stack ten layers and want to keep latency low.

Rotary position embeddings (RoPE)

• Attention uses rotary position embeddings instead of absolute position encodings. RoPE keeps relative position information baked into Q/K, which works well for product titles where patterns like “number → unit” or “brand → model” repeat in many contexts. It also behaves better when you vary sequence length than fixed positional embeddings, which helps when some titles are tiny and others are long.

SwiGLU feed‑forward layers

The feed‑forward layers use SwiGLU, a gated activation that improves gradient flow compared to a plain GELU MLP:

• The gating lets the model gate features in or out dynamically, which increases expressiveness for the same hidden size.
• In practice, SwiGLU tends to train more smoothly in transformer blocks, especially with RMSNorm and higher learning rates.

RMSNorm for all normalisation

Normalisation is done with RMSNorm (scale‑only LayerNorm) throughout:

• It plays nicely with pre‑norm transformers and deep stacks, helping to stabilise training from scratch.
• It’s slightly cheaper than full LayerNorm and avoids some of the issues with shifting means, which is helpful when you run relatively large learning rates on a small model.

DropPath in upper layers (stochastic depth)

Upper layers use DropPath as a regulariser. Instead of dropping individual activations, DropPath randomly drops entire residual branches during training, more often in deeper layers:

• Early layers stay stable and deterministic, good for basic lexical features.
• Deeper layers get stochastic regularisation, which reduces overfitting on high‑level patterns (e.g. brand–category quirks) and improves generalisation to unseen products.

Residual γ‑scaling for stability

Residual connections are scaled by small learnable parameters (gamma) that start near zero. At the beginning of training, each block behaves almost like an identity function. As training progresses, the model learns how much each block should actually change the representation:

• This makes training from scratch far more stable.
• It avoids the “too much change too early” problem you can get when stacking many residual layers.

Mean pooling + small MLP head

Get Ivan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A simple mean pooling over tokens and a small MLP head produce the final category logits:

• Mean pooling matches the intuition that “the whole title” matters; there’s no magic [CLS] token — every contextualised token contributes.
• The head is intentionally small, so most of the work is forced into the encoder. That leads to a clean embedding space where the six product categories are close to linearly separable.

Putting this together, the total parameter count is around 80 million, dominated by the token embedding matrix (thanks to the large Qwen3 vocabulary). That’s small enough for easy deployment and fast inference, but large enough to handle millions of product titles across multiple languages without feeling cramped. You can inspect the full implementation under: category_classification/embedding_classifier.py in github.com/ivan-digital/web-product-data

How the PyTorch encoder reads product titles

Let’s look at what happens to a title like: “Nike Air Zoom Pegasus 40 men’s running shoes”

1. Tokenisation
   The text is tokenised with the Qwen3-Embedding tokenizer. That means both the LoRA baseline and this new encoder see exactly the same input_ids and attention_mask.
2. Embedding
   Each token ID is mapped to a 384‑dimensional vector. These embeddings are scaled and passed through RMSNorm before entering the transformer blocks.
3. Bidirectional rotary attention
   In each block, bidirectional self‑attention lets every token attend to every other token. There is no causal mask, only a padding mask. Rotary position embeddings inject relative position information directly into the attention mechanism, so the model can tell “Air Max 90” from “Air Max 97” while staying robust to sequence length.
4. Grouped key/value heads keep things efficient: six query heads share two sets of keys and values, which reduces memory and compute while preserving expressiveness for these short titles.
5. SwiGLU feed‑forward layers
   After attention, each token goes through a SwiGLU feed‑forward network. SwiGLU is a gated variant of a standard transformer MLP and tends to be more expressive and stable, especially paired with RMSNorm.
6. Stability tricks
   Each block is pre‑norm (RMSNorm before attention and MLP), and both residual branches are scaled by small learnable parameters that start near zero. Combined with DropPath in deeper layers, this makes the encoder very stable even with aggressive learning rates and large batches.
7. Mean pooling and classification
   After 10 blocks, the model applies a masked mean over token embeddings (ignoring padding). There’s no special [CLS] token; the sentence embedding is literally the average of contextualised token vectors. A small 2‑layer MLP then maps this 384‑dimensional embedding to six logits, and cross‑entropy loss does the rest.

Because the classifier head is intentionally small, the encoder is forced to learn a representation where the six product categories are almost linearly separable. That makes the pooled embeddings useful not only for classification but also for retrieval and clustering.

Training pipeline: same LSPC data, different objective

To keep the comparison fair, the training pipeline mirrors the Qwen3 LoRA setup:

• The script loads the same Hugging Face DatasetDict built from LSPC.
• It uses the same text, label and language columns.
• It tokenises with the Qwen3-Embedding tokenizer (right‑padding, fixed max length).
• It builds PyTorch DataLoaders for train, validation and test splits.
• It computes the same metrics: loss, macro‑F1, micro‑F1, accuracy, plus per‑language breakdown.

The key differences are:

• The encoder is trained from scratch, not adapted with LoRA.
• All parameters are updated to optimise 6‑class cross‑entropy.
• Training uses several epochs over the 8.1M training titles.
• Optimisation is done with AdamW, gradient accumulation for large effective batch sizes, cosine learning‑rate decay with warmup, mixed precision (torch.cuda.amp) and gradient clipping.

In other words, it’s a modern, but straightforward, PyTorch training loop. The important part is that data, tokenisation, and evaluation are held constant. Only the model and objective differ.

The training script lives in: category_classification/train_embedding_classifier.py in github.com/ivan-digital/web-product-data

Results on LSPC: accuracy and multilingual performance

On the LSPC V2020 test split, the small PyTorch encoder achieves:

• Macro‑F1: 0.9315
• Accuracy (micro‑F1): 0.9487

The previous Qwen3-Embedding‑0.6B + LoRA baseline scored:

• Macro‑F1: 0.8360
• Accuracy: 0.8791

So the task‑specific encoder improves macro‑F1 by about 9.5 points and accuracy by about 7 points, on exactly the same dataset and splits.

Because the dataset is multilingual, it’s useful to look at per‑language macro‑F1 for languages with enough test samples:

• English: ~0.946
• German: ~0.753
• French: ~0.809
• Spanish: ~0.783
• Japanese: ~0.830

The pattern here is intuitive: English dominates the corpus, so it performs best. Under‑represented languages lag behind, but their scores are still solid given the data imbalance.

The interesting point is that the performance gap is no longer about the architecture. The encoder is clearly capable enough. The remaining headroom is in rebalancing the LSPC dataset or adding more multilingual product data, not in making the model deeper or wider.

Latency and throughput: batch‑1 vs batch‑256

For production use, latency and throughput matter just as much as accuracy.

The Qwen3 + LoRA model from the earlier article achieved roughly 3.3–3.9 ms per title across batch sizes 16–128, with about 299 titles per second at batch size 32 on a single GPU.

The small PyTorch encoder has a slightly different latency profile:

• With batch size 1 (one title at a time), the average latency is about 7.4 ms per example, with a 95th percentile around 7.9 ms and a 99th percentile around 8.6 ms. That keeps single‑request inference comfortably under 10 ms per call.
• With batch size 256, the mean per‑example latency drops to roughly 0.50 ms, and the 95th percentile is about 0.51 ms. That corresponds to ≈2,000 titles per second on one GPU when you can batch tightly.

Why the PyTorch encoder outperforms Qwen3 LoRA

Why does a roughly 80M‑parameter encoder beat a 615M‑parameter Qwen3 Embeddings model adapted with LoRA?

A few reasons:

1. Fully discriminative, end‑to‑end training
   Qwen3 Embeddings must stay good at many tasks — retrieval, similarity, multilingual transfer. LoRA fine‑tuning makes low‑rank updates on top of that. The small encoder has no such constraints: every weight is optimised purely for product category classification on LSPC. With millions of labelled examples, that single‑task focus pays off.
2. Right‑sized capacity
   For a 6‑class classification task with ~8M examples, 615M parameters are overkill. You’re paying for general language understanding you don’t exploit. The 80M‑parameter encoder is in a sweet spot: big enough to model brands, models, and multilingual patterns, small enough to train quickly and generalise well.
3. Modern stabilisation tricks
   RMSNorm, SwiGLU, DropPath and residual gamma scaling are tricks borrowed from large LLMs. They also help at this smaller scale, making optimisation smoother and allowing for larger batches and more aggressive learning rates without instability.
4. Simple head, strong representation
   Because the classifier head is small, almost all the work happens inside the encoder. That pushes the model to learn a clean embedding space where categories are nearly linearly separable, which helps both classification and downstream retrieval.

Takeaways for product category classification and SEO‑friendly ML content

From a machine learning perspective, the main lesson is simple:

For focused tasks like product category classification with plenty of labels, a small, well‑designed, task‑specific encoder can outperform a much larger, generic embedding model adapted with LoRA — while being faster and easier to serve.

From a content perspective, this experiment shows that sharing concrete metrics, open‑source code, and real‑world latency numbers helps other teams evaluate whether your solution fits their stack. If you’re building ML systems for e‑commerce search, recommendation or catalog management, this kind of transparent, reproducible benchmark is far more useful than yet another abstract “we fine‑tuned an LLM” post.