Laya Vision, Inside Out

How a small model reads an eye photo, picks an answer, and learns from 2,721 images. From zero, with moving diagrams and the real code.

I fine-tuned a small vision model to name eye diseases from retinal photos. Before training it was right 27.6% of the time, about the same as guessing. After 98 minutes of training on my laptop it was right 81.3% of the time, on 627 test images it had never seen.

This post explains how that model works, from the very first idea. I did not know what an encoder or a decoder was when I started. So the post starts from zero: numbers, tokens, vectors and attention. Then it shows how the model, Laya Vision, is built, with its real code. Then it shows how fine-tuning works, with the real numbers from my run.

Every diagram in this post is a recording of a live, animated diagram. The numbers are measured unless a caption says “illustration”.

Part A. The basics, from zero

A1. A model only sees numbers

A neural network is a very long calculation. It takes numbers in and gives numbers out. So everything must become numbers first: images, words, questions and answers.

A colour image is a grid of pixels. Each pixel has three numbers: red, green and blue. Each number is from 0 to 255.

A magnifier reads 4 × 4 pixels of a drawn retina. Each pixel is three numbers (R, G, B). A 512 × 512 image has 786,432 numbers.

A2. Text becomes tokens, then vectors

Text is cut into small pieces called tokens. A token can be a word, a part of a word, or a sign. Each token has an id number from a fixed list, the vocabulary. Laya Vision’s vocabulary has 49,280 tokens.

Then each id becomes a vector: a list of numbers that the model learned. In Laya Vision each vector has 576 numbers. The model looks up the vector in a table with 49,280 rows and 576 columns. That table alone holds 28,385,280 numbers.

A sentence becomes token pieces, then id numbers, then vectors. The split and the numbers are an illustration; the real tokenizer splits text in its own way.

# What the lookup is, in PyTorch (sizes from the real checkpoint)
import torch
embed_tokens = torch.nn.Embedding(49280, 576)   # one learned row of 576 numbers per token
ids = torch.tensor([[1214, 309, 2245]])          # token ids for a short text
vectors = embed_tokens(ids)                      # shape [1, 3, 576]: one vector per token

Why vectors? Similar meanings get similar vectors. The model can then compare tokens by comparing their numbers.

A3. An image becomes tokens too

Laya Vision treats an image like text. It cuts the image into small squares called patches. Each patch becomes a vector. Then the vectors join a sentence like words do.

Real sizes: a 512-pixel image, 16-pixel patches, 32 × 32 = 1,024 patches of 768 numbers, then 4 × 4 groups give 64 image tokens of 576 numbers.

1. The image is resized to 512 × 512 pixels.
2. The vision tower cuts it into 16 × 16-pixel patches. That gives 32 × 32 = 1,024 patches. Each patch becomes a vector of 768 numbers.
3. Pixel shuffle joins each 4 × 4 group of neighbour patches into one vector. 1,024 ÷ 16 = 64 vectors, each with 768 × 16 = 12,288 numbers.
4. The connector (one learned matrix of 576 × 12,288 numbers) changes each one to 576 numbers. Now the image is 64 tokens with the same size as word tokens.

# Shapes from the real checkpoint (model.safetensors)
encoder.vision_model.embeddings.patch_embedding.weight     [768, 3, 16, 16]   # patch to 768 numbers
encoder.vision_model.embeddings.position_embedding.weight  [1024, 768]        # where each patch is
encoder.connector.modality_projection.proj.weight           [576, 12288]       # 4x4 patches to 576

A4. Attention: every token looks at the others

One token alone has little meaning. “Pale” means more next to “optic disc”. Attention lets each token collect information from other tokens.

Each token makes three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I give?). The model compares one token’s query with every key. A high match gives a high weight. Softmax changes the matches into weights that add up to 1. Then the token takes a weighted mix of the values.

Each word in turn looks at the others. Thicker lines mean more attention; the percentages add up to 100%. The weights are an illustration.

attention(Q, K, V) = softmax( Q · Kᵀ / √d ) · V

import torch, math
def attention(q, k, v):                     # q, k, v: [tokens, d]
    scores = q @ k.T / math.sqrt(q.size(-1)) # how well each query matches each key
    weights = torch.softmax(scores, dim=-1)  # each row now adds up to 1
    return weights @ v                       # weighted mix of the values

A model runs several attentions side by side, called heads. Each head can look for a different relation. Laya Vision’s language layers have 9 heads. To save memory, the 9 query heads share 3 key and value heads (this is called grouped-query attention).

A5. Encoder and decoder: who may look at whom

The difference between an encoder and a decoder is a rule about attention. The rule is a grid called the attention mask. A row is the token that looks. A column is the token it may look at.

• Encoder (for example BERT, or ModernBERT in text Laya): every token may look at every token, before and after it. This is called bidirectional. It is good for reading and understanding.
• Decoder (for example SmolLM2 or GPT): a token may look only at itself and the tokens before it. This is called causal. A model needs this shape to write text one token at a time, because the future is not written yet.
• Laya Vision uses a decoder, but it never writes. It adds one change: inside the list of options, every option may look at every other option. The code calls this option_attention="block".

Filled squares mean “may look”. The grid fills row by row for the encoder, then the decoder, then Laya Vision (the red box is the option block).

A6. Inside one transformer layer

A transformer layer is one round of thinking. It has two parts. Attention mixes information between tokens. Then a small network, the MLP, changes each token’s vector alone. Each part adds its result to the vector it got (the residual connection), so nothing is lost. A norm keeps the numbers at a stable size before each part.

One Llama-style layer as used by Laya Vision’s language model: width 576, 9 attention heads, MLP 576 → 1,536 → 576. The dashed paths are the residual connections.

A model stacks many layers. Each layer refines every token’s vector a little more. Laya Vision has 12 layers in its vision tower, 20 layers in its language model, and 2 more layers in its decision head.

Part B. How Laya Vision is built

Laya is an open-source decision engine. It does not write text. It reads an input and gives a probability to each option you allow. Laya Vision is a community fork that lets Laya see images. It keeps Laya’s question types, decision head, training objective and calibration. It replaces Laya’s text encoder with SmolVLM-256M, a small vision-language model from Hugging Face, and cuts its language model from 30 to 20 layers. The result has 201,161,350 numbers (parameters).

B1. The whole path, one stage at a time

The eight stages from image to answer. The code under the diagram is the real code for each stage, from laya/vlm.py.

B2. The row that the model reads

For each question, Laya Vision builds one row of tokens. The function build_vlm_inputs() in laya/vlm.py uses these fixed pieces of text:

PREFIX_TEXT   = "<|im_start|>User:"
QUESTION_TEXT = "\n%s question: %s<end_of_utterance>\nAssistant: Options:\n"
OPTION_BULLET = "- "
OPTION_END    = "\n"     # the model reads each option's answer at this token

The real row for my eye question: the start marker, 64 image tokens, the question, and four options. ▼ marks each option’s reading point, the “\n” token that ends the option line.

# laya/vlm.py, build_vlm_inputs() (shortened)
tail = list(head_ids)                  # "\nchoice question: ...\nAssistant: Options:\n"
markers = []
for o in opt_ids:                      # each option: "- cataract: ..." (at most 48 tokens)
    tail.extend(o)
    tail.append(end_id)                # the "\n" that ends the option line
    markers.append(len(tail) - 1)      # remember where it is
ids = prefix["ids"] + state_tokens + tail   # image run + your text + question + options

Why read at the “\n”? In a decoder, a token sees only what comes before it. The “\n” at the end of an option has seen the image, the text, the question and that whole option. So its vector is the best summary of “this option, for this image”. With block attention it has also seen the other options, so it can compare them.

# laya/vlm.py, option_block_mask(): causal everywhere, two-way inside the option span
allowed = query_pos[:, None] >= key_pos[None, :]           # causal: look back only
in_q = (query_pos >= s) & (query_pos < e)                  # query inside the options
in_k = (key_pos >= s) & (key_pos < e)                      # key inside the options
allowed = allowed | (in_q[:, None] & in_k[None, :])        # options see each other

B3. The decision head: from vectors to an answer

The language model gives one vector of 576 numbers per token. The decision head turns the option vectors into an answer. This part is Laya’s own design.

1. Add a learned type embedding (one vector each for choice, score and noul), so the head knows the question type.
2. Run 2 more transformer layers over the whole row.
3. Take only the vectors at the option markers. With 4 options, that is 4 vectors.
4. The scorer (LayerNorm → Linear → GELU → Linear) changes each vector into one number, the logit.
5. Divide by the temperature and apply softmax. Now there is one probability per option.

Option vectors become logits, then probabilities. Then the temperature moves from 1.62 up to 3.0 (softer), down to 0.4 (sharper), and back to 1.618, the real temperature of my fine-tuned model. The logits are an illustration.

# laya/vlm.py, VLMDecisionModel._readout() (shortened)
h = h + self.type_emb(qtype)[:, None, :]          # 1. which question type
for layer in self.head.layers:                     # 2. two transformer layers
    h = layer(h, src_key_padding_mask=pad)
m = torch.gather(h, 1, marker_idx)                 # 3. vectors at the option markers
logits = self.scorer(m).squeeze(-1)                # 4. one number per option
# 5. at answer time: p = softmax(logits / temperature)

B4. Where the 201 million numbers are

Real counts from the fine-tuned checkpoint. Striped parts are the ones my fine-tuning changed.

B5. How to call it

# One call answers every question about the image
import laya
from PIL import Image

agent = laya.load_vlm("models/laya-vision-fundus/final")
state = {"image": Image.open("eye.jpg")}
questions = {"diagnosis": {"type": "choice",
             "instructions": "This is a retinal fundus photograph. Which eye disease does it show?",
             "criteria": {"cataract": "...", "diabetic_retinopathy": "...", "glaucoma": "...", "normal": "..."}}}
result = agent.predict(state, questions)
result["answers"]["diagnosis"]["choice"]            # e.g. "diabetic_retinopathy"

The image is encoded once and reused for every question. Each question then becomes its own row, and all rows go through the model together.

Part C. How fine-tuning works

C1. Learning means changing numbers to make a loss smaller

A loss is one number that says how wrong the model is on some examples. Training changes the model’s numbers a little, many times, so that the loss goes down.

Get Harsh chandrakar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The gradient says, for every number in the model, which direction makes the loss go up. So the model moves each number a small step the other way. The size of that step is the learning rate. Too small, and learning is slow. Too large, and the model jumps over the best point.

First with learning rate 0.35: the ball settles at the bottom. Then with 2.6: it jumps over the bottom and bounces. The curve is an illustration with one number to learn; a real model has millions.

C2. One training step

Forward, loss, backward, update. My run did this loop 3,062 times, with 8 questions in each batch.

# laya/vlm_train.py, train() (shortened)
logits, act = model(batch)                           # 1. forward: answer 8 questions
loss, reward = vlm_loss(logits, batch["target"], ...) # 2. loss: how wrong, how dishonest
opt.zero_grad(set_to_none=True)
loss.backward()                                      # 3. backward: a gradient for each trained number
torch.nn.utils.clip_grad_norm_(params, 1.0)          #    limit very large gradients
opt.step()                                           # 4. AdamW moves the numbers a small step

Backward (also called backpropagation) runs the calculation in reverse. It uses the chain rule from calculus to get the gradient for every trained number in one pass. AdamW is the rule that makes each step. It keeps a running average of past gradients, so the steps are smoother.

C3. The loss Laya Vision uses

The loss has two parts. Both compare the model’s probabilities with the target (here: 1.0 on the right disease, 0 on the others).

• Cross-entropy: −log(probability of the right answer). If the model gives the right answer 0.9, the loss is 0.11. If it gives 0.1, the loss is 2.3. This pulls the right answer up.
• Proper scoring reward (Laya’s RLCD idea): add small random noise to the logits four times, score each noisy guess with a strictly proper scoring rule, and move toward the guesses that scored better than their average. A proper rule gives the best score only to honest probabilities.

Four noisy copies of the model’s guess. Green copies scored above average and pull the model toward them; red copies push it away. Round by round, p(cataract) rises. The numbers are computed live with the same formula as the code.

# laya/vlm_train.py, vlm_loss() (shortened)
eps = torch.randn((4,) + logits.shape) * sigma          # 4 noisy copies, sigma = 0.3
z = logits.detach() + eps
q = torch.softmax(z, -1)
r = proper_reward(q, target)                           # log score + 0.75 x spherical score
adv = (r - r.mean(0)) / (r.std() + 1e-6)               # better or worse than average
logp = -((z - logits) ** 2).sum(-1) / (2 * sigma**2)
loss_rl = -(adv * logp).mean()                         # move toward the better copies
loss_ce = -(target * torch.log_softmax(logits, -1)).sum(-1).mean()
loss = loss_rl + loss_ce

C4. What I trained and what I froze

Training all 201 million numbers needs a lot of memory and data. I used freeze="last_n" with 4 layers. The vision tower and the first 16 language layers stay frozen (they do not change). Only the last 4 language layers, the final norm and the decision head learn. That is 22,623,619 numbers, 11% of the model.

The stack of layers. Highlighted blocks learn; grey blocks are frozen.

# laya/vlm.py, set_trainable(mode="last_n", n_last=4)
enc.requires_grad_(False)                          # freeze the whole backbone
for layer in layers[len(layers) - n_last:]:
    layer.requires_grad_(True)                     # ...then unfreeze the last 4 language layers
norm.requires_grad_(True)                          # ...and the final norm
# the decision head (type_emb, head, scorer, act_head) is never frozen here

Why not train everything? The early layers already know general shapes and colours. The last layers and the head are closest to the decision, so they gain the most from new examples. Fewer trained numbers also means less memory and less risk of forgetting.

C5. Splitting the data honestly

A model can memorise. So it must be tested on images it never saw. I kept related images together: the same patient’s two eyes, the same file-name group, and near-copies (also mirrored or rotated ones). Then I cut whole groups into four sets.

Each dot is about 10 images. The counts are real.

• train, 2,721 images: learning. Each image gives 3 questions, so 8,163 training questions.
• calib, 235 images: fitting the temperatures after training. Never trained on.
• eval, 634 images: checked every 300 steps to choose the best checkpoint.
• test, 627 images: scored once, at the end. The honest result.

C6. The learning-rate schedule

The learning rate is not fixed. It rises for the first 50 steps (warm-up), then falls along a cosine curve to 10% of its start. Large steps help early; small steps help the model settle at the end.

The real schedule of my run: 3,062 steps, head learning rate 1e-4, layer learning rate 2e-5. The ticks show the eval checks every 300 steps.

# laya/vlm_train.py, inside the loop
f = min(1.0, (step + 1) / warmup)                     # warm-up
f *= 0.1 + 0.45 * (1 + math.cos(math.pi * progress))  # cosine from 1.0 down to 0.1
group["lr"] = base_lr * f

C7. What happened in my run

Real eval accuracy for “which disease” (634 eval images), measured every 300 steps. The best checkpoint (step 2,700, 83.4%) was kept.

After training, I fitted the temperatures on the 235 calibration images: 1.618 for choice and 1.559 for yes/no questions. The calibration error on the eval set fell from 0.157 to 0.061. Then I scored the 627 test images once:

Test set: 627 images never used for training, checkpoint choice or calibration. Left: accuracy per disease, before and after. Right: the fine-tuned model’s confusion grid (rows are the true disease).

• Which disease: 27.6% → 81.3% (±3.0%).
• Is any disease present: 44.2% → 90.3%.
• Hardest class: glaucoma (64%). It is confused with cataract and normal.

# finetune_fundus.py (shortened): the whole recipe
agent = laya.load_vlm("thaitea/laya-vision")
train(agent.model, agent.processor, train_ex, steps=3062, batch_size=8,
      freeze="last_n", n_last=4, lr_head=1e-4, lr_backbone=2e-5, warmup=50,
      eval_fn=on_eval, eval_every=300)                 # saves the best eval checkpoint
agent = laya.load_vlm("models/laya-vision-fundus/best")
agent.temperature = fit_temperatures_from(collect_logits(agent.model, agent.processor, calib_ex))
agent.save("models/laya-vision-fundus/final")

On an Apple M3 Pro laptop the training took 98 minutes.

Words used in this post

• Parameter: one learned number in the model. Laya Vision has 201,161,350.
• Token: a small piece of the input: part of a word, or a group of image patches.
• Vector: a list of numbers. Here, 576 numbers describe one token.
• Embedding: the learned vector for a token id, read from a table.
• Patch: a 16 × 16-pixel square of the image.
• Vision tower: the part that turns patches into vectors (SigLIP, 12 layers).
• Pixel shuffle: joining 4 × 4 neighbour patches into one longer vector (1,024 patches to 64 tokens).
• Connector: the matrix that changes image vectors to the language model’s size (12,288 to 576).
• Attention: the step where each token takes information from other tokens.
• Query, key, value: the three vectors each token makes for attention.
• Head: one of several attentions that run side by side.
• Attention mask: the grid that says which token may look at which.
• Encoder: a model whose tokens look in both directions (bidirectional).
• Decoder: a model whose tokens look only back (causal), as needed to write text.
• MLP: the small per-token network inside each transformer layer.
• Residual: adding a part’s output to its input, so earlier information stays.
• Norm: a step that keeps a vector’s numbers at a stable size.
• Logit: the raw score for one option, before softmax.
• Softmax: changes scores into probabilities that add up to 1.
• Temperature: a number that divides the logits to make probabilities softer or sharper.
• Loss: one number that says how wrong the model is.
• Gradient: for each parameter, the direction that increases the loss.
• Learning rate: how big each training step is.
• Backpropagation: the reverse pass that computes all gradients.
• Frozen: a part whose parameters do not change during training.
• Checkpoint: a saved copy of all parameters at one point in training.
• Calibration: making stated confidence match real accuracy.

Credits and notes

• Laya: github.com/NandhaKishorM/laya (Apache-2.0).
• Laya Vision: github.com/r33drichards/laya-vision, commit 9e1e241. Code Apache-2.0; the model weights are CC BY-NC-SA 4.0 (non-commercial). The code snippets in this post are shortened from its laya/vlm.py and laya/vlm_train.py.
• Data: Preprocessed eye diseases fundus images on Kaggle (4,217 images, 4 classes).
• This is a learning project. The model is not a medical device and must not be used for diagnosis.