Hobbyist Day 2: LLM is a specialized ANN + More
13 min readFeb 2, 2026
Press enter or click to view image in full size![]()
· 🟢 Vector Index — Storage & Retrieval
· 🟢 LLM — The GPU Language Model
∘ ANN based — Character by Character
∘ Transformer — Attention Is All You Need
· 🟢 Pre-trained LLM Model vs Tuning vs RAG
· 🟢 Embedding models
∘ How string to vector works?
∘ Embedding models and Language models are different
· 🟢 Conclusion
Well, I have never been a core AI guy. But in my last company, I created vector index. It gave me a glimpse of how AI indexes work. The part which I was not clear of was the HuggingFace library part, or the LLM.
🟢 Vector Index — Storage & Retrieval
Press enter or click to view image in full size![]()
🟢 LLM — The GPU Language Model
ANN based — Character by Character
LLM predicts the next character based on the previous character already printed. This is kind of a high level LLM based on ANN library synaptic.js
npm init -y
npm i synaptic
touch tiny_llm.js
node tiny_llm.js/*
============================================================
TINY LLM (Character-Level) USING SYNAPTIC.JS
============================================================
============================================================
STEP 1 — CORPUS (RAW TEXT)
============================================================
We need enough text + repetition to learn patterns.
Example corpus (toy):
"hello world. this is tiny."
Improved corpus (still small, but richer):
- repeats patterns so the model learns
- includes multiple sentences
Sample output (if we printed it):
corpus length: 350
corpus preview: "hello world. this is a tiny language model.\n..."
*/
const synaptic = require("synaptic");
const { Trainer, Architect } = synaptic;
const corpus =
`
hello world. this is a tiny language model.
hello again world. this model predicts the next character.
the goal is to learn patterns in text and generate new text.
a tiny model can repeat, but it can also remix phrases.
if you give it a seed, it continues the sentence.
hello world. the model learns by predicting the next character.
this is not a real llm. it is a small toy rnn with lstm memory.
try different seeds: hello, the goal, this model, a tiny.
`.trim() + "\n";
/*
============================================================
STEP 2 — VOCABULARY (UNIQUE CHARACTERS)
============================================================
Extract all unique characters from corpus, then sort them.
Example:
Corpus: "hello"
Unique chars = {h,e,l,o}
Sorted chars = [e,h,l,o]
Sample output:
vocab size: 27
vocab chars: "\n .,:abcdefghilmnorstuyw"
*/
const chars = [...new Set([...corpus])].sort();
const V = chars.length;
/*
============================================================
STEP 2.5 — CHAR <-> ID MAPS (YOU ASKED FOR THIS)
============================================================
We create two maps:
charToId: maps character -> integer id
idToChar: maps integer id -> character
This is like a dictionary encoding.
Example:
chars = [' ', '.', 'a', 'b']
charToId:
' ' -> 0
'.' -> 1
'a' -> 2
'b' -> 3
idToChar:
0 -> ' '
1 -> '.'
2 -> 'a'
3 -> 'b'
Sample output (first ~12 entries):
charToId (first 12):
0 -> "\n"
1 -> " "
2 -> "."
3 -> ","
4 -> ":"
5 -> "a"
6 -> "b"
7 -> "c"
8 -> "d"
9 -> "e"
10 -> "f"
11 -> "g"
(Exact chars + indices depend on your corpus.)
*/
const charToId = new Map(chars.map((c, i) => [c, i]));
const idToChar = new Map(chars.map((c, i) => [i, c]));
/*
============================================================
STEP 3 — ONE HOT ENCODING
============================================================
Neural nets don’t take "letters" directly.
We convert each char into a one-hot vector length V.
Sample output (for your corpus, if printed):
oneHot('h') length: 27
oneHot('h') = [0,0,0,0,0,0,0,0,0,1,0,0,...]
*/
function oneHot(ch) {
const v = new Array(V).fill(0);
v[charToId.get(ch)] = 1;
return v;
}
/*
============================================================
STEP 4 — SEQUENCE WINDOW (CONTEXT) + TRAINING SAMPLES
============================================================
We train on sequences, not just single chars.
We choose a window length SEQ.
For each position i in the corpus:
inSeq = text[i .. i+SEQ-1]
outSeq = text[i+1 .. i+SEQ]
So outSeq is inSeq shifted by 1 char forward.
Example:
text: "hello world"
SEQ = 6
inSeq = "hello "
outSeq = "ello w"
Sample output (first 3 samples):
sample[0]:
in : "hello "
out: "ello w"
sample[1]:
in : "ello w"
out: "llo wo"
sample[2]:
in : "llo wo"
out: "lo wor"
Code output
samples[0] = {
input: [
[0,0,1,0,0,...], // 'h'
[0,1,0,0,0,...], // 'e'
[0,0,0,1,0,...], // 'l'
...
32 vectors total
],
output: [
[0,1,0,0,0,...], // 'e'
[0,0,0,1,0,...], // 'l'
...
]
}
*/
const SEQ = 32;
function buildSamples(text) {
const samples = [];
for (let i = 0; i + SEQ < text.length; i++) {
const inSeq = text.slice(i, i + SEQ);
const outSeq = text.slice(i + 1, i + SEQ + 1);
samples.push({
input: [...inSeq].map(oneHot),
output: [...outSeq].map(oneHot),
});
}
return samples;
}
const samples = buildSamples(corpus);
/*
============================================================
STEP 5 — MODEL (LSTM NETWORK)
============================================================
Structure:
[Input size = V] -> [LSTM hidden size = H] -> [Output size = V]
Output is a probability distribution over next char.
Sample output (conceptual):
Given last char='o', net outputs probs:
' ' : 0.42
'r' : 0.14
'l' : 0.06
'.' : 0.05
...
We will sample from this distribution during generation.
*/
const H = 128;
const ann = new Architect.LSTM(V, H, V);
const trainer = new Trainer(ann);
/*
============================================================
STEP 6 — TRAINING
============================================================
Train multiple epochs so it actually learns.
Sample output (printed every 10 epochs):
Training...
epoch 10 error=1.92
epoch 20 error=1.41
epoch 30 error=1.12
epoch 40 error=0.98
epoch 50 error=0.91
epoch 60 error=0.86
epoch 70 error=0.83
epoch 80 error=0.81
(Your numbers will vary.)
*/
console.log("Training...");
for (let e = 1; e <= 80; e++) {
const res = trainer.train(samples, {
rate: 0.03,
iterations: 1,
shuffle: true,
cost: Trainer.cost.CROSS_ENTROPY,
});
if (e % 10 === 0) {
console.log(`epoch ${e} error=${res.error.toFixed(4)}`);
}
}
/*
============================================================
STEP 7 — SAMPLING (TEMPERATURE)
============================================================
Argmax (always picking max) can get stuck.
We sample with a temperature.
probs = [0.50, 0.30, 0.15, 0.05]
char0 = 50%
char1 = 30%
char2 = 15%
char3 = 5%
| Temperature | Effect |
| ----------- | --------------------------- |
| 0.3 | Very deterministic (boring) |
| 0.85 | Balanced |
| 1.5 | Chaotic/random |
What is achieved by temperature?
Instead of
"hello world world world world"
We get
"hello world. this is a tiny model"
*/
function sample(probs, temperature = 0.85) {
const eps = 1e-12;
const logits = probs.map((p) => Math.log(p + eps) / temperature);
const max = Math.max(...logits);
const exps = logits.map((x) => Math.exp(x - max));
const sum = exps.reduce((a, b) => a + b, 0);
const norm = exps.map((x) => x / sum);
let r = Math.random();
for (let i = 0; i < norm.length; i++) {
r -= norm[i];
if (r <= 0) return i;
}
return norm.length - 1;
}
/*
============================================================
STEP 8 — GENERATION LOOP
============================================================
Algorithm:
ann.clear()
warm-up on seed
repeat N times:
probs = ann.activate(oneHot(lastChar))
nextChar = sample(probs, temperature)
append nextChar
*/
function generate(seed, n = 300, temperature = 0.85) {
ann.clear();
for (const ch of seed) ann.activate(oneHot(ch));
let out = seed;
let last = seed[seed.length - 1];
for (let i = 0; i < n; i++) {
const probs = ann.activate(oneHot(last));
const nextId = sample(probs, temperature);
const nextCh = idToChar.get(nextId);
out += nextCh;
last = nextCh;
}
return out;
}
/*
============================================================
FINAL RUN
============================================================
We print:
- RAW OUTPUT
- VISIBLE SPACES (replace " " with "·")
- LENGTH
- PLUS: charToId/idToChar mapping preview (you asked for this)
Sample output (mapping preview):
vocab size: 27
charToId preview (first 12):
0: "\n"
1: " "
2: "."
...
Sample output (generation):
Seed: "hello"
RAW:
"hello world. this is a tiny language model.\nhello again world..."
*/
const seed = "hello";
const result = generate(seed, 420, 0.85);
console.log("\n--- RAW OUTPUT ---");
console.log(result);
console.log("\n--- VISIBLE SPACES ---");
console.log(result.replace(/ /g, "·"));
console.log("\n--- LENGTH ---");
console.log(result.length);Output
Training...
epoch 10 error=NaN
epoch 20 error=NaN
epoch 30 error=NaN
epoch 40 error=NaN
epoch 50 error=NaN
epoch 60 error=NaN
epoch 70 error=NaN
epoch 80 error=NaN
--- RAW OUTPUT ---
helloda
edbh
wx.pdv
bdytshhn,,. di l xsnt.ogtudiuaex v
def:ar,dacvoeralrlv:,emnmomtbill:ydmyxep.geoyeymuhv:
:lgllhvfrxhl:cbosgglpiof:y.vgtvovvehntvuirxitdbfrh
eyxwdx,.trn:gnwvyn: xxcot,t:bnnsmmucl
boa
d:opehadrsoty
xxdicb:vb,lt cevvb,:u,on
frnray:cvfy
wbaam:y:vtplsivmed,xnha:coxrfr bbwl
o:,ivhatiumresuivhsnnn:x.awerr.plc,tctme.mcw tumnwmgxrrbflitshvuib,dg
h
md
urehty
ah
wutgmlg::o.dyrgluwxvuosooc,wuiFor Language Model usecase, I think it is more trained on dictionary words than on characters.
Transformer — Attention Is All You Need
RNN: “I remember what came before.”
Transformer: “I look at everything and decide what’s important.”
https://arxiv.org/abs/1706.03762
Press enter or click to view image in full size![]()
Press enter or click to view image in full size![]()
A sample ChatGPT generated code.
Get 月亮’s stories in your inbox
Join Medium for free to get updates from this writer.
TODO: Will come back to understand this.
/*
============================================================
TOY SELF-ATTENTION (TRANSFORMER CORE) IN PURE JS
============================================================
GOAL:
Show how "Attention Is All You Need" self-attention works,
step-by-step, with real numbers printed.
We implement ONE self-attention head:
Given token embeddings X:
Q = XWq
K = XWk
V = XWv
Attention(Q,K,V):
scores = (Q K^T) / sqrt(dk)
weights = softmax(scores) // row-wise
output = weights V
We DO NOT TRAIN.
We use tiny, hand-picked embeddings and weights so the math is visible.
Run:
node toy_attention.js
Expected output includes:
- tokens
- embeddings
- Q, K, V
- scores (before softmax)
- attention weights (after softmax)
- output/context vectors
============================================================
*/
function dot(a, b) {
let s = 0;
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
return s;
}
function matMul(A, B) {
// A: (n x d), B: (d x m) -> (n x m)
const n = A.length;
const d = A[0].length;
const m = B[0].length;
const out = Array.from({ length: n }, () => new Array(m).fill(0));
for (let i = 0; i < n; i++) {
for (let j = 0; j < m; j++) {
let s = 0;
for (let k = 0; k < d; k++) s += A[i][k] * B[k][j];
out[i][j] = s;
}
}
return out;
}
function transpose(A) {
const n = A.length;
const m = A[0].length;
const out = Array.from({ length: m }, () => new Array(n).fill(0));
for (let i = 0; i < n; i++) for (let j = 0; j < m; j++) out[j][i] = A[i][j];
return out;
}
function softmaxRow(row) {
const max = Math.max(...row);
const exps = row.map((x) => Math.exp(x - max));
const sum = exps.reduce((a, b) => a + b, 0);
return exps.map((e) => e / sum);
}
function softmaxRows(M) {
return M.map(softmaxRow);
}
function scaleMatrix(M, s) {
return M.map((row) => row.map((x) => x * s));
}
function rowWiseMatMul(A, B) {
// A: (n x n) attention weights
// B: (n x dv) values
// out: (n x dv)
const n = A.length;
const dv = B[0].length;
const out = Array.from({ length: n }, () => new Array(dv).fill(0));
for (let i = 0; i < n; i++) {
for (let j = 0; j < dv; j++) {
let s = 0;
for (let k = 0; k < n; k++) s += A[i][k] * B[k][j];
out[i][j] = s;
}
}
return out;
}
function fmtMatrix(name, M, tokens) {
console.log(`\n--- ${name} ---`);
if (tokens && M.length === tokens.length) {
for (let i = 0; i < M.length; i++) {
console.log(tokens[i].padEnd(10), M[i].map((x) => x.toFixed(3)).join(" "));
}
} else {
for (const row of M) console.log(row.map((x) => x.toFixed(3)).join(" "));
}
}
function fmtSquareMatrix(name, M, tokens) {
console.log(`\n--- ${name} ---`);
const header = ["".padEnd(10), ...tokens.map((t) => t.padEnd(10))].join("");
console.log(header);
for (let i = 0; i < M.length; i++) {
const row = [tokens[i].padEnd(10)];
for (let j = 0; j < M[i].length; j++) row.push(M[i][j].toFixed(3).padEnd(10));
console.log(row.join(""));
}
}
/*
============================================================
STEP 1 — TOKENS
============================================================
We pick a small sentence and split into tokens.
*/
const text = "the cat sat";
const tokens = text.split(/\s+/);
console.log("\n[1] TOKENS:", tokens);
/*
============================================================
STEP 2 — EMBEDDINGS X (hand-made)
============================================================
Normally embeddings are learned.
Here we pick tiny vectors just to illustrate attention.
X shape: (n_tokens x d_model)
*/
const dModel = 4;
// Very small, simple embeddings for demo.
const embeddingTable = {
the: [1.0, 0.0, 0.0, 0.0],
cat: [0.0, 1.0, 0.0, 0.0],
sat: [0.0, 0.0, 1.0, 0.0],
};
const X = tokens.map((t) => embeddingTable[t]);
fmtMatrix("[2] X (embeddings)", X, tokens);
/*
============================================================
STEP 3 — WEIGHTS Wq, Wk, Wv (hand-made)
============================================================
In a real transformer these are learned matrices.
Wq, Wk, Wv shapes: (d_model x d_k)
We set d_k = 3 for visibility.
*/
const dK = 3;
// Chosen to make attention values non-trivial but easy-ish.
const Wq = [
[0.9, 0.1, 0.0],
[0.2, 0.8, 0.0],
[0.1, 0.2, 0.7],
[0.0, 0.0, 0.0],
];
const Wk = [
[0.7, 0.0, 0.2],
[0.1, 0.9, 0.0],
[0.0, 0.3, 0.8],
[0.0, 0.0, 0.0],
];
const Wv = [
[1.0, 0.0, 0.0],
[0.0, 1.0, 0.0],
[0.2, 0.2, 0.6],
[0.0, 0.0, 0.0],
];
/*
============================================================
STEP 4 — Q, K, V
============================================================
Q = XWq, K = XWk, V = XWv
*/
const Q = matMul(X, Wq);
const K = matMul(X, Wk);
const Vv = matMul(X, Wv);
fmtMatrix("[4] Q", Q, tokens);
fmtMatrix("[4] K", K, tokens);
fmtMatrix("[4] V", Vv, tokens);
/*
============================================================
STEP 5 — ATTENTION SCORES
============================================================
scores = (Q K^T) / sqrt(dK)
scores shape: (n_tokens x n_tokens)
Interpretation:
scores[i][j] = how much token i "attends to" token j (before softmax)
*/
const KT = transpose(K);
// rawScores = Q * K^T
let rawScores = matMul(Q, KT);
// scale by 1/sqrt(dK)
const scale = 1 / Math.sqrt(dK);
const scores = scaleMatrix(rawScores, scale);
fmtSquareMatrix("[5] scores = QK^T / sqrt(dK)", scores, tokens);
/*
============================================================
STEP 6 — SOFTMAX => ATTENTION WEIGHTS
============================================================
weights = softmax(scores) row-wise
Each row sums to 1:
weights[i] is a distribution over tokens being attended-to by token i.
*/
const weights = softmaxRows(scores);
fmtSquareMatrix("[6] weights = softmax(scores)", weights, tokens);
/*
============================================================
STEP 7 — OUTPUT / CONTEXT VECTORS
============================================================
output = weights * V
output shape: (n_tokens x dK)
Meaning:
each token becomes a weighted mixture of all tokens' Value vectors.
*/
const out = rowWiseMatMul(weights, Vv);
fmtMatrix("[7] output/context = weights * V", out, tokens);
/*
============================================================
FINAL INTUITION
============================================================
If token "cat" attends strongly to "cat" (itself),
its output will resemble the "cat" value vector.
If token "sat" attends strongly to "cat",
its output will carry information from "cat".
This is the fundamental mechanism that replaced RNNs/LSTMs.
============================================================
*/When people say large language model, it is the whole thing
Language Model
├─ Tokenizer
├─ Embeddings
├─ Transformer Layers
│ ├─ Attention Heads
│ │ ├─ Wq
│ │ ├─ Wk
│ │ └─ Wv
│ └─ Feed-Forward
├─ Output Layer
└─ Sampling
The dataset is only used during training.
afterward, the weights are what remain.🟢 Pre-trained LLM Model vs Tuning vs RAG
LLM Model can be download from here https://huggingface.co/models
┌─────────────────────────────┐
│ RAW INTERNET DATA │
│ Books, Code, Articles, etc │
└──────────────┬──────────────┘
│
▼
(PRETRAINING PHASE)
Massive GPUs / Months / $$$$$$
│
▼
┌───────────────────────────────┐
│ BASE LANGUAGE MODEL │
│ (Qwen / Llama / GPT etc.) │
│ │
│ • Knowledge in WEIGHTS │
│ • Can generate text alone │
└──────────────┬────────────────┘
│
┌──────────────────────┼──────────────────────┐
│ │ │
▼ ▼ ▼
(Fine-Tuning) (RAG / Vector DB) (Inference Only)
Weight Changes No Weight Changes Just PromptLlama.cpp is an inference only library
Hugging Face supports inference and training.
For Fine Tuning
┌────────────────────────────┐
│ BASE MODEL (Qwen/Llama) │
└──────────────┬─────────────┘
│
▼
Small Dataset (Domain)
Medical / Legal / Coding / etc
│
▼
Fine-Tuning Methods:
• LoRA
• PEFT
• SFT
│
▼
┌────────────────────────────────┐
│ NEW SPECIALIZED MODEL │
│ Weights Slightly Changed │
└──────────────┬─────────────────┘
│
▼
Export / Convert / RunTo use fine tuned model on inference only engine, do this
┌─────────────────────┐
│ TRAINING WORLD │
│ (PyTorch, HF, etc) │
└─────────┬───────────┘
│
▼
Fine-Tuned Model Weights
│
▼
Convert → GGUF Format
│
▼
┌─────────────────────┐
│ llama.cpp │
│ FAST INFERENCE │
│ CPU / Low Memory │
└─────────────────────┘Full fine-tuning vs LORA / PEFT
FULL FINE-TUNE
Update: 7B params
Cost: Huge
Rare
LORA / PEFT
Update: ~20M params
Cost: Small
CommonDuring export you can do either full model export or delta export
Fine-Tuned Model
│
┌────────┴────────┐
│ │
▼ ▼
Merge LoRA Keep LoRA Separate
Single File Base + Adapter
Faster FlexibleTo summarize
Base LLM = Brain
Fine-Tune = Brain Rewiring
RAG = Notebook
llama.cpp = Efficient Mouth That Talks Fast🟢 Embedding models
How string to vector works?
┌─────────────────────────────────────────────────────────────────────────────┐
│ STRING → VECTOR EMBEDDING │
└─────────────────────────────────────────────────────────────────────────────┘
INPUT STRING
────────────
"Database transactions are hard and require concurrency control"
(1) TOKENIZATION
─────────────────
Model does NOT see characters.
It sees TOKENS.
Example Tokens:
["Database", " transactions", " are", " hard",
" and", " require", " concurrency", " control"]
Token IDs:
[1452, 9983, 21, 777, 88, 6612, 9123, 441]
NOTE:
1 token ≈ 3–4 characters (English average)
(2) TOKEN LIMIT CHECK
─────────────────────
Each embedding model has MAX TOKENS.
Example Limits:
┌──────────────────────────────┐
│ Small Embedding : 512 tokens│
│ Medium Embedding : 2048 │
│ Large Embedding : 8k+ │
└──────────────────────────────┘
If exceeded:
LONG TEXT
│
▼
┌───────────────────┐
│ CHUNKING │
│ 500–1000 tokens │
└───────────────────┘
(3) TOKEN → INITIAL VECTOR LOOKUP
─────────────────────────────────
Embedding Matrix (VocabSize × HiddenDim)
Example:
Token 1452 → [0.12, -0.33, 0.91, ...]
Token 9983 → [0.08, 0.77, -0.44, ...]
Token 21 → [0.51, -0.11, 0.29, ...]
Each token becomes a vector
Dimension typically = 384 / 768 / 1024 / 1536
(4) CONTEXTUAL TRANSFORMATION
─────────────────────────────
Transformer Layers:
• Self Attention
• Feed Forward
• Residual Connections
• Layer Normalization
Meaning becomes CONTEXTUAL:
"bank (river)" ≠ "bank (finance)"
Computational Cost:
Attention = O(n²)
More tokens → exponential compute growth
(5) POOLING → SINGLE VECTOR
───────────────────────────
We must compress MANY token vectors
into ONE sentence/document vector.
Pooling Strategies:
┌──────────────────────────────┐
│ CLS Token │
│ Mean Pooling (Average) │
│ Max Pooling │
│ Attention Weighted Pooling │
└──────────────────────────────┘
(6) FINAL EMBEDDING VECTOR
──────────────────────────
Example Output:
[0.021, -0.771, 0.338, 0.902, ...]
Length = Fixed (e.g., 1536 dims)
IMPORTANT:
Different length inputs
→ SAME output vector size.
(7) VECTOR SPACE SEMANTICS
───────────────────────────
(High-Dimensional Space)
"buy stock"
*
*
*
*
"purchase shares"
FAR AWAY
*
*
*
*
"quantum physics"
Similar meaning → close vectors
Different meaning → far vectors
(8) PRACTICAL VECTOR DB PIPELINE
───────────────────────────────
LONG DOCUMENT
│
▼
SPLIT INTO CHUNKS
│
▼
EMBED EACH CHUNK
│
▼
STORE VECTORS IN DB
│
▼
COSINE / DOT PRODUCT SEARCH
┌─────────────────────────────────────────────────────────────────────────────┐
│ FINAL TAKEAWAY │
├─────────────────────────────────────────────────────────────────────────────┤
│ String → Tokens → Token Vectors → Transformer Context → Pooling → Vector │
│ │
│ Limits exist due to TOKEN COUNT + O(n²) attention cost. │
│ Embedding = Semantic compression of meaning into fixed-length coordinates. │
└─────────────────────────────────────────────────────────────────────────────┘Embedding models and Language models are different
══════════════════════════════════════════════════════════════════════════════
SUMMARY
══════════════════════════════════════════════════════════════════════════════
Language Model
• Goal: Generate Text
• Embeddings: Incidental
• Can Produce Vectors: YES
• Quality for Search: MEDIUM
Embedding Model
• Goal: Semantic Similarity
• Embeddings: Primary Output
• Can Generate Text: NO
• Quality for Search: HIGH
FINAL IDEA:
Language Model = Swiss Army Knife
Embedding Model = Precision Tool🟢 Conclusion
Trying to connect LLM (new) with ANN (old) and make more sense.
