PinnedInTowards AIbyJAIGANESAN·Oct 15Expert Transplant: I Replaced Gemma 3 1B’s FFNs With Another Model’s MoE Experts on a MacBook AirFrom the dense feed-forward layer, to a frozen Mixture of Experts taken from another model, to the two projection matrices that map hidden…A response icon4A response icon4
PinnedInTowards AIbyJAIGANESAN·Oct 4Muon: The Orthogonalized Optimizer Explained Through Equations, Intuition, Code and ResultsFrom SGD, to Adam’s per-coordinate scaling, to the five matrix multiplications that turn a gradient into an orthogonal updateA response icon2A response icon2
PinnedInTowards AIbyJAIGANESAN·Sep 22LatentMoE: NVIDIA’s Latent Mixture of Experts Explained Through Equations, Architecture, Code and…From the dense layer, to the mixture of experts, to the latent space the experts moved into, and the two byte counts that explain whyA response icon3A response icon3
PinnedInTowards AIbyJAIGANESAN·Sep 18DeepSeek’s mHC: Manifold-Constrained Hyper-Connections. Equations, Architecture, code, VisualsFrom the plain skip connection, to Hyper-Connections, to the doubly stochastic matrix that made them trainable at 27BA response icon3A response icon3
InTowards AIbyJAIGANESAN·Oct 21Reflex Drone: I Trained a System 1 (Decision Model) Pilot ‘Arc’ That Flies by Navigating ObstaclesMasked attention over a set of 20 tokens, a beam-search teacher, DAgger and KL-anchored PPO: a 179,847-weight pilot that decides in 0.14 ms
InTowards AIbyJAIGANESAN·Oct 6BERT: In-Depth Exploration of Architecture, Workflow, Code, and Mathematical Foundations [Updated]Embeddings, Masked Language Model Tasks, Attention Mechanisms, and Feed-Forward Networks Explained Through Equations, Matrix…
InTowards AIbyJAIGANESAN·Aug 13, 2024SAM 2 (Segment Anything Model 2) is Amazing, But We Need to understand SAM 1.Explained Segment Anything Model 1 with Architecture, and CodeA response icon1A response icon1
InAI AdvancesbyJAIGANESAN·Jul 31, 2024Understand RNN from Four Perspectives: Architecture, Math, Code, and Visual WorkflowA must read article to understand RNN
InTowards AIbyJAIGANESAN·Jul 22, 2024Multi-task Learning (MTL) and The Role of Activation Functions in Neural Networks [Train MLP With…Two concepts in Deep Learning- Simple and Important.A response icon1A response icon1
InTowards AIbyJAIGANESAN·Jul 17, 2024Quantization: Post Training Quantization, Quantization Error, and Quantization Aware TrainingEfficient Inference in AI Models