Activations Functions->ANNADI ARORA
Introduction
I have chosen Activation functions ReLU , SIGMOID and Tanh. Here I will specifically talk about SIGMOID. They are mathematical tools that help a neuron in attaining a value. What’s fascinating is how a word is interpreted by a machine/ transformer it all depends on the activation function, weights and biases. The activation functions play a major role here as linear transformation using weights and biases gives us an equivalent weight and bias. So activations functions break this linearity.
Equation
Y= W3(W2(W1* x+b1)+b2) +b3
Y =W’ x + b’
SIGMOID
σ(x) = 1 / (1 + e^(-x))
It takes every real x and gives an output between (0,1).
Key Points
->Output range: (0, 1)
->Shape: S-shaped curve (hence “sigmoid”)
Get ANNADI’s stories in your inbox
Join Medium for free to get updates from this writer.
->Centred: The inflection point is at (0, 0.5)
->Derivative: σ’(x) = σ(x) × (1 — σ(x))
Uses of the Sigmoid function
1111 Since it gives output between (0,1) It helps in predicting probability.
Eg you tell ChatGPT “ Write a blog on Sigmoid” Now chatGPT searches the WEB / earlier memory/ Training references to convert your prompt into “Write a blog on the mathematical function Sigmoid etc” So it finds well defined, structured sentences that mean similar to your prompt. So the probability here is SIGMOID. Although ReLU is used now and other mathematical tools that directly give 0 to x <0 or 1 to x >0 etc. So using these functions the Machine Learns better. In the earlier models SIGMOID was prevalent( Visual Text detection, image detector etc)
2222 Sigmoid functions control information flow in recurrent networks, deciding what to forget and what to remember.
INTERESTING
1111 The Relation between Tanh and SIGMOID is tanh(x) = 2σ(2x) — 1.
2222 The Tanh is 0 for x =0 always. Hence it is 0 centred.
3333 The tanh takes values from (-1,1) So a neuron can be assignes -1.
Interesting question is what does it reflect and how these negatives are converted to 0 for ease are all part of LLM/Model training.
4444 Tanh has steeper gradients near 0 in comparison with SIGMOID.
A bit of my Learning Journey
-> I used to watch Numberphile, Harvard STATs and Probability , Some Brainstellar puzzles, 3B1B for visuals and Linear Algebra. For me the hook in AI and ML would be Maths, as it still fascinates me that number and some functions are behind me searching how to write a blog, how to use notion AI, how to use medium etc.
-> Watching 3B1B made me realise AI and ML is much more than ChatGPT. It can be diversifies to sustainability, Health , production and obviously analysis than to generate some image. What I wonder is the wastage of resources a wrong prompt does, Electricity wasted, water wasted, GPU facility waste etc.
-> Also a credit to vectors how the concept of parallels, anti parallels is used. Also I understood how a dimension in a space represents a quality of something. Like heart is related to organ and LOVE.
-> Activation Functions They give neural networks their power to learn complex things(adapting) and non linear relationships. Also ‘0’ Gradient means neuron learns nothing but permanently:-( .
-> In photos I have attached an extra article:-)
(Never mess with prof !!!)
