t-SNE

Manifold learning is a type of nonlinear dimensionality reduction that assumes high-dimensional data actually lies on a lower-dimensional, curved surface (a manifold) embedded in that space, aiming to uncover this intrinsic structure for visualization, noise reduction, and improved analysis by preserving local relationships. One of the popular methods is

t-SNE

t — In t-SNE, the t-distribution is used in the low-dimensional embedding (visualization) space to model similarities, giving heavy tails that allow for better separation of dissimilar clusters, unlike the Gaussian used for high-dimensional similarities, thereby solving the “crowding problem” (we will discuss it later) and making far-apart points stay far apart.

Embedding — An embedding is a way to represent complex data, like words, images, or users, as numerical vectors (lists of numbers) in a lower-dimensional space, capturing their meaning and relationships so machines can process them effectively. Similar items have similar vectors and are placed closer together in this space, allowing AI models to understand context, find similar items, and perform tasks like semantic search or recommendations.

Intro

t-SNE (t-Distributed Stochastic Neighbor Embedding) is often described as a visualization algorithm, but this description hides its real nature. t-SNE is a probabilistic model of neighborhood similarity that happens to output a two-dimensional map. What it preserves is not distance, not angles, and not scale, but the likelihood that two points are neighbors.

Let’s assume we have this dataset:

The objective of t-SNE is to construct a low-dimensional representation:

High-Dimensional Similarities:

For a fixed data point 𝑥_𝑖, t-SNE defines a conditional probability distribution over all other points. This distribution represents the probability that point 𝑥 𝑗​ would be selected as a neighbor of 𝑥 𝑖​ , assuming neighbors are sampled according to a Gaussian centered at 𝑥 𝑖

Conditional Probability (Similarity Score)

Let’s assume we have a dataset shown in the plot below:

import numpy as np
import matplotlib.pyplot as plt

# Data
X = np.array([
    [1, 2, 5],   # P1
    [2, 3, 5],   # P2
    [1, 2, 6],   # P3
    [4, 1, 8],   # P4
    [1, 3, 5],   # P5
    [1, 4, 7]    # P6
])

labels = [f"P{i+1}" for i in range(len(X))]
colors = plt.cm.tab10(np.arange(len(X)))  # distinct colors

# Figure
fig = plt.figure(figsize=(7, 6))
ax = fig.add_subplot(111, projection="3d")

# Scatter
ax.scatter(
    X[:, 0], X[:, 1], X[:, 2],
    c=colors,
    s=120,
    edgecolor="black",
    linewidth=0.8,
    alpha=0.9
)

# Labels on points
for i, (x, y, z) in enumerate(X):
    ax.text(x, y, z + 0.1, labels[i], fontsize=10, weight="bold")

# Axis labels
ax.set_xlabel("X₁", fontsize=12, labelpad=10)
ax.set_ylabel("X₂", fontsize=12, labelpad=10)
ax.set_zlabel("X₃", fontsize=12, labelpad=10)

# Remove grid and panes
ax.grid(False)
ax.xaxis.pane.set_visible(False)
ax.yaxis.pane.set_visible(False)
ax.zaxis.pane.set_visible(False)

# Subtle axis lines
ax.xaxis.line.set_color("black")
ax.yaxis.line.set_color("black")
ax.zaxis.line.set_color("black")

ax.set_title("3D Geometry of Data Points", fontsize=14, pad=15)

plt.tight_layout()
plt.show()

Plot tells us some insights (might be misleading), for example:

Point 1 is much closer to the Point 3 and 5 than it is to Points 2 and 4. Additionally, Point 6 seems the furthest one. To see this analogy on a probabilistic distribution graph from Point 1.

It seems we were mostly correct but wrong about the furthest distance, which is Point 4, NOT 6. Numerical calculation proves it.

Distance P1–P4: 4.358898943540674
p(P4 | P1): 4.680195539613231e-05
Distance P1–P6: 2.8284271247461903
p(P6 | P1): 0.01145206089962347

The plot below provides the conditional probability from the perspective of each point:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

# Your 6-value, 3D dataset
data = np.array([
    [1, 2, 5], [2, 3, 5], [1, 2, 6],
    [4, 1, 8], [1,3,5], [1, 4, 7]
])


def plot_all_conditional_distributions(X, sigma=1.5):
    n = len(X)
    fig, axes = plt.subplots(n, 1, figsize=(10, 15), sharex=True)

    # Precompute all pairwise distances
    sum_X = np.sum(np.square(X), axis=1)
    dist_matrix = np.sqrt(np.abs(np.add(np.add(-2 * np.dot(X, X.T), sum_X).T, sum_X)))

    x_curve = np.linspace(0, 10, 500)
    y_curve = norm.pdf(x_curve, 0, sigma)

    for i in range(n):
        ax = axes[i]
        ax.plot(x_curve, y_curve, 'k-', alpha=0.5)
        ax.fill_between(x_curve, y_curve, color='gray', alpha=0.1)

        # Calculate raw densities for normalization (t-SNE p_j|i)
        densities = norm.pdf(dist_matrix[i], 0, sigma)
        densities[i] = 0  # p_i|i is 0 in t-SNE
        row_sum = np.sum(densities)

        # Plot each point j as seen by point i
        for j in range(n):
            if i == j: continue

            dist = dist_matrix[i, j]
            p_density = densities[j]
            # Normalizing to get the actual conditional probability p_j|i
            p_cond = p_density / row_sum if row_sum > 0 else 0

            color = plt.cm.tab10(j)
            ax.scatter(dist, p_density, color=color, s=80, label=f"to P{j+1}: {p_cond:.2f}")
            ax.vlines(dist, 0, p_density, colors=color, linestyles='--', alpha=0.6)

        ax.set_ylabel(f"P{i+1} as Center")
        ax.legend(loc='upper right', fontsize='small', ncol=2)
        if i == n-1: ax.set_xlabel("Distance from Source Point")

    plt.suptitle("t-SNE Conditional Probabilities: Each Point's Perspective", fontsize=16)
    plt.tight_layout(rect=[0, 0.03, 1, 0.97])
    plt.show()

plot_all_conditional_distributions(data)

The conditional probabilities​ are asymmetric. To obtain a single joint probability distribution over unordered pairs, t-SNE symmetrizes them as

Low-Dimensional Similarity:

t-SNE now introduces the low-dimensional variables, and the similarity between them is calculated using “kind of ” different formula. Distances between points in the embedding space are converted into similarities using a Student-t distribution with one degree of freedom:

This heavy-tailed distribution is a critical design choice. Compared to a Gaussian, it assigns relatively larger probability mass to moderately distant points, which mitigates the crowding problem.

Crowding Problem (briefly)

The crowding problem arises when we try to map high-dimensional data into a low-dimensional space while preserving local neighborhoods.In high dimensions, each point can have many neighbors at similar distances since there is a lot of “volume” available to place those neighbors around a point. However, in low dimensions, there is far less space and many moderately distant neighbors must be placed too close together

To address this issue, t-SNE keeps:

• Gaussian similarities in high-D
• Student’s t-distribution (ν = 1, Cauchy) in low-D

Because the t distribution has heavy tails, Probabilities drop very fast as distance increases. Only very close neighbors get meaningful probability mass

Lastly,

• Gaussian table: small table → everyone ends up shoulder-to-shoulder
• Student-t table: expandable table → close friends sit nearby, strangers stay far

Objective function:

The objective is to minimize the difference between these two distributions using the Kullback-Leibler (KL) divergence formula to measure how well the low-dimensional map represents the high-dimensional data.

Kullback-Leibler divergence

Optimization is done using Gradient Descent:

Get Be Curious !’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The loss function is minimized with respect to the embedding coordinates Y using gradient-based optimization. The gradient of the KL divergence with respect to a point y_i​ is

Please see the code example that follows the math behind the t-SNE by plotting 2 dimensional graph and an animation of how gradient descent moves points.

Simple Code Implementation:

Import libraries and create a dataset:

# Libraries 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler

# Import Data 
X = df[["x1", "x2"]].values   # or high-dimensional version
n = X.shape[0]

Standardize (Mandatory for t-SNE since it is distance-based, like PCA)

X = StandardScaler().fit_transform(X)

Pairwise squared Euclidean distances

D = np.sum((X[:, None, :] - X[None, :, :])**2, axis=2)

Perplexity — Entropy

# Entropy
def entropy(P):
    P = P[P > 0]
    return -np.sum(P * np.log2(P))

# Conditional probabilities for fixed σᵢ

def conditional_probabilities(dist_row, sigma):
    P = np.exp(-dist_row / (2 * sigma**2))
    P[dist_row == 0] = 0  # remove self
    return P / np.sum(P)
# Binary search for σᵢ (core perplexity step)
def find_sigma(dist_row, target_perplexity, tol=1e-5):
    sigma_min, sigma_max = 1e-10, 1000.0
    sigma = 1.0

    for _ in range(50):
        P = conditional_probabilities(dist_row, sigma)
        perp = 2 ** entropy(P)

        if abs(perp - target_perplexity) < tol:
            break
        if perp > target_perplexity:
            sigma_max = sigma
            sigma = (sigma + sigma_min) / 2
        else:
            sigma_min = sigma
            sigma = (sigma + sigma_max) / 2

    return sigma

Full perplexity-based P matrix computation

perplexity = 30
P_cond = np.zeros((n, n))

for i in range(n):
    sigma_i = find_sigma(D[i], perplexity)
    P_cond[i] = conditional_probabilities(D[i], sigma_i)

Symmetrization

P = (P_cond + P_cond.T) / (2 * n)

Initialize low-dimensional embedding

np.random.seed(42)
Y = 1e-4 * np.random.randn(n, 2)

Low-dimensional similarities (Student t distribution)

def compute_Q(Y):
    dist = np.sum((Y[:, None, :] - Y[None, :, :])**2, axis=2)
    num = 1 / (1 + dist)
    np.fill_diagonal(num, 0)
    Q = num / np.sum(num)
    return Q, num

Loss function — KL divergence loss

def kl_divergence(P, Q):
    eps = 1e-12
    return np.sum(P * np.log((P + eps) / (Q + eps)))

Gradient of t-SNE Loss function

def compute_gradient(P, Q, num, Y):
    grad = np.zeros_like(Y)
    for i in range(Y.shape[0]):
        diff = Y[i] - Y
        grad[i] = 4 * np.sum(
            (P[i] - Q[i])[:, None] * num[i][:, None] * diff,
            axis=0
        )
    return grad

Gradient descent optimization

learning_rate = 200
n_iter = 500
losses = []

for t in range(n_iter):
    Q, num = compute_Q(Y)
    loss = kl_divergence(P, Q)
    losses.append(loss)

    grad = compute_gradient(P, Q, num, Y)
    Y -= learning_rate * grad

    if t % 50 == 0:
        print(f"Iter {t}, KL = {loss:.4f}")

Visualization

plt.figure(figsize=(6, 6))
plt.scatter(Y[:, 0], Y[:, 1], s=25)
plt.title("t-SNE embedding (with perplexity)")
plt.grid(True)
plt.show()

Output:

Iter 000 | KL = 15.5419
Iter 050 | KL = 10.5574
Iter 100 | KL = 1.1701
Iter 150 | KL = 0.3767
Iter 200 | KL = 0.3305
Iter 250 | KL = 0.3029
Iter 300 | KL = 0.2841
Iter 350 | KL = 0.2704
Iter 400 | KL = 0.2598
Iter 450 | KL = 0.2513

This animation clearly shows how Gradient descent moves points to merge them as groups.

Key Reasons to Use t-SNE

• Revealing Hidden Clusters: Unlike linear methods like PCA (Principal Component Analysis), t-SNE excels at capturing non-linear relationships, making it easier to see distinct groups or “islands” in complex data like gene expressions or image pixels.
• Solving the “Crowding Problem”: t-SNE uses a Student’s t-distribution in the low-dimensional space. This distribution has “heavier tails” than a normal distribution, which allows points to spread out more naturally and prevents them from all clumping together in the center of the map.
• Validation of Models: It is often used to visualize the internal representations of deep learning models (e.g., CNN feature maps). If a t-SNE plot shows clear separation between classes, it indicates that the model has successfully learned meaningful features.
• Handling Non-Linear Manifolds: It is specifically designed to handle data that lies on several different but related low-dimensional manifolds, such as objects viewed from multiple different angles

Note: The objective function is non-convex, which means that different random initializations of the low-dimensional points can lead to different final visualizations (local minima). It is recommended to run t-SNE multiple times with the same data to select the best visualization, often indicated by the lowest final KL divergence value.

source

Conclusion:

t-SNE first standardizes the data, computes pairwise distances, converts them into adaptive Gaussian neighborhood probabilities using perplexity-controlled bandwidths, symmetrizes these probabilities, initializes a low-dimensional embedding, defines Student-t similarities in the embedding space, and iteratively minimizes the KL divergence between high- and low-dimensional distributions via gradient descent — directly optimizing point locations rather than learning a reusable model.

Check out my blogs on PCA and PCA Implementation

Sources:

L. Maaten, G. Hinton. Visualizing Data using t-SNE, 2008 http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf

StatQuest: t-SNE, Clearly Explained

Géron, A. (2019). Hands-on machine learning with Scikit-Learn, Keras and TensorFlow: Concepts, tools, and techniques to build intelligent systems (2nd ed.). O’Reilly Media.