Sitemap
Dhruv Mali

Dhruv Mali

Cognitive Neuroscience geek.

TTDnet — ThingsThoughtDiffusion Reconstructing Visual Perception from EEG Signals via Multi-Scale Neural Encoding and Latent Diffusion

6 min readMar 2, 2026

--

Overview

TTDnet is a two-stage framework for generating photorealistic images directly from electroencephalography (EEG) recordings. Given a raw multi-channel EEG epoch captured while a subject views an object, TTDnet reconstructs a semantically faithful image of that object. The system bridges the gap between low-SNR neural signals and high-fidelity image generation by chaining:

  1. A self-supervised EEG encoder (InceptSADNet) pre-trained to learn robust temporal-spatial neural representations,
  2. An IP-Adapter–style conditioning bridge that translates EEG latents into the semantic embedding space of a large-scale text-to-image diffusion model, and
  3. A Stable Diffusion XL (SDXL) backbone fine-tuned with LoRA to generate 512 × 512 images conditioned on EEG rather than text.

The pipeline requires no textual descriptions at any point; visual content is decoded entirely from brain activity.

Architecture

Press enter or click to view image in full size

The full architecture is divided into two sequential training stages. Stage A learns a general-purpose EEG representation via masked signal reconstruction; Stage B fine-tunes the frozen encoder together with a conditioning bridge and an SDXL diffusion backbone on paired EEG–image data.

Stage A— Self-Supervised EEG Pre-training

Goal: Learn transferable EEG representations without any image labels, using large-scale unlabelled EEG recordings.

InceptSADEncoder (sc_mbm/incept_encoder.py)

The encoder transforms raw multi-channel EEG into a compact sequence of latent tokens. It consists of the following major blocks:

Press enter or click to view image in full size

Default hyperparameters: embed_dim = 1024, depth (L) = 6, num_heads = 8, F₁ = 8, D = 2.

Temporal Masking Pre-training Wrapper (sc_mbm/incept_pretrain.py)

Instead of the patch-level random masking used in standard MAE, we employ contiguous temporal masking tailored to EEG:

  1. Masking Stage: Multiple random contiguous time segments are zeroed out in the raw EEG input (default: 50% of all time-steps). This forces the encoder to infer missing temporal context from surrounding neural activity.
  2. Encoding: The masked signal passes through the full InceptSADEncoder, producing latent tokens.
  3. Lightweight Decoder: A 4-layer Transformer decoder with sinusoidal positional embeddings and a linear prediction head reconstructs the original (un-masked) signal from the encoder’s output.
  4. Reconstruction Loss: MSE is computed only over the masked time regions, encouraging the encoder to build a holistic understanding of the temporal dynamics rather than merely copying visible segments.

The pre-training loop (stageA1_incept_pretrain.py) trains the encoder + decoder jointly with AdamW (lr = 4e-4, cosine schedule) for 200 epochs on large-scale unlabelled EEG data. After pre-training, only the encoder weights are transferred to Stage B; the decoder is discarded.

Alternative encoder path: The codebase also retains a ViT-based Masked Autoencoder (sc_mbm/mae_for_eeg.py) with 1-D Patch Embedding and standard random patch masking, used with the original EEG dataset and the SD 1.5 backend. The InceptSADEncoder supersedes this for the SDXL pipeline.

Stage B — Generative Fine-tuning (EEG → Image)

Goal: Learn to generate images that match the visual stimulus a subject was viewing, given only their EEG recording.

EEG Conditioning Wrapper (dc_ldm/sdxl_pipeline.py → EEGConditioningWrapper)

A thin wrapper that chains the frozen pre-trained InceptSADEncoder with the IP-Adapter Bridge and exposes a unified conditioning interface. On a forward pass it returns two outputs: (a) cross-attention conditioning tokens for the UNet, and (b) the raw EEG latent for CLIP alignment supervision.

IP-Adapter Bridge (dc_ldm/ip_adapter_bridge.py)

The bridge converts the EEG encoder’s variable-length token sequence into a fixed-size conditioning signal compatible with SDXL’s cross-attention mechanism. It has two sub-modules:

Press enter or click to view image in full size

Diffusion Backbone — Stable Diffusion XL (dc_ldm/sdxl_pipeline.py → EEGtoImageSDXL)

Get Dhruv Mali’s stories in your inbox

Join Medium for free to get updates from this writer.

The generative core is built on SDXL (Stable Diffusion XL Base 1.0), modified for EEG-conditioned generation:

Press enter or click to view image in full size

Training Objective

The total loss during fine-tuning is a weighted sum:

L_total = L_diffusion + 0.5 · L_clip
  • L_diffusion (MSE): Standard ε-prediction loss — the UNet predicts the noise added to the VAE latent, and the MSE between predicted and actual noise is minimized.
  • L_clip (InfoNCE): Symmetric contrastive loss aligning the EEG-derived embedding with the CLIP image embedding in a shared 768-dim space. Computed every 4 batches to reduce CLIP encoder overhead.

Inference Pipeline

At inference, no ground-truth images are needed:

Raw EEG → InceptSADEncoder → EEG Latent Tokens
→ Perceiver Resampler → 16 Conditioning Tokens [B, 16, 2048]
→ Pooled Projection → Pooled Embedding [B, 1280]
Random Gaussian Noise [B, 4, 64, 64]
→ 25 DPM++ Steps (UNet denoising, conditioned via cross-attention)
→ Denoised VAE Latent [B, 4, 64, 64]
→ VAE Decoder
→ Reconstructed Image [B, 3, 512, 512]

Legacy SD 1.5 path: The codebase retains a full Stable Diffusion 1.5 pipeline (dc_ldm/ldm_for_eeg.py → eLDM) with DDIM/PLMS sampling (250 steps), a simpler cond_stage_model using linear projection, and the ViT-based eeg_encoder. This is used when config.model_type = 'sd15'.

Datasets

ThingsEEG (Primary Dataset for finetuning)

Press enter or click to view image in full size

Pre-training Data

Press enter or click to view image in full size

Training Pipeline

Stage A — EEG Pre-training

python stageA1_incept_pretrain.py

Stage B — Generative Fine-tuning

python eeg_ldm.py
Press enter or click to view image in full size

Evaluation

After training, the framework generates multiple samples per test EEG epoch and evaluates them against ground-truth images using:

Press enter or click to view image in full size

Evaluation is implemented in eval_metrics.py and operates in both pair-wise and n-way scoring modes.

Repository Structure

TTDnet/
├── code/
│ ├── config.py # All hyperparameter configurations
│ ├── dataset.py # EEG pre-training & original EEG datasets
│ ├── things_dataset.py # ThingsEEG paired dataset (EEG + images)
│ ├── eeg_ldm.py # Main entry point (training + generation)
│ ├── eval_metrics.py # MSE, PCC, SSIM, LPIPS, FID, n-way accuracy
│ │
│ ├── stageA1_eeg_pretrain.py # Stage A1: ViT-MAE pre-training script
│ ├── stageA1_incept_pretrain.py # Stage A1: InceptSAD pre-training script
│ │
│ ├── preprocess_edf_to_npy.py # Raw .edf → .npy conversion
│ ├── preprocess_things_eeg.py # ThingsEEG preprocessing pipeline
│ │
│ ├── sc_mbm/ # Self-supervised EEG encoders
│ │ ├── InceptSADNet.py # InceptSADNet classification model
│ │ ├── incept_encoder.py # InceptSADEncoder (used in pipeline)
│ │ ├── incept_pretrain.py # Temporal masking pre-training wrapper
│ │ ├── mae_for_eeg.py # ViT-based MAE encoder (legacy)
│ │ ├── trainer.py # MAE training utilities
│ │ └── utils.py # Checkpoint save/load helpers
│ │
│ └── dc_ldm/ # Diffusion & conditioning modules
│ ├── sdxl_pipeline.py # SDXL pipeline (train + generate)
│ ├── ip_adapter_bridge.py # Perceiver Resampler + CLIP alignment
│ ├── ldm_for_eeg.py # SD 1.5 pipeline (legacy)
│ ├── util.py # Config instantiation utilities
│ │
│ ├── models/
│ │ ├── autoencoder.py # VAE (SD 1.5 path)
│ │ └── diffusion/
│ │ ├── ddpm.py # DDPM implementation
│ │ ├── ddim.py # DDIM sampler
│ │ ├── plms.py # PLMS sampler
│ │ └── classifier.py # Classifier-free guidance
│ │
│ └── modules/
│ ├── attention.py # Cross-/Self-/Linear Attention
│ ├── x_transformer.py # Extended Transformer blocks
│ ├── ema.py # Exponential Moving Average
│ ├── diffusionmodules/ # UNet blocks, timestep embed, utils
│ ├── encoders/ # CLIP/FrozenEncoder wrappers
│ ├── distributions/ # Gaussian distributions
│ └── losses/ # Perceptual & VQ losses

Results

Press enter or click to view image in full size

Training

All the training was performed on single workstation, with Nvidia GV100 32GB Vram.
For the pretraining stage, to teach the model how to interpret the brainwaves we used physionet dataset which was around 3.9GB and it took around 8.5 hours to train,
For the second stage which was finetuning we use ThingsEEG dataset of 50.9 GB approx but we only used first 5 subjects because of shortage in resources which was around 8GB and the training time for finetuning process was 42.32 hours.

--

--

Dhruv Mali
Dhruv Mali

Written by Dhruv Mali

Cognitive Neuroscience geek.