ICED: Bringing the TabPFN Revolution to Density Estimation
One frozen transformer, pretrained once on synthetic data, now handles density estimation, OOD detection, anomaly detection, and data augmentation in a single forward pass.
Density estimation quietly sits underneath a lot of unsupervised work. Anomaly detection, out-of-distribution (OOD) detection, and data augmentation all come down to the same question: where does the probability mass actually live? Yet in practice we treat them as separate tasks, and every new dataset means fitting and tuning yet another model from scratch.
On the supervised side, this has already changed. Classification and regression on tabular data were reshaped by meta-learned, in-context learning (ICL) foundation models like TabPFN. Density estimation never caught up. ICED is the paper that sets out to close that gap.
The Core Idea
ICED is a transformer that, like TabPFN, is pretrained just once, entirely on synthetic data. At inference time, you hand it a dataset as context along with a batch of query points, and in a single forward pass it returns an energy for each point, meaning an unnormalized log-density. Conceptually, it works much the same way TabPFN does.
The authors deliberately model energy rather than a normalized likelihood, for two reasons. First, downstream tasks only need relative densities and rankings between points. Second, normalization would require the partition function, Z = ∫ exp(energy) dx, which is intractable here: x is high-dimensional, continuous tabular data, and that integral has no closed-form solution. Following the classic energy-based model formulation p(x) ∝ exp(−E(x)), the model outputs the negative energy, so a larger output corresponds to a higher density.
A Synthetic Prior That Looks Like Real Tables
The key to the data side is a synthetic prior realistic enough to resemble actual tabular data. The authors build it from Gaussian mixtures combined with heavy-tailed mixtures (Student-t, Laplace, and Cauchy). Optionally, they apply an invertible normalizing flow, the same kind of invertible network used in normalizing flows, to add nonlinear warping. They also mix in discrete categorical features, and they perturb a portion of the query points so the model gets exposed to low-density regions off the data manifold.
Training Objective: Regression Plus Ranking
The loss combines a regression term with a ranking term.
Get ComfyAI’s stories in your inbox
Join Medium for free to get updates from this writer.
The regression term uses squared error in high-density regions. For out-of-bounds OOD tails, it applies only a one-sided penalty: it kicks in only when the model predicts such a point’s density so high that it crosses into the range of the normal distribution, even though that density should be low.
The ranking term penalizes ordering mistakes pairwise. It takes two query points at a time, and if the model ranks their densities the wrong way around, it gets penalized.
Architecture
ICED builds on the feature extractor from TabICLv2. Context tokens attend only to each other, while query tokens attend to the context. As a result, the model is insensitive to the order of the data points.
One Frozen Model, Four Tasks
With this setup, a single frozen ICED drives four tasks. It performs density estimation by ranking points according to energy. It uses negative energy as an outlier score for both OOD detection and anomaly detection (the two tasks differ in how the context is set up). And it performs data augmentation by sampling with Langevin dynamics (SGLD) along the energy gradient.
Results
On synthetic data, ICED ranks first for density ranking. On ADBench, it places third out of 26 methods for OOD detection. In anomaly detection with a contaminated context, it lands in the top three. And because it needs only a single forward pass, it runs roughly 50 times faster than TabPFN.
| Find papers faster on arXivSub with AI summary (CVPR、ICCV、ECCV、ICML、ICLR、NeurIPS、AAAI、MICCAI、CoRL、RSS、ACL、EMNLP、ICRA、IROS、IJCAI)
