PatchCore: Rethinking Cold-Start Industrial Anomaly Detection with Patch-Level Memory

How locally aware patch features and coreset memory banks achieve near-perfect recall without training

Introduction

About

Fig 1. Examples from the MVTec benchmark datasets. Here, the orange boundary denotes anomaly contours of actual segmentation maps for anomalies. (Source)

This paper studies cold-start industrial anomaly detection, a setting where models must detect and localize defects using only nominal (non-defective) training images. The authors propose PatchCore, a patch-level, memory-based anomaly detection framework that maximizes nominal coverage at test time while remaining computationally efficient. As shown in Figure 1, PatchCore achieves state-of-the-art detection and localization performance without requiring any task-specific training or adaptation.

Background

Traditional anomaly detection approaches in computer vision primarily aim to model the nominal data distribution and identify anomalies as deviations from this model. Autoencoders, Gaussian mixture models, and GAN-based methods typically detect anomalies via reconstruction error or likelihood estimates. While effective in controlled settings, these approaches often struggle with industrial image data, where anomalies can be extremely subtle and local, and nominal variability is difficult to model accurately. Numerous extensions — such as structural constraints, attention mechanisms, and student–teacher frameworks — have been proposed to improve robustness, but these methods still rely on training procedures that can be sensitive to data scale, hyperparameters, and domain-specific assumptions.

A more recent paradigm leverages ImageNet-pretrained convolutional networks without adapting them to the target domain. Methods such as SPADE and PaDiM perform anomaly detection via feature matching or patch-wise distribution modeling, achieving strong performance without training. However, these approaches expose new limitations: deep features are biased toward ImageNet semantics, high-level representations provide limited nominal context, and large memory banks lead to scalability and inference-time bottlenecks. PatchCore is designed within this paradigm, explicitly targeting these shortcomings by emphasizing mid-level patch features, increasing nominal coverage, and introducing an efficient coreset-based memory reduction strategy.

Method

Fig 2. The brief outline of PatchCore. (Source)

PatchCore consists of three tightly coupled components: locally aware patch-level feature extraction, coreset-based memory bank reduction, and nearest-neighbor–based anomaly scoring for detection and localization. The overall pipeline is illustrated in Figure 2 of the original paper. Importantly, PatchCore operates in a training-free manner, relying solely on pretrained convolutional backbones and nominal data.

Locally Aware Patch Features

PatchCore extracts feature maps from an ImageNet-pretrained convolutional network, such as ResNet or WideResNet. A central design choice is the use of mid-level feature hierarchies, rather than the deepest layers, which are known to be strongly biased toward semantic object categories. Each spatial location in a feature map corresponds to a patch-level representation associated with a receptive field in the input image.

To enhance robustness to small spatial misalignments while preserving resolution, PatchCore introduces local neighborhood aggregation. For a patch located at spatial position (h, w), features from a surrounding neighborhood of size p are aggregated:

The locally aware patch representation is then computed as

where f denotes adaptive average pooling.

This aggregation is a key contribution, as it increases receptive field size without descending deeper into the network, thereby reducing ImageNet bias while preserving fine-grained anomaly sensitivity. Patch-level features from adjacent hierarchies are spatially aligned and combined, and all nominal patch representations are stored in a unified memory bank.

Coreset-Reduced Patch-Feature Memory Bank

Storing patch-level features for all nominal images results in a memory bank that grows rapidly with dataset size, making naive nearest-neighbor search impractical. To address this, PatchCore introduces coreset subsampling to compress the memory bank while preserving its coverage of the nominal feature space.

Memory bank reduction is formulated as a minimax facility location problem, seeking a subset 𝓜꜀​ that best approximates the original memory bank 𝓜:

Fig 3. Comparison of sampling manners. These indicate coreset (top) versus random subsampling (bottom) (red) for 2D data (blue) sampled from (a) multimodal and (b) uniform distributions. (Source)

Since exact optimization is intractable, a greedy approximation is used, iteratively selecting features that are farthest from the current subset. To further improve efficiency, PatchCore applies random linear projections, leveraging the Johnson–Lindenstrauss lemma to reduce dimensionality during coreset selection. As illustrated in Figure 3, this strategy preserves the structure of the nominal feature distribution far more effectively than random subsampling, enabling orders-of-magnitude reduction in memory size with minimal performance loss.

Anomaly Detection with PatchCore

During inference, PatchCore extracts patch-level features from a test image and compares them against the reduced memory bank. For each test patch, the distance to its nearest nominal neighbor is computed, and the image-level anomaly score is defined as the maximum patch distance:

To improve robustness, PatchCore applies a density-aware re-weighting that accounts for how isolated the nearest nominal neighbor is in feature space:

This re-weighting emphasizes anomalies that lie far from dense nominal regions, a crucial design choice for improving recall. Pixel-level anomaly localization is obtained by mapping patch scores back to their spatial locations, followed by interpolation and Gaussian smoothing to produce dense anomaly heatmaps.

Experiment

This section evaluates PatchCore from multiple perspectives: image-level anomaly detection, pixel-level anomaly localization, computational efficiency, and robustness under limited training data. The experiments are designed to reflect realistic industrial inspection settings, emphasizing cold-start scenarios where only nominal data is available. In addition to the widely used MVTec AD benchmark, the authors evaluate PatchCore on more specialized industrial datasets and a non-industrial anomaly localization benchmark to assess generalization.

Dataset

The experimental evaluation spans three datasets with complementary characteristics:

• MVTec Anomaly Detection (AD)
  This is the primary benchmark used throughout the paper. It consists of 15 industrial categories covering textures and objects, with a total of 5,354 images, including 1,725 test images. Training data contains only nominal samples, while test data includes both normal and anomalous images with pixel-level ground truth masks. Images are resized and center-cropped to fixed resolutions (224 × 224 or 256 × 256), following standard protocols.
• Magnetic Tile Defects (MTD)
  This dataset contains 925 defect-free and 392 anomalous images of magnetic tiles, exhibiting high variability in illumination and image size. The dataset is particularly challenging due to its intra-class variability, which resembles anomaly patterns.
• Mini ShanghaiTech Campus (mSTC)
  A subsampled version of the ShanghaiTech Campus dataset, used for unsupervised anomaly localization in video frames. Images are resized to 256×256, and no anomaly supervision is used during training. This dataset serves to test domain transferability beyond industrial imagery.

Model Set-up

PatchCore is implemented using ImageNet-pretrained convolutional backbones, without any fine-tuning on the target datasets. The key configuration choices include:

• Backbone networks: ResNet and WideResNet variants.
• Feature hierarchies: primarily mid-level layers (e.g., levels 2 and 3).
• Patch construction: stride set to 1 with neighborhood size p = 3.
• Memory bank reduction: coreset subsampling to 25%, 10%, and 1% of the original memory.
• Inference: exact nearest-neighbor search unless otherwise specified.

Notably, PatchCore requires no training on the target dataset, making inference speed and memory efficiency critical factors in the design.

Comparison

PatchCore is compared against a broad range of state-of-the-art anomaly detection methods, including:

• SPADE (deep feature correspondence with memory banks),
• PaDiM (patch distribution modeling with Mahalanobis distance),
• PatchSVDD, DifferNet, and other reconstruction- or flow-based baselines.

Where possible, the authors ensure fair comparison by using identical backbone architectures and preprocessing pipelines. Comparisons are performed separately for image-level detection and pixel-level localization, reflecting the different strengths of competing approaches.

Evaluation Metrics

Three primary metrics are used to evaluate performance:

• Image-level AUROC
  Measures the ability to distinguish anomalous images from nominal ones based on anomaly scores. This metric is critical for industrial inspection, where missing a defective sample can be costly.
• Pixel-wise AUROC
  Evaluates anomaly localization accuracy by treating each pixel independently. This metric captures fine-grained localization performance.
• PRO (Per-Region Overlap)
  A region-based metric that measures overlap between predicted and ground-truth anomaly regions, accounting for connected components. PRO is particularly important for industrial defects, which often vary in size and shape.

Results

Anomaly Detection on MVTec AD

Table 1. Anomaly Detection Performance (AUROC) on MVTec AD. (Source)

Table 2. Anomaly Segmentation Performance (pixelwise AUROC) on MVTec AD. (Source)

Tab;e 3. Anomaly Detection Performance on MVTec AD as measured in PRO [%]. (Source)

Table 4. PatchCore-1% with higher resolution/larger backbones/ensembles. (Source)

PatchCore achieves state-of-the-art image-level anomaly detection performance on MVTec AD. As shown in Table 1, PatchCore reaches up to 99.1% AUROC, significantly outperforming prior methods such as PaDiM and SPADE. With the AUROC metric, this manner also achieves state-of-the-art anomaly segmentation, both measured by pixelwise AUROC (Table 2, 98.1 versus 97.5 for PaDiM) and PRO metric (Table 3, 93.5 versus 92.1). Importantly, the improved version (PatchCore−1%) can be also applied n images of higher resolution (e.g. 280/320 instead of 224) and ensemble systems while retaining inferences times less than the default one. Given outcomes of Table 4 summarizes the effectiveness of the proposed subsampling manner.

Get Dong-Keon Kim’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This result is particularly impactful in industrial contexts, where even small reductions in false negatives can have substantial practical value. PatchCore also demonstrates consistent performance across all categories, rather than relying on gains in only a subset of classes.

Inference Time

Table 5. Mean inference time per image on MVTec AD. Note that scores are (image AUROC, pixel AUROC, PRO metric). (Source)

Inference efficiency is analyzed in Table 5, comparing PatchCore against SPADE and PaDiM. Even without subsampling, PatchCore achieves faster inference than SPADE while delivering higher accuracy. With coreset subsampling, PatchCore becomes significantly faster, achieving inference times comparable to or better than PaDiM while maintaining superior detection and localization performance.

The authors also evaluate approximate nearest-neighbor search (IVFPQ) as an alternative speed-up strategy. While this further reduces inference time, it introduces a noticeable drop in detection performance. Coreset subsampling emerges as the more effective and robust solution, striking a better balance between speed and accuracy.

Ablation Study

— Locally Aware Patch Features and Hierarchies

Fig 4. Local awareness and network feature depths vs. detection performance. (Source)

The ablation results in Figure 4 demonstrate the importance of local neighborhood aggregation. Performance peaks at a neighborhood size of p = 3, confirming that moderate local context is critical for accurate patch-level anomaly detection. Additional experiments show that mid-level feature hierarchies outperform both shallow and deep layers, validating the design choice to avoid excessive ImageNet bias.

— Importance of Coreset Subsampling

Fig 5. Performance retention for different subsamplers. (Source)

Figure 5 compares greedy coreset subsampling with random subsampling and learned proxy representations. Coreset selection consistently preserves performance better than alternatives and, in some cases, even improves joint detection and localization accuracy. The authors also show that coreset subsampling dramatically increases memory utilization efficiency, with nearly all retained features contributing during inference. In contrast, reducing memory size via increased stride leads to degraded performance, underscoring the importance of preserving spatial resolution.

— Low-shot Anomaly Detection

Fig 6. Sample-efficiency comparsions according to the 3 evaluation metrics. (Source)

PatchCore exhibits exceptional sample efficiency, as shown in Figure 6. With as few as 20% of nominal training data, PatchCore matches or exceeds the performance of competing methods trained on full datasets. Even in extreme low-shot regimes, such as 1–5 training samples, PatchCore remains competitive, highlighting its robustness and practical applicability in data-scarce industrial scenarios.

— Evaluation on Other Benchmarks

Table 6. Anomaly Segmentation on mSTC and anomaly detection on MTD. (Source)

Results on mSTC and MTD are summarized in Table 6. On mSTC, PatchCore achieves state-of-the-art anomaly localization performance without any dataset-specific tuning, demonstrating strong transferability to non-industrial domains. On MTD, PatchCore slightly outperforms DifferNet, despite the dataset’s high intra-class variability. These results suggest that PatchCore’s design generalizes well beyond rigid industrial inspection tasks.

Conclusion

This paper introduces PatchCore, a training-free, memory-based framework for cold-start anomaly detection that effectively balances performance, efficiency, and scalability. By leveraging locally aware patch-level features and a coreset-reduced memory bank, PatchCore maximizes nominal context while mitigating ImageNet bias and inference bottlenecks. Extensive experiments demonstrate state-of-the-art results in both image-level detection and pixel-level localization across multiple benchmarks. The method’s strong performance in low-shot settings further underscores its practical relevance.

Review

The paper presents a well-motivated and carefully engineered approach to industrial anomaly detection, addressing both methodological and practical limitations of prior work. Its central strength lies in recognizing that pretrained features are highly effective when paired with sufficient nominal context, and in proposing a principled way to retain this context through patch-level representations and coreset selection. The experimental evaluation is comprehensive and convincingly demonstrates improvements not only in accuracy but also in inference efficiency and robustness.

That said, PatchCore’s reliance on pretrained representations also defines its primary limitation. Performance is ultimately constrained by the transferability of ImageNet features, particularly for domains that diverge significantly from natural imagery. While the paper acknowledges this issue, future work could explore hybrid approaches that combine PatchCore with lightweight feature adaptation or self-supervised fine-tuning. Despite this limitation, PatchCore stands out as a highly practical and impactful contribution, with clear relevance for industrial deployment and strong potential as a foundation for future research.

Reference

Towards Total Recall in Industrial Anomaly Detection

Happy new year!